Started by user Snigdha Agrawal (she/her) Rebuilds build #30 Obtained px-backup/custom-pipelines/cloud-byoc-system-test.Jenkinsfile from git git@github.com:portworx/jenkinsfiles.git [Pipeline] Start of Pipeline [Pipeline] node Running on pxbackup-agent04 in /root/workspace/portworx-backup/system-tests/byoc/Cloud_BYOC@2 [Pipeline] { [Pipeline] ws Running in /root/workspace/px-backup-system-tests/iks/iks/4cac76dc [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Checkout SCM) [Pipeline] checkout The recommended git tool is: git using credential github_ssh_key Cloning the remote Git repository Cloning repository git@github.com:portworx/jenkinsfiles.git > git init /root/workspace/px-backup-system-tests/iks/iks/4cac76dc # timeout=10 Fetching upstream changes from git@github.com:portworx/jenkinsfiles.git > git --version # timeout=10 > git --version # 'git version 2.25.1' using GIT_SSH to set credentials SSH key for GitHub > git fetch --tags --force --progress -- git@github.com:portworx/jenkinsfiles.git +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision 2a08bdacdb6bccd01506f37d00e3e643bd243b87 (refs/remotes/origin/snigdha-byoc-roks) Commit message: "BYOC for ROKS" > git config remote.origin.url git@github.com:portworx/jenkinsfiles.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse refs/remotes/origin/snigdha-byoc-roks^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 2a08bdacdb6bccd01506f37d00e3e643bd243b87 # timeout=10 > git rev-list --no-walk 2a08bdacdb6bccd01506f37d00e3e643bd243b87 # timeout=10 The recommended git tool is: git using credential github_ssh_key The recommended git tool is: git using credential github_ssh_key [GitCheckoutListener] Recording commits of 'git git@github.com:portworx/jenkinsfiles.git' [GitCheckoutListener] Found previous build 'portworx-backup/system-tests/byoc/Cloud_BYOC #41' that contains recorded Git commits [GitCheckoutListener] -> Starting recording of new commits since '2a08bda' [GitCheckoutListener] -> Single parent commit found - branch is already descendant of target branch head [GitCheckoutListener] -> Using head commit '2a08bda' as starting point [GitCheckoutListener] -> No new commits found [GitCheckoutListener] -> Git commit decorator successfully obtained 'hudson.plugins.git.browser.GithubWeb@2504ce12' to render commit links [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $DOCKERHUB_CREDS_USR or $DOCKERHUB_CREDS_PSW or $IBM_ACCESS_KEY or $IBM_SECRET_KEY or $SSH_PASSWORD or $PX_BACKUP_MONGODB_USERNAME or $IBM_CLOUD_API_KEY or $FB_SECRET_KEY or $FB_ACCESS_KEY or $SLACK_CREDS or $PX_BACKUP_MONGODB_PASSWORD or $GKE_JSON_KEY [Pipeline] { [Pipeline] withEnv Warning: A secret was passed to "withEnv" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [PX_BACKUP_MONGODB_USERNAME] See https://jenkins.io/redirect/groovy-string-interpolation for details. [Pipeline] { [Pipeline] ansiColor [Pipeline] { [Pipeline] timestamps [Pipeline] { [Pipeline] stage [Pipeline] { (Pipeline Setup) [Pipeline] script [Pipeline] { [Pipeline] load > git rev-parse HEAD^{commit} # timeout=10 [Pipeline] { (scripts/spawn.groovy) [Pipeline] } [Pipeline] // load [Pipeline] load [Pipeline] { (scripts/px-backup.groovy) [Pipeline] } [Pipeline] // load [Pipeline] load [Pipeline] { (scripts/torpedo.groovy) [Pipeline] } [Pipeline] // load [Pipeline] sh [2024-01-18T15:27:11.810Z] Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure. [2024-01-18T15:27:11.810Z] Affected argument(s) used the following variable(s): [DOCKERHUB_CREDS_USR, DOCKERHUB_CREDS_PSW] [2024-01-18T15:27:11.810Z] See https://jenkins.io/redirect/groovy-string-interpolation for details. [2024-01-18T15:27:12.084Z] + docker login -u **** -p **** [2024-01-18T15:27:12.084Z] WARNING! Using --password via the CLI is insecure. Use --password-stdin. [2024-01-18T15:27:12.646Z] WARNING! Your password will be stored unencrypted in /****/.docker/config.json. [2024-01-18T15:27:12.646Z] Configure a credential helper to remove this warning. See [2024-01-18T15:27:12.646Z] https://docs.docker.com/engine/reference/commandline/login/#credentials-store [2024-01-18T15:27:12.646Z] [2024-01-18T15:27:12.646Z] Login Succeeded [Pipeline] sh [2024-01-18T15:27:12.962Z] + mkdir -p clusters [Pipeline] wrap [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] sh [2024-01-18T15:27:13.355Z] + git rev-parse HEAD [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // wrap [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Pull Git Repos and Images) [Pipeline] parallel [Pipeline] { (Branch: Clone Torpedo Repo) [Pipeline] { (Branch: Pull Latest Spawn Image) [Pipeline] stage [Pipeline] { (Clone Torpedo Repo) [Pipeline] stage [Pipeline] { (Pull Latest Spawn Image) [Pipeline] checkout [2024-01-18T15:27:13.746Z] The recommended git tool is: NONE [Pipeline] script [Pipeline] { [2024-01-18T15:27:13.753Z] using credential sudas [2024-01-18T15:27:13.766Z] Cloning the remote Git repository [Pipeline] sh [2024-01-18T15:27:14.060Z] + docker pull portworx/spawn:master [2024-01-18T15:27:43.434Z] Cloning repository https://github.com/portworx/torpedo.git [2024-01-18T15:27:43.435Z] > git init /****/workspace/px-backup-system-tests/iks/iks/4cac76dc/torpedo # timeout=10 [2024-01-18T15:27:43.440Z] Fetching upstream changes from https://github.com/portworx/torpedo.git [2024-01-18T15:27:43.440Z] > git --version # timeout=10 [2024-01-18T15:27:43.443Z] > git --version # 'git version 2.25.1' [2024-01-18T15:27:43.443Z] using GIT_ASKPASS to set credentials Sumit Das PAT for Github [2024-01-18T15:27:43.443Z] > git fetch --tags --force --progress -- https://github.com/portworx/torpedo.git +refs/heads/*:refs/remotes/origin/* # timeout=10 [2024-01-18T15:27:14.621Z] master: Pulling from portworx/spawn [2024-01-18T15:27:14.621Z] Digest: sha256:f5dc3066a5a4baa8666b195c6d0ea7d5f3b62215fda25b22e191132b47692b25 [2024-01-18T15:27:14.621Z] Status: Image is up to date for portworx/spawn:master [2024-01-18T15:27:14.621Z] docker.io/portworx/spawn:master [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // stage [Pipeline] } [2024-01-18T15:27:24.075Z] Avoid second fetch [2024-01-18T15:27:24.080Z] Checking out Revision 978038272e968b2a2f1c094aee9896d683964a5a (origin/ibm-mysql) [2024-01-18T15:27:53.732Z] > git config remote.origin.url https://github.com/portworx/torpedo.git # timeout=10 [2024-01-18T15:27:53.735Z] > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 [2024-01-18T15:27:53.743Z] > git rev-parse origin/ibm-mysql^{commit} # timeout=10 [2024-01-18T15:27:53.748Z] > git config core.sparsecheckout # timeout=10 [2024-01-18T15:27:53.751Z] > git checkout -f 978038272e968b2a2f1c094aee9896d683964a5a # timeout=10 [2024-01-18T15:27:25.175Z] Commit message: "Temp mysql for ibm" [2024-01-18T15:27:54.843Z] > git rev-list --no-walk 978038272e968b2a2f1c094aee9896d683964a5a # timeout=10 [2024-01-18T15:27:25.188Z] The recommended git tool is: NONE [2024-01-18T15:27:25.194Z] using credential sudas [2024-01-18T15:27:25.207Z] The recommended git tool is: NONE [2024-01-18T15:27:25.213Z] using credential sudas [2024-01-18T15:27:25.229Z] [GitCheckoutListener] Recording commits of 'git https://github.com/portworx/torpedo.git' [2024-01-18T15:27:25.229Z] [GitCheckoutListener] Found previous build 'portworx-backup/system-tests/byoc/Cloud_BYOC #41' that contains recorded Git commits [2024-01-18T15:27:25.229Z] [GitCheckoutListener] -> Starting recording of new commits since '440b03b' [2024-01-18T15:27:25.229Z] [GitCheckoutListener] -> Single parent commit found - branch is already descendant of target branch head [2024-01-18T15:27:25.229Z] [GitCheckoutListener] -> Using head commit '9780382' as starting point [2024-01-18T15:27:25.229Z] [GitCheckoutListener] -> Recorded 200 new commits [2024-01-18T15:27:25.229Z] [GitCheckoutListener] -> Git commit decorator successfully obtained 'hudson.plugins.git.browser.GithubWeb@561a1406' to render commit links [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // parallel [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Fetch Kube Configs) Stage "Fetch Kube Configs" skipped due to when conditional [Pipeline] parallel [Pipeline] { (Branch: For Source Cluster) [Pipeline] { (Branch: For Destination Cluster) [Pipeline] stage [Pipeline] { (For Source Cluster) [Pipeline] stage [Pipeline] { (For Destination Cluster) Stage "For Source Cluster" skipped due to when conditional [Pipeline] } Stage "For Destination Cluster" skipped due to when conditional [Pipeline] } [Pipeline] // stage [Pipeline] // stage [Pipeline] } [Pipeline] } [Pipeline] // parallel [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Copy Kube Configs) [Pipeline] script [Pipeline] { [Pipeline] dir [2024-01-18T15:27:25.943Z] Running in /****/workspace/px-backup-system-tests/iks/iks/4cac76dc/clusters [Pipeline] { [Pipeline] withFileParameter [Pipeline] { [Pipeline] sh [2024-01-18T15:27:26.035Z] Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure. [2024-01-18T15:27:26.035Z] Affected argument(s) used the following variable(s): [PX_BACKUP_MONGODB_USERNAME] [2024-01-18T15:27:26.035Z] See https://jenkins.io/redirect/groovy-string-interpolation for details. [2024-01-18T15:27:54.866Z] > git rev-parse HEAD^{commit} # timeout=10 [2024-01-18T15:27:26.307Z] + cp /****/workspace/px-backup-system-tests/iks/iks/4cac76dc/clusters@tmp/SOURCE_CLUSTER_KUBECONFIG9246416918651371079.tmp ./source-config [Pipeline] } [Pipeline] // withFileParameter [Pipeline] withFileParameter [Pipeline] { [Pipeline] sh [2024-01-18T15:27:26.469Z] Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure. [2024-01-18T15:27:26.469Z] Affected argument(s) used the following variable(s): [PX_BACKUP_MONGODB_USERNAME] [2024-01-18T15:27:26.469Z] See https://jenkins.io/redirect/groovy-string-interpolation for details. [2024-01-18T15:27:26.744Z] + cp /****/workspace/px-backup-system-tests/iks/iks/4cac76dc/clusters@tmp/DESTINATION_CLUSTER_KUBECONFIG9263388878321227476.tmp ./destination-config [Pipeline] } [Pipeline] // withFileParameter [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Copy GKE kubeconfig for torpedo) Stage "Copy GKE kubeconfig for torpedo" skipped due to when conditional [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Build and Push PX-Backup Torpedo Image) [Pipeline] node [2024-01-18T15:27:27.169Z] Running on k8sjenkins-agent22 in /****/workspace/portworx-backup/system-tests/byoc/Cloud_BYOC [Pipeline] { [Pipeline] checkout [2024-01-18T15:27:27.276Z] The recommended git tool is: git [2024-01-18T15:27:27.283Z] using credential github_ssh_key [2024-01-18T15:27:27.302Z] Fetching changes from the remote Git repository [2024-01-18T15:26:32.180Z] > git rev-parse --resolve-git-dir /****/workspace/portworx-backup/system-tests/byoc/Cloud_BYOC/.git # timeout=10 [2024-01-18T15:26:32.195Z] > git config remote.origin.url git@github.com:portworx/jenkinsfiles.git # timeout=10 [2024-01-18T15:26:32.203Z] Fetching upstream changes from git@github.com:portworx/jenkinsfiles.git [2024-01-18T15:26:32.203Z] > git --version # timeout=10 [2024-01-18T15:26:32.206Z] > git --version # 'git version 1.8.3.1' [2024-01-18T15:26:32.206Z] using GIT_SSH to set credentials SSH key for GitHub [2024-01-18T15:26:32.211Z] > git fetch --tags --progress git@github.com:portworx/jenkinsfiles.git +refs/heads/*:refs/remotes/origin/* # timeout=10 [2024-01-18T15:27:28.866Z] Checking out Revision 2a08bdacdb6bccd01506f37d00e3e643bd243b87 (refs/remotes/origin/snigdha-byoc-roks) [2024-01-18T15:27:28.920Z] Commit message: "BYOC for ROKS" [2024-01-18T15:27:28.920Z] [GitCheckoutListener] Skipping recording, since SCM 'git git@github.com:portworx/jenkinsfiles.git' already has been processed [Pipeline] withEnv [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] checkout [2024-01-18T15:27:29.025Z] The recommended git tool is: NONE [2024-01-18T15:27:29.032Z] using credential sudas [2024-01-18T15:27:29.043Z] Fetching changes from the remote Git repository [2024-01-18T15:26:33.746Z] > git rev-parse refs/remotes/origin/snigdha-byoc-roks^{commit} # timeout=10 [2024-01-18T15:26:33.753Z] > git config core.sparsecheckout # timeout=10 [2024-01-18T15:26:33.758Z] > git checkout -f 2a08bdacdb6bccd01506f37d00e3e643bd243b87 # timeout=10 [2024-01-18T15:26:33.923Z] > git rev-parse --resolve-git-dir /****/workspace/portworx-backup/system-tests/byoc/Cloud_BYOC/torpedo/.git # timeout=10 [2024-01-18T15:26:33.934Z] > git config remote.origin.url https://github.com/portworx/torpedo.git # timeout=10 [2024-01-18T15:26:33.940Z] Fetching upstream changes from https://github.com/portworx/torpedo.git [2024-01-18T15:26:33.940Z] > git --version # timeout=10 [2024-01-18T15:26:33.944Z] > git --version # 'git version 1.8.3.1' [2024-01-18T15:26:33.944Z] using GIT_ASKPASS to set credentials Sumit Das PAT for Github [2024-01-18T15:26:33.946Z] > git fetch --tags --progress https://github.com/portworx/torpedo.git +refs/heads/*:refs/remotes/origin/* # timeout=10 [2024-01-18T15:27:31.597Z] Checking out Revision 978038272e968b2a2f1c094aee9896d683964a5a (origin/ibm-mysql) [2024-01-18T15:26:36.477Z] > git rev-parse origin/ibm-mysql^{commit} # timeout=10 [2024-01-18T15:26:36.484Z] > git config core.sparsecheckout # timeout=10 [2024-01-18T15:26:36.488Z] > git checkout -f 978038272e968b2a2f1c094aee9896d683964a5a # timeout=10 [2024-01-18T15:27:32.938Z] Commit message: "Temp mysql for ibm" [2024-01-18T15:27:32.938Z] [GitCheckoutListener] Skipping recording, since SCM 'git https://github.com/portworx/torpedo.git' already has been processed [Pipeline] sh [2024-01-18T15:27:33.238Z] + cd torpedo [2024-01-18T15:27:33.238Z] + git rev-parse --short --verify HEAD [Pipeline] sh [2024-01-18T15:27:33.269Z] Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure. [2024-01-18T15:27:33.269Z] Affected argument(s) used the following variable(s): [DOCKERHUB_CREDS_USR, DOCKERHUB_CREDS_PSW] [2024-01-18T15:27:33.269Z] See https://jenkins.io/redirect/groovy-string-interpolation for details. [2024-01-18T15:27:33.540Z] + docker login -u **** -p **** [2024-01-18T15:27:33.540Z] WARNING! Using --password via the CLI is insecure. Use --password-stdin. [2024-01-18T15:27:34.099Z] WARNING! Your password will be stored unencrypted in /****/.docker/config.json. [2024-01-18T15:27:34.100Z] Configure a credential helper to remove this warning. See [2024-01-18T15:27:34.100Z] https://docs.docker.com/engine/reference/commandline/login/#credentials-store [2024-01-18T15:27:34.100Z] [2024-01-18T15:27:34.100Z] Login Succeeded [Pipeline] sh [2024-01-18T15:27:34.404Z] + cd torpedo [2024-01-18T15:27:34.404Z] + make -e DOCKER_HUB_REPO=portworx -e DOCKER_HUB_TORPEDO_IMAGE=torpedo-backup -e DOCKER_HUB_TAG=9780382 container-backup [2024-01-18T15:27:37.663Z] Building backup.test container portworx/torpedo-backup:9780382 [2024-01-18T15:27:37.665Z] sudo DOCKER_BUILDKIT=1 docker build --tag portworx/torpedo-backup:9780382 --build-arg MAKE_TARGET=build-backup -f Dockerfile . [2024-01-18T15:27:37.665Z] #1 [internal] load build definition from Dockerfile [2024-01-18T15:27:37.665Z] #1 sha256:d5aaea6859de3c8574ba063aeaecac18402660a1e1fc31e39d4785caeda7e2ac [2024-01-18T15:27:37.665Z] #1 transferring dockerfile: 3.22kB done [2024-01-18T15:27:37.665Z] #1 DONE 0.0s [2024-01-18T15:27:37.665Z] [2024-01-18T15:27:37.665Z] #2 [internal] load .dockerignore [2024-01-18T15:27:37.665Z] #2 sha256:d0109600a777a840aa693cccab64167b35b48089e1d65e4dcadfd450b81813e9 [2024-01-18T15:27:37.666Z] #2 transferring context: 2B done [2024-01-18T15:27:37.666Z] #2 DONE 0.0s [2024-01-18T15:27:37.666Z] [2024-01-18T15:27:37.666Z] #4 [internal] load metadata for docker.io/library/golang:1.19.5-alpine [2024-01-18T15:27:37.666Z] #4 sha256:724dacd5ee20fa6484eb14a42596f408281a7718d063eaa6d71eb68d178e1087 [2024-01-18T15:27:38.597Z] #4 DONE 1.1s [2024-01-18T15:27:38.597Z] [2024-01-18T15:27:38.597Z] #3 [internal] load metadata for docker.io/library/alpine:3.18.5 [2024-01-18T15:27:38.597Z] #3 sha256:a42fba27eeded8d3308a3078ad435ecc0cfad8773e911a745390cd52f42b9565 [2024-01-18T15:27:38.597Z] #3 DONE 1.2s [2024-01-18T15:27:38.597Z] [2024-01-18T15:27:38.597Z] #21 [build 1/19] FROM docker.io/library/golang:1.19.5-alpine@sha256:a00a03c72ecc1a01e4ca7b7cfb62459063912ff45ccb13011f9fb061e25916eb [2024-01-18T15:27:38.597Z] #21 sha256:31d43f0ad39cc6cf7d49273d478fc2ba4f2f281540cfbbc615757b8d58e9b383 [2024-01-18T15:27:38.597Z] #21 resolve docker.io/library/golang:1.19.5-alpine@sha256:a00a03c72ecc1a01e4ca7b7cfb62459063912ff45ccb13011f9fb061e25916eb 0.0s done [2024-01-18T15:27:38.597Z] #21 ... [2024-01-18T15:27:38.597Z] [2024-01-18T15:27:38.597Z] #5 [stage-1 1/20] FROM docker.io/library/alpine:3.18.5@sha256:34871e7290500828b39e22294660bee86d966bc0017544e848dd9a255cdf59e0 [2024-01-18T15:27:38.597Z] #5 sha256:d61db98444079f6f37991a479a89a981b1f244b35f53e548d0863c205a7be973 [2024-01-18T15:27:38.597Z] #5 resolve docker.io/library/alpine:3.18.5@sha256:34871e7290500828b39e22294660bee86d966bc0017544e848dd9a255cdf59e0 done [2024-01-18T15:27:38.597Z] #5 sha256:d695c3de6fcd8cfe3a6222b0358425d40adfd129a8a47c3416faff1a8aece389 528B / 528B done [2024-01-18T15:27:38.597Z] #5 sha256:b541f2080109ab7b6bf2c06b28184fb750cdd17836c809211127717f48809858 1.47kB / 1.47kB done [2024-01-18T15:27:38.597Z] #5 sha256:34871e7290500828b39e22294660bee86d966bc0017544e848dd9a255cdf59e0 1.64kB / 1.64kB done [2024-01-18T15:27:38.597Z] #5 DONE 0.1s [2024-01-18T15:27:38.597Z] [2024-01-18T15:27:38.597Z] #9 FROM docker.io/lachlanevenson/k8s-kubectl:latest [2024-01-18T15:27:38.597Z] #9 sha256:4cccb2da46f82b9f80ef995cefa1260e725cc135913b1becda047f214d621f2d [2024-01-18T15:27:38.597Z] #9 DONE 0.1s [2024-01-18T15:27:38.597Z] [2024-01-18T15:27:38.597Z] #21 [build 1/19] FROM docker.io/library/golang:1.19.5-alpine@sha256:a00a03c72ecc1a01e4ca7b7cfb62459063912ff45ccb13011f9fb061e25916eb [2024-01-18T15:27:38.597Z] #21 sha256:31d43f0ad39cc6cf7d49273d478fc2ba4f2f281540cfbbc615757b8d58e9b383 [2024-01-18T15:27:38.597Z] #21 sha256:f8e86a052e6c54e2970ebf965b570be1301a6ebc08e5fd9c7b171a3c57a74bff 5.11kB / 5.11kB done [2024-01-18T15:27:38.597Z] #21 sha256:63b65145d645c1250c391b2d16ebe53b3747c295ca8ba2fcb6b0cf064a4dc21c 0B / 3.37MB 0.1s [2024-01-18T15:27:38.597Z] #21 sha256:a2d21d5440ebff5aaaaeb115a003f7a4a3897f1866a87de95bc4a21436fc563c 0B / 284.82kB 0.1s [2024-01-18T15:27:38.597Z] #21 sha256:46eea44c656f6a945ac54680f3b7545ce75ea2a89fdb1ffdfd16f20ec3794fef 0B / 122.33MB 0.1s [2024-01-18T15:27:38.597Z] #21 sha256:a00a03c72ecc1a01e4ca7b7cfb62459063912ff45ccb13011f9fb061e25916eb 1.65kB / 1.65kB done [2024-01-18T15:27:38.597Z] #21 sha256:c82f81ed35e1b92a9ed49d2ee39b18dcee9f38864053c3d0d7c0ff491c08b4dc 1.16kB / 1.16kB done [2024-01-18T15:27:38.855Z] #21 sha256:a2d21d5440ebff5aaaaeb115a003f7a4a3897f1866a87de95bc4a21436fc563c 284.82kB / 284.82kB 0.3s done [2024-01-18T15:27:38.855Z] #21 sha256:f44e73cf898e4378b99153fdeb27594e3b34ed15c7b5a3ce2782a4f1323f4591 0B / 156B 0.3s [2024-01-18T15:27:39.111Z] #21 sha256:63b65145d645c1250c391b2d16ebe53b3747c295ca8ba2fcb6b0cf064a4dc21c 3.15MB / 3.37MB 0.4s [2024-01-18T15:27:39.112Z] #21 sha256:46eea44c656f6a945ac54680f3b7545ce75ea2a89fdb1ffdfd16f20ec3794fef 21.18MB / 122.33MB 0.4s [2024-01-18T15:27:39.112Z] #21 extracting sha256:63b65145d645c1250c391b2d16ebe53b3747c295ca8ba2fcb6b0cf064a4dc21c [2024-01-18T15:27:39.112Z] #21 sha256:63b65145d645c1250c391b2d16ebe53b3747c295ca8ba2fcb6b0cf064a4dc21c 3.37MB / 3.37MB 0.4s done [2024-01-18T15:27:39.112Z] #21 sha256:46eea44c656f6a945ac54680f3b7545ce75ea2a89fdb1ffdfd16f20ec3794fef 54.53MB / 122.33MB 0.6s [2024-01-18T15:27:39.112Z] #21 sha256:f44e73cf898e4378b99153fdeb27594e3b34ed15c7b5a3ce2782a4f1323f4591 156B / 156B 0.5s done [2024-01-18T15:27:39.381Z] #21 sha256:46eea44c656f6a945ac54680f3b7545ce75ea2a89fdb1ffdfd16f20ec3794fef 72.35MB / 122.33MB 0.7s [2024-01-18T15:27:39.381Z] #21 extracting sha256:63b65145d645c1250c391b2d16ebe53b3747c295ca8ba2fcb6b0cf064a4dc21c 0.2s done [2024-01-18T15:27:39.381Z] #21 extracting sha256:a2d21d5440ebff5aaaaeb115a003f7a4a3897f1866a87de95bc4a21436fc563c [2024-01-18T15:27:39.381Z] #21 sha256:46eea44c656f6a945ac54680f3b7545ce75ea2a89fdb1ffdfd16f20ec3794fef 91.23MB / 122.33MB 0.8s [2024-01-18T15:27:39.381Z] #21 sha256:46eea44c656f6a945ac54680f3b7545ce75ea2a89fdb1ffdfd16f20ec3794fef 110.10MB / 122.33MB 0.9s [2024-01-18T15:27:39.381Z] #21 extracting sha256:a2d21d5440ebff5aaaaeb115a003f7a4a3897f1866a87de95bc4a21436fc563c 0.1s done [2024-01-18T15:27:39.652Z] #21 sha256:46eea44c656f6a945ac54680f3b7545ce75ea2a89fdb1ffdfd16f20ec3794fef 122.33MB / 122.33MB 1.0s [2024-01-18T15:27:39.652Z] #21 sha256:46eea44c656f6a945ac54680f3b7545ce75ea2a89fdb1ffdfd16f20ec3794fef 122.33MB / 122.33MB 1.1s done [2024-01-18T15:27:39.910Z] #21 extracting sha256:46eea44c656f6a945ac54680f3b7545ce75ea2a89fdb1ffdfd16f20ec3794fef [2024-01-18T15:27:40.840Z] #21 ... [2024-01-18T15:27:40.840Z] [2024-01-18T15:27:40.840Z] #6 [stage-1 2/20] RUN apk add --no-cache ca-certificates bash curl jq libc6-compat [2024-01-18T15:27:40.841Z] #6 sha256:b0eb8a5cf2989e2d90047f6f0d3460c758d2e1c6e719698ae3b327ca4c21ab02 [2024-01-18T15:27:40.841Z] #6 0.449 fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz [2024-01-18T15:27:40.841Z] #6 0.700 fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz [2024-01-18T15:27:40.841Z] #6 1.014 (1/14) Installing ncurses-terminfo-base (6.4_p20230506-r0) [2024-01-18T15:27:40.841Z] #6 1.041 (2/14) Installing libncursesw (6.4_p20230506-r0) [2024-01-18T15:27:40.841Z] #6 1.068 (3/14) Installing readline (8.2.1-r1) [2024-01-18T15:27:40.841Z] #6 1.094 (4/14) Installing bash (5.2.15-r5) [2024-01-18T15:27:40.841Z] #6 1.133 Executing bash-5.2.15-r5.post-install [2024-01-18T15:27:40.841Z] #6 1.136 (5/14) Installing ca-certificates (20230506-r0) [2024-01-18T15:27:40.841Z] #6 1.176 (6/14) Installing brotli-libs (1.0.9-r14) [2024-01-18T15:27:40.841Z] #6 1.207 (7/14) Installing libunistring (1.1-r1) [2024-01-18T15:27:40.841Z] #6 1.247 (8/14) Installing libidn2 (2.3.4-r1) [2024-01-18T15:27:40.841Z] #6 1.271 (9/14) Installing nghttp2-libs (1.57.0-r0) [2024-01-18T15:27:40.841Z] #6 1.296 (10/14) Installing libcurl (8.5.0-r0) [2024-01-18T15:27:40.841Z] #6 1.325 (11/14) Installing curl (8.5.0-r0) [2024-01-18T15:27:40.841Z] #6 1.350 (12/14) Installing oniguruma (6.9.8-r1) [2024-01-18T15:27:40.841Z] #6 1.379 (13/14) Installing jq (1.6-r4) [2024-01-18T15:27:40.841Z] #6 1.409 (14/14) Installing libc6-compat (1.2.4-r2) [2024-01-18T15:27:40.841Z] #6 1.431 Executing busybox-1.36.1-r5.trigger [2024-01-18T15:27:40.841Z] #6 1.437 Executing ca-certificates-20230506-r0.trigger [2024-01-18T15:27:40.841Z] #6 1.481 OK: 15 MiB in 29 packages [2024-01-18T15:27:40.841Z] #6 DONE 2.0s [2024-01-18T15:27:40.841Z] [2024-01-18T15:27:40.841Z] #21 [build 1/19] FROM docker.io/library/golang:1.19.5-alpine@sha256:a00a03c72ecc1a01e4ca7b7cfb62459063912ff45ccb13011f9fb061e25916eb [2024-01-18T15:27:40.841Z] #21 sha256:31d43f0ad39cc6cf7d49273d478fc2ba4f2f281540cfbbc615757b8d58e9b383 [2024-01-18T15:27:43.374Z] #21 ... [2024-01-18T15:27:43.374Z] [2024-01-18T15:27:43.374Z] #14 [internal] load build context [2024-01-18T15:27:43.374Z] #14 sha256:8831202b575b05fb3d035ba43c92823a8c492debb58fc6cb70075b45a2a5daa9 [2024-01-18T15:27:43.374Z] #14 transferring context: 381.20MB 4.1s done [2024-01-18T15:27:43.374Z] #14 DONE 4.2s [2024-01-18T15:27:43.374Z] [2024-01-18T15:27:43.374Z] #21 [build 1/19] FROM docker.io/library/golang:1.19.5-alpine@sha256:a00a03c72ecc1a01e4ca7b7cfb62459063912ff45ccb13011f9fb061e25916eb [2024-01-18T15:27:43.374Z] #21 sha256:31d43f0ad39cc6cf7d49273d478fc2ba4f2f281540cfbbc615757b8d58e9b383 [2024-01-18T15:27:43.374Z] #21 ... [2024-01-18T15:27:43.374Z] [2024-01-18T15:27:43.374Z] #11 FROM docker.io/alpine/helm:latest [2024-01-18T15:27:43.374Z] #11 sha256:703f1939d88aa9cc7fecd46837ea25a7229b1c84a38c4db2037d9aa7becff6b1 [2024-01-18T15:27:43.374Z] #11 resolve docker.io/alpine/helm:latest 1.4s done [2024-01-18T15:27:43.374Z] #11 sha256:ec7cf28e98d87facdbe4dac11b8a462fc8df17bbc04545b614041fc643962ca2 26.80MB / 26.80MB 0.5s done [2024-01-18T15:27:43.374Z] #11 sha256:0ca7d6e06197b31d20a107d8c160bf02716746003aa40a9d510cf9cacc31ceac 94B / 94B 0.3s done [2024-01-18T15:27:43.374Z] #11 sha256:904fe876ba5eb93dea4c10ff12c44f19c81dd2ae81e61b60e50ea4863b4676c1 1.85kB / 1.85kB done [2024-01-18T15:27:43.374Z] #11 sha256:8d3394089d085310e44a910c1dc06fb29a3e2422ee1d03d67c87f15f7f9ac1c4 895B / 895B done [2024-01-18T15:27:43.374Z] #11 sha256:7cab771630838d8c9dda363f256cb9bc1ee3e9a06ef09cc3aa23cdff8584f833 2.16kB / 2.16kB done [2024-01-18T15:27:43.374Z] #11 sha256:661ff4d9561e3fd050929ee5097067c34bafc523ee60f5294a37fd08056a73ca 3.41MB / 3.41MB 0.3s done [2024-01-18T15:27:43.374Z] #11 extracting sha256:661ff4d9561e3fd050929ee5097067c34bafc523ee60f5294a37fd08056a73ca 0.3s done [2024-01-18T15:27:43.374Z] #11 extracting sha256:ec7cf28e98d87facdbe4dac11b8a462fc8df17bbc04545b614041fc643962ca2 1.9s done [2024-01-18T15:27:43.374Z] #11 extracting sha256:0ca7d6e06197b31d20a107d8c160bf02716746003aa40a9d510cf9cacc31ceac done [2024-01-18T15:27:43.374Z] #11 DONE 3.0s [2024-01-18T15:27:43.374Z] [2024-01-18T15:27:43.374Z] #21 [build 1/19] FROM docker.io/library/golang:1.19.5-alpine@sha256:a00a03c72ecc1a01e4ca7b7cfb62459063912ff45ccb13011f9fb061e25916eb [2024-01-18T15:27:43.374Z] #21 sha256:31d43f0ad39cc6cf7d49273d478fc2ba4f2f281540cfbbc615757b8d58e9b383 [2024-01-18T15:27:43.935Z] #21 ... [2024-01-18T15:27:43.935Z] [2024-01-18T15:27:43.935Z] #7 [stage-1 3/20] RUN apk add --no-cache --update python3 py3-pip [2024-01-18T15:27:43.935Z] #7 sha256:b7075a1b6a8f8b9533ea0c66c93435c81861399644c91495acfa85bdb25c95cf [2024-01-18T15:27:43.935Z] #7 0.513 fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz [2024-01-18T15:27:43.935Z] #7 0.808 fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz [2024-01-18T15:27:43.935Z] #7 1.199 (1/22) Installing libbz2 (1.0.8-r5) [2024-01-18T15:27:43.935Z] #7 1.229 (2/22) Installing libexpat (2.5.0-r1) [2024-01-18T15:27:43.935Z] #7 1.258 (3/22) Installing libffi (3.4.4-r2) [2024-01-18T15:27:43.935Z] #7 1.281 (4/22) Installing gdbm (1.23-r1) [2024-01-18T15:27:43.935Z] #7 1.308 (5/22) Installing xz-libs (5.4.3-r0) [2024-01-18T15:27:43.936Z] #7 1.341 (6/22) Installing libgcc (12.2.1_git20220924-r10) [2024-01-18T15:27:43.936Z] #7 1.369 (7/22) Installing libstdc++ (12.2.1_git20220924-r10) [2024-01-18T15:27:43.936Z] #7 1.424 (8/22) Installing mpdecimal (2.5.1-r2) [2024-01-18T15:27:43.936Z] #7 1.456 (9/22) Installing libpanelw (6.4_p20230506-r0) [2024-01-18T15:27:43.936Z] #7 1.489 (10/22) Installing sqlite-libs (3.41.2-r3) [2024-01-18T15:27:43.936Z] #7 1.524 (11/22) Installing python3 (3.11.6-r0) [2024-01-18T15:27:43.936Z] #7 1.852 (12/22) Installing python3-pycache-pyc0 (3.11.6-r0) [2024-01-18T15:27:43.936Z] #7 2.092 (13/22) Installing pyc (0.1-r0) [2024-01-18T15:27:43.936Z] #7 2.119 (14/22) Installing py3-setuptools-pyc (67.7.2-r0) [2024-01-18T15:27:43.936Z] #7 2.209 (15/22) Installing py3-pip-pyc (23.1.2-r0) [2024-01-18T15:27:43.936Z] #7 2.339 (16/22) Installing py3-parsing (3.0.9-r2) [2024-01-18T15:27:43.936Z] #7 2.371 (17/22) Installing py3-parsing-pyc (3.0.9-r2) [2024-01-18T15:27:43.936Z] #7 2.404 (18/22) Installing py3-packaging-pyc (23.1-r1) [2024-01-18T15:27:43.936Z] #7 2.435 (19/22) Installing python3-pyc (3.11.6-r0) [2024-01-18T15:27:43.936Z] #7 2.461 (20/22) Installing py3-packaging (23.1-r1) [2024-01-18T15:27:43.936Z] #7 2.491 (21/22) Installing py3-setuptools (67.7.2-r0) [2024-01-18T15:27:43.936Z] #7 2.569 (22/22) Installing py3-pip (23.1.2-r0) [2024-01-18T15:27:43.936Z] #7 2.693 Executing busybox-1.36.1-r5.trigger [2024-01-18T15:27:43.936Z] #7 2.702 OK: 82 MiB in 51 packages [2024-01-18T15:27:43.936Z] #7 DONE 3.2s [2024-01-18T15:27:43.936Z] [2024-01-18T15:27:43.936Z] #21 [build 1/19] FROM docker.io/library/golang:1.19.5-alpine@sha256:a00a03c72ecc1a01e4ca7b7cfb62459063912ff45ccb13011f9fb061e25916eb [2024-01-18T15:27:43.936Z] #21 sha256:31d43f0ad39cc6cf7d49273d478fc2ba4f2f281540cfbbc615757b8d58e9b383 [2024-01-18T15:27:44.861Z] #21 extracting sha256:46eea44c656f6a945ac54680f3b7545ce75ea2a89fdb1ffdfd16f20ec3794fef 5.1s [2024-01-18T15:27:47.380Z] #21 extracting sha256:46eea44c656f6a945ac54680f3b7545ce75ea2a89fdb1ffdfd16f20ec3794fef 7.2s done [2024-01-18T15:27:47.380Z] #21 extracting sha256:f44e73cf898e4378b99153fdeb27594e3b34ed15c7b5a3ce2782a4f1323f4591 [2024-01-18T15:27:47.380Z] #21 extracting sha256:f44e73cf898e4378b99153fdeb27594e3b34ed15c7b5a3ce2782a4f1323f4591 done [2024-01-18T15:27:47.380Z] #21 DONE 8.6s [2024-01-18T15:27:47.380Z] [2024-01-18T15:27:47.380Z] #22 [build 2/19] WORKDIR /go/src/github.com/portworx/torpedo [2024-01-18T15:27:47.380Z] #22 sha256:a5fe5ca821b9e292064b16a05be9b7cf3a0ef9b5007fa2dcb7a6bf080b0f48c3 [2024-01-18T15:27:47.637Z] #22 DONE 0.4s [2024-01-18T15:27:47.637Z] [2024-01-18T15:27:47.637Z] #8 [stage-1 4/20] RUN apk add --no-cache --update --virtual=build gcc musl-dev python3-dev libffi-dev openssl-dev cargo make && pip3 install "pyyaml<=5.3.1" && pip3 install --no-cache-dir --prefer-binary azure-cli && apk del build [2024-01-18T15:27:47.637Z] #8 sha256:4efb1b73fb1716746a3ed94677a3924261e9bd638976a6ba84c5bf7e7caeeffd [2024-01-18T15:27:47.637Z] #8 0.494 fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz [2024-01-18T15:27:47.637Z] #8 0.817 fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz [2024-01-18T15:27:47.637Z] #8 1.182 (1/23) Upgrading libcrypto3 (3.1.4-r1 -> 3.1.4-r4) [2024-01-18T15:27:47.637Z] #8 1.250 (2/23) Upgrading libssl3 (3.1.4-r1 -> 3.1.4-r4) [2024-01-18T15:27:47.637Z] #8 1.282 (3/23) Installing zstd-libs (1.5.5-r4) [2024-01-18T15:27:47.637Z] #8 1.316 (4/23) Installing binutils (2.40-r7) [2024-01-18T15:27:47.637Z] #8 1.412 (5/23) Installing libgomp (12.2.1_git20220924-r10) [2024-01-18T15:27:47.637Z] #8 1.441 (6/23) Installing libatomic (12.2.1_git20220924-r10) [2024-01-18T15:27:47.637Z] #8 1.468 (7/23) Installing gmp (6.2.1-r3) [2024-01-18T15:27:47.637Z] #8 1.499 (8/23) Installing isl26 (0.26-r1) [2024-01-18T15:27:47.637Z] #8 1.548 (9/23) Installing mpfr4 (4.2.0_p12-r0) [2024-01-18T15:27:47.637Z] #8 1.586 (10/23) Installing mpc1 (1.3.1-r1) [2024-01-18T15:27:47.637Z] #8 1.618 (11/23) Installing gcc (12.2.1_git20220924-r10) [2024-01-18T15:27:47.637Z] #8 2.602 (12/23) Installing musl-dev (1.2.4-r2) [2024-01-18T15:27:47.637Z] #8 2.705 (13/23) Installing pkgconf (1.9.5-r0) [2024-01-18T15:27:47.637Z] #8 2.734 (14/23) Installing python3-dev (3.11.6-r0) [2024-01-18T15:27:47.637Z] #8 3.248 (15/23) Installing linux-headers (6.3-r0) [2024-01-18T15:27:47.637Z] #8 3.395 (16/23) Installing libffi-dev (3.4.4-r2) [2024-01-18T15:27:47.637Z] #8 3.429 (17/23) Installing openssl-dev (3.1.4-r4) [2024-01-18T15:27:47.637Z] #8 3.477 (18/23) Installing libxml2 (2.11.6-r0) [2024-01-18T15:27:47.637Z] #8 3.514 (19/23) Installing llvm16-libs (16.0.6-r1) [2024-01-18T15:27:48.583Z] #8 4.513 (20/23) Installing rust (1.71.1-r0) [2024-01-18T15:27:51.176Z] #8 ... [2024-01-18T15:27:51.176Z] [2024-01-18T15:27:51.176Z] #23 [build 3/19] RUN apk update && apk add --no-cache bash git gcc musl-dev make curl openssh-client [2024-01-18T15:27:51.176Z] #23 sha256:b4eba0938bcefc8630acfb162ad7570801a1d99ac4ff3f947b0f1808cbeb04a2 [2024-01-18T15:27:51.176Z] #23 0.242 fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/x86_64/APKINDEX.tar.gz [2024-01-18T15:27:51.176Z] #23 0.688 fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/x86_64/APKINDEX.tar.gz [2024-01-18T15:27:51.176Z] #23 1.164 v3.17.6-70-gdf4d5956476 [https://dl-cdn.alpinelinux.org/alpine/v3.17/main] [2024-01-18T15:27:51.176Z] #23 1.164 v3.17.6-69-g91aa3febc78 [https://dl-cdn.alpinelinux.org/alpine/v3.17/community] [2024-01-18T15:27:51.176Z] #23 1.164 OK: 17825 distinct packages available [2024-01-18T15:27:51.176Z] #23 1.222 fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/x86_64/APKINDEX.tar.gz [2024-01-18T15:27:51.176Z] #23 1.517 fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/x86_64/APKINDEX.tar.gz [2024-01-18T15:27:51.176Z] #23 1.819 (1/28) Upgrading musl (1.2.3-r4 -> 1.2.3-r5) [2024-01-18T15:27:51.176Z] #23 1.854 (2/28) Installing ncurses-terminfo-base (6.3_p20221119-r1) [2024-01-18T15:27:51.176Z] #23 1.885 (3/28) Installing ncurses-libs (6.3_p20221119-r1) [2024-01-18T15:27:51.176Z] #23 1.920 (4/28) Installing readline (8.2.0-r0) [2024-01-18T15:27:51.176Z] #23 1.950 (5/28) Installing bash (5.2.15-r0) [2024-01-18T15:27:51.176Z] #23 1.991 Executing bash-5.2.15-r0.post-install [2024-01-18T15:27:51.176Z] #23 1.994 (6/28) Installing brotli-libs (1.0.9-r9) [2024-01-18T15:27:51.176Z] #23 2.030 (7/28) Installing nghttp2-libs (1.51.0-r2) [2024-01-18T15:27:51.176Z] #23 2.059 (8/28) Installing libcurl (8.5.0-r0) [2024-01-18T15:27:51.176Z] #23 2.093 (9/28) Installing curl (8.5.0-r0) [2024-01-18T15:27:51.176Z] #23 2.124 (10/28) Installing libgcc (12.2.1_git20220924-r4) [2024-01-18T15:27:51.176Z] #23 2.152 (11/28) Installing libstdc++ (12.2.1_git20220924-r4) [2024-01-18T15:27:51.176Z] #23 2.198 (12/28) Installing binutils (2.39-r2) [2024-01-18T15:27:51.176Z] #23 2.301 (13/28) Installing libgomp (12.2.1_git20220924-r4) [2024-01-18T15:27:51.176Z] #23 2.332 (14/28) Installing libatomic (12.2.1_git20220924-r4) [2024-01-18T15:27:51.176Z] #23 2.361 (15/28) Installing gmp (6.2.1-r2) [2024-01-18T15:27:51.176Z] #23 2.394 (16/28) Installing isl25 (0.25-r1) [2024-01-18T15:27:51.176Z] #23 2.439 (17/28) Installing mpfr4 (4.1.0-r0) [2024-01-18T15:27:51.176Z] #23 2.490 (18/28) Installing mpc1 (1.2.1-r1) [2024-01-18T15:27:51.176Z] #23 2.518 (19/28) Installing gcc (12.2.1_git20220924-r4) [2024-01-18T15:27:51.176Z] #23 3.500 (20/28) Installing libexpat (2.5.0-r0) [2024-01-18T15:27:51.176Z] #23 3.539 (21/28) Installing pcre2 (10.42-r0) [2024-01-18T15:27:51.176Z] #23 3.589 (22/28) Installing git (2.38.5-r0) [2024-01-18T15:27:51.441Z] #23 3.700 (23/28) Installing make (4.3-r1) [2024-01-18T15:27:51.441Z] #23 3.729 (24/28) Installing musl-dev (1.2.3-r5) [2024-01-18T15:27:51.441Z] #23 3.834 (25/28) Installing openssh-keygen (9.1_p1-r5) [2024-01-18T15:27:51.441Z] #23 3.866 (26/28) Installing libedit (20221030.3.1-r0) [2024-01-18T15:27:51.441Z] #23 3.902 (27/28) Installing openssh-client-common (9.1_p1-r5) [2024-01-18T15:27:51.698Z] #23 3.952 (28/28) Installing openssh-client-default (9.1_p1-r5) [2024-01-18T15:27:51.698Z] #23 3.988 Executing busybox-1.35.0-r29.trigger [2024-01-18T15:27:51.698Z] #23 3.995 OK: 168 MiB in 43 packages [2024-01-18T15:27:52.636Z] #23 DONE 4.8s [2024-01-18T15:27:52.636Z] [2024-01-18T15:27:52.636Z] #24 [build 4/19] RUN GOFLAGS= GO111MODULE=on go install github.com/onsi/ginkgo/ginkgo@v1.16.5 [2024-01-18T15:27:52.636Z] #24 sha256:8b85fd97fb03810c979e0fe3c20d044bf4a3740bc13b5c2a3f4205a8e22efe04 [2024-01-18T15:27:53.212Z] #24 0.566 go: downloading github.com/onsi/ginkgo v1.16.5 [2024-01-18T15:27:56.536Z] #24 4.130 go: downloading github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0 [2024-01-18T15:27:56.793Z] #24 4.142 go: downloading golang.org/x/tools v0.0.0-20201224043029-2b0845dc783e [2024-01-18T15:27:56.793Z] #24 4.157 go: downloading github.com/nxadm/tail v1.4.8 [2024-01-18T15:27:56.793Z] #24 4.157 go: downloading golang.org/x/sys v0.0.0-20210112080510-489259a85091 [2024-01-18T15:27:56.793Z] #24 4.321 go: downloading github.com/fsnotify/fsnotify v1.4.9 [2024-01-18T15:27:56.793Z] #24 4.321 go: downloading gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 [2024-01-18T15:27:59.305Z] #24 ... [2024-01-18T15:27:59.305Z] [2024-01-18T15:27:59.305Z] #8 [stage-1 4/20] RUN apk add --no-cache --update --virtual=build gcc musl-dev python3-dev libffi-dev openssl-dev cargo make && pip3 install "pyyaml<=5.3.1" && pip3 install --no-cache-dir --prefer-binary azure-cli && apk del build [2024-01-18T15:27:59.305Z] #8 sha256:4efb1b73fb1716746a3ed94677a3924261e9bd638976a6ba84c5bf7e7caeeffd [2024-01-18T15:27:59.305Z] #8 9.019 (21/23) Installing cargo (1.71.1-r0) [2024-01-18T15:27:59.305Z] #8 9.175 (22/23) Installing make (4.4.1-r1) [2024-01-18T15:27:59.305Z] #8 9.205 (23/23) Installing build (20240118.152649) [2024-01-18T15:27:59.305Z] #8 9.209 Executing busybox-1.36.1-r5.trigger [2024-01-18T15:27:59.305Z] #8 9.216 Executing ca-certificates-20230506-r0.trigger [2024-01-18T15:27:59.305Z] #8 9.270 OK: 1037 MiB in 72 packages [2024-01-18T15:27:59.305Z] #8 11.13 Collecting pyyaml<=5.3.1 [2024-01-18T15:27:59.305Z] #8 11.23 Downloading PyYAML-5.3.1.tar.gz (269 kB) [2024-01-18T15:27:59.305Z] #8 11.28 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 269.4/269.4 kB 5.4 MB/s eta 0:00:00 [2024-01-18T15:27:59.305Z] #8 11.46 Installing build dependencies: started [2024-01-18T15:27:59.305Z] #8 14.26 Installing build dependencies: finished with status 'done' [2024-01-18T15:27:59.305Z] #8 14.26 Getting requirements to build wheel: started [2024-01-18T15:27:59.305Z] #8 14.54 Getting requirements to build wheel: finished with status 'done' [2024-01-18T15:27:59.305Z] #8 14.54 Preparing metadata (pyproject.toml): started [2024-01-18T15:27:59.305Z] #8 14.86 Preparing metadata (pyproject.toml): finished with status 'done' [2024-01-18T15:27:59.305Z] #8 14.86 Building wheels for collected packages: pyyaml [2024-01-18T15:27:59.305Z] #8 14.86 Building wheel for pyyaml (pyproject.toml): started [2024-01-18T15:27:59.305Z] #8 15.31 Building wheel for pyyaml (pyproject.toml): finished with status 'done' [2024-01-18T15:27:59.305Z] #8 15.32 Created wheel for pyyaml: filename=PyYAML-5.3.1-cp311-cp311-linux_x86_64.whl size=44636 sha256=a4bf452cd8ee6930b867592b657d9cc33105c998d23256dccd132c076e70bea8 [2024-01-18T15:27:59.305Z] #8 15.32 Stored in directory: /****/.cache/pip/wheels/93/d0/25/0923993e11f9273a21be3d272ca3dfe97c757a255a55c56bad [2024-01-18T15:27:59.305Z] #8 15.32 Successfully built pyyaml [2024-01-18T15:27:59.305Z] #8 15.38 Installing collected packages: pyyaml [2024-01-18T15:27:59.305Z] #8 15.46 Successfully installed pyyaml-5.3.1 [2024-01-18T15:27:59.305Z] #8 15.46 WARNING: Running pip as the **** user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [2024-01-18T15:27:59.866Z] #8 ... [2024-01-18T15:27:59.866Z] [2024-01-18T15:27:59.866Z] #24 [build 4/19] RUN GOFLAGS= GO111MODULE=on go install github.com/onsi/ginkgo/ginkgo@v1.16.5 [2024-01-18T15:27:59.866Z] #24 sha256:8b85fd97fb03810c979e0fe3c20d044bf4a3740bc13b5c2a3f4205a8e22efe04 [2024-01-18T15:27:59.866Z] #24 DONE 7.3s [2024-01-18T15:27:59.866Z] [2024-01-18T15:27:59.866Z] #25 [build 5/19] RUN mkdir bin && curl -o aws-iam-authenticator https://amazon-eks.s3.us-west-2.amazonaws.com/1.16.8/2020-04-16/bin/linux/amd64/aws-iam-authenticator && chmod a+x aws-iam-authenticator && mv aws-iam-authenticator bin [2024-01-18T15:27:59.866Z] #25 sha256:bec8c4b6e9731c3a0110c135a636ca0b284084443e17d2fa78a6026898457aa2 [2024-01-18T15:28:00.121Z] #25 0.356 % Total % Received % Xferd Average Speed Time Time Time Current [2024-01-18T15:28:00.376Z] #25 0.357 Dload Upload Total Spent Left Speed [2024-01-18T15:28:05.635Z] #25 0.357 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 1 33.6M 1 360k 0 0 493k 0 0:01:09 --:--:-- 0:01:09 494k 4 33.6M 4 1380k 0 0 802k 0 0:00:42 0:00:01 0:00:41 803k 8 33.6M 8 2859k 0 0 1053k 0 0:00:32 0:00:02 0:00:30 1054k 13 33.6M 13 4797k 0 0 1294k 0 0:00:26 0:00:03 0:00:23 1295k 21 33.6M 21 7279k 0 0 1547k 0 0:00:22 0:00:04 0:00:18 1547k [2024-01-18T15:28:05.635Z] #25 ... [2024-01-18T15:28:05.635Z] [2024-01-18T15:28:05.635Z] #8 [stage-1 4/20] RUN apk add --no-cache --update --virtual=build gcc musl-dev python3-dev libffi-dev openssl-dev cargo make && pip3 install "pyyaml<=5.3.1" && pip3 install --no-cache-dir --prefer-binary azure-cli && apk del build [2024-01-18T15:28:05.635Z] #8 sha256:4efb1b73fb1716746a3ed94677a3924261e9bd638976a6ba84c5bf7e7caeeffd [2024-01-18T15:28:05.635Z] #8 16.33 Collecting azure-cli [2024-01-18T15:28:05.635Z] #8 16.45 Downloading azure_cli-2.56.0-py3-none-any.whl (13.4 MB) [2024-01-18T15:28:05.635Z] #8 16.69 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.4/13.4 MB 120.7 MB/s eta 0:00:00 [2024-01-18T15:28:05.635Z] #8 16.91 Collecting antlr4-python3-runtime~=4.13.1 (from azure-cli) [2024-01-18T15:28:05.635Z] #8 16.94 Downloading antlr4_python3_runtime-4.13.1-py3-none-any.whl (144 kB) [2024-01-18T15:28:05.635Z] #8 16.94 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 144.5/144.5 kB 207.3 MB/s eta 0:00:00 [2024-01-18T15:28:05.635Z] #8 17.00 Collecting azure-appconfiguration~=1.1.1 (from azure-cli) [2024-01-18T15:28:05.635Z] #8 17.03 Downloading azure_appconfiguration-1.1.1-py2.py3-none-any.whl (37 kB) [2024-01-18T15:28:05.635Z] #8 17.06 Collecting azure-batch~=14.0.0 (from azure-cli) [2024-01-18T15:28:05.635Z] #8 17.09 Downloading azure_batch-14.0.0-py3-none-any.whl (234 kB) [2024-01-18T15:28:05.635Z] #8 17.09 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 234.1/234.1 kB 203.8 MB/s eta 0:00:00 [2024-01-18T15:28:05.635Z] #8 17.18 Collecting azure-cli-core==2.56.0 (from azure-cli) [2024-01-18T15:28:05.635Z] #8 17.21 Downloading azure_cli_core-2.56.0-py3-none-any.whl (245 kB) [2024-01-18T15:28:05.635Z] #8 17.21 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 245.9/245.9 kB 157.8 MB/s eta 0:00:00 [2024-01-18T15:28:05.635Z] #8 17.25 Collecting azure-cosmos>=3.0.2,~=3.0 (from azure-cli) [2024-01-18T15:28:05.635Z] #8 17.28 Downloading azure_cosmos-3.2.0-py2.py3-none-any.whl (106 kB) [2024-01-18T15:28:05.635Z] #8 17.29 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 106.6/106.6 kB 135.1 MB/s eta 0:00:00 [2024-01-18T15:28:05.635Z] #8 17.32 Collecting azure-data-tables==12.4.0 (from azure-cli) [2024-01-18T15:28:05.635Z] #8 17.35 Downloading azure_data_tables-12.4.0-py3-none-any.whl (113 kB) [2024-01-18T15:28:05.635Z] #8 17.35 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 113.9/113.9 kB 144.2 MB/s eta 0:00:00 [2024-01-18T15:28:05.636Z] #8 17.39 Collecting azure-datalake-store~=0.0.49 (from azure-cli) [2024-01-18T15:28:05.636Z] #8 17.42 Downloading azure_datalake_store-0.0.53-py2.py3-none-any.whl (55 kB) [2024-01-18T15:28:05.636Z] #8 17.42 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.3/55.3 kB 135.5 MB/s eta 0:00:00 [2024-01-18T15:28:05.636Z] #8 17.46 Collecting azure-graphrbac~=0.60.0 (from azure-cli) [2024-01-18T15:28:05.636Z] #8 17.49 Downloading azure_graphrbac-0.60.0-py2.py3-none-any.whl (139 kB) [2024-01-18T15:28:05.636Z] #8 17.49 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.7/139.7 kB 127.7 MB/s eta 0:00:00 [2024-01-18T15:28:05.636Z] #8 17.52 Collecting azure-keyvault-administration==4.4.0b2 (from azure-cli) [2024-01-18T15:28:05.636Z] #8 17.55 Downloading azure_keyvault_administration-4.4.0b2-py3-none-any.whl (105 kB) [2024-01-18T15:28:05.636Z] #8 17.55 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 105.1/105.1 kB 130.3 MB/s eta 0:00:00 [2024-01-18T15:28:05.636Z] #8 17.58 Collecting azure-keyvault-certificates==4.7.0 (from azure-cli) [2024-01-18T15:28:05.636Z] #8 17.61 Downloading azure_keyvault_certificates-4.7.0-py3-none-any.whl (428 kB) [2024-01-18T15:28:05.636Z] #8 17.62 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 428.1/428.1 kB 173.9 MB/s eta 0:00:00 [2024-01-18T15:28:05.636Z] #8 17.65 Collecting azure-keyvault-keys==4.9.0b3 (from azure-cli) [2024-01-18T15:28:05.636Z] #8 17.68 Downloading azure_keyvault_keys-4.9.0b3-py3-none-any.whl (149 kB) [2024-01-18T15:28:05.636Z] #8 17.69 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 149.5/149.5 kB 145.4 MB/s eta 0:00:00 [2024-01-18T15:28:05.636Z] #8 17.73 Collecting azure-keyvault-secrets==4.7.0 (from azure-cli) [2024-01-18T15:28:05.636Z] #8 17.76 Downloading azure_keyvault_secrets-4.7.0-py3-none-any.whl (348 kB) [2024-01-18T15:28:05.636Z] #8 17.77 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 348.6/348.6 kB 148.7 MB/s eta 0:00:00 [2024-01-18T15:28:05.636Z] #8 17.80 Collecting azure-loganalytics~=0.1.0 (from azure-cli) [2024-01-18T15:28:05.636Z] #8 17.83 Downloading azure_loganalytics-0.1.1-py2.py3-none-any.whl (16 kB) [2024-01-18T15:28:05.636Z] #8 17.85 Collecting azure-mgmt-advisor==9.0.0 (from azure-cli) [2024-01-18T15:28:05.636Z] #8 17.88 Downloading azure_mgmt_advisor-9.0.0-py2.py3-none-any.whl (46 kB) [2024-01-18T15:28:05.636Z] #8 17.89 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.9/46.9 kB 118.1 MB/s eta 0:00:00 [2024-01-18T15:28:05.636Z] #8 17.91 Collecting azure-mgmt-apimanagement==4.0.0 (from azure-cli) [2024-01-18T15:28:05.636Z] #8 17.94 Downloading azure_mgmt_apimanagement-4.0.0-py3-none-any.whl (804 kB) [2024-01-18T15:28:05.636Z] #8 17.95 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 804.5/804.5 kB 149.8 MB/s eta 0:00:00 [2024-01-18T15:28:05.636Z] #8 18.00 Collecting azure-mgmt-appconfiguration==3.0.0 (from azure-cli) [2024-01-18T15:28:05.636Z] #8 18.03 Downloading azure_mgmt_appconfiguration-3.0.0-py3-none-any.whl (252 kB) [2024-01-18T15:28:05.636Z] #8 18.03 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 252.0/252.0 kB 169.3 MB/s eta 0:00:00 [2024-01-18T15:28:05.636Z] #8 18.06 Collecting azure-mgmt-appcontainers==2.0.0 (from azure-cli) [2024-01-18T15:28:05.637Z] #8 18.09 Downloading azure_mgmt_appcontainers-2.0.0-py3-none-any.whl (214 kB) [2024-01-18T15:28:05.637Z] #8 18.09 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 214.1/214.1 kB 161.9 MB/s eta 0:00:00 [2024-01-18T15:28:05.637Z] #8 18.12 Collecting azure-mgmt-applicationinsights~=1.0.0 (from azure-cli) [2024-01-18T15:28:05.637Z] #8 18.15 Downloading azure_mgmt_applicationinsights-1.0.0-py2.py3-none-any.whl (302 kB) [2024-01-18T15:28:05.637Z] #8 18.16 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 303.0/303.0 kB 172.0 MB/s eta 0:00:00 [2024-01-18T15:28:05.637Z] #8 18.19 Collecting azure-mgmt-authorization~=4.0.0 (from azure-cli) [2024-01-18T15:28:05.638Z] #8 18.22 Downloading azure_mgmt_authorization-4.0.0-py3-none-any.whl (1.1 MB) [2024-01-18T15:28:05.638Z] #8 18.23 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 178.3 MB/s eta 0:00:00 [2024-01-18T15:28:05.638Z] #8 18.27 Collecting azure-mgmt-batchai==7.0.0b1 (from azure-cli) [2024-01-18T15:28:05.638Z] #8 18.31 Downloading azure_mgmt_batchai-7.0.0b1-py2.py3-none-any.whl (99 kB) [2024-01-18T15:28:05.638Z] #8 18.31 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 99.1/99.1 kB 156.5 MB/s eta 0:00:00 [2024-01-18T15:28:05.638Z] #8 18.34 Collecting azure-mgmt-batch~=17.0.0 (from azure-cli) [2024-01-18T15:28:05.638Z] #8 18.37 Downloading azure_mgmt_batch-17.0.0-py3-none-any.whl (157 kB) [2024-01-18T15:28:05.638Z] #8 18.38 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 157.5/157.5 kB 163.6 MB/s eta 0:00:00 [2024-01-18T15:28:05.638Z] #8 18.40 Collecting azure-mgmt-billing==6.0.0 (from azure-cli) [2024-01-18T15:28:05.638Z] #8 18.43 Downloading azure_mgmt_billing-6.0.0-py2.py3-none-any.whl (166 kB) [2024-01-18T15:28:05.638Z] #8 18.44 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 167.0/167.0 kB 160.4 MB/s eta 0:00:00 [2024-01-18T15:28:05.638Z] #8 18.47 Collecting azure-mgmt-botservice~=2.0.0b3 (from azure-cli) [2024-01-18T15:28:05.638Z] #8 18.49 Downloading azure_mgmt_botservice-2.0.0-py3-none-any.whl (126 kB) [2024-01-18T15:28:05.638Z] #8 18.50 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 126.5/126.5 kB 151.5 MB/s eta 0:00:00 [2024-01-18T15:28:05.638Z] #8 18.53 Collecting azure-mgmt-cdn==12.0.0 (from azure-cli) [2024-01-18T15:28:05.638Z] #8 18.57 Downloading azure_mgmt_cdn-12.0.0-py3-none-any.whl (239 kB) [2024-01-18T15:28:05.638Z] #8 18.58 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 239.4/239.4 kB 140.8 MB/s eta 0:00:00 [2024-01-18T15:28:05.638Z] #8 18.62 Collecting azure-mgmt-cognitiveservices~=13.5.0 (from azure-cli) [2024-01-18T15:28:05.638Z] #8 18.65 Downloading azure_mgmt_cognitiveservices-13.5.0-py3-none-any.whl (144 kB) [2024-01-18T15:28:05.638Z] #8 18.65 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 144.3/144.3 kB 129.2 MB/s eta 0:00:00 [2024-01-18T15:28:05.638Z] #8 18.70 Collecting azure-mgmt-compute~=30.4.0 (from azure-cli) [2024-01-18T15:28:05.638Z] #8 18.74 Downloading azure_mgmt_compute-30.4.0-py3-none-any.whl (5.6 MB) [2024-01-18T15:28:05.638Z] #8 18.79 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.6/5.6 MB 109.0 MB/s eta 0:00:00 [2024-01-18T15:28:05.638Z] #8 18.85 Collecting azure-mgmt-containerinstance==10.1.0 (from azure-cli) [2024-01-18T15:28:05.638Z] #8 18.88 Downloading azure_mgmt_containerinstance-10.1.0-py3-none-any.whl (87 kB) [2024-01-18T15:28:05.639Z] #8 18.88 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 87.3/87.3 kB 148.7 MB/s eta 0:00:00 [2024-01-18T15:28:05.639Z] #8 18.92 Collecting azure-mgmt-containerregistry==10.1.0 (from azure-cli) [2024-01-18T15:28:05.639Z] #8 18.95 Downloading azure_mgmt_containerregistry-10.1.0-py3-none-any.whl (1.7 MB) [2024-01-18T15:28:05.639Z] #8 18.96 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.7/1.7 MB 167.7 MB/s eta 0:00:00 [2024-01-18T15:28:05.639Z] #8 19.02 Collecting azure-mgmt-containerservice~=28.0.0 (from azure-cli) [2024-01-18T15:28:05.639Z] #8 19.05 Downloading azure_mgmt_containerservice-28.0.0-py3-none-any.whl (8.3 MB) [2024-01-18T15:28:05.639Z] #8 19.13 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.3/8.3 MB 107.5 MB/s eta 0:00:00 [2024-01-18T15:28:05.639Z] #8 19.22 Collecting azure-mgmt-cosmosdb==9.4.0 (from azure-cli) [2024-01-18T15:28:05.639Z] #8 19.25 Downloading azure_mgmt_cosmosdb-9.4.0-py3-none-any.whl (400 kB) [2024-01-18T15:28:05.639Z] #8 19.26 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 401.0/401.0 kB 158.4 MB/s eta 0:00:00 [2024-01-18T15:28:05.639Z] #8 19.29 Collecting azure-mgmt-databoxedge~=1.0.0 (from azure-cli) [2024-01-18T15:28:05.639Z] #8 19.32 Downloading azure_mgmt_databoxedge-1.0.0-py2.py3-none-any.whl (1.3 MB) [2024-01-18T15:28:05.639Z] #8 19.33 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 161.7 MB/s eta 0:00:00 [2024-01-18T15:28:05.639Z] #8 19.37 Collecting azure-mgmt-datalake-store~=0.5.0 (from azure-cli) [2024-01-18T15:28:05.639Z] #8 19.39 Downloading azure_mgmt_datalake_store-0.5.0-py2.py3-none-any.whl (88 kB) [2024-01-18T15:28:05.639Z] #8 19.40 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 88.7/88.7 kB 118.8 MB/s eta 0:00:00 [2024-01-18T15:28:05.639Z] #8 19.44 Collecting azure-mgmt-datamigration~=10.0.0 (from azure-cli) [2024-01-18T15:28:05.639Z] #8 19.47 Downloading azure_mgmt_datamigration-10.0.0-py2.py3-none-any.whl (174 kB) [2024-01-18T15:28:05.639Z] #8 19.47 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 174.5/174.5 kB 202.1 MB/s eta 0:00:00 [2024-01-18T15:28:05.639Z] #8 19.50 Collecting azure-mgmt-devtestlabs~=4.0 (from azure-cli) [2024-01-18T15:28:05.639Z] #8 19.53 Downloading azure_mgmt_devtestlabs-4.0.0-py2.py3-none-any.whl (137 kB) [2024-01-18T15:28:05.639Z] #8 19.53 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 137.0/137.0 kB 196.9 MB/s eta 0:00:00 [2024-01-18T15:28:05.639Z] #8 19.58 Collecting azure-mgmt-dns~=8.0.0 (from azure-cli) [2024-01-18T15:28:05.639Z] #8 19.61 Downloading azure_mgmt_dns-8.0.0-py2.py3-none-any.whl (118 kB) [2024-01-18T15:28:05.639Z] #8 19.61 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.9/118.9 kB 202.6 MB/s eta 0:00:00 [2024-01-18T15:28:05.639Z] #8 19.66 Collecting azure-mgmt-eventgrid==10.2.0b2 (from azure-cli) [2024-01-18T15:28:05.639Z] #8 19.69 Downloading azure_mgmt_eventgrid-10.2.0b2-py3-none-any.whl (248 kB) [2024-01-18T15:28:05.640Z] #8 19.69 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 248.5/248.5 kB 213.1 MB/s eta 0:00:00 [2024-01-18T15:28:05.640Z] #8 19.72 Collecting azure-mgmt-eventhub~=10.1.0 (from azure-cli) [2024-01-18T15:28:05.640Z] #8 19.76 Downloading azure_mgmt_eventhub-10.1.0-py3-none-any.whl (598 kB) [2024-01-18T15:28:05.640Z] #8 19.79 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 598.9/598.9 kB 202.9 MB/s eta 0:00:00 [2024-01-18T15:28:05.640Z] #8 19.82 Collecting azure-mgmt-extendedlocation==1.0.0b2 (from azure-cli) [2024-01-18T15:28:05.640Z] #8 19.85 Downloading azure_mgmt_extendedlocation-1.0.0b2-py2.py3-none-any.whl (37 kB) [2024-01-18T15:28:05.640Z] #8 19.88 Collecting azure-mgmt-hdinsight~=9.0.0 (from azure-cli) [2024-01-18T15:28:05.640Z] #8 19.91 Downloading azure_mgmt_hdinsight-9.0.0-py2.py3-none-any.whl (125 kB) [2024-01-18T15:28:05.640Z] #8 19.92 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 125.7/125.7 kB 223.9 MB/s eta 0:00:00 [2024-01-18T15:28:05.640Z] #8 19.94 Collecting azure-mgmt-imagebuilder~=1.2.0 (from azure-cli) [2024-01-18T15:28:05.640Z] #8 19.97 Downloading azure_mgmt_imagebuilder-1.2.0-py3-none-any.whl (78 kB) [2024-01-18T15:28:05.640Z] #8 19.97 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.1/78.1 kB 199.7 MB/s eta 0:00:00 [2024-01-18T15:28:05.640Z] #8 20.00 Collecting azure-mgmt-iotcentral~=10.0.0b1 (from azure-cli) [2024-01-18T15:28:05.640Z] #8 20.04 Downloading azure_mgmt_iotcentral-10.0.0b2-py3-none-any.whl (55 kB) [2024-01-18T15:28:05.640Z] #8 20.04 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.4/55.4 kB 140.0 MB/s eta 0:00:00 [2024-01-18T15:28:05.640Z] #8 20.07 Collecting azure-mgmt-iothub==3.0.0 (from azure-cli) [2024-01-18T15:28:05.640Z] #8 20.10 Downloading azure_mgmt_iothub-3.0.0-py3-none-any.whl (1.3 MB) [2024-01-18T15:28:05.640Z] #8 20.11 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 166.7 MB/s eta 0:00:00 [2024-01-18T15:28:05.640Z] #8 20.16 Collecting azure-mgmt-iothubprovisioningservices==1.1.0 (from azure-cli) [2024-01-18T15:28:05.640Z] #8 20.19 Downloading azure_mgmt_iothubprovisioningservices-1.1.0-py3-none-any.whl (52 kB) [2024-01-18T15:28:05.640Z] #8 20.19 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 52.5/52.5 kB 121.0 MB/s eta 0:00:00 [2024-01-18T15:28:05.640Z] #8 20.23 Collecting azure-mgmt-keyvault==10.3.0 (from azure-cli) [2024-01-18T15:28:05.640Z] #8 20.26 Downloading azure_mgmt_keyvault-10.3.0-py3-none-any.whl (933 kB) [2024-01-18T15:28:05.640Z] #8 20.27 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 933.0/933.0 kB 156.9 MB/s eta 0:00:00 [2024-01-18T15:28:05.640Z] #8 20.30 Collecting azure-mgmt-kusto~=0.3.0 (from azure-cli) [2024-01-18T15:28:05.640Z] #8 20.33 Downloading azure_mgmt_kusto-0.3.0-py2.py3-none-any.whl (73 kB) [2024-01-18T15:28:05.640Z] #8 20.34 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 73.9/73.9 kB 141.8 MB/s eta 0:00:00 [2024-01-18T15:28:05.640Z] #8 20.37 Collecting azure-mgmt-loganalytics==13.0.0b4 (from azure-cli) [2024-01-18T15:28:05.640Z] #8 20.40 Downloading azure_mgmt_loganalytics-13.0.0b4-py3-none-any.whl (162 kB) [2024-01-18T15:28:05.640Z] #8 20.40 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 162.6/162.6 kB 170.3 MB/s eta 0:00:00 [2024-01-18T15:28:05.640Z] #8 20.43 Collecting azure-mgmt-managedservices~=1.0 (from azure-cli) [2024-01-18T15:28:05.640Z] #8 20.45 Downloading azure_mgmt_managedservices-1.0.0-py2.py3-none-any.whl (38 kB) [2024-01-18T15:28:05.640Z] #8 20.48 Collecting azure-mgmt-managementgroups~=1.0.0 (from azure-cli) [2024-01-18T15:28:05.640Z] #8 20.51 Downloading azure_mgmt_managementgroups-1.0.0-py2.py3-none-any.whl (58 kB) [2024-01-18T15:28:05.640Z] #8 20.51 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.1/58.1 kB 148.6 MB/s eta 0:00:00 [2024-01-18T15:28:05.641Z] #8 20.53 Collecting azure-mgmt-maps~=2.0.0 (from azure-cli) [2024-01-18T15:28:05.641Z] #8 20.56 Downloading azure_mgmt_maps-2.0.0-py2.py3-none-any.whl (38 kB) [2024-01-18T15:28:05.641Z] #8 20.58 Collecting azure-mgmt-marketplaceordering==1.1.0 (from azure-cli) [2024-01-18T15:28:05.641Z] #8 20.62 Downloading azure_mgmt_marketplaceordering-1.1.0-py2.py3-none-any.whl (26 kB) [2024-01-18T15:28:05.641Z] #8 20.65 Collecting azure-mgmt-media~=9.0 (from azure-cli) [2024-01-18T15:28:05.641Z] #8 20.68 Downloading azure_mgmt_media-9.0.0-py3-none-any.whl (211 kB) [2024-01-18T15:28:05.641Z] #8 20.68 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 211.1/211.1 kB 147.3 MB/s eta 0:00:00 [2024-01-18T15:28:05.641Z] #8 20.72 Collecting azure-mgmt-monitor~=5.0.0 (from azure-cli) [2024-01-18T15:28:05.641Z] #8 20.75 Downloading azure_mgmt_monitor-5.0.1-py3-none-any.whl (1.1 MB) [2024-01-18T15:28:05.641Z] #8 20.76 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 154.0 MB/s eta 0:00:00 [2024-01-18T15:28:05.641Z] #8 20.80 Collecting azure-mgmt-msi~=7.0.0 (from azure-cli) [2024-01-18T15:28:05.641Z] #8 20.83 Downloading azure_mgmt_msi-7.0.0-py3-none-any.whl (183 kB) [2024-01-18T15:28:05.641Z] #8 20.83 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 183.4/183.4 kB 153.7 MB/s eta 0:00:00 [2024-01-18T15:28:05.641Z] #8 20.87 Collecting azure-mgmt-netapp~=10.1.0 (from azure-cli) [2024-01-18T15:28:05.641Z] #8 20.90 Downloading azure_mgmt_netapp-10.1.0-py3-none-any.whl (200 kB) [2024-01-18T15:28:05.641Z] #8 20.90 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 200.7/200.7 kB 113.3 MB/s eta 0:00:00 [2024-01-18T15:28:05.641Z] #8 20.93 Collecting azure-mgmt-policyinsights==1.1.0b4 (from azure-cli) [2024-01-18T15:28:05.641Z] #8 20.97 Downloading azure_mgmt_policyinsights-1.1.0b4-py3-none-any.whl (127 kB) [2024-01-18T15:28:05.641Z] #8 20.97 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 127.0/127.0 kB 145.9 MB/s eta 0:00:00 [2024-01-18T15:28:05.641Z] #8 21.01 Collecting azure-mgmt-privatedns~=1.0.0 (from azure-cli) [2024-01-18T15:28:05.641Z] #8 21.03 Downloading azure_mgmt_privatedns-1.0.0-py2.py3-none-any.whl (43 kB) [2024-01-18T15:28:05.641Z] #8 21.04 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.0/44.0 kB 89.1 MB/s eta 0:00:00 [2024-01-18T15:28:05.641Z] #8 21.08 Collecting azure-mgmt-rdbms~=10.2.0b12 (from azure-cli) [2024-01-18T15:28:05.641Z] #8 21.11 Downloading azure_mgmt_rdbms-10.2.0b13-py3-none-any.whl (950 kB) [2024-01-18T15:28:05.641Z] #8 21.12 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 950.7/950.7 kB 151.4 MB/s eta 0:00:00 [2024-01-18T15:28:05.641Z] #8 21.16 Collecting azure-mgmt-recoveryservicesbackup~=7.0.0 (from azure-cli) [2024-01-18T15:28:05.641Z] #8 21.19 Downloading azure_mgmt_recoveryservicesbackup-7.0.0-py3-none-any.whl (564 kB) [2024-01-18T15:28:05.641Z] #8 21.20 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 564.2/564.2 kB 159.6 MB/s eta 0:00:00 [2024-01-18T15:28:05.641Z] #8 21.23 Collecting azure-mgmt-recoveryservices~=2.5.0 (from azure-cli) [2024-01-18T15:28:05.641Z] #8 21.27 Downloading azure_mgmt_recoveryservices-2.5.0-py3-none-any.whl (109 kB) [2024-01-18T15:28:05.641Z] #8 21.27 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 109.5/109.5 kB 131.4 MB/s eta 0:00:00 [2024-01-18T15:28:05.641Z] #8 21.30 Collecting azure-mgmt-redhatopenshift~=1.4.0 (from azure-cli) [2024-01-18T15:28:05.641Z] #8 21.33 Downloading azure_mgmt_redhatopenshift-1.4.0-py3-none-any.whl (345 kB) [2024-01-18T15:28:05.641Z] #8 21.33 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 345.5/345.5 kB 117.6 MB/s eta 0:00:00 [2024-01-18T15:28:05.641Z] #8 21.37 Collecting azure-mgmt-redis~=14.2.0 (from azure-cli) [2024-01-18T15:28:05.641Z] #8 21.40 Downloading azure_mgmt_redis-14.2.0-py3-none-any.whl (111 kB) [2024-01-18T15:28:05.641Z] #8 21.41 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 111.4/111.4 kB 151.8 MB/s eta 0:00:00 [2024-01-18T15:28:05.641Z] #8 21.45 Collecting azure-mgmt-resource==23.1.0b2 (from azure-cli) [2024-01-18T15:28:05.641Z] #8 21.49 Downloading azure_mgmt_resource-23.1.0b2-py3-none-any.whl (2.6 MB) [2024-01-18T15:28:05.641Z] #8 21.51 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.6/2.6 MB 151.7 MB/s eta 0:00:00 [2024-01-18T15:28:05.641Z] #8 21.55 Collecting azure-mgmt-search~=9.0 (from azure-cli) [2024-01-18T15:28:05.641Z] #8 21.58 Downloading azure_mgmt_search-9.1.0-py3-none-any.whl (110 kB) [2024-01-18T15:28:05.641Z] #8 21.59 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 110.3/110.3 kB 149.4 MB/s eta 0:00:00 [2024-01-18T15:28:05.641Z] #8 21.62 Collecting azure-mgmt-security==5.0.0 (from azure-cli) [2024-01-18T15:28:05.641Z] #8 21.65 Downloading azure_mgmt_security-5.0.0-py3-none-any.whl (1.1 MB) [2024-01-18T15:28:05.641Z] #8 21.66 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 182.1 MB/s eta 0:00:00 [2024-01-18T15:28:05.642Z] #8 21.72 Collecting azure-mgmt-servicebus~=8.2.0 (from azure-cli) [2024-01-18T15:28:05.642Z] #8 21.75 Downloading azure_mgmt_servicebus-8.2.0-py3-none-any.whl (940 kB) [2024-01-18T15:28:05.642Z] #8 21.76 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 940.1/940.1 kB 206.1 MB/s eta 0:00:00 [2024-01-18T15:28:05.642Z] #8 21.79 Collecting azure-mgmt-servicefabricmanagedclusters~=1.0.0 (from azure-cli) [2024-01-18T15:28:05.642Z] #8 21.82 Downloading azure_mgmt_servicefabricmanagedclusters-1.0.0-py2.py3-none-any.whl (112 kB) [2024-01-18T15:28:05.897Z] #8 21.83 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 112.5/112.5 kB 205.9 MB/s eta 0:00:00 [2024-01-18T15:28:05.897Z] #8 21.85 Collecting azure-mgmt-servicelinker==1.2.0b1 (from azure-cli) [2024-01-18T15:28:05.897Z] #8 21.88 Downloading azure_mgmt_servicelinker-1.2.0b1-py3-none-any.whl (90 kB) [2024-01-18T15:28:05.897Z] #8 21.88 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 90.7/90.7 kB 175.0 MB/s eta 0:00:00 [2024-01-18T15:28:05.897Z] #8 21.91 Collecting azure-mgmt-servicefabric~=1.0.0 (from azure-cli) [2024-01-18T15:28:05.897Z] #8 21.94 Downloading azure_mgmt_servicefabric-1.0.0-py2.py3-none-any.whl (139 kB) [2024-01-18T15:28:05.897Z] #8 21.94 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.6/139.6 kB 181.6 MB/s eta 0:00:00 [2024-01-18T15:28:05.897Z] #8 21.97 Collecting azure-mgmt-signalr==2.0.0b1 (from azure-cli) [2024-01-18T15:28:05.897Z] #8 22.00 Downloading azure_mgmt_signalr-2.0.0b1-py3-none-any.whl (123 kB) [2024-01-18T15:28:05.897Z] #8 22.01 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 123.6/123.6 kB 185.3 MB/s eta 0:00:00 [2024-01-18T15:28:05.897Z] #8 22.03 Collecting azure-mgmt-sqlvirtualmachine==1.0.0b5 (from azure-cli) [2024-01-18T15:28:05.897Z] #8 22.06 Downloading azure_mgmt_sqlvirtualmachine-1.0.0b5-py3-none-any.whl (95 kB) [2024-01-18T15:28:06.153Z] #8 22.07 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 95.3/95.3 kB 147.8 MB/s eta 0:00:00 [2024-01-18T15:28:06.153Z] #8 22.12 Collecting azure-mgmt-sql==4.0.0b13 (from azure-cli) [2024-01-18T15:28:06.153Z] #8 22.16 Downloading azure_mgmt_sql-4.0.0b13-py3-none-any.whl (1.2 MB) [2024-01-18T15:28:06.153Z] #8 22.17 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 158.6 MB/s eta 0:00:00 [2024-01-18T15:28:06.153Z] #8 22.22 Collecting azure-mgmt-storage==21.1.0 (from azure-cli) [2024-01-18T15:28:06.153Z] #8 22.25 Downloading azure_mgmt_storage-21.1.0-py3-none-any.whl (3.0 MB) [2024-01-18T15:28:06.153Z] #8 22.28 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.0/3.0 MB 121.7 MB/s eta 0:00:00 [2024-01-18T15:28:06.153Z] #8 22.33 Collecting azure-mgmt-synapse==2.1.0b5 (from azure-cli) [2024-01-18T15:28:06.153Z] #8 22.36 Downloading azure_mgmt_synapse-2.1.0b5-py3-none-any.whl (547 kB) [2024-01-18T15:28:06.153Z] #8 22.36 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 547.1/547.1 kB 159.6 MB/s eta 0:00:00 [2024-01-18T15:28:06.153Z] #8 22.39 Collecting azure-mgmt-trafficmanager~=1.0.0 (from azure-cli) [2024-01-18T15:28:06.409Z] #8 22.42 Downloading azure_mgmt_trafficmanager-1.0.0-py3-none-any.whl (49 kB) [2024-01-18T15:28:06.409Z] #8 22.43 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.3/49.3 kB 98.6 MB/s eta 0:00:00 [2024-01-18T15:28:06.409Z] #8 22.46 Collecting azure-mgmt-web==7.2.0 (from azure-cli) [2024-01-18T15:28:06.409Z] #8 22.50 Downloading azure_mgmt_web-7.2.0-py3-none-any.whl (6.0 MB) [2024-01-18T15:28:06.409Z] #8 22.56 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.0/6.0 MB 109.8 MB/s eta 0:00:00 [2024-01-18T15:28:06.409Z] #8 22.62 Collecting azure-multiapi-storage~=1.2.0 (from azure-cli) [2024-01-18T15:28:06.665Z] #8 22.65 Downloading azure_multiapi_storage-1.2.0-py2.py3-none-any.whl (2.9 MB) [2024-01-18T15:28:06.665Z] #8 22.68 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.9/2.9 MB 146.6 MB/s eta 0:00:00 [2024-01-18T15:28:06.665Z] #8 22.75 Collecting azure-storage-common~=1.4 (from azure-cli) [2024-01-18T15:28:06.665Z] #8 22.78 Downloading azure_storage_common-1.4.2-py2.py3-none-any.whl (47 kB) [2024-01-18T15:28:06.665Z] #8 22.78 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47.4/47.4 kB 112.7 MB/s eta 0:00:00 [2024-01-18T15:28:06.665Z] #8 22.81 Collecting azure-synapse-accesscontrol~=0.5.0 (from azure-cli) [2024-01-18T15:28:06.665Z] #8 22.84 Downloading azure_synapse_accesscontrol-0.5.0-py2.py3-none-any.whl (30 kB) [2024-01-18T15:28:06.665Z] #8 22.87 Collecting azure-synapse-artifacts~=0.18.0 (from azure-cli) [2024-01-18T15:28:06.921Z] #8 22.90 Downloading azure_synapse_artifacts-0.18.0-py3-none-any.whl (486 kB) [2024-01-18T15:28:06.921Z] #8 22.91 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 486.7/486.7 kB 167.2 MB/s eta 0:00:00 [2024-01-18T15:28:06.921Z] #8 22.94 Collecting azure-synapse-managedprivateendpoints~=0.4.0 (from azure-cli) [2024-01-18T15:28:06.921Z] #8 22.97 Downloading azure_synapse_managedprivateendpoints-0.4.0-py2.py3-none-any.whl (45 kB) [2024-01-18T15:28:06.921Z] #8 22.97 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.0/46.0 kB 107.2 MB/s eta 0:00:00 [2024-01-18T15:28:06.921Z] #8 23.00 Collecting azure-synapse-spark~=0.2.0 (from azure-cli) [2024-01-18T15:28:06.921Z] #8 23.02 Downloading azure_synapse_spark-0.2.0-py2.py3-none-any.whl (29 kB) [2024-01-18T15:28:06.921Z] #8 23.05 Collecting chardet~=5.2.0 (from azure-cli) [2024-01-18T15:28:06.921Z] #8 23.08 Downloading chardet-5.2.0-py3-none-any.whl (199 kB) [2024-01-18T15:28:07.176Z] #8 23.08 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 199.4/199.4 kB 155.4 MB/s eta 0:00:00 [2024-01-18T15:28:07.176Z] #8 23.13 Collecting colorama~=0.4.4 (from azure-cli) [2024-01-18T15:28:07.176Z] #8 23.16 Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB) [2024-01-18T15:28:07.176Z] #8 23.21 Collecting fabric~=2.4 (from azure-cli) [2024-01-18T15:28:07.176Z] #8 23.24 Downloading fabric-2.7.1-py2.py3-none-any.whl (53 kB) [2024-01-18T15:28:07.176Z] #8 23.24 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.5/53.5 kB 114.2 MB/s eta 0:00:00 [2024-01-18T15:28:07.176Z] #8 23.27 Collecting javaproperties~=0.5.1 (from azure-cli) [2024-01-18T15:28:07.176Z] #8 23.30 Downloading javaproperties-0.5.2-py2.py3-none-any.whl (19 kB) [2024-01-18T15:28:07.176Z] #8 23.33 Collecting jsondiff~=2.0.0 (from azure-cli) [2024-01-18T15:28:07.431Z] #8 23.36 Downloading jsondiff-2.0.0-py3-none-any.whl (6.6 kB) [2024-01-18T15:28:07.431Z] #8 23.36 Requirement already satisfied: packaging>=20.9 in /usr/lib/python3.11/site-packages (from azure-cli) (23.1) [2024-01-18T15:28:07.431Z] #8 23.39 Collecting pycomposefile>=0.0.29 (from azure-cli) [2024-01-18T15:28:07.431Z] #8 23.42 Downloading pycomposefile-0.0.30-py3-none-any.whl (28 kB) [2024-01-18T15:28:07.431Z] #8 23.48 Collecting PyGithub~=1.38 (from azure-cli) [2024-01-18T15:28:07.431Z] #8 23.53 Downloading PyGithub-1.59.1-py3-none-any.whl (342 kB) [2024-01-18T15:28:07.431Z] #8 23.54 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 342.2/342.2 kB 130.8 MB/s eta 0:00:00 [2024-01-18T15:28:07.431Z] #8 23.58 Collecting PyNaCl~=1.5.0 (from azure-cli) [2024-01-18T15:28:07.686Z] #8 23.61 Downloading PyNaCl-1.5.0-cp36-abi3-musllinux_1_1_x86_64.whl (1.1 MB) [2024-01-18T15:28:07.686Z] #8 23.64 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 61.7 MB/s eta 0:00:00 [2024-01-18T15:28:07.686Z] #8 23.67 Collecting scp~=0.13.2 (from azure-cli) [2024-01-18T15:28:07.686Z] #8 23.71 Downloading scp-0.13.6-py2.py3-none-any.whl (8.2 kB) [2024-01-18T15:28:07.686Z] #8 23.75 Collecting semver==2.13.0 (from azure-cli) [2024-01-18T15:28:07.686Z] #8 23.78 Downloading semver-2.13.0-py2.py3-none-any.whl (12 kB) [2024-01-18T15:28:07.686Z] #8 23.81 Collecting six>=1.10.0 (from azure-cli) [2024-01-18T15:28:07.686Z] #8 23.84 Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) [2024-01-18T15:28:07.686Z] #8 23.87 Collecting sshtunnel~=0.1.4 (from azure-cli) [2024-01-18T15:28:07.686Z] #8 23.91 Downloading sshtunnel-0.1.5-py2.py3-none-any.whl (23 kB) [2024-01-18T15:28:07.944Z] #8 23.98 Collecting urllib3 (from azure-cli) [2024-01-18T15:28:07.944Z] #8 24.01 Downloading urllib3-2.1.0-py3-none-any.whl (104 kB) [2024-01-18T15:28:07.944Z] #8 24.02 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 104.6/104.6 kB 135.9 MB/s eta 0:00:00 [2024-01-18T15:28:07.944Z] #8 24.06 Collecting websocket-client~=1.3.1 (from azure-cli) [2024-01-18T15:28:07.944Z] #8 24.09 Downloading websocket_client-1.3.3-py3-none-any.whl (54 kB) [2024-01-18T15:28:07.944Z] #8 24.10 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.3/54.3 kB 139.0 MB/s eta 0:00:00 [2024-01-18T15:28:07.944Z] #8 24.12 Collecting xmltodict~=0.12 (from azure-cli) [2024-01-18T15:28:08.207Z] #8 24.15 Downloading xmltodict-0.13.0-py2.py3-none-any.whl (10.0 kB) [2024-01-18T15:28:08.207Z] #8 24.18 Collecting distro (from azure-cli) [2024-01-18T15:28:08.207Z] #8 24.21 Downloading distro-1.9.0-py3-none-any.whl (20 kB) [2024-01-18T15:28:08.207Z] #8 24.29 Collecting argcomplete~=3.1.1 (from azure-cli-core==2.56.0->azure-cli) [2024-01-18T15:28:08.207Z] #8 24.31 Downloading argcomplete-3.1.6-py3-none-any.whl (41 kB) [2024-01-18T15:28:08.207Z] #8 24.32 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.7/41.7 kB 125.1 MB/s eta 0:00:00 [2024-01-18T15:28:08.207Z] #8 24.35 Collecting azure-cli-telemetry==1.1.0.* (from azure-cli-core==2.56.0->azure-cli) [2024-01-18T15:28:08.207Z] #8 24.38 Downloading azure_cli_telemetry-1.1.0-py3-none-any.whl (11 kB) [2024-01-18T15:28:08.463Z] #8 24.41 Collecting azure-mgmt-core<2,>=1.2.0 (from azure-cli-core==2.56.0->azure-cli) [2024-01-18T15:28:08.463Z] #8 24.44 Downloading azure_mgmt_core-1.4.0-py3-none-any.whl (27 kB) [2024-01-18T15:28:08.718Z] #8 24.71 Collecting cryptography (from azure-cli-core==2.56.0->azure-cli) [2024-01-18T15:28:08.718Z] #8 24.74 Downloading cryptography-41.0.7-cp37-abi3-musllinux_1_1_x86_64.whl (4.4 MB) [2024-01-18T15:28:08.718Z] #8 24.78 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.4/4.4 MB 119.7 MB/s eta 0:00:00 [2024-01-18T15:28:08.718Z] #8 24.90 Collecting humanfriendly~=10.0 (from azure-cli-core==2.56.0->azure-cli) [2024-01-18T15:28:08.973Z] #8 24.93 Downloading humanfriendly-10.0-py2.py3-none-any.whl (86 kB) [2024-01-18T15:28:08.973Z] #8 24.93 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 86.8/86.8 kB 138.4 MB/s eta 0:00:00 [2024-01-18T15:28:08.973Z] #8 24.96 Collecting jmespath (from azure-cli-core==2.56.0->azure-cli) [2024-01-18T15:28:08.973Z] #8 24.99 Downloading jmespath-1.0.1-py3-none-any.whl (20 kB) [2024-01-18T15:28:08.973Z] #8 25.02 Collecting knack~=0.11.0 (from azure-cli-core==2.56.0->azure-cli) [2024-01-18T15:28:08.973Z] #8 25.05 Downloading knack-0.11.0-py3-none-any.whl (60 kB) [2024-01-18T15:28:08.973Z] #8 25.05 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.8/60.8 kB 133.1 MB/s eta 0:00:00 [2024-01-18T15:28:08.973Z] #8 25.08 Collecting msal-extensions~=1.0.0 (from azure-cli-core==2.56.0->azure-cli) [2024-01-18T15:28:08.973Z] #8 25.11 Downloading msal_extensions-1.0.0-py2.py3-none-any.whl (19 kB) [2024-01-18T15:28:08.973Z] #8 25.15 Collecting msal[broker]==1.24.0b2 (from azure-cli-core==2.56.0->azure-cli) [2024-01-18T15:28:09.228Z] #8 25.18 Downloading msal-1.24.0b2-py2.py3-none-any.whl (91 kB) [2024-01-18T15:28:09.228Z] #8 25.18 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 91.1/91.1 kB 146.7 MB/s eta 0:00:00 [2024-01-18T15:28:09.228Z] #8 25.22 Collecting msrestazure~=0.6.4 (from azure-cli-core==2.56.0->azure-cli) [2024-01-18T15:28:09.228Z] #8 25.25 Downloading msrestazure-0.6.4-py2.py3-none-any.whl (40 kB) [2024-01-18T15:28:09.229Z] #8 25.25 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.5/40.5 kB 102.7 MB/s eta 0:00:00 [2024-01-18T15:28:09.229Z] #8 25.31 Collecting paramiko<4.0.0,>=2.0.8 (from azure-cli-core==2.56.0->azure-cli) [2024-01-18T15:28:09.229Z] #8 25.34 Downloading paramiko-3.4.0-py3-none-any.whl (225 kB) [2024-01-18T15:28:09.229Z] #8 25.35 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 225.9/225.9 kB 148.2 MB/s eta 0:00:00 [2024-01-18T15:28:09.229Z] #8 25.39 Collecting pkginfo>=1.5.0.1 (from azure-cli-core==2.56.0->azure-cli) [2024-01-18T15:28:09.484Z] #8 25.42 Downloading pkginfo-1.9.6-py3-none-any.whl (30 kB) [2024-01-18T15:28:09.484Z] #8 25.47 Collecting PyJWT>=2.1.0 (from azure-cli-core==2.56.0->azure-cli) [2024-01-18T15:28:09.484Z] #8 25.50 Downloading PyJWT-2.8.0-py3-none-any.whl (22 kB) [2024-01-18T15:28:09.484Z] #8 25.53 Collecting pyopenssl>=17.1.0 (from azure-cli-core==2.56.0->azure-cli) [2024-01-18T15:28:09.484Z] #8 25.56 Downloading pyOpenSSL-23.3.0-py3-none-any.whl (58 kB) [2024-01-18T15:28:09.484Z] #8 25.57 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.8/58.8 kB 162.2 MB/s eta 0:00:00 [2024-01-18T15:28:09.484Z] #8 25.63 Collecting requests[socks] (from azure-cli-core==2.56.0->azure-cli) [2024-01-18T15:28:09.738Z] #8 25.66 Downloading requests-2.31.0-py3-none-any.whl (62 kB) [2024-01-18T15:28:09.738Z] #8 25.66 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 129.6 MB/s eta 0:00:00 [2024-01-18T15:28:09.738Z] #8 25.80 Collecting psutil~=5.9 (from azure-cli-core==2.56.0->azure-cli) [2024-01-18T15:28:09.739Z] #8 25.83 Downloading psutil-5.9.7.tar.gz (498 kB) [2024-01-18T15:28:09.739Z] #8 25.84 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 498.4/498.4 kB 223.7 MB/s eta 0:00:00 [2024-01-18T15:28:09.993Z] #8 25.95 Installing build dependencies: started [2024-01-18T15:28:11.358Z] #8 ... [2024-01-18T15:28:11.358Z] [2024-01-18T15:28:11.358Z] #25 [build 5/19] RUN mkdir bin && curl -o aws-iam-authenticator https://amazon-eks.s3.us-west-2.amazonaws.com/1.16.8/2020-04-16/bin/linux/amd64/aws-iam-authenticator && chmod a+x aws-iam-authenticator && mv aws-iam-authenticator bin [2024-01-18T15:28:11.358Z] #25 sha256:bec8c4b6e9731c3a0110c135a636ca0b284084443e17d2fa78a6026898457aa2 [2024-01-18T15:28:11.358Z] #25 0.357 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 1 33.6M 1 360k 0 0 493k 0 0:01:09 --:--:-- 0:01:09 494k 4 33.6M 4 1380k 0 0 802k 0 0:00:42 0:00:01 0:00:41 803k 8 33.6M 8 2859k 0 0 1053k 0 0:00:32 0:00:02 0:00:30 1054k 13 33.6M 13 4797k 0 0 1294k 0 0:00:26 0:00:03 0:00:23 1295k 21 33.6M 21 7279k 0 0 1547k 0 0:00:22 0:00:04 0:00:18 1547k 29 33.6M 29 9.8M 0 0 1762k 0 0:00:19 0:00:05 0:00:14 1948k 39 33.6M 39 13.1M 0 0 2003k 0 0:00:17 0:00:06 0:00:11 2416k 49 33.6M 49 16.8M 0 0 2233k 0 0:00:15 0:00:07 0:00:08 2873k 62 33.6M 62 20.8M 0 0 2454k 0 0:00:14 0:00:08 0:00:06 3313k 76 33.6M 76 25.6M 0 0 2697k 0 0:00:12 0:00:09 0:00:03 3771k 93 33.6M 93 31.4M 0 0 3000k 0 0:00:11 0:00:10 0:00:01 4410k 100 33.6M 100 33.6M 0 0 3128k 0 0:00:11 0:00:11 --:--:-- 4884k [2024-01-18T15:28:11.358Z] #25 DONE 11.6s [2024-01-18T15:28:11.358Z] [2024-01-18T15:28:11.358Z] #26 [build 6/19] RUN curl -fsSL https://clis.cloud.ibm.com/install/linux | sh && ibmcloud plugin install -f vpc-infrastructure && ibmcloud plugin install -f container-service [2024-01-18T15:28:11.358Z] #26 sha256:b60976048ddae99848cf6112926d51a9f1f296c63737c1b7e1193e63f8030cbb [2024-01-18T15:28:12.719Z] #26 1.250 Current platform is linux64. Downloading corresponding IBM Cloud CLI... [2024-01-18T15:28:12.719Z] #26 1.255 % Total % Received % Xferd Average Speed Time Time Time Current [2024-01-18T15:28:12.719Z] #26 1.255 Dload Upload Total Spent Left Speed [2024-01-18T15:28:12.974Z] #26 1.255 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 65 8573k 65 5597k 0 0 27.4M 0 --:--:-- --:--:-- --:--:-- 27.6M 100 8573k 100 8573k 0 0 37.1M 0 --:--:-- --:--:-- --:--:-- 37.3M [2024-01-18T15:28:12.974Z] #26 1.483 Download complete. Executing installer... [2024-01-18T15:28:12.974Z] #26 1.539 Bluemix_CLI/ [2024-01-18T15:28:13.229Z] #26 1.848 Bluemix_CLI/bin/ [2024-01-18T15:28:13.229Z] #26 1.848 Bluemix_CLI/bin/ibmcloud [2024-01-18T15:28:13.229Z] #26 1.848 Bluemix_CLI/bin/ibmcloud.sig [2024-01-18T15:28:13.229Z] #26 1.848 Bluemix_CLI/bin/NOTICE [2024-01-18T15:28:13.229Z] #26 1.848 Bluemix_CLI/bin/LICENSE [2024-01-18T15:28:13.229Z] #26 1.848 Bluemix_CLI/bin/CF_CLI_Notices.txt [2024-01-18T15:28:13.229Z] #26 1.848 Bluemix_CLI/bin/CF_CLI_SLC_Notices.txt [2024-01-18T15:28:13.229Z] #26 1.848 Bluemix_CLI/autocomplete/ [2024-01-18T15:28:13.229Z] #26 1.848 Bluemix_CLI/autocomplete/bash_autocomplete [2024-01-18T15:28:13.229Z] #26 1.848 Bluemix_CLI/autocomplete/zsh_autocomplete [2024-01-18T15:28:13.229Z] #26 1.848 Bluemix_CLI/install [2024-01-18T15:28:13.229Z] #26 1.848 Bluemix_CLI/uninstall [2024-01-18T15:28:13.229Z] #26 1.848 Bluemix_CLI/install_bluemix_cli [2024-01-18T15:28:13.229Z] #26 1.897 Install complete. [2024-01-18T15:28:13.229Z] #26 1.985 Looking up 'vpc-infrastructure' from repository 'IBM Cloud'... [2024-01-18T15:28:13.795Z] #26 2.309 Plug-in 'vpc-infrastructure[infrastructure-service/is] 9.0.1' found in repository 'IBM Cloud' [2024-01-18T15:28:13.795Z] #26 2.309 Attempting to download the binary file... [2024-01-18T15:28:14.056Z] #26 2.366 0 B / 22.71 MiB 0.00% 22.71 MiB / 22.71 MiB 100.00% 0s [2024-01-18T15:28:14.056Z] #26 2.549 23810048 bytes downloaded [2024-01-18T15:28:14.056Z] #26 2.633 Installing binary... [2024-01-18T15:28:14.616Z] #26 2.951 OK [2024-01-18T15:28:14.616Z] #26 2.951 Plug-in 'vpc-infrastructure 9.0.1' was successfully installed into /****/.bluemix/plugins/vpc-infrastructure. Use 'ibmcloud plugin show vpc-infrastructure' to show its details. [2024-01-18T15:28:14.616Z] #26 3.321 Looking up 'container-service' from repository 'IBM Cloud'... [2024-01-18T15:28:14.871Z] #26 3.554 Plug-in 'container-service[kubernetes-service/ks] 1.0.579' found in repository 'IBM Cloud' [2024-01-18T15:28:14.871Z] #26 3.554 Attempting to download the binary file... [2024-01-18T15:28:15.126Z] #26 3.609 0 B / 28.76 MiB 0.00% 19.01 MiB / 28.76 MiB 66.10% 28.76 MiB / 28.76 MiB 100.00% 0s [2024-01-18T15:28:15.381Z] #26 3.864 30158848 bytes downloaded [2024-01-18T15:28:15.381Z] #26 3.973 Installing binary... [2024-01-18T15:28:15.641Z] #26 4.140 OK [2024-01-18T15:28:15.641Z] #26 4.140 Plug-in 'container-service 1.0.579' was successfully installed into /****/.bluemix/plugins/container-service. Use 'ibmcloud plugin show container-service' to show its details. [2024-01-18T15:28:15.896Z] #26 DONE 4.3s [2024-01-18T15:28:15.896Z] [2024-01-18T15:28:15.896Z] #27 [build 7/19] RUN curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster [2024-01-18T15:28:15.896Z] #27 sha256:8a400d3892224089ddfe38a4e1ff4a87c6440bf86694ba5b3daf6a6b44a7b8fd [2024-01-18T15:28:16.151Z] #27 0.391 % Total % Received % Xferd Average Speed Time Time Time Current [2024-01-18T15:28:16.151Z] #27 0.391 Dload Upload Total Spent Left Speed [2024-01-18T15:28:16.406Z] #27 0.392 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 [2024-01-18T15:28:16.661Z] #27 0.784 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 [2024-01-18T15:28:17.476Z] #27 1.133 0 61.4M 0 8259 0 0 11108 0 1:36:45 --:--:-- 1:36:45 11108 100 61.4M 100 61.4M 0 0 54.0M 0 0:00:01 0:00:01 --:--:-- 156M [2024-01-18T15:28:17.476Z] #27 DONE 1.8s [2024-01-18T15:28:17.476Z] [2024-01-18T15:28:17.476Z] #28 [build 8/19] COPY vendor vendor [2024-01-18T15:28:17.476Z] #28 sha256:aa95d35c5fffbab79a572036c620e27809deb22a98eb7a961b106a426f2542b4 [2024-01-18T15:28:19.371Z] #28 DONE 1.8s [2024-01-18T15:28:19.371Z] [2024-01-18T15:28:19.371Z] #8 [stage-1 4/20] RUN apk add --no-cache --update --virtual=build gcc musl-dev python3-dev libffi-dev openssl-dev cargo make && pip3 install "pyyaml<=5.3.1" && pip3 install --no-cache-dir --prefer-binary azure-cli && apk del build [2024-01-18T15:28:19.372Z] #8 sha256:4efb1b73fb1716746a3ed94677a3924261e9bd638976a6ba84c5bf7e7caeeffd [2024-01-18T15:28:19.372Z] #8 28.06 Installing build dependencies: finished with status 'done' [2024-01-18T15:28:19.372Z] #8 28.06 Getting requirements to build wheel: started [2024-01-18T15:28:19.372Z] #8 28.44 Getting requirements to build wheel: finished with status 'done' [2024-01-18T15:28:19.372Z] #8 28.45 Preparing metadata (pyproject.toml): started [2024-01-18T15:28:19.372Z] #8 28.80 Preparing metadata (pyproject.toml): finished with status 'done' [2024-01-18T15:28:19.372Z] #8 28.86 Collecting azure-core<2.0.0,>=1.15.0 (from azure-data-tables==12.4.0->azure-cli) [2024-01-18T15:28:19.372Z] #8 28.89 Downloading azure_core-1.29.6-py3-none-any.whl (192 kB) [2024-01-18T15:28:19.372Z] #8 28.90 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 192.5/192.5 kB 130.1 MB/s eta 0:00:00 [2024-01-18T15:28:19.372Z] #8 28.95 Collecting msrest>=0.6.21 (from azure-data-tables==12.4.0->azure-cli) [2024-01-18T15:28:19.372Z] #8 28.98 Downloading msrest-0.7.1-py3-none-any.whl (85 kB) [2024-01-18T15:28:19.372Z] #8 28.98 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 85.4/85.4 kB 140.3 MB/s eta 0:00:00 [2024-01-18T15:28:19.372Z] #8 29.02 Collecting azure-common~=1.1 (from azure-keyvault-administration==4.4.0b2->azure-cli) [2024-01-18T15:28:19.372Z] #8 29.05 Downloading azure_common-1.1.28-py2.py3-none-any.whl (14 kB) [2024-01-18T15:28:19.372Z] #8 29.09 Collecting isodate>=0.6.1 (from azure-keyvault-administration==4.4.0b2->azure-cli) [2024-01-18T15:28:19.372Z] #8 29.11 Downloading isodate-0.6.1-py2.py3-none-any.whl (41 kB) [2024-01-18T15:28:19.372Z] #8 29.12 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.7/41.7 kB 124.9 MB/s eta 0:00:00 [2024-01-18T15:28:19.372Z] #8 29.15 Collecting typing-extensions>=4.0.1 (from azure-keyvault-administration==4.4.0b2->azure-cli) [2024-01-18T15:28:19.372Z] #8 29.19 Downloading typing_extensions-4.9.0-py3-none-any.whl (32 kB) [2024-01-18T15:28:19.372Z] #8 29.53 Collecting applicationinsights<0.12,>=0.11.1 (from azure-cli-telemetry==1.1.0.*->azure-cli-core==2.56.0->azure-cli) [2024-01-18T15:28:19.372Z] #8 29.56 Downloading applicationinsights-0.11.10-py2.py3-none-any.whl (55 kB) [2024-01-18T15:28:19.372Z] #8 29.56 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.1/55.1 kB 119.1 MB/s eta 0:00:00 [2024-01-18T15:28:19.372Z] #8 29.60 Collecting portalocker<3,>=1.6 (from azure-cli-telemetry==1.1.0.*->azure-cli-core==2.56.0->azure-cli) [2024-01-18T15:28:19.372Z] #8 29.63 Downloading portalocker-2.8.2-py3-none-any.whl (17 kB) [2024-01-18T15:28:19.372Z] #8 29.98 Collecting cffi (from azure-datalake-store~=0.0.49->azure-cli) [2024-01-18T15:28:19.372Z] #8 30.01 Downloading cffi-1.16.0-cp311-cp311-musllinux_1_1_x86_64.whl (476 kB) [2024-01-18T15:28:19.372Z] #8 30.02 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 476.4/476.4 kB 167.5 MB/s eta 0:00:00 [2024-01-18T15:28:19.372Z] #8 30.28 Collecting azure-mgmt-datalake-nspkg>=2.0.0 (from azure-mgmt-datalake-store~=0.5.0->azure-cli) [2024-01-18T15:28:19.372Z] #8 30.31 Downloading azure_mgmt_datalake_nspkg-3.0.1-py3-none-any.whl (1.7 kB) [2024-01-18T15:28:19.372Z] #8 30.85 Collecting python-dateutil (from azure-multiapi-storage~=1.2.0->azure-cli) [2024-01-18T15:28:19.372Z] #8 30.87 Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) [2024-01-18T15:28:19.372Z] #8 30.88 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 153.4 MB/s eta 0:00:00 [2024-01-18T15:28:19.372Z] #8 31.12 Collecting invoke<2.0,>=1.3 (from fabric~=2.4->azure-cli) [2024-01-18T15:28:19.372Z] #8 31.15 Downloading invoke-1.7.3-py3-none-any.whl (216 kB) [2024-01-18T15:28:19.372Z] #8 31.15 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 216.4/216.4 kB 169.4 MB/s eta 0:00:00 [2024-01-18T15:28:19.372Z] #8 31.20 Collecting pathlib2 (from fabric~=2.4->azure-cli) [2024-01-18T15:28:19.372Z] #8 31.23 Downloading pathlib2-2.3.7.post1-py2.py3-none-any.whl (18 kB) [2024-01-18T15:28:19.373Z] #8 31.31 Requirement already satisfied: pyyaml in /usr/lib/python3.11/site-packages (from pycomposefile>=0.0.29->azure-cli) (5.3.1) [2024-01-18T15:28:19.373Z] #8 31.36 Collecting deprecated (from PyGithub~=1.38->azure-cli) [2024-01-18T15:28:19.373Z] #8 31.39 Downloading Deprecated-1.2.14-py2.py3-none-any.whl (9.6 kB) [2024-01-18T15:28:19.373Z] #8 31.77 Collecting anyio<5.0,>=3.0 (from azure-core<2.0.0,>=1.15.0->azure-data-tables==12.4.0->azure-cli) [2024-01-18T15:28:19.373Z] #8 31.80 Downloading anyio-4.2.0-py3-none-any.whl (85 kB) [2024-01-18T15:28:19.373Z] #8 31.80 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 85.5/85.5 kB 139.7 MB/s eta 0:00:00 [2024-01-18T15:28:19.373Z] #8 31.94 Collecting azure-mgmt-nspkg>=3.0.0 (from azure-mgmt-datalake-nspkg>=2.0.0->azure-mgmt-datalake-store~=0.5.0->azure-cli) [2024-01-18T15:28:19.373Z] #8 31.97 Downloading azure_mgmt_nspkg-3.0.2-py3-none-any.whl (1.6 kB) [2024-01-18T15:28:19.373Z] #8 32.04 Collecting pycparser (from cffi->azure-datalake-store~=0.0.49->azure-cli) [2024-01-18T15:28:19.373Z] #8 32.06 Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB) [2024-01-18T15:28:19.373Z] #8 32.07 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.7/118.7 kB 136.4 MB/s eta 0:00:00 [2024-01-18T15:28:19.373Z] #8 32.35 Collecting pygments (from knack~=0.11.0->azure-cli-core==2.56.0->azure-cli) [2024-01-18T15:28:19.373Z] #8 32.38 Downloading pygments-2.17.2-py3-none-any.whl (1.2 MB) [2024-01-18T15:28:19.373Z] #8 32.39 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 173.9 MB/s eta 0:00:00 [2024-01-18T15:28:19.373Z] #8 32.42 Collecting tabulate (from knack~=0.11.0->azure-cli-core==2.56.0->azure-cli) [2024-01-18T15:28:19.373Z] #8 32.45 Downloading tabulate-0.9.0-py3-none-any.whl (35 kB) [2024-01-18T15:28:19.373Z] #8 32.65 Collecting certifi>=2017.4.17 (from msrest>=0.6.21->azure-data-tables==12.4.0->azure-cli) [2024-01-18T15:28:19.373Z] #8 32.67 Downloading certifi-2023.11.17-py3-none-any.whl (162 kB) [2024-01-18T15:28:19.373Z] #8 32.68 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 162.5/162.5 kB 202.4 MB/s eta 0:00:00 [2024-01-18T15:28:19.373Z] #8 32.73 Collecting requests-oauthlib>=0.5.0 (from msrest>=0.6.21->azure-data-tables==12.4.0->azure-cli) [2024-01-18T15:28:19.373Z] #8 32.76 Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB) [2024-01-18T15:28:19.373Z] #8 32.87 Collecting adal<2.0.0,>=0.6.0 (from msrestazure~=0.6.4->azure-cli-core==2.56.0->azure-cli) [2024-01-18T15:28:19.373Z] #8 32.90 Downloading adal-1.2.7-py2.py3-none-any.whl (55 kB) [2024-01-18T15:28:19.373Z] #8 32.90 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.5/55.5 kB 179.6 MB/s eta 0:00:00 [2024-01-18T15:28:19.373Z] #8 33.02 Collecting bcrypt>=3.2 (from paramiko<4.0.0,>=2.0.8->azure-cli-core==2.56.0->azure-cli) [2024-01-18T15:28:19.373Z] #8 33.05 Downloading bcrypt-4.1.2-cp39-abi3-musllinux_1_2_x86_64.whl (749 kB) [2024-01-18T15:28:19.373Z] #8 33.06 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 749.7/749.7 kB 238.6 MB/s eta 0:00:00 [2024-01-18T15:28:19.373Z] #8 33.57 Collecting charset-normalizer<4,>=2 (from requests[socks]->azure-cli-core==2.56.0->azure-cli) [2024-01-18T15:28:19.373Z] #8 33.60 Downloading charset_normalizer-3.3.2-cp311-cp311-musllinux_1_1_x86_64.whl (140 kB) [2024-01-18T15:28:19.373Z] #8 33.61 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.9/140.9 kB 159.9 MB/s eta 0:00:00 [2024-01-18T15:28:19.373Z] #8 33.64 Collecting idna<4,>=2.5 (from requests[socks]->azure-cli-core==2.56.0->azure-cli) [2024-01-18T15:28:19.373Z] #8 33.68 Downloading idna-3.6-py3-none-any.whl (61 kB) [2024-01-18T15:28:19.373Z] #8 33.69 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.6/61.6 kB 165.2 MB/s eta 0:00:00 [2024-01-18T15:28:19.373Z] #8 33.94 Collecting wrapt<2,>=1.10 (from deprecated->PyGithub~=1.38->azure-cli) [2024-01-18T15:28:19.373Z] #8 33.97 Downloading wrapt-1.16.0-cp311-cp311-musllinux_1_1_x86_64.whl (85 kB) [2024-01-18T15:28:19.373Z] #8 33.97 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 85.7/85.7 kB 183.8 MB/s eta 0:00:00 [2024-01-18T15:28:19.373Z] #8 34.26 Collecting PySocks!=1.5.7,>=1.5.6 (from requests[socks]->azure-cli-core==2.56.0->azure-cli) [2024-01-18T15:28:19.373Z] #8 34.29 Downloading PySocks-1.7.1-py3-none-any.whl (16 kB) [2024-01-18T15:28:19.373Z] #8 34.45 Collecting sniffio>=1.1 (from anyio<5.0,>=3.0->azure-core<2.0.0,>=1.15.0->azure-data-tables==12.4.0->azure-cli) [2024-01-18T15:28:19.373Z] #8 34.48 Downloading sniffio-1.3.0-py3-none-any.whl (10 kB) [2024-01-18T15:28:19.373Z] #8 34.61 Collecting azure-nspkg>=3.0.0 (from azure-mgmt-nspkg>=3.0.0->azure-mgmt-datalake-nspkg>=2.0.0->azure-mgmt-datalake-store~=0.5.0->azure-cli) [2024-01-18T15:28:19.373Z] #8 34.63 Downloading azure_nspkg-3.0.2-py3-none-any.whl (1.5 kB) [2024-01-18T15:28:19.373Z] #8 35.06 Collecting oauthlib>=3.0.0 (from requests-oauthlib>=0.5.0->msrest>=0.6.21->azure-data-tables==12.4.0->azure-cli) [2024-01-18T15:28:19.374Z] #8 35.09 Downloading oauthlib-3.2.2-py3-none-any.whl (151 kB) [2024-01-18T15:28:19.374Z] #8 35.09 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 151.7/151.7 kB 185.3 MB/s eta 0:00:00 [2024-01-18T15:28:19.630Z] #8 ... [2024-01-18T15:28:19.630Z] [2024-01-18T15:28:19.630Z] #29 [build 9/19] COPY Makefile Makefile [2024-01-18T15:28:19.630Z] #29 sha256:42b4decc42d5fc146d890c1442af7f831d5b0653539311d84d06881842f72959 [2024-01-18T15:28:19.630Z] #29 DONE 0.1s [2024-01-18T15:28:19.630Z] [2024-01-18T15:28:19.630Z] #8 [stage-1 4/20] RUN apk add --no-cache --update --virtual=build gcc musl-dev python3-dev libffi-dev openssl-dev cargo make && pip3 install "pyyaml<=5.3.1" && pip3 install --no-cache-dir --prefer-binary azure-cli && apk del build [2024-01-18T15:28:19.630Z] #8 sha256:4efb1b73fb1716746a3ed94677a3924261e9bd638976a6ba84c5bf7e7caeeffd [2024-01-18T15:28:19.630Z] #8 35.55 Building wheels for collected packages: psutil [2024-01-18T15:28:19.630Z] #8 35.55 Building wheel for psutil (pyproject.toml): started [2024-01-18T15:28:19.630Z] #8 ... [2024-01-18T15:28:19.630Z] [2024-01-18T15:28:19.630Z] #30 [build 10/19] COPY go.mod go.mod [2024-01-18T15:28:19.630Z] #30 sha256:0322fff9a29ce349f68177a7a615abe07e07fe10a7ad9f821caafb4b6d0a90a7 [2024-01-18T15:28:19.630Z] #30 DONE 0.2s [2024-01-18T15:28:19.630Z] [2024-01-18T15:28:19.630Z] #31 [build 11/19] COPY go.sum go.sum [2024-01-18T15:28:19.630Z] #31 sha256:2b8e4404050c23b9e2188d76fdcb844077e788238e6114d97a0aa94a4205a042 [2024-01-18T15:28:19.885Z] #31 DONE 0.1s [2024-01-18T15:28:19.885Z] [2024-01-18T15:28:19.885Z] #32 [build 12/19] COPY pkg pkg [2024-01-18T15:28:19.885Z] #32 sha256:8a92ca449199455e2a172f176dbcfe819656cc96d13149b0f1704f012e06409b [2024-01-18T15:28:20.139Z] #32 DONE 0.1s [2024-01-18T15:28:20.139Z] [2024-01-18T15:28:20.139Z] #33 [build 13/19] COPY porx porx [2024-01-18T15:28:20.140Z] #33 sha256:b4190b219079607821ee33980ebfff01645e49bfdb7b6a6af83b5f67a19c82f0 [2024-01-18T15:28:20.140Z] #33 DONE 0.1s [2024-01-18T15:28:20.140Z] [2024-01-18T15:28:20.140Z] #34 [build 14/19] COPY scripts scripts [2024-01-18T15:28:20.140Z] #34 sha256:708cc787133d92e0c55ef2521b591b2c6ae8c16579e4a68147b37d0aa6dc96b3 [2024-01-18T15:28:20.395Z] #34 DONE 0.1s [2024-01-18T15:28:20.395Z] [2024-01-18T15:28:20.395Z] #35 [build 15/19] COPY drivers drivers [2024-01-18T15:28:20.395Z] #35 sha256:071b1a6aebdc8e6d780d04b75f70907d180baea66464801fd495c694629eb44e [2024-01-18T15:28:20.395Z] #35 DONE 0.2s [2024-01-18T15:28:20.395Z] [2024-01-18T15:28:20.395Z] #36 [build 16/19] COPY deployments deployments [2024-01-18T15:28:20.395Z] #36 sha256:31c144553c4a4b8c068c0319eab2bbf352cd5e1132c808e913e5c7730846b54a [2024-01-18T15:28:20.650Z] #36 DONE 0.1s [2024-01-18T15:28:20.650Z] [2024-01-18T15:28:20.650Z] #37 [build 17/19] COPY .git .git [2024-01-18T15:28:20.650Z] #37 sha256:ab9aad0fc38b78f37119e02572242f4bb9f24ee2e0f077d25c604a2a0c81a225 [2024-01-18T15:28:21.578Z] #37 DONE 0.7s [2024-01-18T15:28:21.578Z] [2024-01-18T15:28:21.578Z] #38 [build 18/19] COPY tests tests [2024-01-18T15:28:21.578Z] #38 sha256:b7c22f4b7293fb76f52936d916a204334dc1188f726385a1ac6d6f9abfb15d81 [2024-01-18T15:28:21.578Z] #38 DONE 0.1s [2024-01-18T15:28:21.578Z] [2024-01-18T15:28:21.578Z] #39 [build 19/19] RUN --mount=type=cache,target=/****/.cache/go-build make build-backup [2024-01-18T15:28:21.578Z] #39 sha256:5acecafccc4f0e05b70bb09b2d722476799083cb626525f912067767c8b30765 [2024-01-18T15:28:26.861Z] #39 5.306 Makefile:46: DOCKER_HUB_REPO not defined, using '****px' instead [2024-01-18T15:28:26.861Z] #39 5.306 Makefile:50: DOCKER_HUB_TAG not defined, using 'latest' instead [2024-01-18T15:28:26.861Z] #39 5.309 mkdir -p /go/src/github.com/portworx/torpedo/bin [2024-01-18T15:28:26.862Z] #39 5.310 go build -tags "daemon" github.com/portworx/torpedo/drivers github.com/portworx/torpedo/drivers/api github.com/portworx/torpedo/drivers/backup github.com/portworx/torpedo/drivers/backup/portworx github.com/portworx/torpedo/drivers/monitor github.com/portworx/torpedo/drivers/monitor/prometheus github.com/portworx/torpedo/drivers/node github.com/portworx/torpedo/drivers/node/aks github.com/portworx/torpedo/drivers/node/aws github.com/portworx/torpedo/drivers/node/gke github.com/portworx/torpedo/drivers/node/ibm github.com/portworx/torpedo/drivers/node/oracle github.com/portworx/torpedo/drivers/node/ssh github.com/portworx/torpedo/drivers/node/vsphere github.com/portworx/torpedo/drivers/objectstore github.com/portworx/torpedo/drivers/pds github.com/portworx/torpedo/drivers/pds/api github.com/portworx/torpedo/drivers/pds/controlplane github.com/portworx/torpedo/drivers/pds/dataservice github.com/portworx/torpedo/drivers/pds/lib github.com/portworx/torpedo/drivers/pds/parameters github.com/portworx/torpedo/drivers/pds/pdsbackup github.com/portworx/torpedo/drivers/pds/pdsrestore github.com/portworx/torpedo/drivers/pds/pdsutils github.com/portworx/torpedo/drivers/pds/targetcluster github.com/portworx/torpedo/drivers/postmanApiLoadDriver github.com/portworx/torpedo/drivers/scheduler github.com/portworx/torpedo/drivers/scheduler/anthos github.com/portworx/torpedo/drivers/scheduler/dcos github.com/portworx/torpedo/drivers/scheduler/k8s github.com/portworx/torpedo/drivers/scheduler/openshift github.com/portworx/torpedo/drivers/scheduler/rke github.com/portworx/torpedo/drivers/scheduler/spec github.com/portworx/torpedo/drivers/vcluster github.com/portworx/torpedo/drivers/volume github.com/portworx/torpedo/drivers/volume/aws github.com/portworx/torpedo/drivers/volume/azure github.com/portworx/torpedo/drivers/volume/gce github.com/portworx/torpedo/drivers/volume/generic_csi github.com/portworx/torpedo/drivers/volume/ibm github.com/portworx/torpedo/drivers/volume/linstor github.com/portworx/torpedo/drivers/volume/ocp github.com/portworx/torpedo/drivers/volume/portworx github.com/portworx/torpedo/drivers/volume/portworx/schedops github.com/portworx/torpedo/drivers/volume/pso github.com/portworx/torpedo/pkg github.com/portworx/torpedo/pkg/aetosutil github.com/portworx/torpedo/pkg/applicationbackup github.com/portworx/torpedo/pkg/asyncdr github.com/portworx/torpedo/pkg/aututils github.com/portworx/torpedo/pkg/email github.com/portworx/torpedo/pkg/errors github.com/portworx/torpedo/pkg/ipv6util github.com/portworx/torpedo/pkg/jirautils github.com/portworx/torpedo/pkg/kvdbutils github.com/portworx/torpedo/pkg/log github.com/portworx/torpedo/pkg/netutil github.com/portworx/torpedo/pkg/osutils github.com/portworx/torpedo/pkg/pureutils github.com/portworx/torpedo/pkg/restutil github.com/portworx/torpedo/pkg/s3utils github.com/portworx/torpedo/pkg/snapshotutils github.com/portworx/torpedo/pkg/stats github.com/portworx/torpedo/pkg/testrailuttils github.com/portworx/torpedo/pkg/units github.com/portworx/torpedo/pkg/vpsutil github.com/portworx/torpedo/porx/px/api [2024-01-18T15:28:31.052Z] #39 ... [2024-01-18T15:28:31.052Z] [2024-01-18T15:28:31.053Z] #8 [stage-1 4/20] RUN apk add --no-cache --update --virtual=build gcc musl-dev python3-dev libffi-dev openssl-dev cargo make && pip3 install "pyyaml<=5.3.1" && pip3 install --no-cache-dir --prefer-binary azure-cli && apk del build [2024-01-18T15:28:31.053Z] #8 sha256:4efb1b73fb1716746a3ed94677a3924261e9bd638976a6ba84c5bf7e7caeeffd [2024-01-18T15:28:31.053Z] #8 37.75 Building wheel for psutil (pyproject.toml): finished with status 'done' [2024-01-18T15:28:31.053Z] #8 37.75 Created wheel for psutil: filename=psutil-5.9.7-cp311-abi3-linux_x86_64.whl size=277372 sha256=5e8f6cfac3b5b13aebf6a66ec4cb344eb843a6377f4c9c71218daa110c779253 [2024-01-18T15:28:31.053Z] #8 37.75 Stored in directory: /tmp/pip-ephem-wheel-cache-sv9god6v/wheels/8a/e0/cf/0387173bc7d441888e75433d029f2d582ecc243b2f03835bc4 [2024-01-18T15:28:31.053Z] #8 37.75 Successfully built psutil [2024-01-18T15:28:31.054Z] #8 38.84 Installing collected packages: jsondiff, invoke, azure-nspkg, azure-common, applicationinsights, antlr4-python3-runtime, xmltodict, wrapt, websocket-client, urllib3, typing-extensions, tabulate, sniffio, six, semver, PySocks, PyJWT, pygments, pycparser, pycomposefile, psutil, portalocker, pkginfo, oauthlib, jmespath, idna, humanfriendly, distro, colorama, charset-normalizer, chardet, certifi, bcrypt, azure-mgmt-nspkg, argcomplete, requests, python-dateutil, pathlib2, knack, javaproperties, isodate, deprecated, cffi, azure-mgmt-datalake-nspkg, azure-cli-telemetry, anyio, requests-oauthlib, PyNaCl, cryptography, azure-cosmos, azure-core, pyopenssl, paramiko, msrest, azure-storage-common, azure-mgmt-core, azure-keyvault-secrets, azure-keyvault-keys, azure-keyvault-certificates, azure-keyvault-administration, adal, sshtunnel, scp, PyGithub, msrestazure, msal, fabric, azure-synapse-spark, azure-synapse-managedprivateendpoints, azure-synapse-artifacts, azure-synapse-accesscontrol, azure-multiapi-storage, azure-mgmt-web, azure-mgmt-trafficmanager, azure-mgmt-synapse, azure-mgmt-storage, azure-mgmt-sqlvirtualmachine, azure-mgmt-sql, azure-mgmt-signalr, azure-mgmt-servicelinker, azure-mgmt-servicefabricmanagedclusters, azure-mgmt-servicefabric, azure-mgmt-servicebus, azure-mgmt-security, azure-mgmt-search, azure-mgmt-resource, azure-mgmt-redis, azure-mgmt-redhatopenshift, azure-mgmt-recoveryservicesbackup, azure-mgmt-recoveryservices, azure-mgmt-rdbms, azure-mgmt-privatedns, azure-mgmt-policyinsights, azure-mgmt-netapp, azure-mgmt-msi, azure-mgmt-monitor, azure-mgmt-media, azure-mgmt-marketplaceordering, azure-mgmt-maps, azure-mgmt-managementgroups, azure-mgmt-loganalytics, azure-mgmt-keyvault, azure-mgmt-iothubprovisioningservices, azure-mgmt-iothub, azure-mgmt-iotcentral, azure-mgmt-imagebuilder, azure-mgmt-hdinsight, azure-mgmt-extendedlocation, azure-mgmt-eventhub, azure-mgmt-eventgrid, azure-mgmt-dns, azure-mgmt-datamigration, azure-mgmt-databoxedge, azure-mgmt-cosmosdb, azure-mgmt-containerservice, azure-mgmt-containerregistry, azure-mgmt-containerinstance, azure-mgmt-compute, azure-mgmt-cognitiveservices, azure-mgmt-cdn, azure-mgmt-botservice, azure-mgmt-billing, azure-mgmt-batchai, azure-mgmt-batch, azure-mgmt-authorization, azure-mgmt-applicationinsights, azure-mgmt-appcontainers, azure-mgmt-appconfiguration, azure-mgmt-apimanagement, azure-mgmt-advisor, azure-loganalytics, azure-data-tables, azure-appconfiguration, msal-extensions, azure-mgmt-managedservices, azure-mgmt-kusto, azure-mgmt-devtestlabs, azure-mgmt-datalake-store, azure-graphrbac, azure-datalake-store, azure-batch, azure-cli-core, azure-cli [2024-01-18T15:30:22.534Z] #8 ... [2024-01-18T15:30:22.534Z] [2024-01-18T15:30:22.534Z] #39 [build 19/19] RUN --mount=type=cache,target=/****/.cache/go-build make build-backup [2024-01-18T15:30:22.534Z] #39 sha256:5acecafccc4f0e05b70bb09b2d722476799083cb626525f912067767c8b30765 [2024-01-18T15:30:22.534Z] #39 111.2 ginkgo build -r ./tests/backup [2024-01-18T15:30:22.534Z] #39 111.2 Compiling backup... [2024-01-18T15:30:25.813Z] #39 ... [2024-01-18T15:30:25.813Z] [2024-01-18T15:30:25.813Z] #8 [stage-1 4/20] RUN apk add --no-cache --update --virtual=build gcc musl-dev python3-dev libffi-dev openssl-dev cargo make && pip3 install "pyyaml<=5.3.1" && pip3 install --no-cache-dir --prefer-binary azure-cli && apk del build [2024-01-18T15:30:25.813Z] #8 sha256:4efb1b73fb1716746a3ed94677a3924261e9bd638976a6ba84c5bf7e7caeeffd [2024-01-18T15:30:25.814Z] #8 161.7 Successfully installed PyGithub-1.59.1 PyJWT-2.8.0 PyNaCl-1.5.0 PySocks-1.7.1 adal-1.2.7 antlr4-python3-runtime-4.13.1 anyio-4.2.0 applicationinsights-0.11.10 argcomplete-3.1.6 azure-appconfiguration-1.1.1 azure-batch-14.0.0 azure-cli-2.56.0 azure-cli-core-2.56.0 azure-cli-telemetry-1.1.0 azure-common-1.1.28 azure-core-1.29.6 azure-cosmos-3.2.0 azure-data-tables-12.4.0 azure-datalake-store-0.0.53 azure-graphrbac-0.60.0 azure-keyvault-administration-4.4.0b2 azure-keyvault-certificates-4.7.0 azure-keyvault-keys-4.9.0b3 azure-keyvault-secrets-4.7.0 azure-loganalytics-0.1.1 azure-mgmt-advisor-9.0.0 azure-mgmt-apimanagement-4.0.0 azure-mgmt-appconfiguration-3.0.0 azure-mgmt-appcontainers-2.0.0 azure-mgmt-applicationinsights-1.0.0 azure-mgmt-authorization-4.0.0 azure-mgmt-batch-17.0.0 azure-mgmt-batchai-7.0.0b1 azure-mgmt-billing-6.0.0 azure-mgmt-botservice-2.0.0 azure-mgmt-cdn-12.0.0 azure-mgmt-cognitiveservices-13.5.0 azure-mgmt-compute-30.4.0 azure-mgmt-containerinstance-10.1.0 azure-mgmt-containerregistry-10.1.0 azure-mgmt-containerservice-28.0.0 azure-mgmt-core-1.4.0 azure-mgmt-cosmosdb-9.4.0 azure-mgmt-databoxedge-1.0.0 azure-mgmt-datalake-nspkg-3.0.1 azure-mgmt-datalake-store-0.5.0 azure-mgmt-datamigration-10.0.0 azure-mgmt-devtestlabs-4.0.0 azure-mgmt-dns-8.0.0 azure-mgmt-eventgrid-10.2.0b2 azure-mgmt-eventhub-10.1.0 azure-mgmt-extendedlocation-1.0.0b2 azure-mgmt-hdinsight-9.0.0 azure-mgmt-imagebuilder-1.2.0 azure-mgmt-iotcentral-10.0.0b2 azure-mgmt-iothub-3.0.0 azure-mgmt-iothubprovisioningservices-1.1.0 azure-mgmt-keyvault-10.3.0 azure-mgmt-kusto-0.3.0 azure-mgmt-loganalytics-13.0.0b4 azure-mgmt-managedservices-1.0.0 azure-mgmt-managementgroups-1.0.0 azure-mgmt-maps-2.0.0 azure-mgmt-marketplaceordering-1.1.0 azure-mgmt-media-9.0.0 azure-mgmt-monitor-5.0.1 azure-mgmt-msi-7.0.0 azure-mgmt-netapp-10.1.0 azure-mgmt-nspkg-3.0.2 azure-mgmt-policyinsights-1.1.0b4 azure-mgmt-privatedns-1.0.0 azure-mgmt-rdbms-10.2.0b13 azure-mgmt-recoveryservices-2.5.0 azure-mgmt-recoveryservicesbackup-7.0.0 azure-mgmt-redhatopenshift-1.4.0 azure-mgmt-redis-14.2.0 azure-mgmt-resource-23.1.0b2 azure-mgmt-search-9.1.0 azure-mgmt-security-5.0.0 azure-mgmt-servicebus-8.2.0 azure-mgmt-servicefabric-1.0.0 azure-mgmt-servicefabricmanagedclusters-1.0.0 azure-mgmt-servicelinker-1.2.0b1 azure-mgmt-signalr-2.0.0b1 azure-mgmt-sql-4.0.0b13 azure-mgmt-sqlvirtualmachine-1.0.0b5 azure-mgmt-storage-21.1.0 azure-mgmt-synapse-2.1.0b5 azure-mgmt-trafficmanager-1.0.0 azure-mgmt-web-7.2.0 azure-multiapi-storage-1.2.0 azure-nspkg-3.0.2 azure-storage-common-1.4.2 azure-synapse-accesscontrol-0.5.0 azure-synapse-artifacts-0.18.0 azure-synapse-managedprivateendpoints-0.4.0 azure-synapse-spark-0.2.0 bcrypt-4.1.2 certifi-2023.11.17 cffi-1.16.0 chardet-5.2.0 charset-normalizer-3.3.2 colorama-0.4.6 cryptography-41.0.7 deprecated-1.2.14 distro-1.9.0 fabric-2.7.1 humanfriendly-10.0 idna-3.6 invoke-1.7.3 isodate-0.6.1 javaproperties-0.5.2 jmespath-1.0.1 jsondiff-2.0.0 knack-0.11.0 msal-1.24.0b2 msal-extensions-1.0.0 msrest-0.7.1 msrestazure-0.6.4 oauthlib-3.2.2 paramiko-3.4.0 pathlib2-2.3.7.post1 pkginfo-1.9.6 portalocker-2.8.2 psutil-5.9.7 pycomposefile-0.0.30 pycparser-2.21 pygments-2.17.2 pyopenssl-23.3.0 python-dateutil-2.8.2 requests-2.31.0 requests-oauthlib-1.3.1 scp-0.13.6 semver-2.13.0 six-1.16.0 sniffio-1.3.0 sshtunnel-0.1.5 tabulate-0.9.0 typing-extensions-4.9.0 urllib3-2.1.0 websocket-client-1.3.3 wrapt-1.16.0 xmltodict-0.13.0 [2024-01-18T15:30:25.814Z] #8 161.7 WARNING: Running pip as the **** user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [2024-01-18T15:30:26.378Z] #8 162.5 WARNING: opening from cache https://dl-cdn.alpinelinux.org/alpine/v3.18/main: No such file or directory [2024-01-18T15:30:26.378Z] #8 162.5 WARNING: opening from cache https://dl-cdn.alpinelinux.org/alpine/v3.18/community: No such file or directory [2024-01-18T15:30:26.378Z] #8 162.5 (1/21) Purging build (20240118.152649) [2024-01-18T15:30:26.378Z] #8 162.5 (2/21) Purging python3-dev (3.11.6-r0) [2024-01-18T15:30:26.378Z] #8 162.5 (3/21) Purging libffi-dev (3.4.4-r2) [2024-01-18T15:30:26.378Z] #8 162.5 (4/21) Purging linux-headers (6.3-r0) [2024-01-18T15:30:26.378Z] #8 162.5 (5/21) Purging openssl-dev (3.1.4-r4) [2024-01-18T15:30:26.378Z] #8 162.5 (6/21) Purging cargo (1.71.1-r0) [2024-01-18T15:30:26.378Z] #8 162.5 (7/21) Purging rust (1.71.1-r0) [2024-01-18T15:30:26.635Z] #8 162.7 (8/21) Purging gcc (12.2.1_git20220924-r10) [2024-01-18T15:30:26.635Z] #8 162.7 (9/21) Purging binutils (2.40-r7) [2024-01-18T15:30:26.635Z] #8 162.7 (10/21) Purging libatomic (12.2.1_git20220924-r10) [2024-01-18T15:30:26.635Z] #8 162.7 (11/21) Purging libgomp (12.2.1_git20220924-r10) [2024-01-18T15:30:26.635Z] #8 162.7 (12/21) Purging musl-dev (1.2.4-r2) [2024-01-18T15:30:26.635Z] #8 162.7 (13/21) Purging make (4.4.1-r1) [2024-01-18T15:30:26.635Z] #8 162.7 (14/21) Purging isl26 (0.26-r1) [2024-01-18T15:30:26.635Z] #8 162.7 (15/21) Purging llvm16-libs (16.0.6-r1) [2024-01-18T15:30:26.635Z] #8 162.7 (16/21) Purging mpc1 (1.3.1-r1) [2024-01-18T15:30:26.635Z] #8 162.7 (17/21) Purging mpfr4 (4.2.0_p12-r0) [2024-01-18T15:30:26.635Z] #8 162.7 (18/21) Purging pkgconf (1.9.5-r0) [2024-01-18T15:30:26.635Z] #8 162.7 (19/21) Purging zstd-libs (1.5.5-r4) [2024-01-18T15:30:26.635Z] #8 162.7 (20/21) Purging gmp (6.2.1-r3) [2024-01-18T15:30:26.635Z] #8 162.7 (21/21) Purging libxml2 (2.11.6-r0) [2024-01-18T15:30:26.635Z] #8 162.7 Executing busybox-1.36.1-r5.trigger [2024-01-18T15:30:26.635Z] #8 162.7 OK: 82 MiB in 51 packages [2024-01-18T15:30:27.568Z] #8 DONE 163.5s [2024-01-18T15:30:27.568Z] [2024-01-18T15:30:27.568Z] #10 [stage-1 5/20] COPY --from=lachlanevenson/k8s-kubectl:latest /usr/local/bin/kubectl /usr/local/bin/kubectl [2024-01-18T15:30:27.568Z] #10 sha256:8d7807f1830622770bf7495a5e4819f9d6f17cf61af9552224aa2d5f5335a261 [2024-01-18T15:30:27.568Z] #10 DONE 0.3s [2024-01-18T15:30:27.568Z] [2024-01-18T15:30:27.568Z] #39 [build 19/19] RUN --mount=type=cache,target=/****/.cache/go-build make build-backup [2024-01-18T15:30:27.568Z] #39 sha256:5acecafccc4f0e05b70bb09b2d722476799083cb626525f912067767c8b30765 [2024-01-18T15:30:27.825Z] #39 ... [2024-01-18T15:30:27.825Z] [2024-01-18T15:30:27.825Z] #12 [stage-1 6/20] COPY --from=alpine/helm:latest /usr/bin/helm /usr/local/bin/helm [2024-01-18T15:30:27.825Z] #12 sha256:a20e85090369584baa57750e8d7a4226d871d8b34c9ced5900810d4a4feb6199 [2024-01-18T15:30:28.081Z] #12 DONE 0.4s [2024-01-18T15:30:28.081Z] [2024-01-18T15:30:28.081Z] #39 [build 19/19] RUN --mount=type=cache,target=/****/.cache/go-build make build-backup [2024-01-18T15:30:28.081Z] #39 sha256:5acecafccc4f0e05b70bb09b2d722476799083cb626525f912067767c8b30765 [2024-01-18T15:30:28.338Z] #39 ... [2024-01-18T15:30:28.338Z] [2024-01-18T15:30:28.338Z] #13 [stage-1 7/20] WORKDIR /torpedo [2024-01-18T15:30:28.338Z] #13 sha256:48a0bd8d431015769e48a2ea6d5a6539a97c53d060030e68d7650afb291f2b4c [2024-01-18T15:30:28.338Z] #13 DONE 0.2s [2024-01-18T15:30:28.338Z] [2024-01-18T15:30:28.338Z] #15 [stage-1 8/20] COPY deployments deployments [2024-01-18T15:30:28.338Z] #15 sha256:f9abdd4b8cc4c9d7118e6b5141c95ae8ae2b9b97ea2495f31fa2ed31526f47a0 [2024-01-18T15:30:28.597Z] #15 DONE 0.2s [2024-01-18T15:30:28.597Z] [2024-01-18T15:30:28.597Z] #16 [stage-1 9/20] COPY scripts scripts [2024-01-18T15:30:28.597Z] #16 sha256:28a2001206a84835375b5ee1b29a2ccc30a6f3907da97127ec01f98ec734d423 [2024-01-18T15:30:28.597Z] #16 DONE 0.2s [2024-01-18T15:30:28.597Z] [2024-01-18T15:30:28.597Z] #39 [build 19/19] RUN --mount=type=cache,target=/****/.cache/go-build make build-backup [2024-01-18T15:30:28.597Z] #39 sha256:5acecafccc4f0e05b70bb09b2d722476799083cb626525f912067767c8b30765 [2024-01-18T15:30:28.856Z] #39 ... [2024-01-18T15:30:28.856Z] [2024-01-18T15:30:28.856Z] #17 [stage-1 10/20] RUN apk update && apk upgrade && apk add --no-cache nodejs npm && rm -rf /var/cache/apk/* [2024-01-18T15:30:28.856Z] #17 sha256:5c9c6ab7f52f155099b7a3a9b59e03bd98ffecf36e152de7b6532243e4656edb [2024-01-18T15:30:29.114Z] #17 0.364 fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz [2024-01-18T15:30:29.372Z] #17 0.736 fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz [2024-01-18T15:30:29.937Z] #17 1.221 v3.18.5-128-g41dea5440d6 [https://dl-cdn.alpinelinux.org/alpine/v3.18/main] [2024-01-18T15:30:29.937Z] #17 1.221 v3.18.5-125-g84553c2552a [https://dl-cdn.alpinelinux.org/alpine/v3.18/community] [2024-01-18T15:30:29.937Z] #17 1.221 OK: 20068 distinct packages available [2024-01-18T15:30:30.502Z] #17 1.620 OK: 82 MiB in 51 packages [2024-01-18T15:30:30.502Z] #17 1.693 fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz [2024-01-18T15:30:30.760Z] #17 1.870 fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz [2024-01-18T15:30:30.760Z] #17 2.179 (1/5) Installing c-ares (1.19.1-r0) [2024-01-18T15:30:31.016Z] #17 2.199 (2/5) Installing icu-data-en (73.2-r2) [2024-01-18T15:30:31.016Z] #17 2.241 Executing icu-data-en-73.2-r2.post-install [2024-01-18T15:30:31.016Z] #17 2.250 * [2024-01-18T15:30:31.016Z] #17 2.250 * If you need ICU with non-English locales and legacy charset support, install [2024-01-18T15:30:31.016Z] #17 2.250 * package icu-data-full. [2024-01-18T15:30:31.016Z] #17 2.250 * [2024-01-18T15:30:31.016Z] #17 2.254 (3/5) Installing icu-libs (73.2-r2) [2024-01-18T15:30:31.016Z] #17 2.314 (4/5) Installing nodejs (18.18.2-r0) [2024-01-18T15:30:31.577Z] #17 2.648 (5/5) Installing npm (9.6.6-r0) [2024-01-18T15:30:31.577Z] #17 2.889 Executing busybox-1.36.1-r5.trigger [2024-01-18T15:30:31.836Z] #17 2.904 OK: 142 MiB in 56 packages [2024-01-18T15:30:32.096Z] #17 DONE 3.2s [2024-01-18T15:30:32.096Z] [2024-01-18T15:30:32.096Z] #18 [stage-1 11/20] RUN npm install -g newman [2024-01-18T15:30:32.096Z] #18 sha256:ab55ec9be5ae124e49d6b89f2b385c12e3b9a093d7115d56ee7915c94b4d9739 [2024-01-18T15:30:38.641Z] #18 5.439 npm WARN deprecated har-validator@5.1.5: this library is no longer supported [2024-01-18T15:30:38.641Z] #18 5.559 npm WARN deprecated uuid@3.4.0: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details. [2024-01-18T15:30:40.029Z] #18 7.659 [2024-01-18T15:30:40.029Z] #18 7.659 added 117 packages in 7s [2024-01-18T15:30:40.029Z] #18 7.659 [2024-01-18T15:30:40.029Z] #18 7.659 9 packages are looking for funding [2024-01-18T15:30:40.029Z] #18 7.659 run `npm fund` for details [2024-01-18T15:30:40.029Z] #18 DONE 8.0s [2024-01-18T15:30:40.029Z] [2024-01-18T15:30:40.029Z] #19 [stage-1 12/20] WORKDIR /go/src/github.com/portworx/torpedo [2024-01-18T15:30:40.029Z] #19 sha256:807b0dd3a1e1bca337e11456ed72560bbfcc6562976f7b14fb107372c4e4e56e [2024-01-18T15:30:40.284Z] #19 DONE 0.1s [2024-01-18T15:30:40.284Z] [2024-01-18T15:30:40.284Z] #20 [stage-1 13/20] RUN apk add --update --no-cache docker [2024-01-18T15:30:40.284Z] #20 sha256:834326aa8614aba962dd157120e980013ce30a18587c089d9f6d7768d1039205 [2024-01-18T15:30:40.539Z] #20 0.291 fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz [2024-01-18T15:30:40.795Z] #20 0.623 fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz [2024-01-18T15:30:41.356Z] #20 1.048 (1/13) Installing libseccomp (2.5.4-r2) [2024-01-18T15:30:41.356Z] #20 1.067 (2/13) Installing runc (1.1.7-r5) [2024-01-18T15:30:41.356Z] #20 1.158 (3/13) Installing containerd (1.7.2-r4) [2024-01-18T15:30:41.916Z] #20 1.793 (4/13) Installing libmnl (1.0.5-r1) [2024-01-18T15:30:41.916Z] #20 1.811 (5/13) Installing libnftnl (1.2.5-r1) [2024-01-18T15:30:41.916Z] #20 1.840 (6/13) Installing iptables (1.8.9-r2) [2024-01-18T15:30:41.916Z] #20 1.882 (7/13) Installing ip6tables (1.8.9-r2) [2024-01-18T15:30:42.171Z] #20 1.903 (8/13) Installing tini-static (0.19.0-r1) [2024-01-18T15:30:42.171Z] #20 1.921 (9/13) Installing device-mapper-libs (2.03.21-r3) [2024-01-18T15:30:42.171Z] #20 1.941 (10/13) Installing docker-engine (23.0.6-r7) [2024-01-18T15:30:42.731Z] #20 2.527 (11/13) Installing docker-cli (23.0.6-r7) [2024-01-18T15:30:42.731Z] #20 2.732 (12/13) Installing docker-cli-buildx (0.10.4-r10) [2024-01-18T15:30:43.291Z] #20 3.125 (13/13) Installing docker (23.0.6-r7) [2024-01-18T15:30:43.291Z] #20 3.144 Executing docker-23.0.6-r7.pre-install [2024-01-18T15:30:43.291Z] #20 3.153 Executing busybox-1.36.1-r5.trigger [2024-01-18T15:30:43.291Z] #20 3.160 OK: 390 MiB in 69 packages [2024-01-18T15:30:43.851Z] #20 DONE 3.6s [2024-01-18T15:30:43.851Z] [2024-01-18T15:30:43.851Z] #39 [build 19/19] RUN --mount=type=cache,target=/****/.cache/go-build make build-backup [2024-01-18T15:30:43.851Z] #39 sha256:5acecafccc4f0e05b70bb09b2d722476799083cb626525f912067767c8b30765 [2024-01-18T15:31:05.755Z] #39 161.4 compiled backup.test [2024-01-18T15:31:05.755Z] #39 161.4 find ./tests/backup -name '*.test' | awk '{cmd="cp "$1" /go/src/github.com/portworx/torpedo/bin"; system(cmd)}' [2024-01-18T15:31:05.755Z] #39 161.6 chmod -R 755 bin/* [2024-01-18T15:31:05.755Z] #39 DONE 162.4s [2024-01-18T15:31:05.755Z] [2024-01-18T15:31:05.755Z] #40 [stage-1 14/20] COPY --from=build /go/bin/ginkgo /bin/ginkgo [2024-01-18T15:31:05.755Z] #40 sha256:53f704fcbb00089f772fbe74b60e54242345cd29b89a569fb2605915d8a2ce3b [2024-01-18T15:31:05.755Z] #40 DONE 0.2s [2024-01-18T15:31:05.755Z] [2024-01-18T15:31:05.755Z] #41 [stage-1 15/20] COPY --from=build /go/src/github.com/portworx/torpedo/bin bin [2024-01-18T15:31:05.755Z] #41 sha256:9fb8f11e4ee7bd6d49c37735ef82098bac4710fec0b79cab1d0e1ecfb95a0151 [2024-01-18T15:31:05.755Z] #41 DONE 0.8s [2024-01-18T15:31:05.755Z] [2024-01-18T15:31:05.755Z] #42 [stage-1 16/20] COPY --from=build /go/src/github.com/portworx/torpedo/bin/aws-iam-authenticator /bin/aws-iam-authenticator [2024-01-18T15:31:05.755Z] #42 sha256:17a7c77c896dd2d46739bfb62f0c85bc2a517d7be3095760410639fb76d71510 [2024-01-18T15:31:06.010Z] #42 DONE 0.4s [2024-01-18T15:31:06.266Z] [2024-01-18T15:31:06.266Z] #43 [stage-1 17/20] COPY --from=build /usr/local/bin/ibmcloud /bin/ibmcloud [2024-01-18T15:31:06.266Z] #43 sha256:7181c08a899608bd0cce7f410cbd354ad291b8f22729bd93125892525855a680 [2024-01-18T15:31:06.521Z] #43 DONE 0.3s [2024-01-18T15:31:06.521Z] [2024-01-18T15:31:06.521Z] #44 [stage-1 18/20] COPY --from=build /usr/local/bin/vcluster /bin/vcluster [2024-01-18T15:31:06.521Z] #44 sha256:ae013f1d83c5a9d2a717fcdaecd493e61a87e98f837bec6b3d291d9935510b0c [2024-01-18T15:31:07.080Z] #44 DONE 0.4s [2024-01-18T15:31:07.080Z] [2024-01-18T15:31:07.080Z] #45 [stage-1 19/20] COPY --from=build /****/.bluemix/plugins /****/.bluemix/plugins [2024-01-18T15:31:07.081Z] #45 sha256:9fa0f770a6d1a8e747847bf8c1142b3eb48868ef6f46a135fe59c21fc17b97a5 [2024-01-18T15:31:07.336Z] #45 DONE 0.4s [2024-01-18T15:31:07.336Z] [2024-01-18T15:31:07.336Z] #46 [stage-1 20/20] COPY drivers drivers [2024-01-18T15:31:07.336Z] #46 sha256:2b263b15180e0f968424c892fd4dc6640039c9c8962c241595715ad4e3109095 [2024-01-18T15:31:07.598Z] #46 DONE 0.3s [2024-01-18T15:31:07.855Z] [2024-01-18T15:31:07.855Z] #47 exporting to image [2024-01-18T15:31:07.855Z] #47 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00 [2024-01-18T15:31:07.855Z] #47 exporting layers [2024-01-18T15:31:20.045Z] #47 exporting layers 11.0s done [2024-01-18T15:31:20.046Z] #47 writing image sha256:b66654a7f54c58583d35fd290f3d75fa59a5b2820cc898ce1f6e4c11964145fa done [2024-01-18T15:31:20.046Z] #47 naming to docker.io/portworx/torpedo-backup:9780382 done [2024-01-18T15:31:20.046Z] #47 DONE 11.0s [Pipeline] sh [2024-01-18T15:31:20.346Z] + docker tag portworx/torpedo-backup:9780382 portworx/torpedo-backup:9780382 [Pipeline] sh [2024-01-18T15:31:20.645Z] + docker push portworx/torpedo-backup:9780382 [2024-01-18T15:31:20.645Z] The push refers to repository [docker.io/portworx/torpedo-backup] [2024-01-18T15:31:20.900Z] fb80f17933ef: Preparing [2024-01-18T15:31:20.900Z] 80669105e3e7: Preparing [2024-01-18T15:31:20.900Z] 024e0d0bf955: Preparing [2024-01-18T15:31:20.900Z] 98520166eb7e: Preparing [2024-01-18T15:31:20.900Z] 70300a914002: Preparing [2024-01-18T15:31:20.900Z] d9f0b38e7d3a: Preparing [2024-01-18T15:31:20.900Z] 210f810f5d54: Preparing [2024-01-18T15:31:20.900Z] 629ae421cf63: Preparing [2024-01-18T15:31:20.900Z] 9d1c36336951: Preparing [2024-01-18T15:31:20.900Z] 2e5035283e85: Preparing [2024-01-18T15:31:20.900Z] 951a068d92dc: Preparing [2024-01-18T15:31:20.900Z] a3f08f586fcc: Preparing [2024-01-18T15:31:20.900Z] 36fdf28779bf: Preparing [2024-01-18T15:31:20.900Z] 50b7c780124d: Preparing [2024-01-18T15:31:20.900Z] 773cfeb5e1e0: Preparing [2024-01-18T15:31:20.900Z] d9f0b38e7d3a: Waiting [2024-01-18T15:31:20.900Z] 2b13bc9f99d5: Preparing [2024-01-18T15:31:20.900Z] 53f1e6d06e0c: Preparing [2024-01-18T15:31:20.900Z] 9d1c36336951: Waiting [2024-01-18T15:31:20.900Z] 210f810f5d54: Waiting [2024-01-18T15:31:20.900Z] 6ae03fcbe71e: Preparing [2024-01-18T15:31:20.900Z] 629ae421cf63: Waiting [2024-01-18T15:31:20.900Z] 7068460b847c: Preparing [2024-01-18T15:31:20.900Z] 9fe9a137fd00: Preparing [2024-01-18T15:31:20.900Z] 2e5035283e85: Waiting [2024-01-18T15:31:20.900Z] 951a068d92dc: Waiting [2024-01-18T15:31:20.900Z] a3f08f586fcc: Waiting [2024-01-18T15:31:20.900Z] 773cfeb5e1e0: Waiting [2024-01-18T15:31:20.900Z] 36fdf28779bf: Waiting [2024-01-18T15:31:20.900Z] 2b13bc9f99d5: Waiting [2024-01-18T15:31:20.900Z] 50b7c780124d: Waiting [2024-01-18T15:31:20.900Z] 53f1e6d06e0c: Waiting [2024-01-18T15:31:20.900Z] 6ae03fcbe71e: Waiting [2024-01-18T15:31:20.900Z] 9fe9a137fd00: Waiting [2024-01-18T15:31:20.900Z] 7068460b847c: Waiting [2024-01-18T15:31:25.082Z] 98520166eb7e: Pushed [2024-01-18T15:31:25.082Z] fb80f17933ef: Pushed [2024-01-18T15:31:26.017Z] 70300a914002: Pushed [2024-01-18T15:31:26.950Z] 80669105e3e7: Pushed [2024-01-18T15:31:27.879Z] 024e0d0bf955: Pushed [2024-01-18T15:31:28.135Z] 210f810f5d54: Pushed [2024-01-18T15:31:29.061Z] 9d1c36336951: Pushed [2024-01-18T15:31:30.957Z] a3f08f586fcc: Pushed [2024-01-18T15:31:32.848Z] 36fdf28779bf: Pushed [2024-01-18T15:31:34.745Z] 951a068d92dc: Pushed [2024-01-18T15:31:34.745Z] 2e5035283e85: Pushed [2024-01-18T15:31:34.745Z] 50b7c780124d: Pushed [2024-01-18T15:31:40.007Z] 2b13bc9f99d5: Pushed [2024-01-18T15:31:40.579Z] 773cfeb5e1e0: Pushed [2024-01-18T15:31:43.100Z] 7068460b847c: Pushed [2024-01-18T15:31:43.659Z] 9fe9a137fd00: Layer already exists [2024-01-18T15:31:45.542Z] d9f0b38e7d3a: Pushed [2024-01-18T15:31:46.102Z] 6ae03fcbe71e: Pushed [2024-01-18T15:31:46.102Z] 629ae421cf63: Pushed [2024-01-18T15:32:18.124Z] 53f1e6d06e0c: Pushed [2024-01-18T15:32:18.124Z] 9780382: digest: sha256:5355b2875fd95a3d243cbdb33dc83ad23f2a29f28502418428de4288563ed4cb size: 4534 [Pipeline] sh [2024-01-18T15:32:18.419Z] + docker system prune -af [2024-01-18T15:32:24.945Z] Deleted Images: [2024-01-18T15:32:24.945Z] untagged: lachlanevenson/k8s-kubectl:latest [2024-01-18T15:32:24.945Z] untagged: lachlanevenson/k8s-kubectl@sha256:6e16ad356091ddfc95cd90fb7145b927531b45798e89d397a4c5e8fb64f8a5f5 [2024-01-18T15:32:24.945Z] deleted: sha256:8c37cace829170f3e776d50ed0ac124b0e0ff8c606c6cdf58dcc65eabd5c598b [2024-01-18T15:32:24.945Z] untagged: portworx/torpedo:master [2024-01-18T15:32:24.945Z] untagged: portworx/torpedo@sha256:fda2a5dbe7c805197737f4d76c2731e845bce3bbd7cddd29e8546af5660c6c2b [2024-01-18T15:32:24.945Z] deleted: sha256:490cf43e5086c61da8ba12d6629ab63826032bc2a18856ceb3c3fd7b13cd18ec [2024-01-18T15:32:24.945Z] deleted: sha256:86367a5c03f7b764433497c40476224d1843e6ece3024491850053a5e96c8790 [2024-01-18T15:32:24.945Z] deleted: sha256:1bec142c169015a9d86f09454f3928c8e249902d48d8cb5a3364988855cca4ba [2024-01-18T15:32:24.945Z] deleted: sha256:40092d66d12d5e4f2ac0f2da8140cdc892a969c5dd8f4a34087f4ebaffdd0835 [2024-01-18T15:32:24.945Z] deleted: sha256:0bbda28cfd44b50c37c9db482efd7a03b95b0fa058af349faa5999af5fef0b55 [2024-01-18T15:32:24.945Z] deleted: sha256:0c8ff74fc04257733cbee469076c15c1f90d24553ae92a92ad705d83a58a57de [2024-01-18T15:32:24.945Z] deleted: sha256:d92fe1f9f0ddf46c81d7e15bbbe18d4d7d31da099f3d4dd32974ce6ac7f252fe [2024-01-18T15:32:24.945Z] deleted: sha256:30527fbcadd114c779ce739ede0aacaf113c49a56693799ed8828b40d76bad4d [2024-01-18T15:32:24.945Z] deleted: sha256:10c88052bb59ae70bdbd3f9c848fa47295187f01a93bfd8cd2a472f3eb577be9 [2024-01-18T15:32:24.945Z] deleted: sha256:aeeb87b3d47732029a8020561f9ce2498b9e8b30efa9533899efc7d4ea28e0ab [2024-01-18T15:32:24.945Z] deleted: sha256:f2afdb7290b41747ba780061479e79cbf71b04d109aadb7be24946585484eb61 [2024-01-18T15:32:24.945Z] deleted: sha256:66cb78e2e0e4053215567c3b2cdcf6418e6304f671cb0adf0e0e4fdb0ade2d42 [2024-01-18T15:32:24.945Z] deleted: sha256:d8e77bed3671374547eea9fed13cba0d2db90ace41995b3a555755fe8bb4252f [2024-01-18T15:32:24.945Z] deleted: sha256:906d92a59cd809794cf88e0a25aa2c5f38a716810222289cb3c09dc6e6d27869 [2024-01-18T15:32:24.945Z] deleted: sha256:49fa5bcf5e041ab9301069c503015f5284fcf16274b819cc663d101d14c39508 [2024-01-18T15:32:24.945Z] deleted: sha256:44bdb5c62ecc8b26833bcbc5aad3541ecf6da263da03b1d5c0acabeb3816aa15 [2024-01-18T15:32:24.945Z] deleted: sha256:3ac955a71e3bd40582ba94e13622ade5940daa6fa04cdbecd1b50794ad54c831 [2024-01-18T15:32:24.945Z] deleted: sha256:76954677b2fcebc870200d21f9bbe2632b992a5f32ea550f49f616d16d6dfd6e [2024-01-18T15:32:24.945Z] deleted: sha256:b2a0ad0dfc92bbcfb0b35c68a21e6d675066f975a862cb6d585d85f206291755 [2024-01-18T15:32:24.945Z] deleted: sha256:17c0a19e79586fdc972c90010070e13ba8a95d427937544e9f112f9e3a300a72 [2024-01-18T15:32:24.945Z] untagged: golang:1.15 [2024-01-18T15:32:24.945Z] untagged: golang@sha256:ea080cc817b02a946461d42c02891bf750e3916c52f7ea8187bccde8f312b59f [2024-01-18T15:32:24.945Z] deleted: sha256:40349a2425efb3650843103390e2ace030c4288c96f3619cafb16320ab021efb [2024-01-18T15:32:24.945Z] deleted: sha256:0e65712b0866a71530b87520496e3b8fc595fd0d34fd2ea94aebce4aef789aa9 [2024-01-18T15:32:24.945Z] deleted: sha256:a0f36ec5803f0417a41c476294fce3a375372eb3ac817cabe8b29cabfecab865 [2024-01-18T15:32:24.945Z] deleted: sha256:a9db896848c0305ce7c6ff993042a61adeecf939a8ba91c6c730b77751fe7d99 [2024-01-18T15:32:24.945Z] deleted: sha256:017b9704876de2443b332b1dfec580d365184b514eb0af43f1d59637e77af9bb [2024-01-18T15:32:24.945Z] deleted: sha256:98fc59c935e697d6375f05f4fa29d0e1ef7e8ece61aed109056926983ada0ef4 [2024-01-18T15:32:24.945Z] deleted: sha256:c21ff68b02e7caf277f5d356e8b323a95e8d3969dd1ab0d9f60e7c8b4a01c874 [2024-01-18T15:32:24.945Z] deleted: sha256:afa3e488a0ee76983343f8aa759e4b7b898db65b715eb90abc81c181388374e3 [2024-01-18T15:32:24.946Z] untagged: portworx/torpedo-backup:9780382 [2024-01-18T15:32:24.946Z] untagged: portworx/torpedo-backup@sha256:5355b2875fd95a3d243cbdb33dc83ad23f2a29f28502418428de4288563ed4cb [2024-01-18T15:32:24.946Z] deleted: sha256:b66654a7f54c58583d35fd290f3d75fa59a5b2820cc898ce1f6e4c11964145fa [2024-01-18T15:32:24.946Z] untagged: portworx/token-generator:0.0.1 [2024-01-18T15:32:24.946Z] untagged: portworx/token-generator@sha256:2d33b5b86725c210bb43df7011c8b2a95fb11fd0334112f9f24e4e8b42958a59 [2024-01-18T15:32:24.946Z] deleted: sha256:fcee67f89d697683c1a0277cfd8928f151bc7d7d4413bf093ba953055e86460d [2024-01-18T15:32:24.946Z] deleted: sha256:ca141d437bde5474557d3e8d087fb8894975d6753f270d0fac9537bdba678c63 [2024-01-18T15:32:24.946Z] deleted: sha256:666868cb74c5b44078f71870c2081d81048820d139fcc81af4c85431ec2a3db0 [2024-01-18T15:32:24.946Z] deleted: sha256:cb381a32b2296e4eb5af3f84092a2e6685e88adbc54ee0768a1a1010ce6376c7 [2024-01-18T15:32:24.946Z] untagged: portworx/spawn:master [2024-01-18T15:32:24.946Z] untagged: portworx/spawn@sha256:f5dc3066a5a4baa8666b195c6d0ea7d5f3b62215fda25b22e191132b47692b25 [2024-01-18T15:32:24.946Z] deleted: sha256:4c7016f10c59ac9fed8c666239bbc92421b73a3e873925f451e62a121cbaa0e1 [2024-01-18T15:32:24.946Z] deleted: sha256:d407010d63f36f9927362438b59d3a0bf8b3e99a64f9b278478da082ac24d484 [2024-01-18T15:32:24.946Z] deleted: sha256:c8ac1c97c3ada79ee26e31e61da2543068752dc498767c654c3555daada68884 [2024-01-18T15:32:24.946Z] deleted: sha256:b81ff35a5151aa33a299cdbfbdbf00a227430fae824199f9e4997db32323f8f5 [2024-01-18T15:32:24.946Z] deleted: sha256:091534d7545600cf0b4500a0ac340969d8d4cd25c1369efa255588955f633f8c [2024-01-18T15:32:24.946Z] deleted: sha256:8ec57f98abf48c7539ec5b3420e9b0bfc202cf281c9202445515ace3bf88a95e [2024-01-18T15:32:24.946Z] deleted: sha256:760a7f7a73bbe00e7ef2c74cd622de4151a16b78ef6bb850ff39e839a163f895 [2024-01-18T15:32:24.946Z] deleted: sha256:c8004e7c19939cd456223b818b4a53c6e6149ccd8b37423e76429297048bc8ab [2024-01-18T15:32:24.946Z] deleted: sha256:1eccb97b06e4929902f14936476f29c1d569a4eb785f6b4d9d1705ed40aa072e [2024-01-18T15:32:24.946Z] deleted: sha256:85e81356cfd86204ad160f24e74c03c81e0c89edf2031767a72afb69a6455647 [2024-01-18T15:32:24.946Z] deleted: sha256:b99d5404371b5c2e803a1b6b6aa1e8e34b375b4f6b9d8abb5992f1d906ca1c70 [2024-01-18T15:32:24.946Z] deleted: sha256:5583f8d47f80013b056ca7cab6cacbd304f14b4b1c121cc4ee031fdb31082da5 [2024-01-18T15:32:24.946Z] deleted: sha256:1b5c71ac5beb9066fc42f110417577eb27aec2d20da4c0d1764e5a4916fd166a [2024-01-18T15:32:24.946Z] deleted: sha256:d48980670ed0e3e8f90df4723b4bd361da140d9bca985bd14bd4072a5bec794a [2024-01-18T15:32:24.946Z] deleted: sha256:3e46a7a149bfb42cd0bf56965601d97bae18b37ef22201b2fcca4a37e4a0c4e7 [2024-01-18T15:32:24.946Z] deleted: sha256:801eaeda859f250ddba484b45742be51f36a08b15e85376a0e10fad4ee113676 [2024-01-18T15:32:24.946Z] deleted: sha256:b8027a0af36d6c300cc5acddd81cd8165a7cbfc9e6aeb24679de9545a933eaf3 [2024-01-18T15:32:24.946Z] deleted: sha256:4e247f0e4b3ae9ee1e54a9a578757c27173ee8c0bd8ac86af24881ab6bfc6572 [2024-01-18T15:32:24.946Z] deleted: sha256:8e87ff28f1b5ff2d5131999ccfa1e674cb252631c50683f5ee43fad59cbea8e1 [2024-01-18T15:32:24.946Z] [2024-01-18T15:32:29.104Z] Deleted build cache objects: [2024-01-18T15:32:29.104Z] vu59rm1e1g7tln8atfxznayv7 [2024-01-18T15:32:29.104Z] nl6c27w30lei7ioxtidnk2zg6 [2024-01-18T15:32:29.104Z] nnq7ans53t7ognqqzkrnqw5k9 [2024-01-18T15:32:29.104Z] in25nczjtzd87xy64c1nsu0v2 [2024-01-18T15:32:29.105Z] prf7ir2r41ks93ket9dp6nao7 [2024-01-18T15:32:29.105Z] 0h9dq3me2ceqw0gvnp164hqr9 [2024-01-18T15:32:29.105Z] 8sd4y115npxl9mmwz8ujqdwvi [2024-01-18T15:32:29.105Z] ylw87oedsg15o46s0t3neb6f3 [2024-01-18T15:32:29.105Z] rrrn9xa6x5cb7hv8c9a466h5t [2024-01-18T15:32:29.105Z] olaavymhrqm8776v3flo12fua [2024-01-18T15:32:29.105Z] 5h1je0vvqks6t3ptckgmprp2j [2024-01-18T15:32:29.105Z] 1hn5uckm5lup0kb17g2lq6xk4 [2024-01-18T15:32:29.105Z] z8e1rvqu7lc33174deuuactqa [2024-01-18T15:32:29.105Z] 09n9uast8jq4mix33c9k3kg11 [2024-01-18T15:32:29.105Z] qcg0w9uqek74ju80xsd3fgyz2 [2024-01-18T15:32:29.105Z] p78y8lzfktwk6qsjhnt2g2wxo [2024-01-18T15:32:29.105Z] vesbjw4fs2p6k39giuu73esps [2024-01-18T15:32:29.105Z] uensqbp9wvxypdskbp7dbqaz0 [2024-01-18T15:32:29.105Z] x0p602rmnpou7jhru8zf83f5e [2024-01-18T15:32:29.105Z] 8pr285734lsswh0eyfvhlgazn [2024-01-18T15:32:29.105Z] i47ixea2klrydi894lpmwq2kl [2024-01-18T15:32:29.105Z] xu2gkgbfhti2g5tmhnmrr0eyr [2024-01-18T15:32:29.105Z] 3rau9z24okalzsdd82liyfqxb [2024-01-18T15:32:29.105Z] mw2n1y7br4yvcb4m1hhzqmjvy [2024-01-18T15:32:29.105Z] wlkxzwimzhmt3iif1dthadj6i [2024-01-18T15:32:29.105Z] ep8u6x3o3cvooohet41wbcfem [2024-01-18T15:32:29.105Z] q2d181s0ut2qx041gt97g0ptf [2024-01-18T15:32:29.105Z] qp4xentht8jdpoooz4oaxug2v [2024-01-18T15:32:29.105Z] p7jm2mgc2ggjbr5qpav9s6vri [2024-01-18T15:32:29.105Z] o0aosywaeyiyjbf4t5ccsfsr4 [2024-01-18T15:32:29.105Z] x4g1b9osm7wnd5xqzkr6cr8bl [2024-01-18T15:32:29.105Z] utlexs5e90ect5g8a2lvieoh8 [2024-01-18T15:32:29.105Z] thkbmrlrwa8m8mesb4gb52i9x [2024-01-18T15:32:29.105Z] mu2kxl1z6u9kpwelmbb6m6lbu [2024-01-18T15:32:29.105Z] 78xm4cvwap5vptifxa8g8pzuf [2024-01-18T15:32:29.105Z] 4bao4tso3qijfwrv3i1g5i7ls [2024-01-18T15:32:29.105Z] uw3j7gxk7yur7bwmfktjksl3d [2024-01-18T15:32:29.105Z] 0v4hjkn88ktd3fbkooia9gg8y [2024-01-18T15:32:29.105Z] lmsjnyg9ir735jrwu3r0ul2g6 [2024-01-18T15:32:29.105Z] gb5yb7o5qdy0ujeslnqiamwmn [2024-01-18T15:32:29.105Z] 2gylzhvz4n0fgburt6h7otaz0 [2024-01-18T15:32:29.105Z] ma9u45bif02kgw0ws7tmxfjyv [2024-01-18T15:32:29.105Z] 31b05bbi70lw07h76comk2nd0 [2024-01-18T15:32:29.105Z] z6jkwoxf4k91ht2aouvp6zwfk [2024-01-18T15:32:29.105Z] qnmub6jw1vnwx4tom3ke3plgj [2024-01-18T15:32:29.105Z] qb9t646aj08jjeacczfz5whqo [2024-01-18T15:32:29.105Z] uf3bhek3ekhspfzki0cefir85 [2024-01-18T15:32:29.105Z] rhglwf6bj2em2zwel5up18ce5 [2024-01-18T15:32:29.105Z] qcu4yw4ysfv41j90l64wwbuib [2024-01-18T15:32:29.105Z] s304b2dtxuhf63ub0vro9rs78 [2024-01-18T15:32:29.105Z] cv7uiix9rzpc15pcb6bzygdfk [2024-01-18T15:32:29.105Z] 8lwz2eyovq6f3iz1rdsjk5wxl [2024-01-18T15:32:29.105Z] [2024-01-18T15:32:29.105Z] Total reclaimed space: 13.96GB [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Configure GCP Credentials) [Pipeline] script [Pipeline] { [Pipeline] withCredentials [2024-01-18T15:32:29.400Z] Masking supported pattern matches of $GCP_SA_KEY_JSON [Pipeline] { [Pipeline] sh [2024-01-18T15:32:29.760Z] + cp **** /****/workspace/px-backup-system-tests/iks/iks/4cac76dc/gcp.json [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Populate cloud credentials from key store) [Pipeline] script [Pipeline] { [Pipeline] sh [2024-01-18T15:32:30.012Z] Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure. [2024-01-18T15:32:30.018Z] Affected argument(s) used the following variable(s): [PX_BACKUP_MONGODB_USERNAME] [2024-01-18T15:32:30.018Z] See https://jenkins.io/redirect/groovy-string-interpolation for details. [2024-01-18T15:32:30.293Z] + touch /****/workspace/px-backup-system-tests/iks/iks/4cac76dc/cloud_config.json [Pipeline] writeJSON [2024-01-18T15:32:30.315Z] Warning: A secret was passed to "writeJSON" using Groovy String interpolation, which is insecure. [2024-01-18T15:32:30.315Z] Affected argument(s) used the following variable(s): [PX_BACKUP_MONGODB_USERNAME] [2024-01-18T15:32:30.315Z] See https://jenkins.io/redirect/groovy-string-interpolation for details. [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Run System Tests) [Pipeline] script [Pipeline] { [Pipeline] echo [2024-01-18T15:32:30.554Z] Warning: A secret was passed to "echo" using Groovy String interpolation, which is insecure. [2024-01-18T15:32:30.554Z] Affected argument(s) used the following variable(s): [SSH_PASSWORD, IBM_ACCESS_KEY, IBM_SECRET_KEY, PX_BACKUP_MONGODB_USERNAME, IBM_CLOUD_API_KEY] [2024-01-18T15:32:30.554Z] See https://jenkins.io/redirect/groovy-string-interpolation for details. [2024-01-18T15:32:30.555Z] Docker Command: docker run --net=host --rm -t -v /var/run/docker.sock:/var/run/docker.sock -v /****/workspace/px-backup-system-tests/iks/iks/4cac76dc/torpedo/deployments:/deployments -v /****/workspace/px-backup-system-tests/iks/iks/4cac76dc/clusters/source-config:/source-cluster/source-config -v /****/workspace/px-backup-system-tests/iks/iks/4cac76dc/clusters/destination-config:/destination-cluster/destination-config -v /****/workspace/px-backup-system-tests/iks/iks/4cac76dc/cloud_config.json:/config/cloud-json -e TORPEDO_IMG=portworx/torpedo-backup:9780382 -e FOCUS_TESTS=BasicBackupCreation,AddMultipleNamespaceLabels,AllNSBackupWithIncludeNewNSOption,BackupLocationWithEncryptionKey,BackupMultipleNsWithSameLabel,BasicSelectiveRestore,CancelAllRunningBackupJobs,CancelAllRunningRestoreJobs,CancelClusterBackupShare,CreateMultipleUsersAndGroups,CustomResourceBackupAndRestore,CustomResourceRestore,DeleteAllBackupObjects,DeleteUsersRole,DifferentAccessSameUser,DuplicateSharedBackup,LicensingCountBeforeAndAfterBackupPodRestart,LicensingCountWithNodeLabelledBeforeClusterAddition,ManualAndScheduleBackupUsingNamespaceLabel,MultipleInPlaceRestoreSameTime,NamespaceLabelledBackupSharedWithDifferentAccessMode,NodeCountForLicensing,RestartBackupPodDuringBackupSharing,RestoreEncryptedAndNonEncryptedBackups,ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress,ScheduleBackupCreationAllNS,ScheduleBackupCreationSingleNS,SetUnsetNSLabelDuringScheduleBackup,ShareAndRemoveBackupLocation,ShareBackupAndEdit,ShareBackupsAndClusterWithUser,ShareBackupWithUsersAndGroups,SharedBackupDelete,SwapShareBackup,ScheduleBackupDeleteAndRecreateNS,BackupScheduleForOldAndNewNS,DeleteNSDeleteClusterRestore,ManualAndScheduledBackupUsingNamespaceAndResourceLabel,ScheduleBackupWithAdditionAndRemovalOfNS,ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit,DeleteBackupOfUserSharedRBAC,UpdatesBackupOfUserFromAdmin,DeleteBackupSharedByMultipleUsersFromAdmin,DeleteSameNameObjectsByMultipleUsersFromAdmin,DeleteSharedBackupOfUserFromAdmin,DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin,DeleteObjectsByMultipleUsersFromNewAdmin,NamespaceLabelledBackupOfEmptyNamespace,VerifyRBACForPxAdmin,VerifyRBACForInfraAdmin,VerifyRBACForAppAdmin,VerifyRBACForAppUser,KubeAndPxNamespacesSkipOnAllNSBackup -e SCALE_FACTOR=1 -e FAIL_FAST=false -e APP_LIST=mysql-ibm -e KUBECONFIGS=/source-cluster/source-config,/destination-cluster/destination-config -e TEST_SUITE=bin/backup.test -e BACKUP_DRIVER=pxb -e VERBOSE=true -e TEST_DESCRIPTION=px-backup-system-test-4cac76dc -e TEST_TYPE=system-tests -e TEST_TAGS=flavour:iks,provisioner:iks,cadence:on-demand -e BRANCH=ibm-mysql -e PRODUCT=px-backup -e SKIP_DIAG_COLLECTION=true -e DRIVER_START_TIMEOUT=30m0s -e CLUSTER_PROVIDER=ibm -e RKE_SOURCE_TOKEN= -e RKE_DESTINATION_TOKEN= -e PROVISIONER=ibm -e FLAVOUR=ibm -e VOLUME_SNAPSHOT_CLASS=ibmc-vpcblock-snapshot -e KUBECONFIG=/source-cluster/source-config -e TORPEDO_SSH_PASSWORD=**** -e BACKUP_TYPE=native_csi -e STORAGE_DRIVER=ibm -e IBMCLOUD_API_KEY=**** -e PROVIDERS=aws -e AWS_ACCESS_KEY_ID=**** -e AWS_SECRET_ACCESS_KEY=**** -e S3_ENDPOINT=s3.us-east.cloud-object-storage.appdomain.cloud -e BUCKET_NAME=px-backup-system-test-4cac76dc -e S3_REGION=us-east-1 -e S3_DISABLE_SSL=FALSE portworx/spawn:master /bin/bash /deployments/deploy-pxb-cloud.sh [Pipeline] sh [2024-01-18T15:32:30.835Z] + docker run --net=host --rm -t -v /var/run/docker.sock:/var/run/docker.sock -v /****/workspace/px-backup-system-tests/iks/iks/4cac76dc/torpedo/deployments:/deployments -v /****/workspace/px-backup-system-tests/iks/iks/4cac76dc/clusters/source-config:/source-cluster/source-config -v /****/workspace/px-backup-system-tests/iks/iks/4cac76dc/clusters/destination-config:/destination-cluster/destination-config -v /****/workspace/px-backup-system-tests/iks/iks/4cac76dc/cloud_config.json:/config/cloud-json -e TORPEDO_IMG=portworx/torpedo-backup:9780382 -e FOCUS_TESTS=BasicBackupCreation,AddMultipleNamespaceLabels,AllNSBackupWithIncludeNewNSOption,BackupLocationWithEncryptionKey,BackupMultipleNsWithSameLabel,BasicSelectiveRestore,CancelAllRunningBackupJobs,CancelAllRunningRestoreJobs,CancelClusterBackupShare,CreateMultipleUsersAndGroups,CustomResourceBackupAndRestore,CustomResourceRestore,DeleteAllBackupObjects,DeleteUsersRole,DifferentAccessSameUser,DuplicateSharedBackup,LicensingCountBeforeAndAfterBackupPodRestart,LicensingCountWithNodeLabelledBeforeClusterAddition,ManualAndScheduleBackupUsingNamespaceLabel,MultipleInPlaceRestoreSameTime,NamespaceLabelledBackupSharedWithDifferentAccessMode,NodeCountForLicensing,RestartBackupPodDuringBackupSharing,RestoreEncryptedAndNonEncryptedBackups,ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress,ScheduleBackupCreationAllNS,ScheduleBackupCreationSingleNS,SetUnsetNSLabelDuringScheduleBackup,ShareAndRemoveBackupLocation,ShareBackupAndEdit,ShareBackupsAndClusterWithUser,ShareBackupWithUsersAndGroups,SharedBackupDelete,SwapShareBackup,ScheduleBackupDeleteAndRecreateNS,BackupScheduleForOldAndNewNS,DeleteNSDeleteClusterRestore,ManualAndScheduledBackupUsingNamespaceAndResourceLabel,ScheduleBackupWithAdditionAndRemovalOfNS,ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit,DeleteBackupOfUserSharedRBAC,UpdatesBackupOfUserFromAdmin,DeleteBackupSharedByMultipleUsersFromAdmin,DeleteSameNameObjectsByMultipleUsersFromAdmin,DeleteSharedBackupOfUserFromAdmin,DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin,DeleteObjectsByMultipleUsersFromNewAdmin,NamespaceLabelledBackupOfEmptyNamespace,VerifyRBACForPxAdmin,VerifyRBACForInfraAdmin,VerifyRBACForAppAdmin,VerifyRBACForAppUser,KubeAndPxNamespacesSkipOnAllNSBackup -e SCALE_FACTOR=1 -e FAIL_FAST=false -e APP_LIST=mysql-ibm -e KUBECONFIGS=/source-cluster/source-config,/destination-cluster/destination-config -e TEST_SUITE=bin/backup.test -e BACKUP_DRIVER=pxb -e VERBOSE=true -e TEST_DESCRIPTION=px-backup-system-test-4cac76dc -e TEST_TYPE=system-tests -e TEST_TAGS=flavour:iks,provisioner:iks,cadence:on-demand -e BRANCH=ibm-mysql -e PRODUCT=px-backup -e SKIP_DIAG_COLLECTION=true -e DRIVER_START_TIMEOUT=30m0s -e CLUSTER_PROVIDER=ibm -e RKE_SOURCE_TOKEN= -e RKE_DESTINATION_TOKEN= -e PROVISIONER=ibm -e FLAVOUR=ibm -e VOLUME_SNAPSHOT_CLASS=ibmc-vpcblock-snapshot -e KUBECONFIG=/source-cluster/source-config -e TORPEDO_SSH_PASSWORD=**** -e BACKUP_TYPE=native_csi -e STORAGE_DRIVER=ibm -e IBMCLOUD_API_KEY=**** -e PROVIDERS=aws -e AWS_ACCESS_KEY_ID=**** -e AWS_SECRET_ACCESS_KEY=**** -e S3_ENDPOINT=s3.us-east.cloud-object-storage.appdomain.cloud -e BUCKET_NAME=px-backup-system-test-4cac76dc -e S3_REGION=us-east-1 -e S3_DISABLE_SSL=FALSE portworx/spawn:master /bin/bash /deployments/deploy-pxb-cloud.sh [2024-01-18T15:32:30.835Z] Using default timeout of 720h0m0s [2024-01-18T15:32:30.835Z] Using default secret type of k8s [2024-01-18T15:32:30.835Z] Using default license expiry timeout of 1h0m0s [2024-01-18T15:32:30.835Z] Using default metering of 10m0s [2024-01-18T15:32:30.835Z] Using default storage node recovery timeout of 35m0s [2024-01-18T15:32:30.835Z] Using default AWS_REGION of us-east-1 [2024-01-18T15:32:30.835Z] checking if we need to override test suite: bin/backup.test [2024-01-18T15:32:30.835Z] Using test suite: bin/backup.test [2024-01-18T15:32:31.760Z] Error from server (NotFound): secrets "torpedo" not found [2024-01-18T15:32:32.324Z] pod "torpedo" deleted [2024-01-18T15:32:32.886Z] Error from server (NotFound): pods "torpedo" not found [2024-01-18T15:32:32.886Z] Create configs for providers,aws [2024-01-18T15:32:33.143Z] error: failed to create configmap: configmaps "cloud-config" already exists [2024-01-18T15:32:33.143Z] Create kubeconfig configmap,/source-cluster/source-config,/destination-cluster/destination-config [2024-01-18T15:32:33.400Z] error: failed to create configmap: configmaps "kubeconfigs" already exists [2024-01-18T15:32:33.400Z] --- [2024-01-18T15:32:33.400Z] apiVersion: v1 [2024-01-18T15:32:33.400Z] kind: ServiceAccount [2024-01-18T15:32:33.400Z] metadata: [2024-01-18T15:32:33.400Z] name: torpedo-account [2024-01-18T15:32:33.400Z] --- [2024-01-18T15:32:33.400Z] kind: ClusterRole [2024-01-18T15:32:33.400Z] apiVersion: rbac.authorization.k8s.io/v1 [2024-01-18T15:32:33.400Z] metadata: [2024-01-18T15:32:33.400Z] name: torpedo-role [2024-01-18T15:32:33.400Z] rules: [2024-01-18T15:32:33.400Z] - [2024-01-18T15:32:33.400Z] apiGroups: [2024-01-18T15:32:33.400Z] # have access to everything except Secrets [2024-01-18T15:32:33.400Z] - "*" [2024-01-18T15:32:33.400Z] resources: ["*"] [2024-01-18T15:32:33.400Z] verbs: ["*"] [2024-01-18T15:32:33.400Z] - nonResourceURLs: ["*"] [2024-01-18T15:32:33.400Z] verbs: ["*"] [2024-01-18T15:32:33.400Z] --- [2024-01-18T15:32:33.400Z] kind: ClusterRoleBinding [2024-01-18T15:32:33.400Z] apiVersion: rbac.authorization.k8s.io/v1 [2024-01-18T15:32:33.400Z] metadata: [2024-01-18T15:32:33.400Z] name: torpedo-role-binding [2024-01-18T15:32:33.400Z] subjects: [2024-01-18T15:32:33.400Z] - kind: ServiceAccount [2024-01-18T15:32:33.400Z] name: torpedo-account [2024-01-18T15:32:33.400Z] namespace: default [2024-01-18T15:32:33.400Z] roleRef: [2024-01-18T15:32:33.400Z] kind: ClusterRole [2024-01-18T15:32:33.400Z] name: torpedo-role [2024-01-18T15:32:33.400Z] apiGroup: rbac.authorization.k8s.io [2024-01-18T15:32:33.400Z] --- [2024-01-18T15:32:33.400Z] kind: PersistentVolumeClaim [2024-01-18T15:32:33.400Z] apiVersion: v1 [2024-01-18T15:32:33.400Z] metadata: [2024-01-18T15:32:33.400Z] name: testresults [2024-01-18T15:32:33.400Z] spec: [2024-01-18T15:32:33.400Z] accessModes: [2024-01-18T15:32:33.400Z] - ReadWriteOnce [2024-01-18T15:32:33.400Z] resources: [2024-01-18T15:32:33.400Z] requests: [2024-01-18T15:32:33.400Z] storage: 10Gi [2024-01-18T15:32:33.400Z] --- [2024-01-18T15:32:33.400Z] apiVersion: v1 [2024-01-18T15:32:33.400Z] kind: Pod [2024-01-18T15:32:33.400Z] metadata: [2024-01-18T15:32:33.400Z] name: torpedo [2024-01-18T15:32:33.400Z] labels: [2024-01-18T15:32:33.400Z] app: torpedo [2024-01-18T15:32:33.400Z] spec: [2024-01-18T15:32:33.400Z] tolerations: [2024-01-18T15:32:33.400Z] - key: node-role.kubernetes.io/master [2024-01-18T15:32:33.400Z] operator: Equal [2024-01-18T15:32:33.400Z] effect: NoSchedule [2024-01-18T15:32:33.400Z] - key: node-role.kubernetes.io/controlplane [2024-01-18T15:32:33.400Z] operator: Equal [2024-01-18T15:32:33.400Z] value: "true" [2024-01-18T15:32:33.400Z] - key: node-role.kubernetes.io/control-plane [2024-01-18T15:32:33.400Z] operator: Exists [2024-01-18T15:32:33.400Z] - key: node-role.kubernetes.io/etcd [2024-01-18T15:32:33.400Z] operator: Equal [2024-01-18T15:32:33.400Z] value: "true" [2024-01-18T15:32:33.400Z] - key: apps [2024-01-18T15:32:33.400Z] operator: Equal [2024-01-18T15:32:33.400Z] value: "false" [2024-01-18T15:32:33.400Z] effect: "NoSchedule" [2024-01-18T15:32:33.400Z] affinity: [2024-01-18T15:32:33.400Z] nodeAffinity: [2024-01-18T15:32:33.400Z] requiredDuringSchedulingIgnoredDuringExecution: [2024-01-18T15:32:33.400Z] nodeSelectorTerms: [2024-01-18T15:32:33.400Z] - matchExpressions: [2024-01-18T15:32:33.400Z] - key: node-role.kubernetes.io/master [2024-01-18T15:32:33.400Z] operator: "Exists" [2024-01-18T15:32:33.400Z] - matchExpressions: [2024-01-18T15:32:33.400Z] - key: node-role.kubernetes.io/control-plane [2024-01-18T15:32:33.400Z] operator: "Exists" [2024-01-18T15:32:33.400Z] - matchExpressions: [2024-01-18T15:32:33.401Z] - key: node-role.kubernetes.io/controlplane [2024-01-18T15:32:33.401Z] operator: "In" [2024-01-18T15:32:33.401Z] values: ["true"] [2024-01-18T15:32:33.401Z] - matchExpressions: [2024-01-18T15:32:33.401Z] - key: px/enabled [2024-01-18T15:32:33.401Z] operator: "In" [2024-01-18T15:32:33.401Z] values: ["false"] [2024-01-18T15:32:33.401Z] initContainers: [2024-01-18T15:32:33.401Z] - name: init-sysctl [2024-01-18T15:32:33.401Z] image: busybox [2024-01-18T15:32:33.401Z] imagePullPolicy: IfNotPresent [2024-01-18T15:32:33.401Z] securityContext: [2024-01-18T15:32:33.401Z] privileged: true [2024-01-18T15:32:33.401Z] command: ["sh", "-c", "mkdir -p /mnt/testresults && chmod 777 /mnt/testresults/"] [2024-01-18T15:32:33.401Z] containers: [2024-01-18T15:32:33.401Z] - name: torpedo [2024-01-18T15:32:33.401Z] image: portworx/torpedo-backup:9780382 [2024-01-18T15:32:33.401Z] imagePullPolicy: Always [2024-01-18T15:32:33.401Z] securityContext: [2024-01-18T15:32:33.401Z] privileged: false [2024-01-18T15:32:33.401Z] command: [ "ginkgo" ] [2024-01-18T15:32:33.401Z] args: [ "--trace", [2024-01-18T15:32:33.401Z] "--timeout", "720h0m0s", [2024-01-18T15:32:33.401Z] "-keepGoing", [2024-01-18T15:32:33.401Z] "--slowSpecThreshold", "600", [2024-01-18T15:32:33.401Z] "--focus={BasicBackupCreation}|{AddMultipleNamespaceLabels}|{AllNSBackupWithIncludeNewNSOption}|{BackupLocationWithEncryptionKey}|{BackupMultipleNsWithSameLabel}|{BasicSelectiveRestore}|{CancelAllRunningBackupJobs}|{CancelAllRunningRestoreJobs}|{CancelClusterBackupShare}|{CreateMultipleUsersAndGroups}|{CustomResourceBackupAndRestore}|{CustomResourceRestore}|{DeleteAllBackupObjects}|{DeleteUsersRole}|{DifferentAccessSameUser}|{DuplicateSharedBackup}|{LicensingCountBeforeAndAfterBackupPodRestart}|{LicensingCountWithNodeLabelledBeforeClusterAddition}|{ManualAndScheduleBackupUsingNamespaceLabel}|{MultipleInPlaceRestoreSameTime}|{NamespaceLabelledBackupSharedWithDifferentAccessMode}|{NodeCountForLicensing}|{RestartBackupPodDuringBackupSharing}|{RestoreEncryptedAndNonEncryptedBackups}|{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}|{ScheduleBackupCreationAllNS}|{ScheduleBackupCreationSingleNS}|{SetUnsetNSLabelDuringScheduleBackup}|{ShareAndRemoveBackupLocation}|{ShareBackupAndEdit}|{ShareBackupsAndClusterWithUser}|{ShareBackupWithUsersAndGroups}|{SharedBackupDelete}|{SwapShareBackup}|{ScheduleBackupDeleteAndRecreateNS}|{BackupScheduleForOldAndNewNS}|{DeleteNSDeleteClusterRestore}|{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}|{ScheduleBackupWithAdditionAndRemovalOfNS}|{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}|{DeleteBackupOfUserSharedRBAC}|{UpdatesBackupOfUserFromAdmin}|{DeleteBackupSharedByMultipleUsersFromAdmin}|{DeleteSameNameObjectsByMultipleUsersFromAdmin}|{DeleteSharedBackupOfUserFromAdmin}|{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}|{DeleteObjectsByMultipleUsersFromNewAdmin}|{NamespaceLabelledBackupOfEmptyNamespace}|{VerifyRBACForPxAdmin}|{VerifyRBACForInfraAdmin}|{VerifyRBACForAppAdmin}|{VerifyRBACForAppUser}|{KubeAndPxNamespacesSkipOnAllNSBackup}", [2024-01-18T15:32:33.401Z] "", [2024-01-18T15:32:33.401Z] bin/backup.test, [2024-01-18T15:32:33.401Z] "--", [2024-01-18T15:32:33.401Z] "--spec-dir", ../drivers/scheduler/k8s/specs, [2024-01-18T15:32:33.401Z] "--app-list", "mysql-ibm", [2024-01-18T15:32:33.401Z] "--deploy-pds-apps=false", [2024-01-18T15:32:33.401Z] "--pds-driver", "", [2024-01-18T15:32:33.401Z] "--secure-apps", "", [2024-01-18T15:32:33.401Z] "--repl1-apps", "", [2024-01-18T15:32:33.401Z] "--csi-app-list", "", [2024-01-18T15:32:33.401Z] "--scheduler", "k8s", [2024-01-18T15:32:33.401Z] "--max-storage-nodes-per-az", "2", [2024-01-18T15:32:33.401Z] "--backup-driver", "pxb", [2024-01-18T15:32:33.401Z] "--log-level", "debug", [2024-01-18T15:32:33.401Z] "--node-driver", "ssh", [2024-01-18T15:32:33.401Z] "--scale-factor", "1", [2024-01-18T15:32:33.401Z] "--hyper-converged=true", [2024-01-18T15:32:33.401Z] "--fail-on-px-pod-restartcount=false", [2024-01-18T15:32:33.401Z] "--minimun-runtime-mins", "0", [2024-01-18T15:32:33.401Z] "--driver-start-timeout", "30m0s", [2024-01-18T15:32:33.401Z] "--chaos-level", "5", [2024-01-18T15:32:33.401Z] "--storagenode-recovery-timeout", "35m0s", [2024-01-18T15:32:33.401Z] "--provisioner", "ibm", [2024-01-18T15:32:33.401Z] "--storage-driver", "ibm", [2024-01-18T15:32:33.401Z] "--config-map", "", [2024-01-18T15:32:33.401Z] "--custom-config", "", [2024-01-18T15:32:33.401Z] "--storage-upgrade-endpoint-url=", [2024-01-18T15:32:33.401Z] "--storage-upgrade-endpoint-version=", [2024-01-18T15:32:33.401Z] "--upgrade-storage-driver-endpoint-list=", [2024-01-18T15:32:33.401Z] "--enable-stork-upgrade=false", [2024-01-18T15:32:33.401Z] "--secret-type=k8s", [2024-01-18T15:32:33.401Z] "--pure-volumes=false", [2024-01-18T15:32:33.402Z] "--pure-fa-snapshot-restore-to-many-test=false", [2024-01-18T15:32:33.402Z] "--pure-san-type=ISCSI", [2024-01-18T15:32:33.402Z] "--vault-addr=", [2024-01-18T15:32:33.402Z] "--vault-token=", [2024-01-18T15:32:33.402Z] "--px-runtime-opts=", [2024-01-18T15:32:33.402Z] "--anthos-ws-node-ip=", [2024-01-18T15:32:33.402Z] "--anthos-inst-path=", [2024-01-18T15:32:33.402Z] "--autopilot-upgrade-version=", [2024-01-18T15:32:33.402Z] "--csi-generic-driver-config-map=", [2024-01-18T15:32:33.402Z] "--sched-upgrade-hops=", [2024-01-18T15:32:33.402Z] "--migration-hops=", [2024-01-18T15:32:33.402Z] "--license_expiry_timeout_hours=1h0m0s", [2024-01-18T15:32:33.402Z] "--metering_interval_mins=10m0s", [2024-01-18T15:32:33.402Z] "--testrail-milestone=", [2024-01-18T15:32:33.402Z] "--testrail-run-name=", [2024-01-18T15:32:33.402Z] "--testrail-run-id=", [2024-01-18T15:32:33.402Z] "--testrail-jeknins-build-url=", [2024-01-18T15:32:33.402Z] "--testrail-host=", [2024-01-18T15:32:33.402Z] "--testrail-username=", [2024-01-18T15:32:33.402Z] "--testrail-password=", [2024-01-18T15:32:33.402Z] "--jira-username=", [2024-01-18T15:32:33.402Z] "--jira-token=", [2024-01-18T15:32:33.402Z] "--jira-account-id=", [2024-01-18T15:32:33.402Z] "--user=", [2024-01-18T15:32:33.402Z] "--enable-dash=true", [2024-01-18T15:32:33.402Z] "--test-desc=px-backup-system-test-4cac76dc", [2024-01-18T15:32:33.402Z] "--test-type=system-tests", [2024-01-18T15:32:33.402Z] "--test-tags=flavour:iks,provisioner:iks,cadence:on-demand", [2024-01-18T15:32:33.402Z] "--testset-id=0", [2024-01-18T15:32:33.402Z] "--branch=ibm-mysql", [2024-01-18T15:32:33.402Z] "--product=px-backup", [2024-01-18T15:32:33.402Z] "--torpedo-job-name=torpedo-daily-job", [2024-01-18T15:32:33.402Z] "--torpedo-job-type=functional", [2024-01-18T15:32:33.402Z] "", [2024-01-18T15:32:33.402Z] ] [2024-01-18T15:32:33.402Z] tty: true [2024-01-18T15:32:33.402Z] volumeMounts: [{ "name": "testresults", "mountPath": "/testresults/" }] [2024-01-18T15:32:33.402Z] env: [2024-01-18T15:32:33.402Z] - name: NODE_NAME [2024-01-18T15:32:33.402Z] valueFrom: [2024-01-18T15:32:33.402Z] fieldRef: [2024-01-18T15:32:33.402Z] fieldPath: spec.nodeName [2024-01-18T15:32:33.402Z] - name: K8S_VENDOR [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: TORPEDO_SSH_USER [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: TORPEDO_SSH_PASSWORD [2024-01-18T15:32:33.402Z] value: "****" [2024-01-18T15:32:33.402Z] - name: TORPEDO_SSH_KEY [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: AZURE_TENANT_ID [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: VOLUME_PROVIDER [2024-01-18T15:32:33.402Z] value: "none" [2024-01-18T15:32:33.402Z] - name: OBJECT_STORE_PROVIDER [2024-01-18T15:32:33.402Z] value: "aws" [2024-01-18T15:32:33.402Z] - name: AZURE_CLIENT_ID [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: AZURE_CLIENT_SECRET [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: AZURE_ACCOUNT_NAME [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: AZURE_ACCOUNT_KEY [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: AZURE_SUBSCRIPTION_ID [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: AZURE_CLUSTER_NAME [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: AWS_ACCESS_KEY_ID [2024-01-18T15:32:33.402Z] value: "****" [2024-01-18T15:32:33.402Z] - name: AWS_SECRET_ACCESS_KEY [2024-01-18T15:32:33.402Z] value: "****" [2024-01-18T15:32:33.402Z] - name: AWS_REGION [2024-01-18T15:32:33.402Z] value: "us-east-1" [2024-01-18T15:32:33.402Z] - name: KUBECONFIGS [2024-01-18T15:32:33.402Z] value: "source-config,destination-config" [2024-01-18T15:32:33.402Z] - name: S3_ENDPOINT [2024-01-18T15:32:33.402Z] value: "s3.us-east.cloud-object-storage.appdomain.cloud" [2024-01-18T15:32:33.402Z] - name: S3_AWS_ACCESS_KEY_ID [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: S3_AWS_SECRET_ACCESS_KEY [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: S3_REGION [2024-01-18T15:32:33.402Z] value: "us-east-1" [2024-01-18T15:32:33.402Z] - name: BUCKET_NAME [2024-01-18T15:32:33.402Z] value: "px-backup-system-test-4cac76dc" [2024-01-18T15:32:33.402Z] - name: LOCKED_BUCKET_NAME [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: S3_DISABLE_SSL [2024-01-18T15:32:33.402Z] value: "FALSE" [2024-01-18T15:32:33.402Z] - name: DIAGS_BUCKET [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: PROVIDERS [2024-01-18T15:32:33.402Z] value: "aws" [2024-01-18T15:32:33.402Z] - name: CLUSTER_PROVIDER [2024-01-18T15:32:33.402Z] value: "ibm" [2024-01-18T15:32:33.402Z] - name: INTERNAL_DOCKER_REGISTRY [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: IMAGE_PULL_SERVER [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: IMAGE_PULL_USERNAME [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: IMAGE_PULL_PASSWORD [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: VSPHERE_USER [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: VSPHERE_PWD [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: VSPHERE_HOST_IP [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: IBMCLOUD_API_KEY [2024-01-18T15:32:33.402Z] value: "****" [2024-01-18T15:32:33.402Z] - name: CONTROL_PLANE_URL [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: DS_VERSION [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: DS_BUILD [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: NAMESPACE [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: NO_OF_NODES [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: DATA_SERVICE [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: DEPLOY_ALL_VERSIONS [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.402Z] - name: DEPLOY_ALL_IMAGES [2024-01-18T15:32:33.402Z] value: "" [2024-01-18T15:32:33.403Z] - name: DEPLOY_ALL_DATASERVICE [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] - name: GCP_PROJECT_ID [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] - name: PDS_USERNAME [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] - name: PDS_PASSWORD [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] - name: PDS_CLIENT_SECRET [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] - name: PDS_CLIENT_ID [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] - name: PDS_PARAM_CM [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] - name: PDS_ISSUER_URL [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] - name: CLUSTER_TYPE [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] - name: TARGET_KUBECONFIG [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] - name: TARGET_CLUSTER_NAME [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] - name: PX_ORACLE_user_ocid [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] - name: PX_ORACLE_fingerprint [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] - name: PX_ORACLE_private_key_path [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] - name: INSTANCE_GROUP [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] - name: LOGGLY_API_TOKEN [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] - name: PODMETRIC_METERING_INTERVAL_MINUTES [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] - name: TARGET_PXBACKUP_VERSION [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] - name: TARGET_STORK_VERSION [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] - name: PX_BACKUP_HELM_REPO_BRANCH [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] - name: BACKUP_TYPE [2024-01-18T15:32:33.403Z] value: "native_csi" [2024-01-18T15:32:33.403Z] - name: NFS_SERVER_ADDR [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] - name: NFS_SUB_PATH [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] - name: NFS_MOUNT_OPTION [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] - name: NFS_PATH [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] - name: SKIP_PX_OPERATOR_UPGRADE [2024-01-18T15:32:33.403Z] value: "false" [2024-01-18T15:32:33.403Z] - name: VOLUME_SNAPSHOT_CLASS [2024-01-18T15:32:33.403Z] value: "ibmc-vpcblock-snapshot" [2024-01-18T15:32:33.403Z] - name: S3_SSE_TYPE [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] - name: S3_POLICY_SID [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] - name: S3_ENCRYPTION_POLICY [2024-01-18T15:32:33.403Z] value: "" [2024-01-18T15:32:33.403Z] volumes: [{ "name": "testresults", "persistentVolumeClaim": {"claimName": "testresults"}},{ "name": "aws-volume", "configMap": { "name": "aws-cm", "items": [{"key": "credentials", "path": "credentials"}, {"key": "config", "path": "config"}]} }] [2024-01-18T15:32:33.403Z] restartPolicy: Never [2024-01-18T15:32:33.403Z] serviceAccountName: torpedo-account [2024-01-18T15:32:33.403Z] Deploying torpedo pod... [2024-01-18T15:32:34.334Z] serviceaccount/torpedo-account unchanged [2024-01-18T15:32:34.334Z] clusterrole.rbac.authorization.k8s.io/torpedo-role unchanged [2024-01-18T15:32:34.591Z] clusterrolebinding.rbac.authorization.k8s.io/torpedo-role-binding unchanged [2024-01-18T15:32:34.591Z] persistentvolumeclaim/testresults unchanged [2024-01-18T15:32:34.847Z] pod/torpedo created [2024-01-18T15:32:34.847Z] Waiting for torpedo to start running [2024-01-18T15:32:34.847Z] Iteration: 1 [2024-01-18T15:32:36.215Z] Iteration: 2 [2024-01-18T15:32:38.102Z] Iteration: 3 [2024-01-18T15:32:39.467Z] Iteration: 4 [2024-01-18T15:32:40.835Z] Iteration: 5 [2024-01-18T15:32:42.202Z] Iteration: 6 [2024-01-18T15:32:44.090Z] Iteration: 7 [2024-01-18T15:32:45.458Z] Iteration: 8 [2024-01-18T15:32:46.824Z] Iteration: 9 [2024-01-18T15:32:48.193Z] Iteration: 10 [2024-01-18T15:32:50.082Z] Iteration: 11 [2024-01-18T15:32:51.447Z] Iteration: 12 [2024-01-18T15:32:52.816Z] Iteration: 13 [2024-01-18T15:32:54.184Z] Iteration: 14 [2024-01-18T15:32:55.547Z] Iteration: 15 [2024-01-18T15:32:57.433Z] Iteration: 16 [2024-01-18T15:32:58.797Z] Iteration: 17 [2024-01-18T15:33:00.166Z] Iteration: 18 [2024-01-18T15:33:01.529Z] Iteration: 19 [2024-01-18T15:33:03.417Z] Iteration: 20 [2024-01-18T15:33:04.790Z] Iteration: 21 [2024-01-18T15:33:06.159Z] Iteration: 22 [2024-01-18T15:33:07.526Z] Iteration: 23 [2024-01-18T15:33:09.413Z] Iteration: 24 [2024-01-18T15:33:10.776Z] Iteration: 25 [2024-01-18T15:33:12.143Z] Iteration: 26 [2024-01-18T15:33:13.509Z] Iteration: 27 [2024-01-18T15:33:15.411Z] Iteration: 28 [2024-01-18T15:33:16.774Z] Iteration: 29 [2024-01-18T15:33:18.136Z] Iteration: 30 [2024-01-18T15:33:19.499Z] Iteration: 31 [2024-01-18T15:33:21.387Z] Iteration: 32 [2024-01-18T15:33:22.751Z] Iteration: 33 [2024-01-18T15:33:24.114Z] Iteration: 34 [2024-01-18T15:33:25.476Z] Iteration: 35 [2024-01-18T15:33:26.841Z] Iteration: 36 [2024-01-18T15:33:28.211Z] Iteration: 37 [2024-01-18T15:33:30.101Z] Iteration: 38 [2024-01-18T15:33:31.476Z] Iteration: 39 [2024-01-18T15:33:32.842Z] Iteration: 40 [2024-01-18T15:33:34.213Z] Iteration: 41 [2024-01-18T15:33:36.098Z] Iteration: 42 [2024-01-18T15:33:37.463Z] Iteration: 43 [2024-01-18T15:33:38.826Z] Iteration: 44 [2024-01-18T15:33:40.714Z] Iteration: 45 [2024-01-18T15:33:42.082Z] Iteration: 46 [2024-01-18T15:33:43.968Z] Iteration: 47 [2024-01-18T15:33:45.335Z] Iteration: 48 [2024-01-18T15:33:46.701Z] Iteration: 49 [2024-01-18T15:33:48.588Z] Iteration: 50 [2024-01-18T15:33:49.957Z] Iteration: 51 [2024-01-18T15:33:51.324Z] Iteration: 52 [2024-01-18T15:33:52.693Z] Iteration: 53 [2024-01-18T15:33:54.584Z] Iteration: 54 [2024-01-18T15:33:55.946Z] Iteration: 55 [2024-01-18T15:33:57.308Z] Iteration: 56 [2024-01-18T15:33:59.201Z] Iteration: 57 [2024-01-18T15:34:00.566Z] Iteration: 58 [2024-01-18T15:34:01.938Z] Iteration: 59 [2024-01-18T15:34:02.498Z] Logs from first iteration [2024-01-18T15:34:03.426Z] Creating new Dashboard instance. [2024-01-18T15:34:03.426Z] 2024-01-18 15:34:00 +0000:[INFO] [monitor.Register:#55] - Registering driver name: prometheus [2024-01-18T15:34:03.426Z] 2024-01-18 15:34:00 +0000:[INFO] [dataservice.init.0:#542] - Registering pds driver [2024-01-18T15:34:03.426Z] 2024-01-18 15:34:00 +0000:[INFO] [pso.init.0:#123] - Registering pso driver [2024-01-18T15:34:03.426Z] 2024-01-18 15:34:00 +0000:[INFO] [ibm.init.0:#99] - Registering pso driver [2024-01-18T15:34:03.426Z] 2024-01-18 15:34:00 +0000:[INFO] [ocp.init.0:#105] - Registering ocp driver [2024-01-18T15:34:03.426Z] INFO[2024-01-18 15:34:00] Registering auth plugin: gcp [2024-01-18T15:34:03.426Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.ParseFlags:#6100] - The default scheduler is k8s [2024-01-18T15:34:03.426Z] 2024-01-18 15:34:00 +0000:[INFO] Log Dir: /testresults//torpedo.log [2024-01-18T15:34:03.426Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.ParseFlags:#6258] - Backup driver name pxb [2024-01-18T15:34:03.426Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.ParseFlags:#6263] - Backup driver found pxb [2024-01-18T15:34:03.426Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.ParseFlags:#6267] - Pds driver name [2024-01-18T15:34:03.426Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.isDashboardReachable:#6435] - Checking URL: https://aetos.pwx.purestorage.com/datamodel/about [2024-01-18T15:34:03.426Z] 2024-01-18 15:34:00 +0000:[WARNING] [tests.isDashboardReachable:#6439] - Get "https://aetos.pwx.purestorage.com/datamodel/about": dial tcp: lookup aetos.pwx.purestorage.com on 172.21.0.10:53: no such host [2024-01-18T15:34:03.427Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.ParseFlags:#6279] - Aetos Dashboard is not reachable. Disabling dashboard reporting. [2024-01-18T15:34:03.427Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags:#6413] - ********Torpedo Command******** [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags:#6414] - /go/src/github.com/portworx/torpedo/bin/backup.test --test.timeout=720h0m0s --ginkgo.seed=1705592040 --ginkgo.focus={BasicBackupCreation}|{AddMultipleNamespaceLabels}|{AllNSBackupWithIncludeNewNSOption}|{BackupLocationWithEncryptionKey}|{BackupMultipleNsWithSameLabel}|{BasicSelectiveRestore}|{CancelAllRunningBackupJobs}|{CancelAllRunningRestoreJobs}|{CancelClusterBackupShare}|{CreateMultipleUsersAndGroups}|{CustomResourceBackupAndRestore}|{CustomResourceRestore}|{DeleteAllBackupObjects}|{DeleteUsersRole}|{DifferentAccessSameUser}|{DuplicateSharedBackup}|{LicensingCountBeforeAndAfterBackupPodRestart}|{LicensingCountWithNodeLabelledBeforeClusterAddition}|{ManualAndScheduleBackupUsingNamespaceLabel}|{MultipleInPlaceRestoreSameTime}|{NamespaceLabelledBackupSharedWithDifferentAccessMode}|{NodeCountForLicensing}|{RestartBackupPodDuringBackupSharing}|{RestoreEncryptedAndNonEncryptedBackups}|{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}|{ScheduleBackupCreationAllNS}|{ScheduleBackupCreationSingleNS}|{SetUnsetNSLabelDuringScheduleBackup}|{ShareAndRemoveBackupLocation}|{ShareBackupAndEdit}|{ShareBackupsAndClusterWithUser}|{ShareBackupWithUsersAndGroups}|{SharedBackupDelete}|{SwapShareBackup}|{ScheduleBackupDeleteAndRecreateNS}|{BackupScheduleForOldAndNewNS}|{DeleteNSDeleteClusterRestore}|{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}|{ScheduleBackupWithAdditionAndRemovalOfNS}|{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}|{DeleteBackupOfUserSharedRBAC}|{UpdatesBackupOfUserFromAdmin}|{DeleteBackupSharedByMultipleUsersFromAdmin}|{DeleteSameNameObjectsByMultipleUsersFromAdmin}|{DeleteSharedBackupOfUserFromAdmin}|{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}|{DeleteObjectsByMultipleUsersFromNewAdmin}|{NamespaceLabelledBackupOfEmptyNamespace}|{VerifyRBACForPxAdmin}|{VerifyRBACForInfraAdmin}|{VerifyRBACForAppAdmin}|{VerifyRBACForAppUser}|{KubeAndPxNamespacesSkipOnAllNSBackup} --ginkgo.slowSpecThreshold=600.00000 --ginkgo.trace --spec-dir ../drivers/scheduler/k8s/specs --app-list mysql-ibm --deploy-pds-apps=false --pds-driver --secure-apps --repl1-apps --csi-app-list --scheduler k8s --max-storage-nodes-per-az 2 --backup-driver pxb --log-level debug --node-driver ssh --scale-factor 1 --hyper-converged=true --fail-on-px-pod-restartcount=false --minimun-runtime-mins 0 --driver-start-timeout 30m0s --chaos-level 5 --storagenode-recovery-timeout 35m0s --provisioner ibm --storage-driver ibm --config-map --custom-config --storage-upgrade-endpoint-url= --storage-upgrade-endpoint-version= --upgrade-storage-driver-endpoint-list= --enable-stork-upgrade=false --secret-type=k8s --pure-volumes=false --pure-fa-snapshot-restore-to-many-test=false --pure-san-type=ISCSI --vault-addr= --vault-token= --px-runtime-opts= --anthos-ws-node-ip= --anthos-inst-path= --autopilot-upgrade-version= --csi-generic-driver-config-map= --sched-upgrade-hops= --migration-hops= --license_expiry_timeout_hours=1h0m0s --metering_interval_mins=10m0s --testrail-milestone= --testrail-run-name= --testrail-run-id= --testrail-jeknins-build-url= --testrail-host= --testrail-username= --testrail-password= --jira-username= --jira-token= --jira-account-id= --user= --enable-dash=true --test-desc=px-backup-system-test-4cac76dc --test-type=system-tests --test-tags=flavour:iks,provisioner:iks,cadence:on-demand --testset-id=0 --branch=ibm-mysql --product=px-backup --torpedo-job-name=torpedo-daily-job --torpedo-job-type=functional [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags:#6415] - ****************************** [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags:#6417] - *********Parsed Args********** [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - alsologtostderr: false [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - anthos-inst-path: [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - anthos-ws-node-ip: [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - app-list: mysql-ibm [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - autopilot-upgrade-version: [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - backup-driver: pxb [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - branch: ibm-mysql [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - bundle-location: /var/cores [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - chaos-level: 5 [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - config-map: [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - csi-app-list: [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - csi-generic-driver-config-map: [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - custom-config: [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - data-integrity-validation-tests: [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - deploy-pds-apps: false [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - destroy-app-timeout: 5m0s [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - driver-start-timeout: 30m0s [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - enable-dash: false [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - enable-stork-upgrade: false [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - fail-on-px-pod-restartcount: false [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - ginkgo.debug: false [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - ginkgo.dryRun: false [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - ginkgo.failFast: false [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - ginkgo.failOnPending: false [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - ginkgo.flakeAttempts: 1 [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - ginkgo.focus: [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - ginkgo.noColor: false [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - ginkgo.noisyPendings: true [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - ginkgo.noisySkippings: true [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - ginkgo.parallel.node: 1 [2024-01-18T15:34:03.428Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - ginkgo.parallel.streamhost: [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - ginkgo.parallel.synchost: [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - ginkgo.parallel.total: 1 [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - ginkgo.progress: false [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - ginkgo.randomizeAllSpecs: false [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - ginkgo.regexScansFilePath: false [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - ginkgo.reportFile: [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - ginkgo.reportPassed: false [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - ginkgo.seed: 1705592040 [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - ginkgo.skip: [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - ginkgo.skipMeasurements: false [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - ginkgo.slowSpecThreshold: 600 [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - ginkgo.succinct: false [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - ginkgo.trace: true [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - ginkgo.v: false [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - hyper-converged: true [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - jira-account-id: [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - jira-token: [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - jira-username: [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - license_expiry_timeout_hours: 1h0m0s [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - log-level: debug [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - log-location: /testresults/ [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - log_backtrace_at: [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - log_dir: [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - log_link: [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - logbuflevel: 0 [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - logtostderr: false [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - max-storage-nodes-per-az: 2 [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - metering_interval_mins: 10m0s [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - migration-hops: [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - minimun-runtime-mins: 0 [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - monitor-driver: prometheus [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - node-driver: ssh [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - pds-driver: [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - product: px-backup [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - provisioner: ibm [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - pure-fa-snapshot-restore-to-many-test: false [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - pure-san-type: ISCSI [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - pure-volumes: false [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - px-runtime-opts: [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - repl1-apps: [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - scale-factor: 1 [2024-01-18T15:34:03.429Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - sched-upgrade-hops: [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - scheduler: k8s [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - secret-type: k8s [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - secure-apps: [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - spec-dir: ../drivers/scheduler/k8s/specs [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - stderrthreshold: 2 [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - storage-driver: ibm [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - storage-upgrade-endpoint-url: [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - storage-upgrade-endpoint-version: [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - storagenode-recovery-timeout: 35m0s [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test-desc: px-backup-system-test-4cac76dc [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test-tags: flavour:iks,provisioner:iks,cadence:on-demand [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test-type: system-tests [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.bench: [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.benchmem: false [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.benchtime: 1s [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.blockprofile: [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.blockprofilerate: 1 [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.count: 1 [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.coverprofile: [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.cpu: [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.cpuprofile: [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.failfast: false [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.fuzz: [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.fuzzcachedir: [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.fuzzminimizetime: 1m0s [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.fuzztime: 0s [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.fuzzworker: false [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.list: [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.memprofile: [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.memprofilerate: 0 [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.mutexprofile: [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.mutexprofilefraction: 1 [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.outputdir: [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.paniconexit0: false [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.parallel: 4 [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.run: [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.short: false [2024-01-18T15:34:03.430Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.shuffle: off [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.testlogfile: [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.timeout: 720h0m0s [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.trace: [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - test.v: false [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - testrail-host: [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - testrail-jeknins-build-url: [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - testrail-milestone: [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - testrail-password: [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - testrail-run-id: [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - testrail-run-name: [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - testrail-username: [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - testset-id: 0 [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - torpedo-job-name: torpedo-daily-job [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - torpedo-job-type: functional [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - torpedo-skip-system-checks: false [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - upgrade-storage-driver-endpoint-list: [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - user: [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - v: 0 [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - vault-addr: [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - vault-token: [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags.func1:#6419] - vmodule: [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [tests.printFlags:#6421] - ****************************** [2024-01-18T15:34:03.431Z] Running Suite: Torpedo : Backup [2024-01-18T15:34:03.431Z] =============================== [2024-01-18T15:34:03.431Z] Random Seed: 1705592040 [2024-01-18T15:34:03.431Z] Will run 51 of 100 specs [2024-01-18T15:34:03.431Z] [2024-01-18T15:34:03.431Z] INFO[2024-01-18 15:34:00] Dash is disabled [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [backup.glob..func1:#172] - Backup Init instance [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [backup.BackupInitInstance:#101] - Inside BackupInitInstance [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:00 +0000:[INFO] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:01 +0000:[INFO] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:01 +0000:[INFO] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:02 +0000:[INFO] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:34:03.431Z] 2024-01-18 15:34:02 +0000:[INFO] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/aggr-mysql... [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/aggr-mysql/px-mysql-app.yaml [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/aggr-mysql/px-mysql-storage.yaml [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/aggr-mysql/px-mysql-app.yaml ../drivers/scheduler/k8s/specs/aggr-mysql/px-mysql-storage.yaml] [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/aggr-mysql/px-mysql-app.yaml [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/aggr-mysql/px-mysql-storage.yaml [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: aggr-mysql [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/aut-postgres... [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/aut-postgres/aut-postgres-app.yaml [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/aut-postgres/aut-postgres-storage.yaml [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/aut-postgres/aut-postgres-app.yaml ../drivers/scheduler/k8s/specs/aut-postgres/aut-postgres-storage.yaml] [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/aut-postgres/aut-postgres-app.yaml [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/aut-postgres/aut-postgres-storage.yaml [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 8: did not find expected key [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: aut-postgres [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/aut-postgres-onlyvol... [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/aut-postgres-onlyvol/aut-postgres-storage.yaml [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/aut-postgres-onlyvol/aut-postgres-storage.yaml] [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/aut-postgres-onlyvol/aut-postgres-storage.yaml [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: aut-postgres-onlyvol [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/aut-postgres-sharedv4... [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.432Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.433Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/aut-postgres-sharedv4/aut-postgres-app.yaml [2024-01-18T15:34:03.433Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/aut-postgres-sharedv4/aut-postgres-storage.yaml [2024-01-18T15:34:03.433Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/aut-postgres-sharedv4/aut-postgres-app.yaml ../drivers/scheduler/k8s/specs/aut-postgres-sharedv4/aut-postgres-storage.yaml] [2024-01-18T15:34:03.433Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/aut-postgres-sharedv4/aut-postgres-app.yaml [2024-01-18T15:34:03.433Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/aut-postgres-sharedv4/aut-postgres-storage.yaml [2024-01-18T15:34:03.433Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 8: did not find expected key [2024-01-18T15:34:03.433Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: aut-postgres-sharedv4 [2024-01-18T15:34:03.433Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/aut-vol-only... [2024-01-18T15:34:03.433Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.433Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.433Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/aut-vol-only/aut-postgres-storage.yaml [2024-01-18T15:34:03.433Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/aut-vol-only/aut-postgres-storage.yaml] [2024-01-18T15:34:03.433Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/aut-vol-only/aut-postgres-storage.yaml [2024-01-18T15:34:03.433Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 8: did not find expected key [2024-01-18T15:34:03.433Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: aut-vol-only [2024-01-18T15:34:03.433Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/bonnie-sharedv4... [2024-01-18T15:34:03.433Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.433Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.433Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/bonnie-sharedv4/px-bonnie-app.yml [2024-01-18T15:34:03.433Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/bonnie-sharedv4/px-bonnie-storage.yml [2024-01-18T15:34:03.433Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/bonnie-sharedv4/px-bonnie-app.yml ../drivers/scheduler/k8s/specs/bonnie-sharedv4/px-bonnie-storage.yml] [2024-01-18T15:34:03.433Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/bonnie-sharedv4/px-bonnie-app.yml [2024-01-18T15:34:03.433Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 6: did not find expected key [2024-01-18T15:34:03.433Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/bonnie-sharedv4/px-bonnie-storage.yml [2024-01-18T15:34:03.433Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 8: did not find expected key [2024-01-18T15:34:03.433Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: bonnie-sharedv4 [2024-01-18T15:34:03.434Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/bonnie-sv4-svc... [2024-01-18T15:34:03.434Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.434Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.434Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/bonnie-sv4-svc/px-bonnie-app.yml [2024-01-18T15:34:03.434Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/bonnie-sv4-svc/px-bonnie-storage.yml [2024-01-18T15:34:03.434Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/bonnie-sv4-svc/px-bonnie-app.yml ../drivers/scheduler/k8s/specs/bonnie-sv4-svc/px-bonnie-storage.yml] [2024-01-18T15:34:03.434Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/bonnie-sv4-svc/px-bonnie-app.yml [2024-01-18T15:34:03.434Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 6: did not find expected key [2024-01-18T15:34:03.434Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/bonnie-sv4-svc/px-bonnie-storage.yml [2024-01-18T15:34:03.434Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 8: did not find expected key [2024-01-18T15:34:03.434Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: bonnie-sv4-svc [2024-01-18T15:34:03.434Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/cassandra... [2024-01-18T15:34:03.434Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.434Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.434Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/cassandra/cassandra-stress.yaml [2024-01-18T15:34:03.434Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/cassandra/cassandra.yaml [2024-01-18T15:34:03.434Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/cassandra/cassandra-stress.yaml ../drivers/scheduler/k8s/specs/cassandra/cassandra.yaml] [2024-01-18T15:34:03.434Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/cassandra/cassandra-stress.yaml [2024-01-18T15:34:03.434Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/cassandra/cassandra.yaml [2024-01-18T15:34:03.434Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: cassandra [2024-01-18T15:34:03.434Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/cassandra-pso... [2024-01-18T15:34:03.434Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.435Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.435Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/cassandra-pso/cassandra-stress.yaml [2024-01-18T15:34:03.435Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/cassandra-pso/cassandra.yaml [2024-01-18T15:34:03.435Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/cassandra-pso/cassandra-stress.yaml ../drivers/scheduler/k8s/specs/cassandra-pso/cassandra.yaml] [2024-01-18T15:34:03.435Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/cassandra-pso/cassandra-stress.yaml [2024-01-18T15:34:03.435Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/cassandra-pso/cassandra.yaml [2024-01-18T15:34:03.435Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: cassandra-pso [2024-01-18T15:34:03.435Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/debug... [2024-01-18T15:34:03.435Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.435Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.435Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/debug/debug.yaml [2024-01-18T15:34:03.435Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/debug/debug.yaml] [2024-01-18T15:34:03.435Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/debug/debug.yaml [2024-01-18T15:34:03.435Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: debug [2024-01-18T15:34:03.435Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/elasticsearch... [2024-01-18T15:34:03.435Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.435Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.435Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/elasticsearch/px-elasticdata-app.yaml [2024-01-18T15:34:03.435Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/elasticsearch/px-elasticdata-app.yaml] [2024-01-18T15:34:03.435Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/elasticsearch/px-elasticdata-app.yaml [2024-01-18T15:34:03.435Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: elasticsearch [2024-01-18T15:34:03.436Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/elasticsearch-crd-webhook... [2024-01-18T15:34:03.436Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.436Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.436Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/elasticsearch-crd-webhook/cr-elasticsearch.yaml [2024-01-18T15:34:03.436Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/elasticsearch-crd-webhook/crds.yaml [2024-01-18T15:34:03.436Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/elasticsearch-crd-webhook/operator.yaml [2024-01-18T15:34:03.436Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/elasticsearch-crd-webhook/cr-elasticsearch.yaml ../drivers/scheduler/k8s/specs/elasticsearch-crd-webhook/crds.yaml ../drivers/scheduler/k8s/specs/elasticsearch-crd-webhook/operator.yaml] [2024-01-18T15:34:03.436Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/elasticsearch-crd-webhook/cr-elasticsearch.yaml [2024-01-18T15:34:03.436Z] 2024-01-18 15:34:03 +0000:[WARNING] [k8s.(*K8s).ParseSpecs:#504] - custom res: &{../drivers/scheduler/k8s/specs/elasticsearch-crd-webhook/cr-elasticsearch.yaml } [2024-01-18T15:34:03.436Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/elasticsearch-crd-webhook/crds.yaml [2024-01-18T15:34:03.436Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/elasticsearch-crd-webhook/operator.yaml [2024-01-18T15:34:03.436Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: elasticsearch-crd-webhook [2024-01-18T15:34:03.436Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/elasticsearch-rally... [2024-01-18T15:34:03.436Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.436Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.436Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/elasticsearch-rally/es-rally.yaml [2024-01-18T15:34:03.436Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/elasticsearch-rally/px-elasticdata-app.yaml [2024-01-18T15:34:03.436Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/elasticsearch-rally/es-rally.yaml ../drivers/scheduler/k8s/specs/elasticsearch-rally/px-elasticdata-app.yaml] [2024-01-18T15:34:03.436Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/elasticsearch-rally/es-rally.yaml [2024-01-18T15:34:03.436Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/elasticsearch-rally/px-elasticdata-app.yaml [2024-01-18T15:34:03.436Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: elasticsearch-rally [2024-01-18T15:34:03.436Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/elk-stack... [2024-01-18T15:34:03.436Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.436Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.436Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/elk-stack/es-exporter.yaml [2024-01-18T15:34:03.436Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/elk-stack/esread.yaml [2024-01-18T15:34:03.437Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/elk-stack/logging-app.yaml [2024-01-18T15:34:03.437Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/elk-stack/logstash-app.yaml [2024-01-18T15:34:03.437Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/elk-stack/px-elastic-app.yaml [2024-01-18T15:34:03.437Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/elk-stack/px-elk-sc.yaml [2024-01-18T15:34:03.437Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/elk-stack/px-kibana-app.yaml [2024-01-18T15:34:03.437Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/elk-stack/servicemonitor.yaml [2024-01-18T15:34:03.437Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/elk-stack/es-exporter.yaml ../drivers/scheduler/k8s/specs/elk-stack/esread.yaml ../drivers/scheduler/k8s/specs/elk-stack/logging-app.yaml ../drivers/scheduler/k8s/specs/elk-stack/logstash-app.yaml ../drivers/scheduler/k8s/specs/elk-stack/px-elastic-app.yaml ../drivers/scheduler/k8s/specs/elk-stack/px-elk-sc.yaml ../drivers/scheduler/k8s/specs/elk-stack/px-kibana-app.yaml ../drivers/scheduler/k8s/specs/elk-stack/servicemonitor.yaml] [2024-01-18T15:34:03.437Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/elk-stack/es-exporter.yaml [2024-01-18T15:34:03.437Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/elk-stack/esread.yaml [2024-01-18T15:34:03.437Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/elk-stack/logging-app.yaml [2024-01-18T15:34:03.437Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/elk-stack/logstash-app.yaml [2024-01-18T15:34:03.437Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/elk-stack/px-elastic-app.yaml [2024-01-18T15:34:03.437Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/elk-stack/px-elk-sc.yaml [2024-01-18T15:34:03.437Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/elk-stack/px-kibana-app.yaml [2024-01-18T15:34:03.437Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/elk-stack/servicemonitor.yaml [2024-01-18T15:34:03.437Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: elk-stack [2024-01-18T15:34:03.437Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/fio... [2024-01-18T15:34:03.437Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.437Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.437Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio/fio-config-map.yaml [2024-01-18T15:34:03.437Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio/fio.yaml [2024-01-18T15:34:03.437Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/fio/fio-config-map.yaml ../drivers/scheduler/k8s/specs/fio/fio.yaml] [2024-01-18T15:34:03.437Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio/fio-config-map.yaml [2024-01-18T15:34:03.437Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio/fio.yaml [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 8: could not find expected ':' [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: fio [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/fio-aggr-cloudsnap... [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-aggr-cloudsnap/fio-config-map.yaml [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-aggr-cloudsnap/fio.yaml [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/fio-aggr-cloudsnap/fio-config-map.yaml ../drivers/scheduler/k8s/specs/fio-aggr-cloudsnap/fio.yaml] [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-aggr-cloudsnap/fio-config-map.yaml [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-aggr-cloudsnap/fio.yaml [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 8: could not find expected ':' [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: fio-aggr-cloudsnap [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/fio-cloudsnap... [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-cloudsnap/fio-config-map.yaml [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-cloudsnap/fio.yaml [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/fio-cloudsnap/fio-config-map.yaml ../drivers/scheduler/k8s/specs/fio-cloudsnap/fio.yaml] [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-cloudsnap/fio-config-map.yaml [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-cloudsnap/fio.yaml [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: fio-cloudsnap [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/fio-fa-cloud-drives... [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-fa-cloud-drives/fio-config-map.yaml [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-fa-cloud-drives/fio.yaml [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-fa-cloud-drives/px-storage-class.yaml [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/fio-fa-cloud-drives/fio-config-map.yaml ../drivers/scheduler/k8s/specs/fio-fa-cloud-drives/fio.yaml ../drivers/scheduler/k8s/specs/fio-fa-cloud-drives/px-storage-class.yaml] [2024-01-18T15:34:03.438Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-fa-cloud-drives/fio-config-map.yaml [2024-01-18T15:34:03.439Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-fa-cloud-drives/fio.yaml [2024-01-18T15:34:03.439Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 63: did not find expected key [2024-01-18T15:34:03.439Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-fa-cloud-drives/px-storage-class.yaml [2024-01-18T15:34:03.439Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 8: did not find expected key [2024-01-18T15:34:03.439Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: fio-fa-cloud-drives [2024-01-18T15:34:03.439Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/fio-fa-davol... [2024-01-18T15:34:03.439Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.439Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.439Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-fa-davol/fio-config-map.yaml [2024-01-18T15:34:03.439Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-fa-davol/fio.yaml [2024-01-18T15:34:03.439Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-fa-davol/px-storage-class.yaml [2024-01-18T15:34:03.439Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/fio-fa-davol/fio-config-map.yaml ../drivers/scheduler/k8s/specs/fio-fa-davol/fio.yaml ../drivers/scheduler/k8s/specs/fio-fa-davol/px-storage-class.yaml] [2024-01-18T15:34:03.439Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-fa-davol/fio-config-map.yaml [2024-01-18T15:34:03.440Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-fa-davol/fio.yaml [2024-01-18T15:34:03.440Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 8: could not find expected ':' [2024-01-18T15:34:03.440Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-fa-davol/px-storage-class.yaml [2024-01-18T15:34:03.440Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: fio-fa-davol [2024-01-18T15:34:03.440Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/fio-fastpath... [2024-01-18T15:34:03.440Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.440Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.440Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-fastpath/fio-config-map.yaml [2024-01-18T15:34:03.440Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-fastpath/fio.yaml [2024-01-18T15:34:03.440Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/fio-fastpath/fio-config-map.yaml ../drivers/scheduler/k8s/specs/fio-fastpath/fio.yaml] [2024-01-18T15:34:03.440Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-fastpath/fio-config-map.yaml [2024-01-18T15:34:03.440Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-fastpath/fio.yaml [2024-01-18T15:34:03.440Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: fio-fastpath [2024-01-18T15:34:03.440Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/fio-fastpath-repl1... [2024-01-18T15:34:03.440Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.440Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.440Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-fastpath-repl1/fio-config-map.yaml [2024-01-18T15:34:03.440Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-fastpath-repl1/fio.yaml [2024-01-18T15:34:03.440Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/fio-fastpath-repl1/fio-config-map.yaml ../drivers/scheduler/k8s/specs/fio-fastpath-repl1/fio.yaml] [2024-01-18T15:34:03.440Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-fastpath-repl1/fio-config-map.yaml [2024-01-18T15:34:03.440Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-fastpath-repl1/fio.yaml [2024-01-18T15:34:03.440Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: fio-fastpath-repl1 [2024-01-18T15:34:03.441Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/fio-fb-davol... [2024-01-18T15:34:03.441Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.441Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.441Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-fb-davol/fio-config-map.yaml [2024-01-18T15:34:03.441Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-fb-davol/fio.yaml [2024-01-18T15:34:03.441Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-fb-davol/px-storage-class.yaml [2024-01-18T15:34:03.441Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/fio-fb-davol/fio-config-map.yaml ../drivers/scheduler/k8s/specs/fio-fb-davol/fio.yaml ../drivers/scheduler/k8s/specs/fio-fb-davol/px-storage-class.yaml] [2024-01-18T15:34:03.441Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-fb-davol/fio-config-map.yaml [2024-01-18T15:34:03.441Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-fb-davol/fio.yaml [2024-01-18T15:34:03.441Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 8: could not find expected ':' [2024-01-18T15:34:03.441Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-fb-davol/px-storage-class.yaml [2024-01-18T15:34:03.441Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: fio-fb-davol [2024-01-18T15:34:03.441Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/fio-fstrim... [2024-01-18T15:34:03.441Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.441Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.441Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-fstrim/fio-config-map.yaml [2024-01-18T15:34:03.441Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-fstrim/fio.yaml [2024-01-18T15:34:03.441Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-fstrim/px-storage-class.yaml [2024-01-18T15:34:03.441Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/fio-fstrim/fio-config-map.yaml ../drivers/scheduler/k8s/specs/fio-fstrim/fio.yaml ../drivers/scheduler/k8s/specs/fio-fstrim/px-storage-class.yaml] [2024-01-18T15:34:03.441Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-fstrim/fio-config-map.yaml [2024-01-18T15:34:03.441Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-fstrim/fio.yaml [2024-01-18T15:34:03.441Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-fstrim/px-storage-class.yaml [2024-01-18T15:34:03.441Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 8: did not find expected key [2024-01-18T15:34:03.442Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: fio-fstrim [2024-01-18T15:34:03.442Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/fio-io-bw... [2024-01-18T15:34:03.442Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.442Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.442Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-io-bw/fio-config-map.yaml [2024-01-18T15:34:03.442Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-io-bw/fio.yaml [2024-01-18T15:34:03.442Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-io-bw/px-storage-class.yaml [2024-01-18T15:34:03.442Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/fio-io-bw/fio-config-map.yaml ../drivers/scheduler/k8s/specs/fio-io-bw/fio.yaml ../drivers/scheduler/k8s/specs/fio-io-bw/px-storage-class.yaml] [2024-01-18T15:34:03.442Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-io-bw/fio-config-map.yaml [2024-01-18T15:34:03.442Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-io-bw/fio.yaml [2024-01-18T15:34:03.442Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-io-bw/px-storage-class.yaml [2024-01-18T15:34:03.442Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 8: did not find expected key [2024-01-18T15:34:03.442Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: fio-io-bw [2024-01-18T15:34:03.442Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/fio-io-throttle... [2024-01-18T15:34:03.442Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.442Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.442Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-io-throttle/fio-config-map.yaml [2024-01-18T15:34:03.442Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-io-throttle/fio-pod.yaml [2024-01-18T15:34:03.442Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-io-throttle/fio-sc.yaml [2024-01-18T15:34:03.442Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/fio-io-throttle/fio-config-map.yaml ../drivers/scheduler/k8s/specs/fio-io-throttle/fio-pod.yaml ../drivers/scheduler/k8s/specs/fio-io-throttle/fio-sc.yaml] [2024-01-18T15:34:03.442Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-io-throttle/fio-config-map.yaml [2024-01-18T15:34:03.443Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-io-throttle/fio-pod.yaml [2024-01-18T15:34:03.443Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-io-throttle/fio-sc.yaml [2024-01-18T15:34:03.443Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: fio-io-throttle [2024-01-18T15:34:03.443Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/fio-localsnap... [2024-01-18T15:34:03.443Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.443Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.443Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-localsnap/fio-config-map.yaml [2024-01-18T15:34:03.443Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-localsnap/fio.yaml [2024-01-18T15:34:03.443Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/fio-localsnap/fio-config-map.yaml ../drivers/scheduler/k8s/specs/fio-localsnap/fio.yaml] [2024-01-18T15:34:03.443Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-localsnap/fio-config-map.yaml [2024-01-18T15:34:03.443Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-localsnap/fio.yaml [2024-01-18T15:34:03.443Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: fio-localsnap [2024-01-18T15:34:03.443Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/fio-low-io... [2024-01-18T15:34:03.443Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.443Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.443Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-low-io/fio-config-map.yaml [2024-01-18T15:34:03.443Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-low-io/fio-low-io.yaml [2024-01-18T15:34:03.443Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/fio-low-io/fio-config-map.yaml ../drivers/scheduler/k8s/specs/fio-low-io/fio-low-io.yaml] [2024-01-18T15:34:03.443Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-low-io/fio-config-map.yaml [2024-01-18T15:34:03.443Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-low-io/fio-low-io.yaml [2024-01-18T15:34:03.444Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 8: could not find expected ':' [2024-01-18T15:34:03.444Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: fio-low-io [2024-01-18T15:34:03.444Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/fio-overlapping-io... [2024-01-18T15:34:03.444Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.444Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.444Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-overlapping-io/px-fio-app.yaml [2024-01-18T15:34:03.444Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-overlapping-io/px-fio-storage.yaml [2024-01-18T15:34:03.444Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/fio-overlapping-io/px-fio-app.yaml ../drivers/scheduler/k8s/specs/fio-overlapping-io/px-fio-storage.yaml] [2024-01-18T15:34:03.444Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-overlapping-io/px-fio-app.yaml [2024-01-18T15:34:03.444Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-overlapping-io/px-fio-storage.yaml [2024-01-18T15:34:03.444Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: fio-overlapping-io [2024-01-18T15:34:03.444Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/fio-pod... [2024-01-18T15:34:03.444Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.444Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.444Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-pod/fio-config-map.yaml [2024-01-18T15:34:03.444Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-pod/px-fio-app.yaml [2024-01-18T15:34:03.444Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-pod/px-fio-storage.yaml [2024-01-18T15:34:03.444Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/fio-pod/fio-config-map.yaml ../drivers/scheduler/k8s/specs/fio-pod/px-fio-app.yaml ../drivers/scheduler/k8s/specs/fio-pod/px-fio-storage.yaml] [2024-01-18T15:34:03.444Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-pod/fio-config-map.yaml [2024-01-18T15:34:03.444Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-pod/px-fio-app.yaml [2024-01-18T15:34:03.444Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-pod/px-fio-storage.yaml [2024-01-18T15:34:03.445Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: fio-pod [2024-01-18T15:34:03.445Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/fio-raw-block... [2024-01-18T15:34:03.445Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.445Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.445Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-raw-block/fio-config.yaml [2024-01-18T15:34:03.445Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-raw-block/fio.yaml [2024-01-18T15:34:03.445Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-raw-block/px-storage-class.yaml [2024-01-18T15:34:03.445Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/fio-raw-block/fio-config.yaml ../drivers/scheduler/k8s/specs/fio-raw-block/fio.yaml ../drivers/scheduler/k8s/specs/fio-raw-block/px-storage-class.yaml] [2024-01-18T15:34:03.445Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-raw-block/fio-config.yaml [2024-01-18T15:34:03.445Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-raw-block/fio.yaml [2024-01-18T15:34:03.445Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 8: could not find expected ':' [2024-01-18T15:34:03.445Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-raw-block/px-storage-class.yaml [2024-01-18T15:34:03.445Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: fio-raw-block [2024-01-18T15:34:03.445Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/fio-sharedv4... [2024-01-18T15:34:03.445Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.445Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.445Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-sharedv4/fio-config-map.yaml [2024-01-18T15:34:03.445Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-sharedv4/px-fio-app.yaml [2024-01-18T15:34:03.445Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-sharedv4/px-fio-storage.yaml [2024-01-18T15:34:03.445Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/fio-sharedv4/fio-config-map.yaml ../drivers/scheduler/k8s/specs/fio-sharedv4/px-fio-app.yaml ../drivers/scheduler/k8s/specs/fio-sharedv4/px-fio-storage.yaml] [2024-01-18T15:34:03.445Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-sharedv4/fio-config-map.yaml [2024-01-18T15:34:03.445Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-sharedv4/px-fio-app.yaml [2024-01-18T15:34:03.445Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 11: could not find expected ':' [2024-01-18T15:34:03.446Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-sharedv4/px-fio-storage.yaml [2024-01-18T15:34:03.446Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: fio-sharedv4 [2024-01-18T15:34:03.446Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/fio-storagepool... [2024-01-18T15:34:03.446Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.446Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.446Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-storagepool/fio-config-map.yaml [2024-01-18T15:34:03.446Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-storagepool/fio.yaml [2024-01-18T15:34:03.448Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/fio-storagepool/fio-config-map.yaml ../drivers/scheduler/k8s/specs/fio-storagepool/fio.yaml] [2024-01-18T15:34:03.448Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-storagepool/fio-config-map.yaml [2024-01-18T15:34:03.448Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-storagepool/fio.yaml [2024-01-18T15:34:03.448Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 8: could not find expected ':' [2024-01-18T15:34:03.448Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: fio-storagepool [2024-01-18T15:34:03.448Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/fio-sv4-svc... [2024-01-18T15:34:03.448Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.448Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.448Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-sv4-svc/fio-config-map.yaml [2024-01-18T15:34:03.448Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-sv4-svc/px-fio-app.yaml [2024-01-18T15:34:03.449Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-sv4-svc/px-fio-storage.yaml [2024-01-18T15:34:03.449Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/fio-sv4-svc/fio-config-map.yaml ../drivers/scheduler/k8s/specs/fio-sv4-svc/px-fio-app.yaml ../drivers/scheduler/k8s/specs/fio-sv4-svc/px-fio-storage.yaml] [2024-01-18T15:34:03.449Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-sv4-svc/fio-config-map.yaml [2024-01-18T15:34:03.449Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-sv4-svc/px-fio-app.yaml [2024-01-18T15:34:03.449Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 9: did not find expected key [2024-01-18T15:34:03.449Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-sv4-svc/px-fio-storage.yaml [2024-01-18T15:34:03.449Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: fio-sv4-svc [2024-01-18T15:34:03.449Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/fio-writes... [2024-01-18T15:34:03.449Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.449Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.449Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-writes/fio-config-map.yaml [2024-01-18T15:34:03.449Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/fio-writes/fio.yaml [2024-01-18T15:34:03.449Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/fio-writes/fio-config-map.yaml ../drivers/scheduler/k8s/specs/fio-writes/fio.yaml] [2024-01-18T15:34:03.449Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-writes/fio-config-map.yaml [2024-01-18T15:34:03.449Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/fio-writes/fio.yaml [2024-01-18T15:34:03.449Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 8: could not find expected ':' [2024-01-18T15:34:03.449Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: fio-writes [2024-01-18T15:34:03.449Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/kafka-stack... [2024-01-18T15:34:03.449Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.449Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.449Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kafka-stack/es-client.yaml [2024-01-18T15:34:03.449Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kafka-stack/es-data.yaml [2024-01-18T15:34:03.450Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kafka-stack/es-exporter.yaml [2024-01-18T15:34:03.450Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kafka-stack/es-master.yaml [2024-01-18T15:34:03.450Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kafka-stack/kafka-es-connect.yaml [2024-01-18T15:34:03.450Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kafka-stack/kafka-exporter.yaml [2024-01-18T15:34:03.450Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kafka-stack/kafka-producer.yaml [2024-01-18T15:34:03.450Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kafka-stack/kafka.yaml [2024-01-18T15:34:03.450Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kafka-stack/zookeeper.yaml [2024-01-18T15:34:03.450Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/kafka-stack/es-client.yaml ../drivers/scheduler/k8s/specs/kafka-stack/es-data.yaml ../drivers/scheduler/k8s/specs/kafka-stack/es-exporter.yaml ../drivers/scheduler/k8s/specs/kafka-stack/es-master.yaml ../drivers/scheduler/k8s/specs/kafka-stack/kafka-es-connect.yaml ../drivers/scheduler/k8s/specs/kafka-stack/kafka-exporter.yaml ../drivers/scheduler/k8s/specs/kafka-stack/kafka-producer.yaml ../drivers/scheduler/k8s/specs/kafka-stack/kafka.yaml ../drivers/scheduler/k8s/specs/kafka-stack/zookeeper.yaml] [2024-01-18T15:34:03.450Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kafka-stack/es-client.yaml [2024-01-18T15:34:03.450Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kafka-stack/es-data.yaml [2024-01-18T15:34:03.450Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kafka-stack/es-exporter.yaml [2024-01-18T15:34:03.450Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kafka-stack/es-master.yaml [2024-01-18T15:34:03.450Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kafka-stack/kafka-es-connect.yaml [2024-01-18T15:34:03.450Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kafka-stack/kafka-exporter.yaml [2024-01-18T15:34:03.450Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kafka-stack/kafka-producer.yaml [2024-01-18T15:34:03.450Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kafka-stack/kafka.yaml [2024-01-18T15:34:03.450Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kafka-stack/zookeeper.yaml [2024-01-18T15:34:03.450Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: kafka-stack [2024-01-18T15:34:03.451Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/kubevirt-app... [2024-01-18T15:34:03.451Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.451Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.451Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-app/kubevirt-app.yaml [2024-01-18T15:34:03.451Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/kubevirt-app/kubevirt-app.yaml] [2024-01-18T15:34:03.451Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-app/kubevirt-app.yaml [2024-01-18T15:34:03.451Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: kubevirt-app [2024-01-18T15:34:03.451Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/kubevirt-cloudsnap... [2024-01-18T15:34:03.451Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.451Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.451Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-cloudsnap/pvc-longevity-cloudsnap.yaml [2024-01-18T15:34:03.451Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-cloudsnap/vm-longevity-cloudsnap.yaml [2024-01-18T15:34:03.451Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/kubevirt-cloudsnap/pvc-longevity-cloudsnap.yaml ../drivers/scheduler/k8s/specs/kubevirt-cloudsnap/vm-longevity-cloudsnap.yaml] [2024-01-18T15:34:03.451Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-cloudsnap/pvc-longevity-cloudsnap.yaml [2024-01-18T15:34:03.452Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-cloudsnap/vm-longevity-cloudsnap.yaml [2024-01-18T15:34:03.452Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: kubevirt-cloudsnap [2024-01-18T15:34:03.452Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/kubevirt-dv-app... [2024-01-18T15:34:03.452Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.452Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.452Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-dv-app/kubevirt-dv-app.yaml [2024-01-18T15:34:03.452Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/kubevirt-dv-app/kubevirt-dv-app.yaml] [2024-01-18T15:34:03.452Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-dv-app/kubevirt-dv-app.yaml [2024-01-18T15:34:03.452Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: kubevirt-dv-app [2024-01-18T15:34:03.452Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/kubevirt-localsnap... [2024-01-18T15:34:03.452Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.452Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.452Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-localsnap/pvc-longevity-localsnap.yaml [2024-01-18T15:34:03.452Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-localsnap/vm-longevity-localsnap.yaml [2024-01-18T15:34:03.452Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/kubevirt-localsnap/pvc-longevity-localsnap.yaml ../drivers/scheduler/k8s/specs/kubevirt-localsnap/vm-longevity-localsnap.yaml] [2024-01-18T15:34:03.452Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-localsnap/pvc-longevity-localsnap.yaml [2024-01-18T15:34:03.452Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-localsnap/vm-longevity-localsnap.yaml [2024-01-18T15:34:03.452Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: kubevirt-localsnap [2024-01-18T15:34:03.452Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/kubevirt-multi-disk... [2024-01-18T15:34:03.452Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.452Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.452Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-multi-disk/pvc-multi-disk.yaml [2024-01-18T15:34:03.452Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-multi-disk/vm-multi-disk.yaml [2024-01-18T15:34:03.452Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/kubevirt-multi-disk/pvc-multi-disk.yaml ../drivers/scheduler/k8s/specs/kubevirt-multi-disk/vm-multi-disk.yaml] [2024-01-18T15:34:03.452Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-multi-disk/pvc-multi-disk.yaml [2024-01-18T15:34:03.453Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-multi-disk/vm-multi-disk.yaml [2024-01-18T15:34:03.453Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: kubevirt-multi-disk [2024-01-18T15:34:03.453Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/kubevirt-multi-vm... [2024-01-18T15:34:03.453Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.453Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.453Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/pvc-cdi-app-ubuntu.yaml [2024-01-18T15:34:03.453Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/pvc-cdi-ubuntu.yaml [2024-01-18T15:34:03.453Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/pvc-plane.yaml [2024-01-18T15:34:03.453Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/svc-clusterip.yaml [2024-01-18T15:34:03.453Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/svc-loadbalancer.yaml [2024-01-18T15:34:03.453Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/svc-nodeport.yaml [2024-01-18T15:34:03.453Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/vm-ubuntu-containerdisk-pvc.yaml [2024-01-18T15:34:03.453Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/vm-ubuntu-containerdisk.yaml [2024-01-18T15:34:03.453Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/vm-ubuntu-pvc-datavolume.yaml [2024-01-18T15:34:03.453Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/vm-ubuntu-pvc.yaml [2024-01-18T15:34:03.453Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/kubevirt-multi-vm/pvc-cdi-app-ubuntu.yaml ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/pvc-cdi-ubuntu.yaml ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/pvc-plane.yaml ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/svc-clusterip.yaml ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/svc-loadbalancer.yaml ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/svc-nodeport.yaml ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/vm-ubuntu-containerdisk-pvc.yaml ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/vm-ubuntu-containerdisk.yaml ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/vm-ubuntu-pvc-datavolume.yaml ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/vm-ubuntu-pvc.yaml] [2024-01-18T15:34:03.453Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/pvc-cdi-app-ubuntu.yaml [2024-01-18T15:34:03.453Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/pvc-cdi-ubuntu.yaml [2024-01-18T15:34:03.458Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/pvc-plane.yaml [2024-01-18T15:34:03.458Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/svc-clusterip.yaml [2024-01-18T15:34:03.459Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/svc-loadbalancer.yaml [2024-01-18T15:34:03.459Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/svc-nodeport.yaml [2024-01-18T15:34:03.459Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/vm-ubuntu-containerdisk-pvc.yaml [2024-01-18T15:34:03.459Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/vm-ubuntu-containerdisk.yaml [2024-01-18T15:34:03.459Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/vm-ubuntu-pvc-datavolume.yaml [2024-01-18T15:34:03.459Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-multi-vm/vm-ubuntu-pvc.yaml [2024-01-18T15:34:03.459Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: kubevirt-multi-vm [2024-01-18T15:34:03.459Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/kubevirt-simple-vm... [2024-01-18T15:34:03.459Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.459Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.459Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-simple-vm/kubevirt-app.yaml [2024-01-18T15:34:03.459Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/kubevirt-simple-vm/kubevirt-app.yaml] [2024-01-18T15:34:03.459Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-simple-vm/kubevirt-app.yaml [2024-01-18T15:34:03.459Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: kubevirt-simple-vm [2024-01-18T15:34:03.459Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/kubevirt-ssie-io... [2024-01-18T15:34:03.459Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.459Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.459Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-ssie-io/pvc-longevity-auto-journal-io-load.yaml [2024-01-18T15:34:03.459Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-ssie-io/pvc-longevity-io-load.yaml [2024-01-18T15:34:03.459Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-ssie-io/vm-longevity-auto-journal-io-load.yaml [2024-01-18T15:34:03.459Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-ssie-io/vm-longevity-io-load.yaml [2024-01-18T15:34:03.459Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/kubevirt-ssie-io/pvc-longevity-auto-journal-io-load.yaml ../drivers/scheduler/k8s/specs/kubevirt-ssie-io/pvc-longevity-io-load.yaml ../drivers/scheduler/k8s/specs/kubevirt-ssie-io/vm-longevity-auto-journal-io-load.yaml ../drivers/scheduler/k8s/specs/kubevirt-ssie-io/vm-longevity-io-load.yaml] [2024-01-18T15:34:03.459Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-ssie-io/pvc-longevity-auto-journal-io-load.yaml [2024-01-18T15:34:03.459Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-ssie-io/pvc-longevity-io-load.yaml [2024-01-18T15:34:03.459Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-ssie-io/vm-longevity-auto-journal-io-load.yaml [2024-01-18T15:34:03.460Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-ssie-io/vm-longevity-io-load.yaml [2024-01-18T15:34:03.460Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: kubevirt-ssie-io [2024-01-18T15:34:03.460Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/kubevirt-ssie-vm... [2024-01-18T15:34:03.460Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.460Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.460Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-ssie-vm/pvc-longevity-io.yaml [2024-01-18T15:34:03.460Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-ssie-vm/pvc-longevity-simple-vm.yaml [2024-01-18T15:34:03.460Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-ssie-vm/vm-longevity-rwo-io.yaml [2024-01-18T15:34:03.460Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-ssie-vm/vm-ubuntu-longevity-simple-vm.yaml [2024-01-18T15:34:03.460Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/kubevirt-ssie-vm/pvc-longevity-io.yaml ../drivers/scheduler/k8s/specs/kubevirt-ssie-vm/pvc-longevity-simple-vm.yaml ../drivers/scheduler/k8s/specs/kubevirt-ssie-vm/vm-longevity-rwo-io.yaml ../drivers/scheduler/k8s/specs/kubevirt-ssie-vm/vm-ubuntu-longevity-simple-vm.yaml] [2024-01-18T15:34:03.460Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-ssie-vm/pvc-longevity-io.yaml [2024-01-18T15:34:03.460Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-ssie-vm/pvc-longevity-simple-vm.yaml [2024-01-18T15:34:03.460Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-ssie-vm/vm-longevity-rwo-io.yaml [2024-01-18T15:34:03.460Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-ssie-vm/vm-ubuntu-longevity-simple-vm.yaml [2024-01-18T15:34:03.460Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: kubevirt-ssie-vm [2024-01-18T15:34:03.460Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/kubevirt-vm-pvc... [2024-01-18T15:34:03.460Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.460Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.460Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-vm-pvc/kubevirt-app.yaml [2024-01-18T15:34:03.460Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/kubevirt-vm-pvc/storage.yaml [2024-01-18T15:34:03.460Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/kubevirt-vm-pvc/kubevirt-app.yaml ../drivers/scheduler/k8s/specs/kubevirt-vm-pvc/storage.yaml] [2024-01-18T15:34:03.460Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-vm-pvc/kubevirt-app.yaml [2024-01-18T15:34:03.460Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/kubevirt-vm-pvc/storage.yaml [2024-01-18T15:34:03.460Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: kubevirt-vm-pvc [2024-01-18T15:34:03.461Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/ml-workload-continuous-training... [2024-01-18T15:34:03.461Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.461Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.461Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/ml-workload-continuous-training/continuous_retraining.yaml [2024-01-18T15:34:03.461Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/ml-workload-continuous-training/continuous_retraining.yaml] [2024-01-18T15:34:03.461Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/ml-workload-continuous-training/continuous_retraining.yaml [2024-01-18T15:34:03.461Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: ml-workload-continuous-training [2024-01-18T15:34:03.461Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/ml-workload-preprocess-rwx... [2024-01-18T15:34:03.461Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.461Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.461Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/ml-workload-preprocess-rwx/ml-workload-pvc.yaml [2024-01-18T15:34:03.461Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/ml-workload-preprocess-rwx/preprocess_and_training.yaml [2024-01-18T15:34:03.461Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/ml-workload-preprocess-rwx/ml-workload-pvc.yaml ../drivers/scheduler/k8s/specs/ml-workload-preprocess-rwx/preprocess_and_training.yaml] [2024-01-18T15:34:03.461Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/ml-workload-preprocess-rwx/ml-workload-pvc.yaml [2024-01-18T15:34:03.461Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/ml-workload-preprocess-rwx/preprocess_and_training.yaml [2024-01-18T15:34:03.461Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: ml-workload-preprocess-rwx [2024-01-18T15:34:03.461Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/ml-workload-rwx... [2024-01-18T15:34:03.461Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.461Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.461Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/ml-workload-rwx/query-ml-workload.yaml [2024-01-18T15:34:03.461Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/ml-workload-rwx/query-ml-workload.yaml] [2024-01-18T15:34:03.461Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/ml-workload-rwx/query-ml-workload.yaml [2024-01-18T15:34:03.461Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: ml-workload-rwx [2024-01-18T15:34:03.461Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/mongo-vps... [2024-01-18T15:34:03.462Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.462Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.462Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/mongo-vps/px-mongo-statefulset.yaml [2024-01-18T15:34:03.462Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/mongo-vps/px-storage-class.yaml [2024-01-18T15:34:03.462Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/mongo-vps/px-mongo-statefulset.yaml ../drivers/scheduler/k8s/specs/mongo-vps/px-storage-class.yaml] [2024-01-18T15:34:03.462Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/mongo-vps/px-mongo-statefulset.yaml [2024-01-18T15:34:03.462Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/mongo-vps/px-storage-class.yaml [2024-01-18T15:34:03.462Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: mongo-vps [2024-01-18T15:34:03.462Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/mongodb... [2024-01-18T15:34:03.462Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.462Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.462Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/mongodb/px-mongo-app.yaml [2024-01-18T15:34:03.462Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/mongodb/px-mongo-storage.yaml [2024-01-18T15:34:03.462Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/mongodb/ycsb.yaml [2024-01-18T15:34:03.462Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/mongodb/px-mongo-app.yaml ../drivers/scheduler/k8s/specs/mongodb/px-mongo-storage.yaml ../drivers/scheduler/k8s/specs/mongodb/ycsb.yaml] [2024-01-18T15:34:03.462Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/mongodb/px-mongo-app.yaml [2024-01-18T15:34:03.462Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/mongodb/px-mongo-storage.yaml [2024-01-18T15:34:03.462Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/mongodb/ycsb.yaml [2024-01-18T15:34:03.462Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: mongodb [2024-01-18T15:34:03.462Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/mysql... [2024-01-18T15:34:03.462Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.463Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.463Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/mysql/px-mysql-app.yaml [2024-01-18T15:34:03.463Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/mysql/px-mysql-storage.yaml [2024-01-18T15:34:03.463Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/mysql/px-mysql-app.yaml ../drivers/scheduler/k8s/specs/mysql/px-mysql-storage.yaml] [2024-01-18T15:34:03.463Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/mysql/px-mysql-app.yaml [2024-01-18T15:34:03.463Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/mysql/px-mysql-storage.yaml [2024-01-18T15:34:03.463Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: mysql [2024-01-18T15:34:03.463Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/mysql-backup... [2024-01-18T15:34:03.463Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.463Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.463Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/mysql-backup/px-mysql-app.yaml [2024-01-18T15:34:03.463Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/mysql-backup/px-mysql-storage.yaml [2024-01-18T15:34:03.463Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/mysql-backup/px-mysql-app.yaml ../drivers/scheduler/k8s/specs/mysql-backup/px-mysql-storage.yaml] [2024-01-18T15:34:03.463Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/mysql-backup/px-mysql-app.yaml [2024-01-18T15:34:03.463Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/mysql-backup/px-mysql-storage.yaml [2024-01-18T15:34:03.463Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: mysql-backup [2024-01-18T15:34:03.463Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/mysql-ibm... [2024-01-18T15:34:03.463Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.463Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.463Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/mysql-ibm/ibm/ibm-storage-class.yaml [2024-01-18T15:34:03.463Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/mysql-ibm/mysql-app.yaml [2024-01-18T15:34:03.463Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/mysql-ibm/mysql-storage.yaml [2024-01-18T15:34:03.463Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/mysql-ibm/ibm/ibm-storage-class.yaml ../drivers/scheduler/k8s/specs/mysql-ibm/mysql-app.yaml ../drivers/scheduler/k8s/specs/mysql-ibm/mysql-storage.yaml] [2024-01-18T15:34:03.463Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/mysql-ibm/ibm/ibm-storage-class.yaml [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/mysql-ibm/mysql-app.yaml [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/mysql-ibm/mysql-storage.yaml [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: mysql-ibm [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/mysql-without-enc... [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/mysql-without-enc/px-mysql-app.yaml [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/mysql-without-enc/px-mysql-storage.yaml [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/mysql-without-enc/px-mysql-app.yaml ../drivers/scheduler/k8s/specs/mysql-without-enc/px-mysql-storage.yaml] [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/mysql-without-enc/px-mysql-app.yaml [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/mysql-without-enc/px-mysql-storage.yaml [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: mysql-without-enc [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/nginx... [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/nginx/px-nginx-app.yaml [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/nginx/px-nginx-storage.yaml [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/nginx/px-nginx-app.yaml ../drivers/scheduler/k8s/specs/nginx/px-nginx-storage.yaml] [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/nginx/px-nginx-app.yaml [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/nginx/px-nginx-storage.yaml [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: nginx [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/nginx-fa-darawvol... [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/nginx-fa-darawvol/rawblock-pod.yaml [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/nginx-fa-darawvol/rawblock-pvc.yaml [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/nginx-fa-darawvol/rawblock-pod.yaml ../drivers/scheduler/k8s/specs/nginx-fa-darawvol/rawblock-pvc.yaml] [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/nginx-fa-darawvol/rawblock-pod.yaml [2024-01-18T15:34:03.464Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/nginx-fa-darawvol/rawblock-pvc.yaml [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: nginx-fa-darawvol [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/nginx-fa-davol... [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/nginx-fa-davol/px-pod.yaml [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/nginx-fa-davol/px-pvc.yaml [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/nginx-fa-davol/px-pod.yaml ../drivers/scheduler/k8s/specs/nginx-fa-davol/px-pvc.yaml] [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/nginx-fa-davol/px-pod.yaml [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/nginx-fa-davol/px-pvc.yaml [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: nginx-fa-davol [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/nginx-fada-deploy... [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/nginx-fada-deploy/px-deploy.yaml [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/nginx-fada-deploy/px-pvc.yaml [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/nginx-fada-deploy/px-deploy.yaml ../drivers/scheduler/k8s/specs/nginx-fada-deploy/px-pvc.yaml] [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/nginx-fada-deploy/px-deploy.yaml [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/nginx-fada-deploy/px-pvc.yaml [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: nginx-fada-deploy [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/nginx-fada-repl-vps... [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/nginx-fada-repl-vps/px-pod.yaml [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/nginx-fada-repl-vps/px-pvc.yaml [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/nginx-fada-repl-vps/px-pod.yaml ../drivers/scheduler/k8s/specs/nginx-fada-repl-vps/px-pvc.yaml] [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/nginx-fada-repl-vps/px-pod.yaml [2024-01-18T15:34:03.465Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/nginx-fada-repl-vps/px-pvc.yaml [2024-01-18T15:34:03.466Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: nginx-fada-repl-vps [2024-01-18T15:34:03.466Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/nginx-proxy-deployment... [2024-01-18T15:34:03.466Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.466Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.466Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/nginx-proxy-deployment/px-nginx-app.yaml [2024-01-18T15:34:03.466Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/nginx-proxy-deployment/px-nginx-storage.yaml [2024-01-18T15:34:03.466Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/nginx-proxy-deployment/px-nginx-app.yaml ../drivers/scheduler/k8s/specs/nginx-proxy-deployment/px-nginx-storage.yaml] [2024-01-18T15:34:03.466Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/nginx-proxy-deployment/px-nginx-app.yaml [2024-01-18T15:34:03.466Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/nginx-proxy-deployment/px-nginx-storage.yaml [2024-01-18T15:34:03.466Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: nginx-proxy-deployment [2024-01-18T15:34:03.466Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/nginx-secure-sc... [2024-01-18T15:34:03.466Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.466Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.466Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/nginx-secure-sc/px-nginx-app.yaml [2024-01-18T15:34:03.466Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/nginx-secure-sc/px-nginx-storage.yaml [2024-01-18T15:34:03.466Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/nginx-secure-sc/px-storage-class.yaml [2024-01-18T15:34:03.466Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/nginx-secure-sc/px-nginx-app.yaml ../drivers/scheduler/k8s/specs/nginx-secure-sc/px-nginx-storage.yaml ../drivers/scheduler/k8s/specs/nginx-secure-sc/px-storage-class.yaml] [2024-01-18T15:34:03.466Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/nginx-secure-sc/px-nginx-app.yaml [2024-01-18T15:34:03.466Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/nginx-secure-sc/px-nginx-storage.yaml [2024-01-18T15:34:03.466Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/nginx-secure-sc/px-storage-class.yaml [2024-01-18T15:34:03.466Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: nginx-secure-sc [2024-01-18T15:34:03.466Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/nginx-shared... [2024-01-18T15:34:03.466Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.466Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.466Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/nginx-shared/px-nginx-app.yaml [2024-01-18T15:34:03.466Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/nginx-shared/px-nginx-storage.yaml [2024-01-18T15:34:03.467Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/nginx-shared/px-nginx-app.yaml ../drivers/scheduler/k8s/specs/nginx-shared/px-nginx-storage.yaml] [2024-01-18T15:34:03.467Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/nginx-shared/px-nginx-app.yaml [2024-01-18T15:34:03.467Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/nginx-shared/px-nginx-storage.yaml [2024-01-18T15:34:03.467Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: nginx-shared [2024-01-18T15:34:03.467Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/nginx-sharedv4... [2024-01-18T15:34:03.467Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.467Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.467Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/nginx-sharedv4/px-nginx-app.yaml [2024-01-18T15:34:03.467Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/nginx-sharedv4/px-nginx-storage.yaml [2024-01-18T15:34:03.467Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/nginx-sharedv4/px-nginx-app.yaml ../drivers/scheduler/k8s/specs/nginx-sharedv4/px-nginx-storage.yaml] [2024-01-18T15:34:03.467Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/nginx-sharedv4/px-nginx-app.yaml [2024-01-18T15:34:03.467Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/nginx-sharedv4/px-nginx-storage.yaml [2024-01-18T15:34:03.467Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: nginx-sharedv4 [2024-01-18T15:34:03.467Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/nginx-sv4-svc... [2024-01-18T15:34:03.467Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.467Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.467Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/nginx-sv4-svc/px-nginx-app.yaml [2024-01-18T15:34:03.467Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/nginx-sv4-svc/px-nginx-storage.yaml [2024-01-18T15:34:03.467Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/nginx-sv4-svc/px-nginx-app.yaml ../drivers/scheduler/k8s/specs/nginx-sv4-svc/px-nginx-storage.yaml] [2024-01-18T15:34:03.467Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/nginx-sv4-svc/px-nginx-app.yaml [2024-01-18T15:34:03.467Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/nginx-sv4-svc/px-nginx-storage.yaml [2024-01-18T15:34:03.467Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: nginx-sv4-svc [2024-01-18T15:34:03.467Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/nginx-sv4-svc-1pv... [2024-01-18T15:34:03.467Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.467Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/nginx-sv4-svc-1pv/px-nginx-app.yaml [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/nginx-sv4-svc-1pv/px-nginx-storage.yaml [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/nginx-sv4-svc-1pv/px-nginx-app.yaml ../drivers/scheduler/k8s/specs/nginx-sv4-svc-1pv/px-nginx-storage.yaml] [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/nginx-sv4-svc-1pv/px-nginx-app.yaml [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/nginx-sv4-svc-1pv/px-nginx-storage.yaml [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: nginx-sv4-svc-1pv [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/nginx-without-enc... [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/nginx-without-enc/px-nginx-app.yaml [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/nginx-without-enc/px-nginx-storage.yaml [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/nginx-without-enc/px-nginx-app.yaml ../drivers/scheduler/k8s/specs/nginx-without-enc/px-nginx-storage.yaml] [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/nginx-without-enc/px-nginx-app.yaml [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/nginx-without-enc/px-nginx-storage.yaml [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: nginx-without-enc [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/pgbench... [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/pgbench/px-pgbench-app.yaml [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/pgbench/px-pgbench-storage.yaml [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/pgbench/px-pgbench-app.yaml ../drivers/scheduler/k8s/specs/pgbench/px-pgbench-storage.yaml] [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/pgbench/px-pgbench-app.yaml [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/pgbench/px-pgbench-storage.yaml [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: pgbench [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/pgbench-io-portion... [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.468Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/pgbench-io-portion/px-pgbench-app.yaml [2024-01-18T15:34:03.469Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/pgbench-io-portion/px-pgbench-storage.yaml [2024-01-18T15:34:03.469Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/pgbench-io-portion/px-pgbench-app.yaml ../drivers/scheduler/k8s/specs/pgbench-io-portion/px-pgbench-storage.yaml] [2024-01-18T15:34:03.469Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/pgbench-io-portion/px-pgbench-app.yaml [2024-01-18T15:34:03.469Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/pgbench-io-portion/px-pgbench-storage.yaml [2024-01-18T15:34:03.469Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 10: could not find expected ':' [2024-01-18T15:34:03.469Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: pgbench-io-portion [2024-01-18T15:34:03.469Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/pgbench-low-io... [2024-01-18T15:34:03.469Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.469Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.469Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/pgbench-low-io/px-pgbench-app.yaml [2024-01-18T15:34:03.469Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/pgbench-low-io/px-pgbench-storage.yaml [2024-01-18T15:34:03.469Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/pgbench-low-io/px-pgbench-app.yaml ../drivers/scheduler/k8s/specs/pgbench-low-io/px-pgbench-storage.yaml] [2024-01-18T15:34:03.469Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/pgbench-low-io/px-pgbench-app.yaml [2024-01-18T15:34:03.469Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/pgbench-low-io/px-pgbench-storage.yaml [2024-01-18T15:34:03.725Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: pgbench-low-io [2024-01-18T15:34:03.725Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/postgres... [2024-01-18T15:34:03.725Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.725Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.725Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/postgres/px-postgres-app.yaml [2024-01-18T15:34:03.725Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/postgres/storage.yaml [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/postgres/px-postgres-app.yaml ../drivers/scheduler/k8s/specs/postgres/storage.yaml] [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/postgres/px-postgres-app.yaml [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/postgres/storage.yaml [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: postgres [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/postgres-backup... [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/postgres-backup/px-postgres-app.yaml [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/postgres-backup/storage.yaml [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/postgres-backup/px-postgres-app.yaml ../drivers/scheduler/k8s/specs/postgres-backup/storage.yaml] [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/postgres-backup/px-postgres-app.yaml [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/postgres-backup/storage.yaml [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: postgres-backup [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/postgres-cephfs-csi... [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/postgres-cephfs-csi/postgres-csi-app.yaml [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/postgres-cephfs-csi/storage.yaml [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/postgres-cephfs-csi/postgres-csi-app.yaml ../drivers/scheduler/k8s/specs/postgres-cephfs-csi/storage.yaml] [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/postgres-cephfs-csi/postgres-csi-app.yaml [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/postgres-cephfs-csi/storage.yaml [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: postgres-cephfs-csi [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/postgres-csi... [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/postgres-csi/ibm/ibm-storage-class.yaml [2024-01-18T15:34:03.726Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/postgres-csi/postgres-csi-app.yaml [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/postgres-csi/storage.yaml [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/postgres-csi/ibm/ibm-storage-class.yaml ../drivers/scheduler/k8s/specs/postgres-csi/postgres-csi-app.yaml ../drivers/scheduler/k8s/specs/postgres-csi/storage.yaml] [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/postgres-csi/ibm/ibm-storage-class.yaml [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/postgres-csi/postgres-csi-app.yaml [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/postgres-csi/storage.yaml [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: postgres-csi [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/postgres-gce-pd... [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/postgres-gce-pd/postgres-csi-app.yaml [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/postgres-gce-pd/storage.yaml [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/postgres-gce-pd/postgres-csi-app.yaml ../drivers/scheduler/k8s/specs/postgres-gce-pd/storage.yaml] [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/postgres-gce-pd/postgres-csi-app.yaml [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/postgres-gce-pd/storage.yaml [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: postgres-gce-pd [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/postgres-pd-csi-storage-gke... [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/postgres-pd-csi-storage-gke/postgres-csi-app.yaml [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/postgres-pd-csi-storage-gke/storage.yaml [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/postgres-pd-csi-storage-gke/postgres-csi-app.yaml ../drivers/scheduler/k8s/specs/postgres-pd-csi-storage-gke/storage.yaml] [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/postgres-pd-csi-storage-gke/postgres-csi-app.yaml [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/postgres-pd-csi-storage-gke/storage.yaml [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: postgres-pd-csi-storage-gke [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/postgres-rbd-csi... [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.727Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/postgres-rbd-csi/postgres-csi-app.yaml [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/postgres-rbd-csi/storage.yaml [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/postgres-rbd-csi/postgres-csi-app.yaml ../drivers/scheduler/k8s/specs/postgres-rbd-csi/storage.yaml] [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/postgres-rbd-csi/postgres-csi-app.yaml [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/postgres-rbd-csi/storage.yaml [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: postgres-rbd-csi [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/postgres-rgw-csi... [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/postgres-rgw-csi/postgres-csi-app.yaml [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/postgres-rgw-csi/storage.yaml [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/postgres-rgw-csi/postgres-csi-app.yaml ../drivers/scheduler/k8s/specs/postgres-rgw-csi/storage.yaml] [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/postgres-rgw-csi/postgres-csi-app.yaml [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/postgres-rgw-csi/storage.yaml [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: postgres-rgw-csi [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/px-central... [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/px-central/px-central-repo.yaml [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/px-central/px-central-repo.yaml] [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/px-central/px-central-repo.yaml [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[INFO] [k8s.(*K8s).IsAppHelmChartType:#607] - Helm chart was found in file: [../drivers/scheduler/k8s/specs/px-central/px-central-repo.yaml] [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: px-central [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/px-license-server... [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/px-license-server/px-license-server-repo.yaml [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/px-license-server/px-license-server-repo.yaml] [2024-01-18T15:34:03.728Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/px-license-server/px-license-server-repo.yaml [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[INFO] [k8s.(*K8s).IsAppHelmChartType:#607] - Helm chart was found in file: [../drivers/scheduler/k8s/specs/px-license-server/px-license-server-repo.yaml] [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: px-license-server [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/px-monitor... [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/px-monitor/px-monitor-repo.yaml [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/px-monitor/px-monitor-repo.yaml] [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/px-monitor/px-monitor-repo.yaml [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[INFO] [k8s.(*K8s).IsAppHelmChartType:#607] - Helm chart was found in file: [../drivers/scheduler/k8s/specs/px-monitor/px-monitor-repo.yaml] [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: px-monitor [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/rabbitmq... [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/rabbitmq/px-rabbitmq-app.yml [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/rabbitmq/px-rabbitmq-app.yml] [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/rabbitmq/px-rabbitmq-app.yml [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: rabbitmq [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/scylladb... [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/scylladb/cassandra-stress-app.yaml [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/scylladb/px-scylladb-storage.yaml [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/scylladb/scylla.yaml [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/scylladb/scylladb-app.yaml [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/scylladb/cassandra-stress-app.yaml ../drivers/scheduler/k8s/specs/scylladb/px-scylladb-storage.yaml ../drivers/scheduler/k8s/specs/scylladb/scylla.yaml ../drivers/scheduler/k8s/specs/scylladb/scylladb-app.yaml] [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/scylladb/cassandra-stress-app.yaml [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/scylladb/px-scylladb-storage.yaml [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/scylladb/scylla.yaml [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/scylladb/scylladb-app.yaml [2024-01-18T15:34:03.729Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: scylladb [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/snap-mysql... [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/snap-mysql/px-mysql-app.yaml [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/snap-mysql/px-mysql-snapshots.yaml [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/snap-mysql/px-mysql-storage.yaml [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/snap-mysql/px-mysql-app.yaml ../drivers/scheduler/k8s/specs/snap-mysql/px-mysql-snapshots.yaml ../drivers/scheduler/k8s/specs/snap-mysql/px-mysql-storage.yaml] [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/snap-mysql/px-mysql-app.yaml [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/snap-mysql/px-mysql-snapshots.yaml [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/snap-mysql/px-mysql-storage.yaml [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: snap-mysql [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/sysbench... [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/sysbench/px-sysbench-app.yml [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/sysbench/px-sysbench-storage.yml [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/sysbench/px-sysbench-app.yml ../drivers/scheduler/k8s/specs/sysbench/px-sysbench-storage.yml] [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/sysbench/px-sysbench-app.yml [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/sysbench/px-sysbench-storage.yml [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: sysbench [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/sysbench-heavyload... [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/sysbench-heavyload/px-sysbench-app.yml [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/sysbench-heavyload/px-sysbench-storage.yml [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/sysbench-heavyload/px-sysbench-app.yml ../drivers/scheduler/k8s/specs/sysbench-heavyload/px-sysbench-storage.yml] [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/sysbench-heavyload/px-sysbench-app.yml [2024-01-18T15:34:03.730Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/sysbench-heavyload/px-sysbench-storage.yml [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: sysbench-heavyload [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/sysbench-postgres-fstrim... [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/sysbench-postgres-fstrim/px-sysbench-app.yml [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/sysbench-postgres-fstrim/px-sysbench-storage.yml [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/sysbench-postgres-fstrim/storage-class.yaml [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/sysbench-postgres-fstrim/px-sysbench-app.yml ../drivers/scheduler/k8s/specs/sysbench-postgres-fstrim/px-sysbench-storage.yml ../drivers/scheduler/k8s/specs/sysbench-postgres-fstrim/storage-class.yaml] [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/sysbench-postgres-fstrim/px-sysbench-app.yml [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/sysbench-postgres-fstrim/px-sysbench-storage.yml [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/sysbench-postgres-fstrim/storage-class.yaml [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 10: could not find expected ':' [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: sysbench-postgres-fstrim [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/sysbench-st... [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/sysbench-st/px-sysbench-app.yml [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/sysbench-st/px-sysbench-storage.yml [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/sysbench-st/px-sysbench-app.yml ../drivers/scheduler/k8s/specs/sysbench-st/px-sysbench-storage.yml] [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/sysbench-st/px-sysbench-app.yml [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 56: did not find expected key [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/sysbench-st/px-sysbench-storage.yml [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 10: did not find expected key [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: sysbench-st [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/test-sharedv4... [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.731Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/test-sharedv4/pod.yaml [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/test-sharedv4/pvc.yaml [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/test-sharedv4/pod.yaml ../drivers/scheduler/k8s/specs/test-sharedv4/pvc.yaml] [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/test-sharedv4/pod.yaml [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 6: did not find expected key [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/test-sharedv4/pvc.yaml [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: test-sharedv4 [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/test-sv4-svc... [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/test-sv4-svc/pod.yaml [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/test-sv4-svc/pvc.yaml [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/test-sv4-svc/pod.yaml ../drivers/scheduler/k8s/specs/test-sv4-svc/pvc.yaml] [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/test-sv4-svc/pod.yaml [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 6: did not find expected key [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/test-sv4-svc/pvc.yaml [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 10: could not find expected ':' [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: test-sv4-svc [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/test-sv4-svc-2... [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/test-sv4-svc-2/pod.yaml [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/test-sv4-svc-2/pvc.yaml [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/test-sv4-svc-2/pod.yaml ../drivers/scheduler/k8s/specs/test-sv4-svc-2/pvc.yaml] [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/test-sv4-svc-2/pod.yaml [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 6: did not find expected key [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/test-sv4-svc-2/pvc.yaml [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: test-sv4-svc-2 [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/test-sv4-svc-enc... [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.732Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.733Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/test-sv4-svc-enc/pod.yaml [2024-01-18T15:34:03.733Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/test-sv4-svc-enc/pvc.yaml [2024-01-18T15:34:03.733Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/test-sv4-svc-enc/pod.yaml ../drivers/scheduler/k8s/specs/test-sv4-svc-enc/pvc.yaml] [2024-01-18T15:34:03.733Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/test-sv4-svc-enc/pod.yaml [2024-01-18T15:34:03.733Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 6: did not find expected key [2024-01-18T15:34:03.733Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/test-sv4-svc-enc/pvc.yaml [2024-01-18T15:34:03.733Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: test-sv4-svc-enc [2024-01-18T15:34:03.733Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/vdbench-fastpath... [2024-01-18T15:34:03.733Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.733Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.733Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/vdbench-fastpath/px-vdbench-app.yml [2024-01-18T15:34:03.733Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/vdbench-fastpath/px-vdbench-storage.yml [2024-01-18T15:34:03.733Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/vdbench-fastpath/px-vdbench-app.yml ../drivers/scheduler/k8s/specs/vdbench-fastpath/px-vdbench-storage.yml] [2024-01-18T15:34:03.733Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/vdbench-fastpath/px-vdbench-app.yml [2024-01-18T15:34:03.733Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/vdbench-fastpath/px-vdbench-storage.yml [2024-01-18T15:34:03.733Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: vdbench-fastpath [2024-01-18T15:34:03.733Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/vdbench-fb-vols... [2024-01-18T15:34:03.733Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.733Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.733Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/vdbench-fb-vols/pure-fb-storage-class.yaml [2024-01-18T15:34:03.733Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/vdbench-fb-vols/pure-vdbench-storage.yaml [2024-01-18T15:34:03.733Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/vdbench-fb-vols/px-vdbench-app.yaml [2024-01-18T15:34:03.733Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/vdbench-fb-vols/pure-fb-storage-class.yaml ../drivers/scheduler/k8s/specs/vdbench-fb-vols/pure-vdbench-storage.yaml ../drivers/scheduler/k8s/specs/vdbench-fb-vols/px-vdbench-app.yaml] [2024-01-18T15:34:03.733Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/vdbench-fb-vols/pure-fb-storage-class.yaml [2024-01-18T15:34:03.733Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/vdbench-fb-vols/pure-vdbench-storage.yaml [2024-01-18T15:34:03.733Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/vdbench-fb-vols/px-vdbench-app.yaml [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: vdbench-fb-vols [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/vdbench-heavyload... [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/vdbench-heavyload/px-vdbench-app.yml [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/vdbench-heavyload/px-vdbench-storage.yml [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/vdbench-heavyload/px-vdbench-app.yml ../drivers/scheduler/k8s/specs/vdbench-heavyload/px-vdbench-storage.yml] [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/vdbench-heavyload/px-vdbench-app.yml [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/vdbench-heavyload/px-vdbench-storage.yml [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 8: did not find expected key [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: vdbench-heavyload [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/vdbench-sharedv4... [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/vdbench-sharedv4/px-vdbench-app.yml [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/vdbench-sharedv4/px-vdbench-storage.yml [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/vdbench-sharedv4/px-vdbench-app.yml ../drivers/scheduler/k8s/specs/vdbench-sharedv4/px-vdbench-storage.yml] [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/vdbench-sharedv4/px-vdbench-app.yml [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/vdbench-sharedv4/px-vdbench-storage.yml [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: vdbench-sharedv4 [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/vdbench-sv4-multivol... [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/vdbench-sv4-multivol/px-vdbench-app.yml [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/vdbench-sv4-multivol/px-vdbench-pvc.yaml [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/vdbench-sv4-multivol/px-vdbench-storage.yml [2024-01-18T15:34:03.734Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/vdbench-sv4-multivol/px-vdbench-app.yml ../drivers/scheduler/k8s/specs/vdbench-sv4-multivol/px-vdbench-pvc.yaml ../drivers/scheduler/k8s/specs/vdbench-sv4-multivol/px-vdbench-storage.yml] [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/vdbench-sv4-multivol/px-vdbench-app.yml [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 30: could not find expected ':' [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/vdbench-sv4-multivol/px-vdbench-pvc.yaml [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: did not find expected node content [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/vdbench-sv4-multivol/px-vdbench-storage.yml [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 8: did not find expected key [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: vdbench-sv4-multivol [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/vdbench-sv4-r2-mvol... [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/vdbench-sv4-r2-mvol/px-vdbench-app.yml [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/vdbench-sv4-r2-mvol/px-vdbench-storage.yml [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/vdbench-sv4-r2-mvol/px-vdbench-app.yml ../drivers/scheduler/k8s/specs/vdbench-sv4-r2-mvol/px-vdbench-storage.yml] [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/vdbench-sv4-r2-mvol/px-vdbench-app.yml [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/vdbench-sv4-r2-mvol/px-vdbench-storage.yml [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: vdbench-sv4-r2-mvol [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/vdbench-sv4-svc... [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/vdbench-sv4-svc/px-vdbench-app.yml [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/vdbench-sv4-svc/px-vdbench-storage.yml [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/vdbench-sv4-svc/px-vdbench-app.yml ../drivers/scheduler/k8s/specs/vdbench-sv4-svc/px-vdbench-storage.yml] [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/vdbench-sv4-svc/px-vdbench-app.yml [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/vdbench-sv4-svc/px-vdbench-storage.yml [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: vdbench-sv4-svc [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/vdbench-sv4-svc-multivol... [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.735Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/vdbench-sv4-svc-multivol/px-vdbench-app.yml [2024-01-18T15:34:03.736Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/vdbench-sv4-svc-multivol/px-vdbench-pvc.yaml [2024-01-18T15:34:03.736Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/vdbench-sv4-svc-multivol/px-vdbench-storage.yml [2024-01-18T15:34:03.736Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/vdbench-sv4-svc-multivol/px-vdbench-app.yml ../drivers/scheduler/k8s/specs/vdbench-sv4-svc-multivol/px-vdbench-pvc.yaml ../drivers/scheduler/k8s/specs/vdbench-sv4-svc-multivol/px-vdbench-storage.yml] [2024-01-18T15:34:03.736Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/vdbench-sv4-svc-multivol/px-vdbench-app.yml [2024-01-18T15:34:03.736Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 30: could not find expected ':' [2024-01-18T15:34:03.736Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/vdbench-sv4-svc-multivol/px-vdbench-pvc.yaml [2024-01-18T15:34:03.736Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: did not find expected node content [2024-01-18T15:34:03.736Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/vdbench-sv4-svc-multivol/px-vdbench-storage.yml [2024-01-18T15:34:03.736Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 8: did not find expected key [2024-01-18T15:34:03.736Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: vdbench-sv4-svc-multivol [2024-01-18T15:34:03.736Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/vdbench-v4-fstrim... [2024-01-18T15:34:03.736Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.736Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.736Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/vdbench-v4-fstrim/vd-sharedv4-st.yaml [2024-01-18T15:34:03.736Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/vdbench-v4-fstrim/vdbench-v4.yaml [2024-01-18T15:34:03.736Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/vdbench-v4-fstrim/vd-sharedv4-st.yaml ../drivers/scheduler/k8s/specs/vdbench-v4-fstrim/vdbench-v4.yaml] [2024-01-18T15:34:03.736Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/vdbench-v4-fstrim/vd-sharedv4-st.yaml [2024-01-18T15:34:03.736Z] 2024-01-18 15:34:03 +0000:[ERROR] [k8s.(*K8s).IsAppHelmChartType:#601] - Ignoring the yaml unmarshalling failure , err: yaml: line 13: could not find expected ':' [2024-01-18T15:34:03.736Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/vdbench-v4-fstrim/vdbench-v4.yaml [2024-01-18T15:34:03.736Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: vdbench-v4-fstrim [2024-01-18T15:34:03.736Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/wordpress... [2024-01-18T15:34:03.736Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.736Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.736Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/wordpress/load-driver.yaml [2024-01-18T15:34:03.736Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/wordpress/mysql.yaml [2024-01-18T15:34:03.736Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/wordpress/storage.yaml [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/wordpress/wordpress.yaml [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/wordpress/load-driver.yaml ../drivers/scheduler/k8s/specs/wordpress/mysql.yaml ../drivers/scheduler/k8s/specs/wordpress/storage.yaml ../drivers/scheduler/k8s/specs/wordpress/wordpress.yaml] [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/wordpress/load-driver.yaml [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/wordpress/mysql.yaml [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/wordpress/storage.yaml [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/wordpress/wordpress.yaml [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: wordpress [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/wordpress-shared... [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/wordpress-shared/load-driver.yaml [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/wordpress-shared/mysql.yaml [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/wordpress-shared/storage.yaml [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/wordpress-shared/wordpress.yaml [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/wordpress-shared/load-driver.yaml ../drivers/scheduler/k8s/specs/wordpress-shared/mysql.yaml ../drivers/scheduler/k8s/specs/wordpress-shared/storage.yaml ../drivers/scheduler/k8s/specs/wordpress-shared/wordpress.yaml] [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/wordpress-shared/load-driver.yaml [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/wordpress-shared/mysql.yaml [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/wordpress-shared/storage.yaml [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/wordpress-shared/wordpress.yaml [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: wordpress-shared [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/wordpress-sharedv4... [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/wordpress-sharedv4/load-driver.yaml [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/wordpress-sharedv4/mysql.yaml [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/wordpress-sharedv4/storage.yaml [2024-01-18T15:34:03.737Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/wordpress-sharedv4/wordpress.yaml [2024-01-18T15:34:03.738Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/wordpress-sharedv4/load-driver.yaml ../drivers/scheduler/k8s/specs/wordpress-sharedv4/mysql.yaml ../drivers/scheduler/k8s/specs/wordpress-sharedv4/storage.yaml ../drivers/scheduler/k8s/specs/wordpress-sharedv4/wordpress.yaml] [2024-01-18T15:34:03.738Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/wordpress-sharedv4/load-driver.yaml [2024-01-18T15:34:03.738Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/wordpress-sharedv4/mysql.yaml [2024-01-18T15:34:03.738Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/wordpress-sharedv4/storage.yaml [2024-01-18T15:34:03.738Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/wordpress-sharedv4/wordpress.yaml [2024-01-18T15:34:03.738Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: wordpress-sharedv4 [2024-01-18T15:34:03.738Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#79] - Parsing: ../drivers/scheduler/k8s/specs/wordpress-sv4-svc... [2024-01-18T15:34:03.738Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.NewFactory:#80] - Storage provisioner ibm [2024-01-18T15:34:03.738Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#470] - ParseSpecs k.CustomConfig = map[] [2024-01-18T15:34:03.738Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/wordpress-sv4-svc/load-driver.yaml [2024-01-18T15:34:03.738Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/wordpress-sv4-svc/mysql.yaml [2024-01-18T15:34:03.738Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/wordpress-sv4-svc/storage.yaml [2024-01-18T15:34:03.738Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs.func1:#475] - add filepath: ../drivers/scheduler/k8s/specs/wordpress-sv4-svc/wordpress.yaml [2024-01-18T15:34:03.738Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).ParseSpecs:#485] - fileList: [../drivers/scheduler/k8s/specs/wordpress-sv4-svc/load-driver.yaml ../drivers/scheduler/k8s/specs/wordpress-sv4-svc/mysql.yaml ../drivers/scheduler/k8s/specs/wordpress-sv4-svc/storage.yaml ../drivers/scheduler/k8s/specs/wordpress-sv4-svc/wordpress.yaml] [2024-01-18T15:34:03.738Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/wordpress-sv4-svc/load-driver.yaml [2024-01-18T15:34:03.738Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/wordpress-sv4-svc/mysql.yaml [2024-01-18T15:34:03.738Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/wordpress-sv4-svc/storage.yaml [2024-01-18T15:34:03.738Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).IsAppHelmChartType:#591] - Reading file: ../drivers/scheduler/k8s/specs/wordpress-sv4-svc/wordpress.yaml [2024-01-18T15:34:03.738Z] 2024-01-18 15:34:03 +0000:[DEBUG] [spec.(*Factory).register:#23] - Registering new app: wordpress-sv4-svc [2024-01-18T15:34:03.738Z] 2024-01-18 15:34:03 +0000:[INFO] [k8s.(*K8s).collectEvents:#6345] - Started collecting events [2024-01-18T15:34:03.738Z] 2024-01-18 15:34:03 +0000:[DEBUG] [k8s.(*K8s).getKubeClient:#6400] - will use in-cluster config to create k8s client [2024-01-18T15:34:03.738Z] 2024-01-18 15:34:03 +0000:[WARNING] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T15:34:03.738Z] 2024-01-18 15:34:03 +0000:[WARNING] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T15:34:03.738Z] 2024-01-18 15:34:03 +0000:[INFO] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T15:34:03.738Z] 2024-01-18 15:34:03 +0000:[INFO] [ibm.(*ibm).Init:#30] - Using the IBM volume driver with provisioner ibm under scheduler: k8s [2024-01-18T15:34:03.738Z] 2024-01-18 15:34:03 +0000:[INFO] [portworx.(*portworx).Init:#126] - using portworx backup driver under scheduler: k8s [2024-01-18T15:34:03.994Z] 2024-01-18 15:34:03 +0000:[INFO] [portworx.(*portworx).testAndSetEndpoint:#195] - Using 172.21.1.93:10002 as endpoint for portworx backup driver [2024-01-18T15:34:03.994Z] 2024-01-18 15:34:03 +0000:[DEBUG] [tests.SetupTestRail:#8177] - Not all information to connect to testrail is provided, skipping updates to testrail [2024-01-18T15:34:03.994Z] 2024-01-18 15:34:03 +0000:[WARNING] [ibm.(*ibm).GetDriverVersion:#69] - GetDriverVersion function has not been implemented for volume driver - ibm [2024-01-18T15:34:07.261Z] 2024-01-18 15:34:06 +0000:[INFO] [backup.BackupInitInstance:#150] - Dumping source and destination kubeconfig to file system path [2024-01-18T15:34:07.261Z] INFO[2024-01-18 15:34:06] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T15:34:07.261Z] INFO[2024-01-18 15:34:06] Actual:true, Expected: true [2024-01-18T15:34:07.261Z] INFO[2024-01-18 15:34:06] Verifying : Description : minimum 2 kubeconfigs are required for source and destination cluster [2024-01-18T15:34:07.261Z] INFO[2024-01-18 15:34:06] Actual:false, Expected: false [2024-01-18T15:34:07.261Z] 2024-01-18 15:34:06 +0000:[INFO] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T15:34:07.261Z] 2024-01-18 15:34:06 +0000:[INFO] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T15:34:07.261Z] 2024-01-18 15:34:06 +0000:[INFO] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T15:34:07.261Z] 2024-01-18 15:34:06 +0000:[INFO] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T15:34:07.261Z] INFO[2024-01-18 15:34:06] Verifying : Description : verfiy getting kubeconfigs [[source-config destination-config]] from configmap [kubeconfigs] [2024-01-18T15:34:07.261Z] INFO[2024-01-18 15:34:06] Actual:true, Expected: true [2024-01-18T15:34:07.261Z] 2024-01-18 15:34:06 +0000:[INFO] Log Dir: /testresults//Setup buckets.log [2024-01-18T15:34:07.261Z] INFO[2024-01-18 15:34:06] --------Test Start------ [2024-01-18T15:34:07.261Z] INFO[2024-01-18 15:34:06] #Test: Setup buckets [2024-01-18T15:34:07.261Z] INFO[2024-01-18 15:34:06] #Description: Creating one generic bucket to be used in all cases [2024-01-18T15:34:07.261Z] INFO[2024-01-18 15:34:06] ------------------------ [2024-01-18T15:34:07.262Z] 2024-01-18 15:34:07 +0000:[INFO] [backup.GetConfigObj:#4154] - Configmap with name cloud-config found in the Configmaps list [2024-01-18T15:34:07.517Z] 2024-01-18 15:34:07 +0000:[INFO] [backup.GetConfigObj:#4160] - Fetch the cloud-config from the Configmap [2024-01-18T15:34:07.517Z] INFO[2024-01-18 15:34:07] Verifying : Description : Fetching the cloud config details and persisting into globalConfig struct [2024-01-18T15:34:07.517Z] INFO[2024-01-18 15:34:07] Actual:true, Expected: true [2024-01-18T15:34:08.883Z] 2024-01-18 15:34:08 +0000:[INFO] [backup.glob..func1:#187] - Bucket created with name - global-aws-px-backup-system-test-4cac76dc [2024-01-18T15:34:08.883Z] INFO[2024-01-18 15:34:08] --------Test End------ [2024-01-18T15:34:08.883Z] INFO[2024-01-18 15:34:08] #Test: [2024-01-18T15:34:08.883Z] INFO[2024-01-18 15:34:08] #Description: [2024-01-18T15:34:08.883Z] INFO[2024-01-18 15:34:08] ------------------------ [2024-01-18T15:34:08.883Z] S2024-01-18 15:34:08 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Log Dir: /testresults//DeleteObjectsByMultipleUsersFromNewAdmin.log [2024-01-18T15:34:08.883Z] INFO[2024-01-18 15:34:08] --------Test Start------ [2024-01-18T15:34:08.883Z] INFO[2024-01-18 15:34:08] #Test: DeleteObjectsByMultipleUsersFromNewAdmin [2024-01-18T15:34:08.883Z] INFO[2024-01-18 15:34:08] #Description: Delete backups, backup schedules, restore and cluster objects created by multiple user from the new admin [2024-01-18T15:34:08.883Z] INFO[2024-01-18 15:34:08] ------------------------ [2024-01-18T15:34:08.883Z] 2024-01-18 15:34:08 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T15:34:08.883Z] 2024-01-18 15:34:08 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T15:34:08.883Z] 2024-01-18 15:34:08 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T15:34:08.883Z] 2024-01-18 15:34:08 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T15:34:08.883Z] 2024-01-18 15:34:08 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T15:34:08.883Z] 2024-01-18 15:34:08 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T15:34:08.884Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T15:34:08.884Z] 2024-01-18 15:34:08 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T15:34:08.884Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T15:34:08.884Z] 2024-01-18 15:34:08 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T15:34:08.884Z] 2024-01-18 15:34:08 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Scheduled application namespaces: [mysql-ibm-pxb-0-87567-01-18-15h34m08s] [2024-01-18T15:34:08.884Z] 2024-01-18 15:34:08 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating applications [2024-01-18T15:34:08.884Z] 2024-01-18 15:34:08 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validate applications [2024-01-18T15:34:08.884Z] 2024-01-18 15:34:08 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating mysql-ibm app [2024-01-18T15:34:08.884Z] 2024-01-18 15:34:08 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating mysql-ibm app's volumes [2024-01-18T15:34:08.884Z] 2024-01-18 15:34:08 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T15:34:08.886Z] 2024/01/18 15:34:08 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T15:34:21.055Z] 2024/01/18 15:34:18 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T15:34:29.152Z] 2024-01-18 15:34:28 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-87567-01-18-15h34m08s [2024-01-18T15:34:29.152Z] 2024-01-18 15:34:28 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T15:34:29.152Z] 2024-01-18 15:34:28 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T15:34:29.152Z] 2024-01-18 15:34:28 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] wait for mysql-ibm app to start running [2024-01-18T15:34:29.152Z] 2024-01-18 15:34:28 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T15:34:29.152Z] 2024/01/18 15:34:29 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:34:29.152Z] pod name:mysql-7d766cd65c-vx4z9 namespace:mysql-ibm-pxb-0-87567-01-18-15h34m08s running:true ready:false node:10.240.64.9 [2024-01-18T15:34:29.152Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:34:41.336Z] 2024/01/18 15:34:39 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:34:41.336Z] pod name:mysql-7d766cd65c-vx4z9 namespace:mysql-ibm-pxb-0-87567-01-18-15h34m08s running:true ready:false node:10.240.64.9 [2024-01-18T15:34:41.336Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:34:49.418Z] 2024/01/18 15:34:49 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:34:49.418Z] pod name:mysql-7d766cd65c-vx4z9 namespace:mysql-ibm-pxb-0-87567-01-18-15h34m08s running:true ready:false node:10.240.64.9 [2024-01-18T15:34:49.418Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:34:59.367Z] 2024/01/18 15:34:59 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:34:59.367Z] pod name:mysql-7d766cd65c-vx4z9 namespace:mysql-ibm-pxb-0-87567-01-18-15h34m08s running:true ready:false node:10.240.64.9 [2024-01-18T15:34:59.367Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:35:11.535Z] 2024/01/18 15:35:09 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:35:11.535Z] pod name:mysql-7d766cd65c-vx4z9 namespace:mysql-ibm-pxb-0-87567-01-18-15h34m08s running:true ready:false node:10.240.64.9 [2024-01-18T15:35:11.535Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:35:19.618Z] 2024/01/18 15:35:19 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:35:19.618Z] pod name:mysql-7d766cd65c-vx4z9 namespace:mysql-ibm-pxb-0-87567-01-18-15h34m08s running:true ready:false node:10.240.64.9 [2024-01-18T15:35:19.618Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:35:29.593Z] 2024/01/18 15:35:29 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:35:29.593Z] pod name:mysql-7d766cd65c-vx4z9 namespace:mysql-ibm-pxb-0-87567-01-18-15h34m08s running:true ready:false node:10.240.64.9 [2024-01-18T15:35:29.593Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:35:41.770Z] 2024-01-18 15:35:39 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T15:35:41.770Z] 2024-01-18 15:35:39 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T15:35:41.770Z] 2024-01-18 15:35:39 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] validate if mysql-ibm app's volumes are setup [2024-01-18T15:35:41.770Z] 2024-01-18 15:35:39 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T15:35:41.770Z] 2024-01-18 15:35:39 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T15:35:41.770Z] 2024-01-18 15:35:39 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T15:35:41.770Z] 2024-01-18 15:35:39 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T15:35:41.770Z] 2024-01-18 15:35:39 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Creating 2 users with px-backup-infra.admin role [2024-01-18T15:35:41.770Z] 2024-01-18 15:35:39 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Creating 2 users [2024-01-18T15:35:41.770Z] 2024-01-18 15:35:39 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.AddUser:#728] - Response for user [testuser1-1705592139] creation - [2024-01-18T15:35:41.770Z] 2024-01-18 15:35:39 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.AddUser:#733] - User [testuser1-1705592139] created [2024-01-18T15:35:41.770Z] INFO[2024-01-18 15:35:39] Verifying : Description : Creating user - testuser1-1705592139 [2024-01-18T15:35:41.770Z] INFO[2024-01-18 15:35:39] Actual:true, Expected: true [2024-01-18T15:35:41.770Z] 2024-01-18 15:35:40 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.AddUser:#728] - Response for user [testuser2-1705592139] creation - [2024-01-18T15:35:41.770Z] 2024-01-18 15:35:40 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.AddUser:#733] - User [testuser2-1705592139] created [2024-01-18T15:35:41.770Z] INFO[2024-01-18 15:35:40] Verifying : Description : Creating user - testuser2-1705592139 [2024-01-18T15:35:41.770Z] INFO[2024-01-18 15:35:40] Actual:true, Expected: true [2024-01-18T15:35:41.770Z] 2024-01-18 15:35:40 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705592139 - 440145f6-bec2-4e3b-9044-2ef575982f50 [2024-01-18T15:35:43.664Z] 2024-01-18 15:35:43 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2-1705592139 - 9d6edf84-500b-4282-9e37-fa729b1cb3bb [2024-01-18T15:35:45.028Z] 2024-01-18 15:35:44 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Creating cloud credential and backup location from the user testuser2-1705592139 [2024-01-18T15:35:45.028Z] 2024-01-18 15:35:44 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Creating cloud credential and backup location from the user testuser1-1705592139 [2024-01-18T15:35:45.589Z] 2024-01-18 15:35:45 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-testuser2-1705592139-ilsac] for org [default] with [aws] as provider [2024-01-18T15:35:45.589Z] 2024-01-18 15:35:45 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-18T15:35:45.589Z] 2024-01-18 15:35:45 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-testuser2-1705592139-ilsac] is present [2024-01-18T15:35:45.589Z] 2024-01-18 15:35:45 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-testuser1-1705592139-wounz] for org [default] with [aws] as provider [2024-01-18T15:35:45.589Z] 2024-01-18 15:35:45 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-18T15:35:45.844Z] 2024-01-18 15:35:45 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-testuser1-1705592139-wounz] is present [2024-01-18T15:35:48.366Z] 2024-01-18 15:35:47 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Creating source and destination cluster from the user testuser1-1705592139 [2024-01-18T15:35:48.366Z] INFO[2024-01-18 15:35:48] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T15:35:48.366Z] INFO[2024-01-18 15:35:48] Actual:true, Expected: true [2024-01-18T15:35:48.367Z] 2024-01-18 15:35:48 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T15:35:48.367Z] 2024-01-18 15:35:48 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T15:35:48.367Z] 2024-01-18 15:35:48 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T15:35:48.367Z] 2024-01-18 15:35:48 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T15:35:48.367Z] 2024-01-18 15:35:48 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Create cluster [source-cluster] in org [default] [2024-01-18T15:35:48.367Z] 2024-01-18 15:35:48 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T15:35:48.367Z] 2024-01-18 15:35:48 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T15:35:48.367Z] 2024-01-18 15:35:48 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-18T15:35:48.367Z] 2024-01-18 15:35:48 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T15:35:48.367Z] 2024-01-18 15:35:48 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T15:35:48.367Z] 2024-01-18 15:35:48 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-fkfzj] for IBM [2024-01-18T15:35:48.367Z] 2024-01-18 15:35:48 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-fkfzj] for org [default] with [ibm] as provider [2024-01-18T15:35:48.367Z] 2024-01-18 15:35:48 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T15:35:48.367Z] 2024-01-18 15:35:48 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T15:35:48.367Z] 2024-01-18 15:35:48 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-fkfzj] is present [2024-01-18T15:35:48.368Z] 2024-01-18 15:35:48 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Create cluster [source-cluster] in org [default] [2024-01-18T15:35:48.930Z] 2024-01-18 15:35:48 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-oxkoo] for IBM [2024-01-18T15:35:48.930Z] 2024-01-18 15:35:48 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-oxkoo] for org [default] with [ibm] as provider [2024-01-18T15:35:48.930Z] 2024-01-18 15:35:48 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T15:35:48.930Z] 2024-01-18 15:35:48 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T15:35:48.931Z] 2024-01-18 15:35:48 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-oxkoo] is present [2024-01-18T15:35:48.931Z] 2024-01-18 15:35:48 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-18T15:35:49.862Z] INFO[2024-01-18 15:35:49] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-18T15:35:49.862Z] INFO[2024-01-18 15:35:49] Actual:Online, Expected: Online [2024-01-18T15:35:49.862Z] INFO[2024-01-18 15:35:49] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-18T15:35:49.862Z] INFO[2024-01-18 15:35:49] Actual:true, Expected: true [2024-01-18T15:35:49.862Z] INFO[2024-01-18 15:35:49] Verifying : Description : Fetching [destination-cluster] cluster uid [2024-01-18T15:35:49.862Z] INFO[2024-01-18 15:35:49] Actual:true, Expected: true [2024-01-18T15:35:49.862Z] 2024-01-18 15:35:49 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Taking backup of applications from the user testuser1-1705592139 [2024-01-18T15:35:49.862Z] 2024-01-18 15:35:49 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T15:35:49.862Z] 2024-01-18 15:35:49 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Volume snapshot classes found in the cluster - [2024-01-18T15:35:49.862Z] 2024-01-18 15:35:49 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] ibmc-vpcblock-snapshot [2024-01-18T15:35:49.862Z] 2024-01-18 15:35:49 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T15:35:50.424Z] 2024/01/18 15:35:50 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T15:35:51.353Z] 2024-01-18 15:35:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Creating source and destination cluster from the user testuser2-1705592139 [2024-01-18T15:35:51.353Z] INFO[2024-01-18 15:35:51] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T15:35:51.353Z] INFO[2024-01-18 15:35:51] Actual:true, Expected: true [2024-01-18T15:35:51.353Z] 2024-01-18 15:35:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T15:35:51.353Z] 2024-01-18 15:35:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T15:35:51.353Z] 2024-01-18 15:35:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T15:35:51.353Z] 2024-01-18 15:35:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T15:35:51.353Z] 2024-01-18 15:35:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Create cluster [source-cluster] in org [default] [2024-01-18T15:35:51.353Z] 2024-01-18 15:35:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T15:35:51.353Z] 2024-01-18 15:35:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T15:35:51.353Z] 2024-01-18 15:35:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-18T15:35:51.353Z] 2024-01-18 15:35:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T15:35:51.353Z] 2024-01-18 15:35:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T15:35:51.353Z] 2024-01-18 15:35:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-jyrnu] for IBM [2024-01-18T15:35:51.353Z] 2024-01-18 15:35:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-jyrnu] for org [default] with [ibm] as provider [2024-01-18T15:35:51.353Z] 2024-01-18 15:35:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T15:35:51.353Z] 2024-01-18 15:35:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T15:35:51.609Z] 2024-01-18 15:35:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-jyrnu] is present [2024-01-18T15:35:51.610Z] 2024-01-18 15:35:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Create cluster [source-cluster] in org [default] [2024-01-18T15:35:51.867Z] 2024-01-18 15:35:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-zzzrt] for IBM [2024-01-18T15:35:51.867Z] 2024-01-18 15:35:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-zzzrt] for org [default] with [ibm] as provider [2024-01-18T15:35:51.867Z] 2024-01-18 15:35:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T15:35:51.867Z] 2024-01-18 15:35:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T15:35:52.123Z] 2024-01-18 15:35:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-zzzrt] is present [2024-01-18T15:35:52.124Z] 2024-01-18 15:35:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-18T15:35:52.380Z] INFO[2024-01-18 15:35:52] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-18T15:35:52.380Z] INFO[2024-01-18 15:35:52] Actual:Online, Expected: Online [2024-01-18T15:35:52.380Z] INFO[2024-01-18 15:35:52] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-18T15:35:52.380Z] INFO[2024-01-18 15:35:52] Actual:true, Expected: true [2024-01-18T15:35:52.380Z] INFO[2024-01-18 15:35:52] Verifying : Description : Fetching [destination-cluster] cluster uid [2024-01-18T15:35:52.380Z] INFO[2024-01-18 15:35:52] Actual:true, Expected: true [2024-01-18T15:35:52.380Z] 2024-01-18 15:35:52 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Taking backup of applications from the user testuser2-1705592139 [2024-01-18T15:35:52.638Z] 2024-01-18 15:35:52 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T15:35:52.638Z] 2024-01-18 15:35:52 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Volume snapshot classes found in the cluster - [2024-01-18T15:35:52.638Z] 2024-01-18 15:35:52 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] ibmc-vpcblock-snapshot [2024-01-18T15:35:52.638Z] 2024-01-18 15:35:52 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T15:35:53.201Z] 2024/01/18 15:35:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T15:36:25.230Z] 2024/01/18 15:36:20 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T15:36:25.230Z] 2024/01/18 15:36:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T15:36:52.146Z] 2024/01/18 15:36:50 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T15:36:53.081Z] 2024/01/18 15:36:53 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T15:37:25.112Z] 2024/01/18 15:37:20 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T15:37:25.112Z] 2024/01/18 15:37:23 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T15:37:51.610Z] 2024-01-18 15:37:50 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.CreateBackup:#313] - Backup [tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc] created successfully [2024-01-18T15:37:51.610Z] 2024-01-18 15:37:50 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating backup [tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc] in org [default] [2024-01-18T15:37:51.610Z] 2024-01-18 15:37:50 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc] [2024-01-18T15:37:51.610Z] 2024-01-18 15:37:50 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87567-01-18-15h34m08s] in backup [tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc] [2024-01-18T15:37:51.610Z] 2024-01-18 15:37:50 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87567-01-18-15h34m08s] in backup [tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc] [2024-01-18T15:37:51.610Z] 2024-01-18 15:37:50 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87567-01-18-15h34m08s] [2024-01-18T15:37:51.610Z] 2024-01-18 15:37:50 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T15:37:51.610Z] 2024-01-18 15:37:50 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T15:37:51.610Z] 2024-01-18 15:37:50 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87567-01-18-15h34m08s] are [[mysql-data]] [2024-01-18T15:37:51.610Z] 2024-01-18 15:37:50 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T15:37:51.610Z] 2024-01-18 15:37:50 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating CR cleanup [2024-01-18T15:37:51.610Z] 2024-01-18 15:37:50 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T15:37:51.611Z] 2024-01-18 15:37:50 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] All CRs in [mysql-ibm-pxb-0-87567-01-18-15h34m08s] are [[]] [2024-01-18T15:37:51.611Z] INFO[2024-01-18 15:37:50] Verifying : Description : Verifying creation and validation of backup [tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc] of namespace (scheduled Context) [mysql-ibm-pxb-0-87567-01-18-15h34m08s] [2024-01-18T15:37:51.611Z] INFO[2024-01-18 15:37:50] Actual:true, Expected: true [2024-01-18T15:37:51.611Z] 2024-01-18 15:37:50 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.glob..func28.2.3.3:#810] - The list of user backups taken are: map[testuser1-1705592139:map[tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc:mysql-ibm-pxb-0-87567-01-18-15h34m08s] testuser2-1705592139:map[]] [2024-01-18T15:37:51.611Z] 2024-01-18 15:37:50 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Creating schedule policy from the user testuser1-1705592139 [2024-01-18T15:37:51.611Z] INFO[2024-01-18 15:37:50] Verifying : Description : Verifying creation schedule policy periodic-testuser1-1705592139-rkpsy [2024-01-18T15:37:51.611Z] INFO[2024-01-18 15:37:50] Actual:true, Expected: true [2024-01-18T15:37:51.611Z] 2024-01-18 15:37:50 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Taking schedule backup of applications from the user testuser1-1705592139 [2024-01-18T15:37:51.611Z] 2024-01-18 15:37:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-18T15:37:51.611Z] 2024-01-18 15:37:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Volume snapshot classes found in the cluster - [2024-01-18T15:37:51.611Z] 2024-01-18 15:37:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] ibmc-vpcblock-snapshot [2024-01-18T15:37:51.611Z] 2024-01-18 15:37:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T15:37:53.512Z] 2024-01-18 15:37:53 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.CreateBackup:#313] - Backup [tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa] created successfully [2024-01-18T15:37:53.512Z] 2024-01-18 15:37:53 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating backup [tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa] in org [default] [2024-01-18T15:37:53.512Z] 2024-01-18 15:37:53 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa] [2024-01-18T15:37:53.512Z] 2024-01-18 15:37:53 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87567-01-18-15h34m08s] in backup [tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa] [2024-01-18T15:37:53.512Z] 2024-01-18 15:37:53 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87567-01-18-15h34m08s] in backup [tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa] [2024-01-18T15:37:53.512Z] 2024-01-18 15:37:53 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87567-01-18-15h34m08s] [2024-01-18T15:37:53.512Z] 2024-01-18 15:37:53 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T15:37:53.512Z] 2024-01-18 15:37:53 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T15:37:53.512Z] 2024-01-18 15:37:53 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87567-01-18-15h34m08s] are [[mysql-data]] [2024-01-18T15:37:53.512Z] 2024-01-18 15:37:53 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T15:37:53.512Z] 2024-01-18 15:37:53 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating CR cleanup [2024-01-18T15:37:53.512Z] 2024-01-18 15:37:53 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T15:37:53.512Z] 2024-01-18 15:37:53 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] All CRs in [mysql-ibm-pxb-0-87567-01-18-15h34m08s] are [[backup-schedule-testuser1-1705592139-cxghr-6807d4b-interval-2024-01-18-153751]] [2024-01-18T15:37:53.513Z] INFO[2024-01-18 15:37:53] Verifying : Description : Verifying creation and validation of backup [tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa] of namespace (scheduled Context) [mysql-ibm-pxb-0-87567-01-18-15h34m08s] [2024-01-18T15:37:53.513Z] INFO[2024-01-18 15:37:53] Actual:true, Expected: true [2024-01-18T15:37:53.513Z] 2024-01-18 15:37:53 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.glob..func28.2.3.3:#810] - The list of user backups taken are: map[testuser1-1705592139:map[tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc:mysql-ibm-pxb-0-87567-01-18-15h34m08s] testuser2-1705592139:map[tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa:mysql-ibm-pxb-0-87567-01-18-15h34m08s]] [2024-01-18T15:37:53.513Z] 2024-01-18 15:37:53 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Creating schedule policy from the user testuser2-1705592139 [2024-01-18T15:37:53.513Z] INFO[2024-01-18 15:37:53] Verifying : Description : Verifying creation schedule policy periodic-testuser2-1705592139-kuuph [2024-01-18T15:37:53.513Z] INFO[2024-01-18 15:37:53] Actual:true, Expected: true [2024-01-18T15:37:53.513Z] 2024-01-18 15:37:53 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Taking schedule backup of applications from the user testuser2-1705592139 [2024-01-18T15:37:53.769Z] 2024-01-18 15:37:53 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-18T15:37:53.769Z] 2024-01-18 15:37:53 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Volume snapshot classes found in the cluster - [2024-01-18T15:37:53.769Z] 2024-01-18 15:37:53 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] ibmc-vpcblock-snapshot [2024-01-18T15:37:53.769Z] 2024-01-18 15:37:53 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T15:39:01.429Z] 2024-01-18 15:38:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] first schedule backup for schedule name [backup-schedule-testuser1-1705592139-cxghr] is [backup-schedule-testuser1-1705592139-cxghr-6807d4b-interval-2024-01-18-153751] [2024-01-18T15:39:01.429Z] 2024/01/18 15:38:51 DoRetryWithTimeout - Error: {backup status for [backup-schedule-testuser1-1705592139-cxghr-6807d4b-interval-2024-01-18-153751] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T15:39:01.429Z] 2024-01-18 15:38:54 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] first schedule backup for schedule name [backup-schedule-testuser2-1705592139-sjwvt] is [backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754] [2024-01-18T15:39:01.430Z] 2024/01/18 15:38:54 DoRetryWithTimeout - Error: {backup status for [backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T15:39:23.318Z] 2024/01/18 15:39:21 DoRetryWithTimeout - Error: {backup status for [backup-schedule-testuser1-1705592139-cxghr-6807d4b-interval-2024-01-18-153751] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T15:39:24.681Z] 2024/01/18 15:39:24 DoRetryWithTimeout - Error: {backup status for [backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T15:39:56.766Z] 2024-01-18 15:39:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating backup [backup-schedule-testuser1-1705592139-cxghr-6807d4b-interval-2024-01-18-153751] in org [default] [2024-01-18T15:39:56.766Z] 2024-01-18 15:39:51 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [backup-schedule-testuser1-1705592139-cxghr-6807d4b-interval-2024-01-18-153751] [2024-01-18T15:39:56.766Z] 2024-01-18 15:39:52 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87567-01-18-15h34m08s] in backup [backup-schedule-testuser1-1705592139-cxghr-6807d4b-interval-2024-01-18-153751] [2024-01-18T15:39:56.766Z] 2024-01-18 15:39:52 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87567-01-18-15h34m08s] in backup [backup-schedule-testuser1-1705592139-cxghr-6807d4b-interval-2024-01-18-153751] [2024-01-18T15:39:56.767Z] 2024-01-18 15:39:52 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87567-01-18-15h34m08s] [2024-01-18T15:39:56.767Z] 2024-01-18 15:39:52 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T15:39:56.767Z] 2024-01-18 15:39:52 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T15:39:56.767Z] 2024-01-18 15:39:52 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87567-01-18-15h34m08s] are [[mysql-data]] [2024-01-18T15:39:56.767Z] 2024-01-18 15:39:52 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T15:39:56.767Z] 2024-01-18 15:39:52 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating CR cleanup [2024-01-18T15:39:56.767Z] 2024-01-18 15:39:52 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T15:39:56.767Z] 2024-01-18 15:39:52 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] All CRs in [mysql-ibm-pxb-0-87567-01-18-15h34m08s] are [[backup-schedule-testuser1-1705592139-cxghr-6807d4b-interval-2024-01-18-153751 backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]] [2024-01-18T15:39:56.767Z] 2024-01-18 15:39:52 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] CR found for [[backup-schedule-testuser1-1705592139-cxghr-6807d4b-interval-2024-01-18-153751 backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]] under [mysql-ibm-pxb-0-87567-01-18-15h34m08s] namespace [2024-01-18T15:39:56.767Z] 2024/01/18 15:39:52 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-testuser1-1705592139-cxghr-6807d4b-interval-2024-01-18-153751]}, Next try in [5s], timeout [5m0s] [2024-01-18T15:39:56.768Z] 2024-01-18 15:39:54 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating backup [backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754] in org [default] [2024-01-18T15:39:56.768Z] 2024-01-18 15:39:54 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754] [2024-01-18T15:39:56.768Z] 2024-01-18 15:39:54 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87567-01-18-15h34m08s] in backup [backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754] [2024-01-18T15:39:56.768Z] 2024-01-18 15:39:54 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87567-01-18-15h34m08s] in backup [backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754] [2024-01-18T15:39:56.768Z] 2024-01-18 15:39:54 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87567-01-18-15h34m08s] [2024-01-18T15:39:56.768Z] 2024-01-18 15:39:54 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T15:39:56.768Z] 2024-01-18 15:39:54 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T15:39:56.768Z] 2024-01-18 15:39:54 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87567-01-18-15h34m08s] are [[mysql-data]] [2024-01-18T15:39:56.768Z] 2024-01-18 15:39:54 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T15:39:56.768Z] 2024-01-18 15:39:54 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating CR cleanup [2024-01-18T15:39:56.768Z] 2024-01-18 15:39:54 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T15:39:56.768Z] 2024-01-18 15:39:54 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] All CRs in [mysql-ibm-pxb-0-87567-01-18-15h34m08s] are [[backup-schedule-testuser1-1705592139-cxghr-6807d4b-interval-2024-01-18-153751 backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]] [2024-01-18T15:39:56.768Z] 2024-01-18 15:39:54 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] CR found for [[backup-schedule-testuser1-1705592139-cxghr-6807d4b-interval-2024-01-18-153751 backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]] under [mysql-ibm-pxb-0-87567-01-18-15h34m08s] namespace [2024-01-18T15:39:56.768Z] 2024/01/18 15:39:54 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]}, Next try in [5s], timeout [5m0s] [2024-01-18T15:39:57.330Z] 2024-01-18 15:39:57 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] All CRs in [mysql-ibm-pxb-0-87567-01-18-15h34m08s] are [[backup-schedule-testuser1-1705592139-cxghr-6807d4b-interval-2024-01-18-153751 backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]] [2024-01-18T15:39:57.331Z] 2024-01-18 15:39:57 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] CR found for [[backup-schedule-testuser1-1705592139-cxghr-6807d4b-interval-2024-01-18-153751 backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]] under [mysql-ibm-pxb-0-87567-01-18-15h34m08s] namespace [2024-01-18T15:39:57.331Z] 2024/01/18 15:39:57 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-testuser1-1705592139-cxghr-6807d4b-interval-2024-01-18-153751]}, Next try in [5s], timeout [5m0s] [2024-01-18T15:39:59.847Z] 2024-01-18 15:39:59 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] All CRs in [mysql-ibm-pxb-0-87567-01-18-15h34m08s] are [[backup-schedule-testuser1-1705592139-cxghr-6807d4b-interval-2024-01-18-153751 backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]] [2024-01-18T15:39:59.847Z] 2024-01-18 15:39:59 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] CR found for [[backup-schedule-testuser1-1705592139-cxghr-6807d4b-interval-2024-01-18-153751 backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]] under [mysql-ibm-pxb-0-87567-01-18-15h34m08s] namespace [2024-01-18T15:39:59.847Z] 2024/01/18 15:39:59 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]}, Next try in [5s], timeout [5m0s] [2024-01-18T15:40:02.363Z] 2024-01-18 15:40:02 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] All CRs in [mysql-ibm-pxb-0-87567-01-18-15h34m08s] are [[backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]] [2024-01-18T15:40:02.363Z] INFO[2024-01-18 15:40:02] Verifying : Description : Verifying creation and validation of schedule backup with schedule name [periodic-testuser1-1705592139-rkpsy] [2024-01-18T15:40:02.363Z] INFO[2024-01-18 15:40:02] Actual:true, Expected: true [2024-01-18T15:40:02.619Z] INFO[2024-01-18 15:40:02] Verifying : Description : Suspending Backup Schedule [periodic-testuser1-1705592139-rkpsy] for user [testuser1-1705592139] [2024-01-18T15:40:02.619Z] INFO[2024-01-18 15:40:02] Actual:true, Expected: true [2024-01-18T15:40:05.134Z] 2024-01-18 15:40:04 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] All CRs in [mysql-ibm-pxb-0-87567-01-18-15h34m08s] are [[backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]] [2024-01-18T15:40:05.134Z] 2024-01-18 15:40:04 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] CR found for [[backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]] under [mysql-ibm-pxb-0-87567-01-18-15h34m08s] namespace [2024-01-18T15:40:05.134Z] 2024/01/18 15:40:04 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]}, Next try in [5s], timeout [5m0s] [2024-01-18T15:40:10.396Z] 2024-01-18 15:40:09 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] All CRs in [mysql-ibm-pxb-0-87567-01-18-15h34m08s] are [[backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]] [2024-01-18T15:40:10.396Z] 2024-01-18 15:40:09 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] CR found for [[backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]] under [mysql-ibm-pxb-0-87567-01-18-15h34m08s] namespace [2024-01-18T15:40:10.396Z] 2024/01/18 15:40:09 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]}, Next try in [5s], timeout [5m0s] [2024-01-18T15:40:15.670Z] 2024-01-18 15:40:14 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] All CRs in [mysql-ibm-pxb-0-87567-01-18-15h34m08s] are [[backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]] [2024-01-18T15:40:15.670Z] 2024-01-18 15:40:14 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] CR found for [[backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]] under [mysql-ibm-pxb-0-87567-01-18-15h34m08s] namespace [2024-01-18T15:40:15.670Z] 2024/01/18 15:40:14 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]}, Next try in [5s], timeout [5m0s] [2024-01-18T15:40:19.866Z] 2024-01-18 15:40:19 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] All CRs in [mysql-ibm-pxb-0-87567-01-18-15h34m08s] are [[backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]] [2024-01-18T15:40:19.866Z] 2024-01-18 15:40:19 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] CR found for [[backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]] under [mysql-ibm-pxb-0-87567-01-18-15h34m08s] namespace [2024-01-18T15:40:19.866Z] 2024/01/18 15:40:19 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]}, Next try in [5s], timeout [5m0s] [2024-01-18T15:40:25.115Z] 2024-01-18 15:40:24 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] All CRs in [mysql-ibm-pxb-0-87567-01-18-15h34m08s] are [[backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]] [2024-01-18T15:40:25.115Z] 2024-01-18 15:40:24 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] CR found for [[backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]] under [mysql-ibm-pxb-0-87567-01-18-15h34m08s] namespace [2024-01-18T15:40:25.116Z] 2024/01/18 15:40:24 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]}, Next try in [5s], timeout [5m0s] [2024-01-18T15:40:30.363Z] 2024-01-18 15:40:29 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] All CRs in [mysql-ibm-pxb-0-87567-01-18-15h34m08s] are [[backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]] [2024-01-18T15:40:30.363Z] 2024-01-18 15:40:29 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] CR found for [[backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]] under [mysql-ibm-pxb-0-87567-01-18-15h34m08s] namespace [2024-01-18T15:40:30.363Z] 2024/01/18 15:40:29 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]}, Next try in [5s], timeout [5m0s] [2024-01-18T15:40:35.664Z] 2024-01-18 15:40:34 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] All CRs in [mysql-ibm-pxb-0-87567-01-18-15h34m08s] are [[backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]] [2024-01-18T15:40:35.664Z] 2024-01-18 15:40:34 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] CR found for [[backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]] under [mysql-ibm-pxb-0-87567-01-18-15h34m08s] namespace [2024-01-18T15:40:35.664Z] 2024/01/18 15:40:34 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-testuser2-1705592139-sjwvt-aef0bcc-interval-2024-01-18-153754]}, Next try in [5s], timeout [5m0s] [2024-01-18T15:40:40.912Z] 2024-01-18 15:40:39 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] All CRs in [mysql-ibm-pxb-0-87567-01-18-15h34m08s] are [[]] [2024-01-18T15:40:40.912Z] INFO[2024-01-18 15:40:39] Verifying : Description : Verifying creation and validation of schedule backup with schedule name [periodic-testuser2-1705592139-kuuph] [2024-01-18T15:40:40.912Z] INFO[2024-01-18 15:40:39] Actual:true, Expected: true [2024-01-18T15:40:40.912Z] INFO[2024-01-18 15:40:40] Verifying : Description : Suspending Backup Schedule [periodic-testuser2-1705592139-kuuph] for user [testuser2-1705592139] [2024-01-18T15:40:40.912Z] INFO[2024-01-18 15:40:40] Actual:true, Expected: true [2024-01-18T15:40:40.912Z] 2024-01-18 15:40:40 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Taking restore of backups from the user testuser1-1705592139 [2024-01-18T15:40:40.912Z] 2024/01/18 15:40:40 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T15:41:12.946Z] 2024-01-18 15:41:10 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.CreateRestore:#936] - Restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc] created successfully [2024-01-18T15:41:12.946Z] 2024-01-18 15:41:10 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Switching cluster context to cluster [destination-cluster] [2024-01-18T15:41:12.946Z] 2024-01-18 15:41:10 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Switching context to [/tmp/destination-config] [2024-01-18T15:41:12.946Z] 2024-01-18 15:41:10 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:41:12.946Z] 2024-01-18 15:41:10 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:41:12.946Z] 2024-01-18 15:41:11 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:41:12.946Z] 2024-01-18 15:41:11 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:41:12.946Z] 2024-01-18 15:41:11 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:41:12.946Z] 2024-01-18 15:41:11 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:41:12.946Z] 2024-01-18 15:41:11 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:41:12.946Z] 2024-01-18 15:41:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:41:12.946Z] 2024-01-18 15:41:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:41:12.946Z] 2024-01-18 15:41:12 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T15:41:12.946Z] 2024-01-18 15:41:12 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T15:41:12.946Z] 2024-01-18 15:41:12 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T15:41:12.946Z] 2024-01-18 15:41:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T15:41:12.946Z] 2024-01-18 15:41:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Switched context to [/tmp/destination-config] [2024-01-18T15:41:12.946Z] 2024-01-18 15:41:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-87567-01-18-15h34m08s] with namespace mapping [map[mysql-ibm-pxb-0-87567-01-18-15h34m08s:custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz]] and storage Class Mapping [map[]] [2024-01-18T15:41:12.947Z] 2024-01-18 15:41:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-87567-01-18-15h34m08s] [2024-01-18T15:41:12.947Z] 2024-01-18 15:41:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T15:41:12.947Z] 2024-01-18 15:41:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T15:41:12.947Z] 2024-01-18 15:41:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc] in org [default] [2024-01-18T15:41:12.947Z] 2024-01-18 15:41:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc] [2024-01-18T15:41:12.947Z] 2024-01-18 15:41:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc] [2024-01-18T15:41:12.947Z] 2024-01-18 15:41:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating specs for the namespace (restoredAppContext) [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz] in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc] [2024-01-18T15:41:12.947Z] 2024-01-18 15:41:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz]) was found in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc], as expected by presence in expectedRestoredAppContext [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz] [2024-01-18T15:41:13.204Z] 2024-01-18 15:41:13 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz]) found in the restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc] was also present on the cluster/namespace [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz] [2024-01-18T15:41:13.204Z] 2024-01-18 15:41:13 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz]) was found in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc], as expected by presence in expectedRestoredAppContext [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz] [2024-01-18T15:41:13.461Z] 2024-01-18 15:41:13 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz]) found in the restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc] was also present on the cluster/namespace [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz] [2024-01-18T15:41:13.461Z] 2024-01-18 15:41:13 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz]) was found in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc], as expected by presence in expectedRestoredAppContext [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz] [2024-01-18T15:41:13.461Z] 2024-01-18 15:41:13 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz]) found in the restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc] was also present on the cluster/namespace [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz] [2024-01-18T15:41:13.461Z] 2024-01-18 15:41:13 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz]) was found in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc], as expected by presence in expectedRestoredAppContext [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz] [2024-01-18T15:41:13.461Z] 2024-01-18 15:41:13 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz]) found in the restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc] was also present on the cluster/namespace [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz] [2024-01-18T15:41:13.461Z] 2024-01-18 15:41:13 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating Restored Volumes for the namespace (restoredAppContext) [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz] in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc] [2024-01-18T15:41:13.461Z] 2024-01-18 15:41:13 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz] in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc] [2024-01-18T15:41:13.461Z] 2024-01-18 15:41:13 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T15:41:13.461Z] 2024-01-18 15:41:13 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T15:41:13.461Z] 2024-01-18 15:41:13 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz] are [[mysql-data]] [2024-01-18T15:41:13.461Z] 2024-01-18 15:41:13 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateRestore:#2240] - in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc], the status of the restored volume [pvc-8a396cd4-c3c8-4ce2-acab-c893aad64b36] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-18T15:41:13.461Z] 2024-01-18 15:41:13 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validate applications in restored namespace [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz] due to restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc] [2024-01-18T15:41:13.461Z] 2024-01-18 15:41:13 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating mysql-ibm app [2024-01-18T15:41:13.462Z] 2024-01-18 15:41:13 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] wait for mysql-ibm app to start running [2024-01-18T15:41:13.717Z] 2024-01-18 15:41:13 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T15:41:13.717Z] 2024/01/18 15:41:13 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:41:13.717Z] pod name:mysql-7d766cd65c-5ksdw namespace:custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz running:false ready:false node:10.240.64.7 [2024-01-18T15:41:13.717Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:41:25.929Z] 2024/01/18 15:41:23 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:41:25.929Z] pod name:mysql-7d766cd65c-5ksdw namespace:custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz running:false ready:false node:10.240.64.7 [2024-01-18T15:41:25.929Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:41:34.018Z] 2024/01/18 15:41:33 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:41:34.018Z] pod name:mysql-7d766cd65c-5ksdw namespace:custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz running:false ready:false node:10.240.64.7 [2024-01-18T15:41:34.018Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:41:43.963Z] 2024/01/18 15:41:43 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:41:43.963Z] pod name:mysql-7d766cd65c-5ksdw namespace:custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz running:false ready:false node:10.240.64.7 [2024-01-18T15:41:43.963Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:41:56.132Z] 2024/01/18 15:41:53 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:41:56.133Z] pod name:mysql-7d766cd65c-5ksdw namespace:custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz running:true ready:false node:10.240.64.7 [2024-01-18T15:41:56.133Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:42:04.217Z] 2024/01/18 15:42:04 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:42:04.217Z] pod name:mysql-7d766cd65c-5ksdw namespace:custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz running:true ready:false node:10.240.64.7 [2024-01-18T15:42:04.217Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:42:14.158Z] 2024/01/18 15:42:14 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:42:14.158Z] pod name:mysql-7d766cd65c-5ksdw namespace:custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz running:true ready:false node:10.240.64.7 [2024-01-18T15:42:14.158Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:42:26.330Z] 2024/01/18 15:42:24 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:42:26.330Z] pod name:mysql-7d766cd65c-5ksdw namespace:custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz running:true ready:false node:10.240.64.7 [2024-01-18T15:42:26.330Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:42:34.414Z] 2024/01/18 15:42:34 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:42:34.414Z] pod name:mysql-7d766cd65c-5ksdw namespace:custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz running:true ready:false node:10.240.64.7 [2024-01-18T15:42:34.414Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:42:44.355Z] 2024/01/18 15:42:44 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:42:44.355Z] pod name:mysql-7d766cd65c-5ksdw namespace:custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz running:true ready:false node:10.240.64.7 [2024-01-18T15:42:44.355Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:42:56.521Z] 2024/01/18 15:42:54 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:42:56.521Z] pod name:mysql-7d766cd65c-5ksdw namespace:custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz running:true ready:false node:10.240.64.7 [2024-01-18T15:42:56.521Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:43:04.611Z] 2024-01-18 15:43:04 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T15:43:04.611Z] 2024-01-18 15:43:04 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T15:43:04.867Z] 2024-01-18 15:43:04 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating CR cleanup [2024-01-18T15:43:04.867Z] 2024-01-18 15:43:04 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T15:43:04.867Z] 2024-01-18 15:43:04 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] All CRs in [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz] are [[]] [2024-01-18T15:43:04.867Z] 2024-01-18 15:43:04 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Switching cluster context back to cluster path [] [2024-01-18T15:43:04.867Z] 2024-01-18 15:43:04 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Switching context to [default] [2024-01-18T15:43:04.867Z] 2024-01-18 15:43:04 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:43:04.867Z] 2024-01-18 15:43:04 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:43:05.123Z] 2024-01-18 15:43:04 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:43:05.123Z] 2024-01-18 15:43:04 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:43:05.123Z] 2024-01-18 15:43:04 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:43:05.379Z] 2024-01-18 15:43:05 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:43:05.636Z] 2024-01-18 15:43:05 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:43:06.201Z] 2024-01-18 15:43:05 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:43:06.458Z] 2024-01-18 15:43:06 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:43:06.713Z] 2024-01-18 15:43:06 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T15:43:06.970Z] 2024-01-18 15:43:06 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T15:43:06.970Z] 2024-01-18 15:43:06 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T15:43:06.970Z] 2024-01-18 15:43:06 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T15:43:06.970Z] 2024-01-18 15:43:06 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Switched context to [default] [2024-01-18T15:43:06.970Z] INFO[2024-01-18 15:43:06] Verifying : Description : Verifying creation of restore tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc of backup tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc [2024-01-18T15:43:06.970Z] INFO[2024-01-18 15:43:06] Actual:true, Expected: true [2024-01-18T15:43:06.970Z] 2024-01-18 15:43:06 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.glob..func28.2.4:#878] - The list of user restores taken are: map[testuser1-1705592139:map[7f7259e8-da7b-4e37-9c74-1989b849848b:tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc]] [2024-01-18T15:43:08.337Z] 2024-01-18 15:43:08 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Verifying backups of the user testuser1-1705592139 from the admin [2024-01-18T15:43:08.899Z] 2024-01-18 15:43:08 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Verifying backup schedules of the user testuser1-1705592139 from the admin [2024-01-18T15:43:09.157Z] 2024-01-18 15:43:09 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Verifying restores of the user testuser1-1705592139 from the admin [2024-01-18T15:43:09.744Z] 2024-01-18 15:43:09 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Taking restore of backups from the user testuser2-1705592139 [2024-01-18T15:43:10.307Z] 2024/01/18 15:43:10 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T15:43:42.331Z] 2024/01/18 15:43:40 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T15:44:04.210Z] INFO[2024-01-18 15:44:00] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T15:44:04.210Z] INFO[2024-01-18 15:44:00] sched stats: tick interval: 600 < 1.2s (100%) [2024-01-18T15:44:04.210Z] INFO[2024-01-18 15:44:00] sched stats: tick process durations: 599 < 500ms (100%) [2024-01-18T15:44:10.746Z] 2024-01-18 15:44:10 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.CreateRestore:#936] - Restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa] created successfully [2024-01-18T15:44:10.746Z] 2024-01-18 15:44:10 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Switching cluster context to cluster [destination-cluster] [2024-01-18T15:44:10.746Z] 2024-01-18 15:44:10 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Switching context to [/tmp/destination-config] [2024-01-18T15:44:10.746Z] 2024-01-18 15:44:10 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:44:10.746Z] 2024-01-18 15:44:10 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:44:10.746Z] 2024-01-18 15:44:10 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:44:10.746Z] 2024-01-18 15:44:10 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:44:10.746Z] 2024-01-18 15:44:10 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:44:11.003Z] 2024-01-18 15:44:10 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:44:11.259Z] 2024-01-18 15:44:11 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:44:11.820Z] 2024-01-18 15:44:11 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:44:12.079Z] 2024-01-18 15:44:11 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:44:12.334Z] 2024-01-18 15:44:12 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T15:44:12.590Z] 2024-01-18 15:44:12 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T15:44:12.590Z] 2024-01-18 15:44:12 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T15:44:12.590Z] 2024-01-18 15:44:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T15:44:12.590Z] 2024-01-18 15:44:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Switched context to [/tmp/destination-config] [2024-01-18T15:44:12.590Z] 2024-01-18 15:44:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-87567-01-18-15h34m08s] with namespace mapping [map[mysql-ibm-pxb-0-87567-01-18-15h34m08s:custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh]] and storage Class Mapping [map[]] [2024-01-18T15:44:12.590Z] 2024-01-18 15:44:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-87567-01-18-15h34m08s] [2024-01-18T15:44:12.590Z] 2024-01-18 15:44:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T15:44:12.590Z] 2024-01-18 15:44:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T15:44:12.590Z] 2024-01-18 15:44:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa] in org [default] [2024-01-18T15:44:12.590Z] 2024-01-18 15:44:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa] [2024-01-18T15:44:12.590Z] 2024-01-18 15:44:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa] [2024-01-18T15:44:12.590Z] 2024-01-18 15:44:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating specs for the namespace (restoredAppContext) [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh] in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa] [2024-01-18T15:44:12.591Z] 2024-01-18 15:44:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh]) was found in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa], as expected by presence in expectedRestoredAppContext [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh] [2024-01-18T15:44:12.591Z] 2024-01-18 15:44:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh]) found in the restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa] was also present on the cluster/namespace [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh] [2024-01-18T15:44:12.591Z] 2024-01-18 15:44:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh]) was found in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa], as expected by presence in expectedRestoredAppContext [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh] [2024-01-18T15:44:12.847Z] 2024-01-18 15:44:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh]) found in the restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa] was also present on the cluster/namespace [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh] [2024-01-18T15:44:12.847Z] 2024-01-18 15:44:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh]) was found in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa], as expected by presence in expectedRestoredAppContext [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh] [2024-01-18T15:44:12.847Z] 2024-01-18 15:44:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh]) found in the restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa] was also present on the cluster/namespace [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh] [2024-01-18T15:44:12.847Z] 2024-01-18 15:44:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh]) was found in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa], as expected by presence in expectedRestoredAppContext [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh] [2024-01-18T15:44:12.847Z] 2024-01-18 15:44:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh]) found in the restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa] was also present on the cluster/namespace [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh] [2024-01-18T15:44:12.847Z] 2024-01-18 15:44:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating Restored Volumes for the namespace (restoredAppContext) [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh] in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa] [2024-01-18T15:44:12.847Z] 2024-01-18 15:44:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh] in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa] [2024-01-18T15:44:13.103Z] 2024-01-18 15:44:12 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T15:44:13.103Z] 2024-01-18 15:44:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T15:44:13.103Z] 2024-01-18 15:44:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh] are [[mysql-data]] [2024-01-18T15:44:13.104Z] 2024-01-18 15:44:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.ValidateRestore:#2240] - in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa], the status of the restored volume [pvc-9bc9110c-2d4e-4795-801a-6504cf1e62a9] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-18T15:44:13.104Z] 2024-01-18 15:44:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validate applications in restored namespace [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh] due to restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa] [2024-01-18T15:44:13.104Z] 2024-01-18 15:44:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating mysql-ibm app [2024-01-18T15:44:13.104Z] 2024-01-18 15:44:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] wait for mysql-ibm app to start running [2024-01-18T15:44:13.359Z] 2024-01-18 15:44:13 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T15:44:13.359Z] 2024/01/18 15:44:13 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:44:13.359Z] pod name:mysql-7d766cd65c-bzrxv namespace:custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh running:true ready:false node:10.240.64.7 [2024-01-18T15:44:13.359Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:44:23.317Z] 2024/01/18 15:44:23 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:44:23.317Z] pod name:mysql-7d766cd65c-bzrxv namespace:custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh running:true ready:false node:10.240.64.7 [2024-01-18T15:44:23.317Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:44:35.491Z] 2024/01/18 15:44:33 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:44:35.491Z] pod name:mysql-7d766cd65c-bzrxv namespace:custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh running:true ready:false node:10.240.64.7 [2024-01-18T15:44:35.491Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:44:43.578Z] 2024/01/18 15:44:43 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:44:43.578Z] pod name:mysql-7d766cd65c-bzrxv namespace:custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh running:true ready:false node:10.240.64.7 [2024-01-18T15:44:43.578Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:44:53.524Z] 2024/01/18 15:44:53 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:44:53.524Z] pod name:mysql-7d766cd65c-bzrxv namespace:custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh running:true ready:false node:10.240.64.7 [2024-01-18T15:44:53.524Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:45:05.687Z] 2024-01-18 15:45:03 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T15:45:05.687Z] 2024-01-18 15:45:03 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T15:45:05.687Z] 2024-01-18 15:45:03 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Validating CR cleanup [2024-01-18T15:45:05.687Z] 2024-01-18 15:45:03 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T15:45:05.687Z] 2024-01-18 15:45:03 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] All CRs in [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh] are [[]] [2024-01-18T15:45:05.687Z] 2024-01-18 15:45:03 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Switching cluster context back to cluster path [] [2024-01-18T15:45:05.687Z] 2024-01-18 15:45:03 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Switching context to [default] [2024-01-18T15:45:05.687Z] 2024-01-18 15:45:03 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:45:05.687Z] 2024-01-18 15:45:03 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:45:05.687Z] 2024-01-18 15:45:03 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:45:05.688Z] 2024-01-18 15:45:03 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:45:05.688Z] 2024-01-18 15:45:04 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:45:05.688Z] 2024-01-18 15:45:04 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:45:05.688Z] 2024-01-18 15:45:04 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:45:05.688Z] 2024-01-18 15:45:05 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:45:05.688Z] 2024-01-18 15:45:05 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:45:05.688Z] 2024-01-18 15:45:05 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T15:45:05.944Z] 2024-01-18 15:45:05 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T15:45:05.944Z] 2024-01-18 15:45:05 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T15:45:05.944Z] 2024-01-18 15:45:05 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T15:45:05.944Z] 2024-01-18 15:45:05 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Switched context to [default] [2024-01-18T15:45:05.944Z] INFO[2024-01-18 15:45:05] Verifying : Description : Verifying creation of restore tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa of backup tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa [2024-01-18T15:45:05.944Z] INFO[2024-01-18 15:45:05] Actual:true, Expected: true [2024-01-18T15:45:05.944Z] 2024-01-18 15:45:05 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.glob..func28.2.4:#878] - The list of user restores taken are: map[testuser1-1705592139:map[7f7259e8-da7b-4e37-9c74-1989b849848b:tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc] testuser2-1705592139:map[221e1871-3401-4fae-9c3c-49d16b724b4f:tp-restore-tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa]] [2024-01-18T15:45:07.310Z] 2024-01-18 15:45:07 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Verifying backups of the user testuser2-1705592139 from the admin [2024-01-18T15:45:07.870Z] 2024-01-18 15:45:07 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Verifying backup schedules of the user testuser2-1705592139 from the admin [2024-01-18T15:45:08.436Z] 2024-01-18 15:45:08 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Verifying restores of the user testuser2-1705592139 from the admin [2024-01-18T15:45:08.694Z] 2024-01-18 15:45:08 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Adding new user to px-admin-group group [2024-01-18T15:45:08.694Z] 2024-01-18 15:45:08 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Creating 1 users [2024-01-18T15:45:09.625Z] 2024-01-18 15:45:09 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.AddUser:#728] - Response for user [testuser1-1705592708] creation - [2024-01-18T15:45:09.625Z] 2024-01-18 15:45:09 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.AddUser:#733] - User [testuser1-1705592708] created [2024-01-18T15:45:09.625Z] INFO[2024-01-18 15:45:09] Verifying : Description : Creating user - testuser1-1705592708 [2024-01-18T15:45:09.625Z] INFO[2024-01-18 15:45:09] Actual:true, Expected: true [2024-01-18T15:45:10.209Z] 2024-01-18 15:45:10 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group px-admin-group - 766fec58-5d4f-4d94-b505-6c3b3c0ee59b [2024-01-18T15:45:11.136Z] 2024-01-18 15:45:10 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705592708 - 157b6684-6119-46de-ac11-165e33ccc7ac [2024-01-18T15:45:12.064Z] 2024-01-18 15:45:11 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T15:45:12.064Z] 2024-01-18 15:45:11 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.AddGroupToUser:#1119] - Added User [testuser1-1705592708] to Group [px-admin-group] [2024-01-18T15:45:12.064Z] INFO[2024-01-18 15:45:11] Verifying : Description : Verifying addition of user testuser1-1705592708 to the group px-admin-group [2024-01-18T15:45:12.064Z] INFO[2024-01-18 15:45:11] Actual:true, Expected: true [2024-01-18T15:45:12.319Z] 2024-01-18 15:45:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Verifying backups of the user testuser2-1705592139 from new admin testuser1-1705592708 [2024-01-18T15:45:12.319Z] 2024-01-18 15:45:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Verifying backups of the user testuser1-1705592139 from new admin testuser1-1705592708 [2024-01-18T15:45:12.882Z] 2024-01-18 15:45:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Verifying backup schedules of the user testuser2-1705592139 from the new admin testuser1-1705592708 [2024-01-18T15:45:13.139Z] 2024-01-18 15:45:12 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Verifying backup schedules of the user testuser1-1705592139 from the new admin testuser1-1705592708 [2024-01-18T15:45:13.709Z] 2024-01-18 15:45:13 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Verifying restores of the user testuser2-1705592139 from the new admin testuser1-1705592708 [2024-01-18T15:45:13.972Z] 2024-01-18 15:45:13 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Verifying restores of the user testuser1-1705592139 from the new admin testuser1-1705592708 [2024-01-18T15:45:14.533Z] 2024-01-18 15:45:14 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Deleting user testuser2-1705592139 schedule backups, backup schedule and schedule policy from the admin [2024-01-18T15:45:14.533Z] 2024-01-18 15:45:14 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Deleting user testuser1-1705592139 schedule backups, backup schedule and schedule policy from the admin [2024-01-18T15:45:15.461Z] 2024/01/18 15:45:15 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-testuser2-1705592139-sjwvt] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T15:45:15.718Z] 2024/01/18 15:45:15 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-testuser1-1705592139-cxghr] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T15:45:25.654Z] 2024/01/18 15:45:25 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-testuser2-1705592139-sjwvt] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T15:45:25.654Z] 2024/01/18 15:45:25 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-testuser1-1705592139-cxghr] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T15:45:35.590Z] 2024-01-18 15:45:35 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Deleting user testuser2-1705592139 backups from the admin [2024-01-18T15:45:35.590Z] 2024/01/18 15:45:35 DoRetryWithTimeout - Error: {backupObject [tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-18T15:45:35.590Z] 2024-01-18 15:45:35 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Deleting user testuser1-1705592139 backups from the admin [2024-01-18T15:45:35.846Z] 2024/01/18 15:45:35 DoRetryWithTimeout - Error: {backupObject [tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-18T15:46:08.441Z] 2024/01/18 15:46:05 DoRetryWithTimeout - Error: {backupObject [tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser2-1705592139-lowwa] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-18T15:46:08.441Z] 2024/01/18 15:46:05 DoRetryWithTimeout - Error: {backupObject [tp-backup-mysql-ibm-pxb-0-87567-01-18-15h34m08s-0-testuser1-1705592139-deqgc] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-18T15:46:40.455Z] 2024-01-18 15:46:35 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Deleting user testuser2-1705592139 restores from the admin [2024-01-18T15:46:40.455Z] 2024-01-18 15:46:35 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Deleting user testuser2-1705592139 source and destination cluster from the admin [2024-01-18T15:46:40.455Z] 2024-01-18 15:46:35 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Deleting user testuser1-1705592139 restores from the admin [2024-01-18T15:46:40.455Z] 2024-01-18 15:46:35 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Deleting user testuser1-1705592139 source and destination cluster from the admin [2024-01-18T15:46:40.455Z] INFO[2024-01-18 15:46:35] Verifying : Description : Verifying deletion of cluster [source-cluster] of the user testuser2-1705592139 [2024-01-18T15:46:40.455Z] INFO[2024-01-18 15:46:35] Actual:true, Expected: true [2024-01-18T15:46:40.456Z] INFO[2024-01-18 15:46:35] Verifying : Description : Verifying deletion of cluster [source-cluster] of the user testuser1-1705592139 [2024-01-18T15:46:40.456Z] INFO[2024-01-18 15:46:35] Actual:true, Expected: true [2024-01-18T15:46:40.456Z] INFO[2024-01-18 15:46:35] Verifying : Description : Verifying deletion of cluster [destination-cluster] of the user testuser2-1705592139 [2024-01-18T15:46:40.456Z] INFO[2024-01-18 15:46:35] Actual:true, Expected: true [2024-01-18T15:46:40.456Z] INFO[2024-01-18 15:46:35] Verifying : Description : Verifying deletion of cluster [destination-cluster] of the user testuser1-1705592139 [2024-01-18T15:46:40.456Z] INFO[2024-01-18 15:46:35] Actual:true, Expected: true [2024-01-18T15:46:40.456Z] 2024-01-18 15:46:35 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Destroying the scheduled applications [2024-01-18T15:46:40.456Z] 2024-01-18 15:46:35 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Destroying apps [2024-01-18T15:46:40.456Z] 2024-01-18 15:46:35 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T15:46:40.456Z] 2024-01-18 15:46:35 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T15:46:40.456Z] 2024-01-18 15:46:35 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] start destroying mysql-ibm app [2024-01-18T15:46:40.456Z] 2024-01-18 15:46:36 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T15:46:40.456Z] 2024-01-18 15:46:36 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T15:46:40.456Z] 2024-01-18 15:46:36 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T15:46:40.456Z] INFO[2024-01-18 15:46:36] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T15:46:40.456Z] INFO[2024-01-18 15:46:36] Actual:true, Expected: true [2024-01-18T15:46:40.456Z] 2024-01-18 15:46:36 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Cleaning backup locations in map [map[747785ab-24a7-4e9b-9b45-c2ff1bb25262:autogenerated-backup-location-testuser1-1705592139-wuhtk]], cloud credential [autogenerated-cred-testuser1-1705592139-wounz], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T15:46:40.457Z] INFO[2024-01-18 15:46:36] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-testuser1-1705592139-wuhtk] [2024-01-18T15:46:40.457Z] INFO[2024-01-18 15:46:36] Actual:true, Expected: true [2024-01-18T15:46:40.457Z] 2024-01-18 15:46:36 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-testuser1-1705592139-wuhtk] is present [2024-01-18T15:46:40.457Z] 2024/01/18 15:46:36 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-testuser1-1705592139-wuhtk is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T15:47:06.954Z] 2024-01-18 15:47:06 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-testuser1-1705592139-wuhtk] is present [2024-01-18T15:47:06.954Z] 2024/01/18 15:47:06 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-testuser1-1705592139-wuhtk is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T15:47:39.016Z] 2024-01-18 15:47:36 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [] [2024-01-18T15:47:39.016Z] INFO[2024-01-18 15:47:36] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-testuser1-1705592139-wuhtk [2024-01-18T15:47:39.016Z] INFO[2024-01-18 15:47:36] Actual:true, Expected: true [2024-01-18T15:47:39.016Z] 2024-01-18 15:47:36 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-testuser1-1705592139-wounz] is present [2024-01-18T15:47:39.016Z] INFO[2024-01-18 15:47:36] Verifying : Description : Verifying if cloud cred [autogenerated-cred-testuser1-1705592139-wounz] is present [2024-01-18T15:47:39.016Z] INFO[2024-01-18 15:47:36] Actual:true, Expected: true [2024-01-18T15:47:39.016Z] INFO[2024-01-18 15:47:36] Verifying : Description : Verifying deletion of cloud cred [autogenerated-cred-testuser1-1705592139-wounz] [2024-01-18T15:47:39.016Z] INFO[2024-01-18 15:47:36] Actual:true, Expected: true [2024-01-18T15:47:39.016Z] INFO[2024-01-18 15:47:36] Verifying : Description : Deleting cloud cred autogenerated-cred-testuser1-1705592139-wounz [2024-01-18T15:47:39.016Z] INFO[2024-01-18 15:47:36] Actual:true, Expected: true [2024-01-18T15:47:39.016Z] INFO[2024-01-18 15:47:36] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T15:47:39.016Z] INFO[2024-01-18 15:47:36] Actual:true, Expected: true [2024-01-18T15:47:39.016Z] 2024-01-18 15:47:36 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705592139 - 440145f6-bec2-4e3b-9044-2ef575982f50 [2024-01-18T15:47:39.016Z] 2024-01-18 15:47:37 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.DeleteUser:#755] - Response for user [testuser1-1705592139] deletion - [2024-01-18T15:47:39.016Z] 2024-01-18 15:47:37 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.DeleteUser:#759] - Deleted User - testuser1-1705592139 [2024-01-18T15:47:39.017Z] 2024-01-18 15:47:37 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Cleaning backup locations in map [map[b72cdc83-3717-41e1-8fe7-d7c57e438bee:autogenerated-backup-location-testuser2-1705592139-sezll]], cloud credential [autogenerated-cred-testuser2-1705592139-ilsac], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T15:47:39.017Z] INFO[2024-01-18 15:47:37] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-testuser2-1705592139-sezll] [2024-01-18T15:47:39.017Z] INFO[2024-01-18 15:47:37] Actual:true, Expected: true [2024-01-18T15:47:39.017Z] 2024-01-18 15:47:37 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-testuser2-1705592139-sezll] is present [2024-01-18T15:47:39.017Z] 2024/01/18 15:47:37 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-testuser2-1705592139-sezll is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T15:48:11.051Z] 2024-01-18 15:48:07 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-testuser2-1705592139-sezll] is present [2024-01-18T15:48:11.051Z] 2024/01/18 15:48:07 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-testuser2-1705592139-sezll is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T15:48:37.548Z] 2024-01-18 15:48:37 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [] [2024-01-18T15:48:37.548Z] INFO[2024-01-18 15:48:37] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-testuser2-1705592139-sezll [2024-01-18T15:48:37.548Z] INFO[2024-01-18 15:48:37] Actual:true, Expected: true [2024-01-18T15:48:37.548Z] 2024-01-18 15:48:37 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-testuser2-1705592139-ilsac] is present [2024-01-18T15:48:37.548Z] INFO[2024-01-18 15:48:37] Verifying : Description : Verifying if cloud cred [autogenerated-cred-testuser2-1705592139-ilsac] is present [2024-01-18T15:48:37.548Z] INFO[2024-01-18 15:48:37] Actual:true, Expected: true [2024-01-18T15:48:37.805Z] INFO[2024-01-18 15:48:37] Verifying : Description : Verifying deletion of cloud cred [autogenerated-cred-testuser2-1705592139-ilsac] [2024-01-18T15:48:37.805Z] INFO[2024-01-18 15:48:37] Actual:true, Expected: true [2024-01-18T15:48:37.805Z] INFO[2024-01-18 15:48:37] Verifying : Description : Deleting cloud cred autogenerated-cred-testuser2-1705592139-ilsac [2024-01-18T15:48:37.805Z] INFO[2024-01-18 15:48:37] Actual:true, Expected: true [2024-01-18T15:48:38.061Z] INFO[2024-01-18 15:48:37] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T15:48:38.061Z] INFO[2024-01-18 15:48:37] Actual:true, Expected: true [2024-01-18T15:48:38.061Z] 2024-01-18 15:48:37 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2-1705592139 - 9d6edf84-500b-4282-9e37-fa729b1cb3bb [2024-01-18T15:48:38.316Z] 2024-01-18 15:48:38 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.DeleteUser:#755] - Response for user [testuser2-1705592139] deletion - [2024-01-18T15:48:38.316Z] 2024-01-18 15:48:38 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [backup.DeleteUser:#759] - Deleted User - testuser2-1705592139 [2024-01-18T15:48:38.316Z] INFO[2024-01-18 15:48:38] --------Test End------ [2024-01-18T15:48:38.316Z] INFO[2024-01-18 15:48:38] #Test: [2024-01-18T15:48:38.316Z] INFO[2024-01-18 15:48:38] #Description: [2024-01-18T15:48:38.316Z] INFO[2024-01-18 15:48:38] ------------------------ [2024-01-18T15:48:38.316Z] 2024-01-18 15:48:38 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-87567-01-18-15h34m08s] [2024-01-18T15:48:38.316Z] 2024-01-18 15:48:38 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T15:48:38.316Z] 2024-01-18 15:48:38 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Switching context to [/tmp/destination-config] [2024-01-18T15:48:38.572Z] 2024-01-18 15:48:38 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:48:38.572Z] 2024-01-18 15:48:38 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:48:38.572Z] 2024-01-18 15:48:38 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:48:38.572Z] 2024-01-18 15:48:38 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:48:38.827Z] 2024-01-18 15:48:38 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:48:38.827Z] 2024-01-18 15:48:38 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:48:39.396Z] 2024-01-18 15:48:39 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:48:39.652Z] 2024-01-18 15:48:39 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:48:40.213Z] 2024-01-18 15:48:39 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:48:40.213Z] 2024-01-18 15:48:40 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T15:48:40.470Z] 2024-01-18 15:48:40 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T15:48:40.470Z] 2024-01-18 15:48:40 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T15:48:40.471Z] 2024-01-18 15:48:40 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T15:48:40.471Z] 2024-01-18 15:48:40 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Switched context to [/tmp/destination-config] [2024-01-18T15:48:40.730Z] 2024-01-18 15:48:40 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-fsvz] [2024-01-18T15:48:40.730Z] 2024-01-18 15:48:40 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [custom-mysql-ibm-pxb-0-87567-01-18-15h34m08s-wjlh] [2024-01-18T15:48:40.986Z] 2024-01-18 15:48:40 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {DeleteObjectsByMultipleUsersFromNewAdmin} Deletes backups, backup schedules, restore and cluster objects created by multiple user from the new admin [2024-01-18T15:48:40.986Z] 2024-01-18 15:48:40 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-18T15:48:40.986Z] 2024/01/18 15:48:40 DoRetryWithTimeout - Error: {dial tcp 10.240.64.29:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T15:48:46.249Z] 2024/01/18 15:48:45 DoRetryWithTimeout - Error: {dial tcp 10.240.64.29:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T15:48:51.496Z] 2024/01/18 15:48:50 DoRetryWithTimeout - Error: {dial tcp 10.240.64.29:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T15:48:56.743Z] 2024/01/18 15:48:55 DoRetryWithTimeout - Error: {dial tcp 10.240.64.29:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T15:49:01.982Z] 2024/01/18 15:49:00 DoRetryWithTimeout - Error: {dial tcp 10.240.64.29:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T15:49:06.145Z] 2024/01/18 15:49:05 DoRetryWithTimeout - Error: {dial tcp 10.240.64.29:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T15:49:11.388Z] 2024/01/18 15:49:10 DoRetryWithTimeout - Error: {dial tcp 10.240.64.29:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T15:49:16.632Z] 2024/01/18 15:49:15 DoRetryWithTimeout - Error: {dial tcp 10.240.64.29:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T15:49:21.878Z] 2024/01/18 15:49:20 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.64.29. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.29 10.240.64.29]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.29:22: connect: connection refused} [2024-01-18T15:49:21.878Z] {dial tcp 10.240.64.29:22: connect: connection refused} [2024-01-18T15:49:21.878Z] {dial tcp 10.240.64.29:22: connect: connection refused} [2024-01-18T15:49:21.878Z] {dial tcp 10.240.64.29:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-18T15:49:26.047Z] 2024-01-18 15:49:25 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.64.29. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.29 10.240.64.29]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.29:22: connect: connection refused} [2024-01-18T15:49:26.047Z] {dial tcp 10.240.64.29:22: connect: connection refused} [2024-01-18T15:49:26.047Z] {dial tcp 10.240.64.29:22: connect: connection refused} [2024-01-18T15:49:26.047Z] {dial tcp 10.240.64.29:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T15:49:26.047Z] 2024-01-18 15:49:25 +0000:[ERROR] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.64.29. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.29 10.240.64.29]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.29:22: connect: connection refused} [2024-01-18T15:49:26.047Z] {dial tcp 10.240.64.29:22: connect: connection refused} [2024-01-18T15:49:26.047Z] {dial tcp 10.240.64.29:22: connect: connection refused} [2024-01-18T15:49:26.047Z] {dial tcp 10.240.64.29:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T15:49:26.047Z] 2024-01-18 15:49:25 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T15:49:26.047Z] 2024-01-18 15:49:25 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Switching context to [/tmp/source-config] [2024-01-18T15:49:26.303Z] 2024-01-18 15:49:26 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:49:26.303Z] 2024-01-18 15:49:26 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:49:26.303Z] 2024-01-18 15:49:26 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:49:26.558Z] 2024-01-18 15:49:26 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:49:26.558Z] 2024-01-18 15:49:26 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:49:26.558Z] 2024-01-18 15:49:26 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:49:26.814Z] 2024-01-18 15:49:26 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:49:27.380Z] 2024-01-18 15:49:27 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:49:27.636Z] 2024-01-18 15:49:27 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:49:27.892Z] 2024-01-18 15:49:27 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T15:49:28.148Z] 2024-01-18 15:49:27 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T15:49:28.148Z] 2024-01-18 15:49:27 +0000:[WARNING] [{DeleteObjectsByMultipleUsersFromNewAdmin}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T15:49:28.149Z] 2024-01-18 15:49:27 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T15:49:28.149Z] 2024-01-18 15:49:27 +0000:[INFO] [{DeleteObjectsByMultipleUsersFromNewAdmin}] Switched context to [/tmp/source-config] [2024-01-18T15:49:28.149Z] [2024-01-18T15:49:28.149Z] ------------------------------ [2024-01-18T15:49:28.149Z] • [SLOW TEST:919.434 seconds] [2024-01-18T15:49:28.149Z] {DeleteObjectsByMultipleUsersFromNewAdmin} [2024-01-18T15:49:28.149Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_orphaned_test.go:696 [2024-01-18T15:49:28.149Z] Deletes backups, backup schedules, restore and cluster objects created by multiple user from the new admin [2024-01-18T15:49:28.149Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_orphaned_test.go:735 [2024-01-18T15:49:28.149Z] ------------------------------ [2024-01-18T15:49:28.149Z] 2024-01-18 15:49:27 +0000:[INFO] [{AddMultipleNamespaceLabels}] Log Dir: /testresults//AddMultipleNamespaceLabels.log [2024-01-18T15:49:28.149Z] INFO[2024-01-18 15:49:27] --------Test Start------ [2024-01-18T15:49:28.149Z] INFO[2024-01-18 15:49:27] #Test: AddMultipleNamespaceLabels [2024-01-18T15:49:28.149Z] INFO[2024-01-18 15:49:27] #Description: Add multiple labels to namespaces, perform manual backup, schedule backup using namespace labels and restore [2024-01-18T15:49:28.149Z] INFO[2024-01-18 15:49:27] ------------------------ [2024-01-18T15:49:28.149Z] 2024-01-18 15:49:27 +0000:[INFO] [{AddMultipleNamespaceLabels}] Deploy applications [2024-01-18T15:49:28.149Z] 2024-01-18 15:49:27 +0000:[INFO] [{AddMultipleNamespaceLabels}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T15:49:28.149Z] 2024-01-18 15:49:27 +0000:[INFO] [{AddMultipleNamespaceLabels}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T15:49:28.407Z] 2024-01-18 15:49:28 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T15:49:28.407Z] 2024-01-18 15:49:28 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T15:49:28.664Z] 2024-01-18 15:49:28 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T15:49:28.664Z] 2024-01-18 15:49:28 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T15:49:28.665Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T15:49:28.920Z] 2024-01-18 15:49:28 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T15:49:28.921Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T15:49:28.921Z] 2024-01-18 15:49:28 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T15:49:28.921Z] 2024-01-18 15:49:28 +0000:[INFO] [{AddMultipleNamespaceLabels}] Validate applications [2024-01-18T15:49:28.921Z] 2024-01-18 15:49:28 +0000:[INFO] [{AddMultipleNamespaceLabels}] Validate applications [2024-01-18T15:49:28.921Z] 2024-01-18 15:49:28 +0000:[INFO] [{AddMultipleNamespaceLabels}] Validating mysql-ibm app [2024-01-18T15:49:28.921Z] 2024-01-18 15:49:28 +0000:[INFO] [{AddMultipleNamespaceLabels}] Validating mysql-ibm app's volumes [2024-01-18T15:49:28.921Z] 2024-01-18 15:49:28 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T15:49:29.180Z] 2024/01/18 15:49:28 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T15:49:39.124Z] 2024/01/18 15:49:38 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T15:49:49.078Z] 2024-01-18 15:49:48 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s [2024-01-18T15:49:49.335Z] 2024-01-18 15:49:49 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T15:49:49.335Z] 2024-01-18 15:49:49 +0000:[WARNING] [{AddMultipleNamespaceLabels}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T15:49:49.335Z] 2024-01-18 15:49:49 +0000:[INFO] [{AddMultipleNamespaceLabels}] wait for mysql-ibm app to start running [2024-01-18T15:49:49.335Z] 2024-01-18 15:49:49 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T15:49:49.335Z] 2024/01/18 15:49:49 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:49:49.335Z] pod name:mysql-7d766cd65c-g47b5 namespace:mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s running:false ready:false node:10.240.64.9 [2024-01-18T15:49:49.335Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:50:01.500Z] 2024/01/18 15:49:59 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:50:01.500Z] pod name:mysql-7d766cd65c-g47b5 namespace:mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s running:true ready:false node:10.240.64.9 [2024-01-18T15:50:01.500Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:50:09.607Z] 2024/01/18 15:50:09 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:50:09.607Z] pod name:mysql-7d766cd65c-g47b5 namespace:mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s running:true ready:false node:10.240.64.9 [2024-01-18T15:50:09.607Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:50:21.775Z] 2024/01/18 15:50:19 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:50:21.775Z] pod name:mysql-7d766cd65c-g47b5 namespace:mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s running:true ready:false node:10.240.64.9 [2024-01-18T15:50:21.775Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:50:29.859Z] 2024/01/18 15:50:29 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:50:29.859Z] pod name:mysql-7d766cd65c-g47b5 namespace:mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s running:true ready:false node:10.240.64.9 [2024-01-18T15:50:29.859Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:50:42.028Z] 2024/01/18 15:50:39 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:50:42.028Z] pod name:mysql-7d766cd65c-g47b5 namespace:mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s running:true ready:false node:10.240.64.9 [2024-01-18T15:50:42.028Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:50:50.113Z] 2024/01/18 15:50:49 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:50:50.113Z] pod name:mysql-7d766cd65c-g47b5 namespace:mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s running:true ready:false node:10.240.64.9 [2024-01-18T15:50:50.113Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:51:02.288Z] 2024/01/18 15:51:00 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:51:02.288Z] pod name:mysql-7d766cd65c-g47b5 namespace:mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s running:true ready:false node:10.240.64.9 [2024-01-18T15:51:02.288Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:51:10.378Z] 2024-01-18 15:51:10 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T15:51:10.378Z] 2024-01-18 15:51:10 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T15:51:10.636Z] 2024-01-18 15:51:10 +0000:[INFO] [{AddMultipleNamespaceLabels}] validate if mysql-ibm app's volumes are setup [2024-01-18T15:51:10.637Z] 2024-01-18 15:51:10 +0000:[WARNING] [{AddMultipleNamespaceLabels}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T15:51:10.637Z] 2024-01-18 15:51:10 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T15:51:10.637Z] 2024-01-18 15:51:10 +0000:[INFO] [{AddMultipleNamespaceLabels}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T15:51:10.637Z] 2024-01-18 15:51:10 +0000:[WARNING] [{AddMultipleNamespaceLabels}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T15:51:10.637Z] 2024-01-18 15:51:10 +0000:[INFO] [{AddMultipleNamespaceLabels}] Adding labels to namespaces in multiple of 10 until 1000 [2024-01-18T15:51:10.892Z] INFO[2024-01-18 15:51:10] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:10.892Z] INFO[2024-01-18 15:51:10] Actual:true, Expected: true [2024-01-18T15:51:10.892Z] INFO[2024-01-18 15:51:10] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:10.892Z] INFO[2024-01-18 15:51:10] Actual:true, Expected: true [2024-01-18T15:51:10.892Z] INFO[2024-01-18 15:51:10] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:10.892Z] INFO[2024-01-18 15:51:10] Actual:true, Expected: true [2024-01-18T15:51:10.892Z] INFO[2024-01-18 15:51:10] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:10.892Z] INFO[2024-01-18 15:51:10] Actual:true, Expected: true [2024-01-18T15:51:11.457Z] INFO[2024-01-18 15:51:11] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:11.457Z] INFO[2024-01-18 15:51:11] Actual:true, Expected: true [2024-01-18T15:51:11.713Z] INFO[2024-01-18 15:51:11] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:11.713Z] INFO[2024-01-18 15:51:11] Actual:true, Expected: true [2024-01-18T15:51:12.280Z] INFO[2024-01-18 15:51:11] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:12.280Z] INFO[2024-01-18 15:51:11] Actual:true, Expected: true [2024-01-18T15:51:12.535Z] INFO[2024-01-18 15:51:12] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:12.535Z] INFO[2024-01-18 15:51:12] Actual:true, Expected: true [2024-01-18T15:51:12.791Z] INFO[2024-01-18 15:51:12] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:12.791Z] INFO[2024-01-18 15:51:12] Actual:true, Expected: true [2024-01-18T15:51:13.352Z] INFO[2024-01-18 15:51:13] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:13.352Z] INFO[2024-01-18 15:51:13] Actual:true, Expected: true [2024-01-18T15:51:13.608Z] INFO[2024-01-18 15:51:13] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:13.608Z] INFO[2024-01-18 15:51:13] Actual:true, Expected: true [2024-01-18T15:51:14.177Z] INFO[2024-01-18 15:51:13] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:14.177Z] INFO[2024-01-18 15:51:13] Actual:true, Expected: true [2024-01-18T15:51:14.438Z] INFO[2024-01-18 15:51:14] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:14.438Z] INFO[2024-01-18 15:51:14] Actual:true, Expected: true [2024-01-18T15:51:14.997Z] INFO[2024-01-18 15:51:14] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:14.997Z] INFO[2024-01-18 15:51:14] Actual:true, Expected: true [2024-01-18T15:51:15.254Z] INFO[2024-01-18 15:51:15] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:15.254Z] INFO[2024-01-18 15:51:15] Actual:true, Expected: true [2024-01-18T15:51:15.816Z] INFO[2024-01-18 15:51:15] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:15.816Z] INFO[2024-01-18 15:51:15] Actual:true, Expected: true [2024-01-18T15:51:16.071Z] INFO[2024-01-18 15:51:15] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:16.071Z] INFO[2024-01-18 15:51:15] Actual:true, Expected: true [2024-01-18T15:51:16.632Z] INFO[2024-01-18 15:51:16] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:16.632Z] INFO[2024-01-18 15:51:16] Actual:true, Expected: true [2024-01-18T15:51:16.888Z] INFO[2024-01-18 15:51:16] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:16.888Z] INFO[2024-01-18 15:51:16] Actual:true, Expected: true [2024-01-18T15:51:17.449Z] INFO[2024-01-18 15:51:17] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:17.449Z] INFO[2024-01-18 15:51:17] Actual:true, Expected: true [2024-01-18T15:51:17.725Z] INFO[2024-01-18 15:51:17] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:17.725Z] INFO[2024-01-18 15:51:17] Actual:true, Expected: true [2024-01-18T15:51:17.981Z] INFO[2024-01-18 15:51:17] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:17.981Z] INFO[2024-01-18 15:51:17] Actual:true, Expected: true [2024-01-18T15:51:18.542Z] INFO[2024-01-18 15:51:18] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:18.542Z] INFO[2024-01-18 15:51:18] Actual:true, Expected: true [2024-01-18T15:51:18.797Z] INFO[2024-01-18 15:51:18] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:18.797Z] INFO[2024-01-18 15:51:18] Actual:true, Expected: true [2024-01-18T15:51:19.368Z] INFO[2024-01-18 15:51:19] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:19.368Z] INFO[2024-01-18 15:51:19] Actual:true, Expected: true [2024-01-18T15:51:19.632Z] INFO[2024-01-18 15:51:19] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:19.632Z] INFO[2024-01-18 15:51:19] Actual:true, Expected: true [2024-01-18T15:51:20.193Z] INFO[2024-01-18 15:51:19] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:20.193Z] INFO[2024-01-18 15:51:19] Actual:true, Expected: true [2024-01-18T15:51:20.449Z] INFO[2024-01-18 15:51:20] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:20.449Z] INFO[2024-01-18 15:51:20] Actual:true, Expected: true [2024-01-18T15:51:21.010Z] INFO[2024-01-18 15:51:20] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:21.010Z] INFO[2024-01-18 15:51:20] Actual:true, Expected: true [2024-01-18T15:51:21.265Z] INFO[2024-01-18 15:51:21] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:21.265Z] INFO[2024-01-18 15:51:21] Actual:true, Expected: true [2024-01-18T15:51:21.826Z] INFO[2024-01-18 15:51:21] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:21.826Z] INFO[2024-01-18 15:51:21] Actual:true, Expected: true [2024-01-18T15:51:22.081Z] INFO[2024-01-18 15:51:21] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:22.081Z] INFO[2024-01-18 15:51:21] Actual:true, Expected: true [2024-01-18T15:51:22.640Z] INFO[2024-01-18 15:51:22] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:22.640Z] INFO[2024-01-18 15:51:22] Actual:true, Expected: true [2024-01-18T15:51:22.897Z] INFO[2024-01-18 15:51:22] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:22.897Z] INFO[2024-01-18 15:51:22] Actual:true, Expected: true [2024-01-18T15:51:23.466Z] INFO[2024-01-18 15:51:23] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:23.466Z] INFO[2024-01-18 15:51:23] Actual:true, Expected: true [2024-01-18T15:51:23.722Z] INFO[2024-01-18 15:51:23] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:23.722Z] INFO[2024-01-18 15:51:23] Actual:true, Expected: true [2024-01-18T15:51:24.283Z] INFO[2024-01-18 15:51:23] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:24.283Z] INFO[2024-01-18 15:51:23] Actual:true, Expected: true [2024-01-18T15:51:24.538Z] INFO[2024-01-18 15:51:24] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:24.538Z] INFO[2024-01-18 15:51:24] Actual:true, Expected: true [2024-01-18T15:51:25.099Z] INFO[2024-01-18 15:51:24] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:25.099Z] INFO[2024-01-18 15:51:24] Actual:true, Expected: true [2024-01-18T15:51:25.361Z] INFO[2024-01-18 15:51:25] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:25.361Z] INFO[2024-01-18 15:51:25] Actual:true, Expected: true [2024-01-18T15:51:25.618Z] INFO[2024-01-18 15:51:25] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:25.618Z] INFO[2024-01-18 15:51:25] Actual:true, Expected: true [2024-01-18T15:51:26.179Z] INFO[2024-01-18 15:51:25] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:26.179Z] INFO[2024-01-18 15:51:25] Actual:true, Expected: true [2024-01-18T15:51:26.434Z] INFO[2024-01-18 15:51:26] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:26.434Z] INFO[2024-01-18 15:51:26] Actual:true, Expected: true [2024-01-18T15:51:26.995Z] INFO[2024-01-18 15:51:26] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:26.995Z] INFO[2024-01-18 15:51:26] Actual:true, Expected: true [2024-01-18T15:51:27.251Z] INFO[2024-01-18 15:51:27] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:27.251Z] INFO[2024-01-18 15:51:27] Actual:true, Expected: true [2024-01-18T15:51:27.813Z] INFO[2024-01-18 15:51:27] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:27.813Z] INFO[2024-01-18 15:51:27] Actual:true, Expected: true [2024-01-18T15:51:28.069Z] INFO[2024-01-18 15:51:27] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:28.069Z] INFO[2024-01-18 15:51:27] Actual:true, Expected: true [2024-01-18T15:51:28.630Z] INFO[2024-01-18 15:51:28] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:28.630Z] INFO[2024-01-18 15:51:28] Actual:true, Expected: true [2024-01-18T15:51:28.885Z] INFO[2024-01-18 15:51:28] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:28.885Z] INFO[2024-01-18 15:51:28] Actual:true, Expected: true [2024-01-18T15:51:29.447Z] INFO[2024-01-18 15:51:29] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:29.447Z] INFO[2024-01-18 15:51:29] Actual:true, Expected: true [2024-01-18T15:51:29.703Z] INFO[2024-01-18 15:51:29] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:29.703Z] INFO[2024-01-18 15:51:29] Actual:true, Expected: true [2024-01-18T15:51:30.264Z] INFO[2024-01-18 15:51:29] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:30.264Z] INFO[2024-01-18 15:51:29] Actual:true, Expected: true [2024-01-18T15:51:30.523Z] INFO[2024-01-18 15:51:30] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:30.523Z] INFO[2024-01-18 15:51:30] Actual:true, Expected: true [2024-01-18T15:51:31.085Z] INFO[2024-01-18 15:51:30] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:31.085Z] INFO[2024-01-18 15:51:30] Actual:true, Expected: true [2024-01-18T15:51:31.341Z] INFO[2024-01-18 15:51:31] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:31.341Z] INFO[2024-01-18 15:51:31] Actual:true, Expected: true [2024-01-18T15:51:31.600Z] INFO[2024-01-18 15:51:31] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:31.600Z] INFO[2024-01-18 15:51:31] Actual:true, Expected: true [2024-01-18T15:51:32.161Z] INFO[2024-01-18 15:51:31] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:32.161Z] INFO[2024-01-18 15:51:31] Actual:true, Expected: true [2024-01-18T15:51:32.424Z] INFO[2024-01-18 15:51:32] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:32.424Z] INFO[2024-01-18 15:51:32] Actual:true, Expected: true [2024-01-18T15:51:32.987Z] INFO[2024-01-18 15:51:32] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:32.988Z] INFO[2024-01-18 15:51:32] Actual:true, Expected: true [2024-01-18T15:51:33.244Z] INFO[2024-01-18 15:51:33] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:33.244Z] INFO[2024-01-18 15:51:33] Actual:true, Expected: true [2024-01-18T15:51:33.806Z] INFO[2024-01-18 15:51:33] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:33.806Z] INFO[2024-01-18 15:51:33] Actual:true, Expected: true [2024-01-18T15:51:34.061Z] INFO[2024-01-18 15:51:33] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:34.061Z] INFO[2024-01-18 15:51:33] Actual:true, Expected: true [2024-01-18T15:51:34.620Z] INFO[2024-01-18 15:51:34] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:34.620Z] INFO[2024-01-18 15:51:34] Actual:true, Expected: true [2024-01-18T15:51:34.875Z] INFO[2024-01-18 15:51:34] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:34.875Z] INFO[2024-01-18 15:51:34] Actual:true, Expected: true [2024-01-18T15:51:35.436Z] INFO[2024-01-18 15:51:35] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:35.436Z] INFO[2024-01-18 15:51:35] Actual:true, Expected: true [2024-01-18T15:51:35.693Z] INFO[2024-01-18 15:51:35] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:35.693Z] INFO[2024-01-18 15:51:35] Actual:true, Expected: true [2024-01-18T15:51:36.254Z] INFO[2024-01-18 15:51:35] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:36.254Z] INFO[2024-01-18 15:51:35] Actual:true, Expected: true [2024-01-18T15:51:36.510Z] INFO[2024-01-18 15:51:36] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:36.510Z] INFO[2024-01-18 15:51:36] Actual:true, Expected: true [2024-01-18T15:51:37.074Z] INFO[2024-01-18 15:51:36] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:37.074Z] INFO[2024-01-18 15:51:36] Actual:true, Expected: true [2024-01-18T15:51:37.329Z] INFO[2024-01-18 15:51:37] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:37.329Z] INFO[2024-01-18 15:51:37] Actual:true, Expected: true [2024-01-18T15:51:37.893Z] INFO[2024-01-18 15:51:37] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:37.893Z] INFO[2024-01-18 15:51:37] Actual:true, Expected: true [2024-01-18T15:51:38.149Z] INFO[2024-01-18 15:51:37] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:38.149Z] INFO[2024-01-18 15:51:37] Actual:true, Expected: true [2024-01-18T15:51:38.404Z] INFO[2024-01-18 15:51:38] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:38.405Z] INFO[2024-01-18 15:51:38] Actual:true, Expected: true [2024-01-18T15:51:38.966Z] INFO[2024-01-18 15:51:38] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:38.966Z] INFO[2024-01-18 15:51:38] Actual:true, Expected: true [2024-01-18T15:51:39.222Z] INFO[2024-01-18 15:51:39] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:39.222Z] INFO[2024-01-18 15:51:39] Actual:true, Expected: true [2024-01-18T15:51:39.791Z] INFO[2024-01-18 15:51:39] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:39.791Z] INFO[2024-01-18 15:51:39] Actual:true, Expected: true [2024-01-18T15:51:40.047Z] INFO[2024-01-18 15:51:39] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:40.047Z] INFO[2024-01-18 15:51:39] Actual:true, Expected: true [2024-01-18T15:51:40.612Z] INFO[2024-01-18 15:51:40] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:40.612Z] INFO[2024-01-18 15:51:40] Actual:true, Expected: true [2024-01-18T15:51:40.867Z] INFO[2024-01-18 15:51:40] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:40.867Z] INFO[2024-01-18 15:51:40] Actual:true, Expected: true [2024-01-18T15:51:41.427Z] INFO[2024-01-18 15:51:41] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:41.427Z] INFO[2024-01-18 15:51:41] Actual:true, Expected: true [2024-01-18T15:51:41.683Z] INFO[2024-01-18 15:51:41] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:41.683Z] INFO[2024-01-18 15:51:41] Actual:true, Expected: true [2024-01-18T15:51:42.255Z] INFO[2024-01-18 15:51:41] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:42.255Z] INFO[2024-01-18 15:51:41] Actual:true, Expected: true [2024-01-18T15:51:42.510Z] INFO[2024-01-18 15:51:42] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:42.510Z] INFO[2024-01-18 15:51:42] Actual:true, Expected: true [2024-01-18T15:51:43.076Z] INFO[2024-01-18 15:51:42] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:43.076Z] INFO[2024-01-18 15:51:42] Actual:true, Expected: true [2024-01-18T15:51:43.332Z] INFO[2024-01-18 15:51:43] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:43.332Z] INFO[2024-01-18 15:51:43] Actual:true, Expected: true [2024-01-18T15:51:43.893Z] INFO[2024-01-18 15:51:43] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:43.893Z] INFO[2024-01-18 15:51:43] Actual:true, Expected: true [2024-01-18T15:51:44.148Z] INFO[2024-01-18 15:51:43] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:44.148Z] INFO[2024-01-18 15:51:43] Actual:true, Expected: true [2024-01-18T15:51:44.711Z] INFO[2024-01-18 15:51:44] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:44.711Z] INFO[2024-01-18 15:51:44] Actual:true, Expected: true [2024-01-18T15:51:44.969Z] INFO[2024-01-18 15:51:44] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:44.969Z] INFO[2024-01-18 15:51:44] Actual:true, Expected: true [2024-01-18T15:51:45.225Z] INFO[2024-01-18 15:51:45] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:45.225Z] INFO[2024-01-18 15:51:45] Actual:true, Expected: true [2024-01-18T15:51:45.785Z] INFO[2024-01-18 15:51:45] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:45.785Z] INFO[2024-01-18 15:51:45] Actual:true, Expected: true [2024-01-18T15:51:46.040Z] INFO[2024-01-18 15:51:45] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:46.040Z] INFO[2024-01-18 15:51:45] Actual:true, Expected: true [2024-01-18T15:51:46.610Z] INFO[2024-01-18 15:51:46] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:46.610Z] INFO[2024-01-18 15:51:46] Actual:true, Expected: true [2024-01-18T15:51:46.868Z] INFO[2024-01-18 15:51:46] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:46.868Z] INFO[2024-01-18 15:51:46] Actual:true, Expected: true [2024-01-18T15:51:47.429Z] INFO[2024-01-18 15:51:47] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:47.429Z] INFO[2024-01-18 15:51:47] Actual:true, Expected: true [2024-01-18T15:51:47.684Z] INFO[2024-01-18 15:51:47] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:47.684Z] INFO[2024-01-18 15:51:47] Actual:true, Expected: true [2024-01-18T15:51:48.244Z] INFO[2024-01-18 15:51:47] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:48.244Z] INFO[2024-01-18 15:51:47] Actual:true, Expected: true [2024-01-18T15:51:48.501Z] INFO[2024-01-18 15:51:48] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:48.501Z] INFO[2024-01-18 15:51:48] Actual:true, Expected: true [2024-01-18T15:51:49.061Z] INFO[2024-01-18 15:51:48] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:49.061Z] INFO[2024-01-18 15:51:48] Actual:true, Expected: true [2024-01-18T15:51:49.316Z] INFO[2024-01-18 15:51:49] Verifying : Description : Adding [10] labels to namespaces [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:49.316Z] INFO[2024-01-18 15:51:49] Actual:true, Expected: true [2024-01-18T15:51:49.316Z] 2024-01-18 15:51:49 +0000:[INFO] [{AddMultipleNamespaceLabels}] Verifying number of labels added to namespace [2024-01-18T15:51:49.571Z] INFO[2024-01-18 15:51:49] Verifying : Description : Fetching the labels added to namespace mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s [2024-01-18T15:51:49.571Z] INFO[2024-01-18 15:51:49] Actual:true, Expected: true [2024-01-18T15:51:49.571Z] INFO[2024-01-18 15:51:49] Verifying : Description : Fetching labels 1004 for namespace mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s [2024-01-18T15:51:49.571Z] INFO[2024-01-18 15:51:49] Actual:true, Expected: true [2024-01-18T15:51:49.571Z] INFO[2024-01-18 15:51:49] Verifying : Description : Verifying number of added labels to desired labels for namespace mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s [2024-01-18T15:51:49.572Z] ERRO[2024-01-18 15:51:49] Actual:1004, Expected: 1000 [2024-01-18T15:51:50.945Z] INFO[2024-01-18 15:51:50] Verifying : Description : Verification of deleting backup schedule - [2024-01-18T15:51:50.945Z] ERRO[2024-01-18 15:51:50] backup schedule with name '' not found for org 'default' [2024-01-18T15:51:52.308Z] INFO[2024-01-18 15:51:52] Verifying : Description : Deleting backup schedule policies [] [2024-01-18T15:51:52.308Z] ERRO[2024-01-18 15:51:52] Failed to delete schedule policy with error [Unable to find schedule policy Uid] [2024-01-18T15:51:52.308Z] 2024-01-18 15:51:52 +0000:[INFO] [{AddMultipleNamespaceLabels}] Deleting labels from namespaces - [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:52.870Z] INFO[2024-01-18 15:51:52] Verifying : Description : Deleting labels [map[0-d276e4fe-05de-4776-a185-b73b226f2007:d276e4fe-05de-4776-a185-b73b226f2007 1-d276e4fe-05de-4776-a185-b73b226f2007:d276e4fe-05de-4776-a185-b73b226f2007 2-d276e4fe-05de-4776-a185-b73b226f2007:d276e4fe-05de-4776-a185-b73b226f2007 3-d276e4fe-05de-4776-a185-b73b226f2007:d276e4fe-05de-4776-a185-b73b226f2007 4-d276e4fe-05de-4776-a185-b73b226f2007:d276e4fe-05de-4776-a185-b73b226f2007 5-d276e4fe-05de-4776-a185-b73b226f2007:d276e4fe-05de-4776-a185-b73b226f2007 6-d276e4fe-05de-4776-a185-b73b226f2007:d276e4fe-05de-4776-a185-b73b226f2007 7-d276e4fe-05de-4776-a185-b73b226f2007:d276e4fe-05de-4776-a185-b73b226f2007 8-d276e4fe-05de-4776-a185-b73b226f2007:d276e4fe-05de-4776-a185-b73b226f2007 9-d276e4fe-05de-4776-a185-b73b226f2007:d276e4fe-05de-4776-a185-b73b226f2007]] from namespaces [[mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s]] [2024-01-18T15:51:52.870Z] INFO[2024-01-18 15:51:52] Actual:true, Expected: true [2024-01-18T15:51:52.870Z] 2024-01-18 15:51:52 +0000:[INFO] [{AddMultipleNamespaceLabels}] Deleting deployed namespaces - [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:52.870Z] 2024-01-18 15:51:52 +0000:[INFO] [{AddMultipleNamespaceLabels}] Destroying apps [2024-01-18T15:51:52.870Z] 2024-01-18 15:51:52 +0000:[INFO] [{AddMultipleNamespaceLabels}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T15:51:53.131Z] 2024-01-18 15:51:52 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T15:51:53.131Z] 2024-01-18 15:51:52 +0000:[INFO] [{AddMultipleNamespaceLabels}] start destroying mysql-ibm app [2024-01-18T15:51:53.386Z] 2024-01-18 15:51:53 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T15:51:53.386Z] 2024-01-18 15:51:53 +0000:[INFO] [{AddMultipleNamespaceLabels}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T15:51:53.386Z] 2024-01-18 15:51:53 +0000:[WARNING] [{AddMultipleNamespaceLabels}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T15:51:53.386Z] INFO[2024-01-18 15:51:53] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T15:51:53.386Z] INFO[2024-01-18 15:51:53] Actual:true, Expected: true [2024-01-18T15:51:53.386Z] 2024-01-18 15:51:53 +0000:[INFO] [{AddMultipleNamespaceLabels}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T15:51:53.386Z] INFO[2024-01-18 15:51:53] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T15:51:53.386Z] INFO[2024-01-18 15:51:53] Actual:true, Expected: true [2024-01-18T15:51:53.386Z] INFO[2024-01-18 15:51:53] --------Test End------ [2024-01-18T15:51:53.386Z] INFO[2024-01-18 15:51:53] #Test: [2024-01-18T15:51:53.386Z] INFO[2024-01-18 15:51:53] #Description: [2024-01-18T15:51:53.386Z] INFO[2024-01-18 15:51:53] ------------------------ [2024-01-18T15:51:53.386Z] 2024-01-18 15:51:53 +0000:[INFO] [{AddMultipleNamespaceLabels}] [tests.EndPxBackupTorpedoTest:#7261] - >>>> FAILED TEST: {AddMultipleNamespaceLabels} Add multiple labels to namespaces, perform manual backup, schedule backup using namespace label and restore [2024-01-18T15:51:53.642Z] 2024-01-18 15:51:53 +0000:[INFO] [{AddMultipleNamespaceLabels}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s] [2024-01-18T15:51:53.642Z] 2024-01-18 15:51:53 +0000:[INFO] [{AddMultipleNamespaceLabels}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T15:51:53.642Z] 2024-01-18 15:51:53 +0000:[INFO] [{AddMultipleNamespaceLabels}] Switching context to [/tmp/destination-config] [2024-01-18T15:51:53.897Z] 2024-01-18 15:51:53 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:51:53.897Z] 2024-01-18 15:51:53 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:51:53.897Z] 2024-01-18 15:51:53 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:51:53.897Z] 2024-01-18 15:51:53 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:51:54.153Z] 2024-01-18 15:51:53 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:51:54.153Z] 2024-01-18 15:51:53 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:51:54.714Z] 2024-01-18 15:51:54 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:51:54.969Z] 2024-01-18 15:51:54 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:51:55.226Z] 2024-01-18 15:51:55 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:51:55.485Z] 2024-01-18 15:51:55 +0000:[WARNING] [{AddMultipleNamespaceLabels}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T15:51:55.742Z] 2024-01-18 15:51:55 +0000:[WARNING] [{AddMultipleNamespaceLabels}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T15:51:55.742Z] 2024-01-18 15:51:55 +0000:[WARNING] [{AddMultipleNamespaceLabels}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T15:51:55.742Z] 2024-01-18 15:51:55 +0000:[INFO] [{AddMultipleNamespaceLabels}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T15:51:55.742Z] 2024-01-18 15:51:55 +0000:[INFO] [{AddMultipleNamespaceLabels}] Switched context to [/tmp/destination-config] [2024-01-18T15:51:55.998Z] 2024-01-18 15:51:55 +0000:[INFO] [{AddMultipleNamespaceLabels}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {AddMultipleNamespaceLabels} Add multiple labels to namespaces, perform manual backup, schedule backup using namespace label and restore [2024-01-18T15:51:55.998Z] 2024-01-18 15:51:55 +0000:[INFO] [{AddMultipleNamespaceLabels}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-18T15:51:55.998Z] 2024/01/18 15:51:55 DoRetryWithTimeout - Error: {dial tcp 10.240.128.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T15:52:01.240Z] 2024/01/18 15:52:00 DoRetryWithTimeout - Error: {dial tcp 10.240.128.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T15:52:06.477Z] 2024/01/18 15:52:05 DoRetryWithTimeout - Error: {dial tcp 10.240.128.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T15:52:11.723Z] 2024/01/18 15:52:10 DoRetryWithTimeout - Error: {dial tcp 10.240.128.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T15:52:15.888Z] 2024/01/18 15:52:15 DoRetryWithTimeout - Error: {dial tcp 10.240.128.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T15:52:21.169Z] 2024/01/18 15:52:20 DoRetryWithTimeout - Error: {dial tcp 10.240.128.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T15:52:26.415Z] 2024/01/18 15:52:25 DoRetryWithTimeout - Error: {dial tcp 10.240.128.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T15:52:31.660Z] 2024/01/18 15:52:30 DoRetryWithTimeout - Error: {dial tcp 10.240.128.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T15:52:36.922Z] 2024/01/18 15:52:35 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.128.4. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.4 10.240.128.4]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T15:52:36.922Z] {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T15:52:36.922Z] {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T15:52:36.922Z] {dial tcp 10.240.128.4:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-18T15:52:41.089Z] 2024-01-18 15:52:40 +0000:[WARNING] [{AddMultipleNamespaceLabels}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.4. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.4 10.240.128.4]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T15:52:41.089Z] {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T15:52:41.089Z] {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T15:52:41.089Z] {dial tcp 10.240.128.4:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T15:52:41.089Z] 2024-01-18 15:52:40 +0000:[ERROR] [{AddMultipleNamespaceLabels}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.4. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.4 10.240.128.4]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T15:52:41.089Z] {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T15:52:41.089Z] {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T15:52:41.089Z] {dial tcp 10.240.128.4:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T15:52:41.089Z] 2024-01-18 15:52:40 +0000:[INFO] [{AddMultipleNamespaceLabels}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T15:52:41.089Z] 2024-01-18 15:52:40 +0000:[INFO] [{AddMultipleNamespaceLabels}] Switching context to [/tmp/source-config] [2024-01-18T15:52:41.089Z] 2024-01-18 15:52:40 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:52:41.089Z] 2024-01-18 15:52:40 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:52:41.089Z] 2024-01-18 15:52:41 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:52:41.089Z] 2024-01-18 15:52:41 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:52:41.345Z] 2024-01-18 15:52:41 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:52:41.345Z] 2024-01-18 15:52:41 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:52:41.909Z] 2024-01-18 15:52:41 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:52:42.164Z] 2024-01-18 15:52:42 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:52:42.725Z] 2024-01-18 15:52:42 +0000:[INFO] [{AddMultipleNamespaceLabels}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:52:42.725Z] 2024-01-18 15:52:42 +0000:[WARNING] [{AddMultipleNamespaceLabels}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T15:52:42.981Z] 2024-01-18 15:52:42 +0000:[WARNING] [{AddMultipleNamespaceLabels}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T15:52:42.981Z] 2024-01-18 15:52:42 +0000:[WARNING] [{AddMultipleNamespaceLabels}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T15:52:42.981Z] 2024-01-18 15:52:42 +0000:[INFO] [{AddMultipleNamespaceLabels}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T15:52:42.981Z] 2024-01-18 15:52:42 +0000:[INFO] [{AddMultipleNamespaceLabels}] Switched context to [/tmp/source-config] [2024-01-18T15:52:42.981Z] • Failure [194.893 seconds] [2024-01-18T15:52:42.981Z] {AddMultipleNamespaceLabels} [2024-01-18T15:52:42.981Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:1666 [2024-01-18T15:52:42.981Z] Add multiple labels to namespaces, perform manual backup, schedule backup using namespace label and restore [It] [2024-01-18T15:52:42.981Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:1712 [2024-01-18T15:52:42.981Z] [2024-01-18T15:52:42.981Z] Unexpected error: [2024-01-18T15:52:42.981Z] <*errors.errorString | 0xc000fb83d0>: { [2024-01-18T15:52:42.981Z] s: "error: 1004, Description: Verifying number of added labels to desired labels for namespace mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s", [2024-01-18T15:52:42.981Z] } [2024-01-18T15:52:42.981Z] error: 1004, Description: Verifying number of added labels to desired labels for namespace mysql-ibm-pxb-0-85583-85583-01-18-15h49m27s [2024-01-18T15:52:42.981Z] occurred [2024-01-18T15:52:42.981Z] [2024-01-18T15:52:42.981Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-18T15:52:42.981Z] [2024-01-18T15:52:42.981Z] Full Stack Trace [2024-01-18T15:52:42.981Z] github.com/portworx/torpedo/pkg/aetosutil.(*Dashboard).VerifyFatal(0x3e8?, {0x6cb97a0?, 0xc00212c5a8}, {0x6cb97a0?, 0xc00212c5c0}, {0xc0000ca9a0, 0x6c}) [2024-01-18T15:52:42.981Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 +0x143 [2024-01-18T15:52:42.981Z] github.com/portworx/torpedo/tests/backup.glob..func64.2.3() [2024-01-18T15:52:42.981Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:1732 +0x2f0 [2024-01-18T15:52:42.981Z] github.com/portworx/torpedo/tests/backup.glob..func64.2() [2024-01-18T15:52:42.981Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:1726 +0x3d8 [2024-01-18T15:52:42.981Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-18T15:52:42.981Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-18T15:52:42.981Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-18T15:52:42.981Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-18T15:52:42.981Z] created by testing.(*T).Run [2024-01-18T15:52:42.981Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-18T15:52:42.981Z] ------------------------------ [2024-01-18T15:52:42.981Z] STEP: schedule applications [2024-01-18T15:52:42.981Z] STEP: Validate applications [2024-01-18T15:52:42.981Z] STEP: validate applications [2024-01-18T15:52:42.981Z] STEP: validate mysql-ibm app's volumes [2024-01-18T15:52:42.981Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T15:52:42.981Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T15:52:42.981Z] STEP: get mysql-ibm app's volume: pvc-c4b25ca9-820c-4d21-a116-f1d68a984b92 inspected by the volume driver [2024-01-18T15:52:42.981Z] STEP: wait for mysql-ibm app to start running [2024-01-18T15:52:42.981Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T15:52:42.981Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T15:52:42.981Z] STEP: Validate Px pod restart count [2024-01-18T15:52:42.981Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T15:52:42.981Z] STEP: Adding labels to namespaces in multiple of 10 until 1000 [2024-01-18T15:52:42.981Z] STEP: Verifying labels added to namespace [2024-01-18T15:52:42.981Z] STEP: destroy apps [2024-01-18T15:52:42.981Z] STEP: destroy the mysql-ibm app's volumes [2024-01-18T15:52:42.981Z] STEP: start destroying mysql-ibm app [2024-01-18T15:52:42.981Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T15:52:42.982Z] SSSS2024-01-18 15:52:42 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Log Dir: /testresults//RestoreEncryptedAndNonEncryptedBackups.log [2024-01-18T15:52:42.982Z] INFO[2024-01-18 15:52:42] --------Test Start------ [2024-01-18T15:52:42.982Z] INFO[2024-01-18 15:52:42] #Test: RestoreEncryptedAndNonEncryptedBackups [2024-01-18T15:52:42.982Z] INFO[2024-01-18 15:52:42] #Description: Restore encrypted and non encrypted backups [2024-01-18T15:52:42.982Z] INFO[2024-01-18 15:52:42] ------------------------ [2024-01-18T15:52:42.982Z] 2024-01-18 15:52:42 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Deploy applications [2024-01-18T15:52:42.982Z] 2024-01-18 15:52:42 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T15:52:42.982Z] 2024-01-18 15:52:42 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T15:52:43.237Z] 2024-01-18 15:52:43 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T15:52:43.237Z] 2024-01-18 15:52:43 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T15:52:43.492Z] 2024-01-18 15:52:43 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T15:52:43.747Z] 2024-01-18 15:52:43 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T15:52:43.748Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T15:52:43.748Z] 2024-01-18 15:52:43 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T15:52:43.748Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T15:52:43.749Z] 2024-01-18 15:52:43 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T15:52:43.749Z] 2024-01-18 15:52:43 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Creating bucket, encrypted and non-encrypted backup location [2024-01-18T15:52:45.111Z] 2024-01-18 15:52:44 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [cred-aws-1705593163] for org [default] with [aws] as provider [2024-01-18T15:52:45.111Z] 2024-01-18 15:52:44 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-18T15:52:45.111Z] 2024-01-18 15:52:44 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cred-aws-1705593163] is present [2024-01-18T15:52:45.111Z] INFO[2024-01-18 15:52:44] Verifying : Description : Verifying creation of cloud credential named [cred-aws-1705593163] for org [default] with [aws] as provider [2024-01-18T15:52:45.111Z] INFO[2024-01-18 15:52:44] Actual:true, Expected: true [2024-01-18T15:52:48.382Z] INFO[2024-01-18 15:52:48] Verifying : Description : Creating backup location location-aws [2024-01-18T15:52:48.382Z] INFO[2024-01-18 15:52:48] Actual:true, Expected: true [2024-01-18T15:52:51.643Z] INFO[2024-01-18 15:52:50] Verifying : Description : Creating backup location encryption-location-aws-1705593163 [2024-01-18T15:52:51.643Z] INFO[2024-01-18 15:52:50] Actual:true, Expected: true [2024-01-18T15:52:51.644Z] 2024-01-18 15:52:50 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validate applications [2024-01-18T15:52:51.644Z] 2024-01-18 15:52:50 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating mysql-ibm app [2024-01-18T15:52:51.644Z] 2024-01-18 15:52:50 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating mysql-ibm app's volumes [2024-01-18T15:52:51.644Z] 2024-01-18 15:52:50 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T15:52:51.644Z] 2024/01/18 15:52:51 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T15:53:01.586Z] 2024-01-18 15:53:01 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-79915-01-18-15h52m42s [2024-01-18T15:53:01.586Z] 2024-01-18 15:53:01 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T15:53:01.586Z] 2024-01-18 15:53:01 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T15:53:01.586Z] 2024-01-18 15:53:01 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] wait for mysql-ibm app to start running [2024-01-18T15:53:01.586Z] 2024-01-18 15:53:01 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T15:53:01.586Z] 2024/01/18 15:53:01 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:53:01.586Z] pod name:mysql-7d766cd65c-56bqp namespace:mysql-ibm-pxb-0-79915-01-18-15h52m42s running:false ready:false node:10.240.64.9 [2024-01-18T15:53:01.586Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:53:13.746Z] 2024/01/18 15:53:11 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:53:13.746Z] pod name:mysql-7d766cd65c-56bqp namespace:mysql-ibm-pxb-0-79915-01-18-15h52m42s running:true ready:false node:10.240.64.9 [2024-01-18T15:53:13.746Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:53:21.823Z] 2024/01/18 15:53:21 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:53:21.823Z] pod name:mysql-7d766cd65c-56bqp namespace:mysql-ibm-pxb-0-79915-01-18-15h52m42s running:true ready:false node:10.240.64.9 [2024-01-18T15:53:21.824Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:53:31.764Z] 2024/01/18 15:53:31 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:53:31.764Z] pod name:mysql-7d766cd65c-56bqp namespace:mysql-ibm-pxb-0-79915-01-18-15h52m42s running:true ready:false node:10.240.64.9 [2024-01-18T15:53:31.764Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:53:43.934Z] 2024/01/18 15:53:41 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:53:43.934Z] pod name:mysql-7d766cd65c-56bqp namespace:mysql-ibm-pxb-0-79915-01-18-15h52m42s running:true ready:false node:10.240.64.9 [2024-01-18T15:53:43.934Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:53:52.017Z] 2024/01/18 15:53:51 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:53:52.017Z] pod name:mysql-7d766cd65c-56bqp namespace:mysql-ibm-pxb-0-79915-01-18-15h52m42s running:true ready:false node:10.240.64.9 [2024-01-18T15:53:52.017Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:54:01.953Z] INFO[2024-01-18 15:54:00] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T15:54:01.953Z] INFO[2024-01-18 15:54:00] sched stats: tick interval: 1200 < 1.2s (100%) [2024-01-18T15:54:01.953Z] INFO[2024-01-18 15:54:00] sched stats: tick process durations: 1199 < 500ms (100%) [2024-01-18T15:54:02.210Z] 2024/01/18 15:54:02 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:54:02.210Z] pod name:mysql-7d766cd65c-56bqp namespace:mysql-ibm-pxb-0-79915-01-18-15h52m42s running:true ready:false node:10.240.64.9 [2024-01-18T15:54:02.210Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:54:14.412Z] 2024/01/18 15:54:12 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:54:14.412Z] pod name:mysql-7d766cd65c-56bqp namespace:mysql-ibm-pxb-0-79915-01-18-15h52m42s running:true ready:false node:10.240.64.9 [2024-01-18T15:54:14.412Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:54:22.502Z] 2024-01-18 15:54:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T15:54:22.502Z] 2024-01-18 15:54:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T15:54:22.757Z] 2024-01-18 15:54:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] validate if mysql-ibm app's volumes are setup [2024-01-18T15:54:22.757Z] 2024-01-18 15:54:22 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T15:54:22.757Z] 2024-01-18 15:54:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T15:54:22.757Z] 2024-01-18 15:54:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T15:54:22.757Z] 2024-01-18 15:54:22 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T15:54:22.757Z] 2024-01-18 15:54:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Register clusters for backup [2024-01-18T15:54:23.015Z] INFO[2024-01-18 15:54:22] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T15:54:23.015Z] INFO[2024-01-18 15:54:22] Actual:true, Expected: true [2024-01-18T15:54:23.015Z] 2024-01-18 15:54:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T15:54:23.015Z] 2024-01-18 15:54:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T15:54:23.015Z] 2024-01-18 15:54:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T15:54:23.015Z] 2024-01-18 15:54:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T15:54:23.015Z] 2024-01-18 15:54:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Create cluster [source-cluster] in org [default] [2024-01-18T15:54:23.015Z] 2024-01-18 15:54:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T15:54:23.015Z] 2024-01-18 15:54:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T15:54:23.015Z] 2024-01-18 15:54:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Create cluster [destination-cluster] in org [default] [2024-01-18T15:54:23.015Z] 2024-01-18 15:54:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T15:54:23.015Z] 2024-01-18 15:54:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T15:54:23.015Z] 2024-01-18 15:54:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-odnat] for IBM [2024-01-18T15:54:23.015Z] 2024-01-18 15:54:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-odnat] for org [default] with [ibm] as provider [2024-01-18T15:54:23.015Z] 2024-01-18 15:54:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T15:54:23.015Z] 2024-01-18 15:54:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T15:54:23.271Z] 2024-01-18 15:54:23 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-odnat] is present [2024-01-18T15:54:23.271Z] 2024-01-18 15:54:23 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Create cluster [source-cluster] in org [default] [2024-01-18T15:54:23.835Z] 2024-01-18 15:54:23 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-sipgz] for IBM [2024-01-18T15:54:23.835Z] 2024-01-18 15:54:23 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-sipgz] for org [default] with [ibm] as provider [2024-01-18T15:54:23.835Z] 2024-01-18 15:54:23 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T15:54:23.835Z] 2024-01-18 15:54:23 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T15:54:23.835Z] 2024-01-18 15:54:23 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-sipgz] is present [2024-01-18T15:54:23.835Z] 2024-01-18 15:54:23 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Create cluster [destination-cluster] in org [default] [2024-01-18T15:54:24.760Z] INFO[2024-01-18 15:54:24] Verifying : Description : Creating source and destination cluster [2024-01-18T15:54:24.760Z] INFO[2024-01-18 15:54:24] Actual:true, Expected: true [2024-01-18T15:54:24.760Z] INFO[2024-01-18 15:54:24] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-18T15:54:24.760Z] INFO[2024-01-18 15:54:24] Actual:Online, Expected: Online [2024-01-18T15:54:24.760Z] INFO[2024-01-18 15:54:24] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-18T15:54:24.760Z] INFO[2024-01-18 15:54:24] Actual:true, Expected: true [2024-01-18T15:54:24.760Z] 2024-01-18 15:54:24 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Taking encrypted and no-encrypted backups [2024-01-18T15:54:24.760Z] 2024-01-18 15:54:24 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T15:54:24.761Z] 2024-01-18 15:54:24 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Volume snapshot classes found in the cluster - [2024-01-18T15:54:24.761Z] 2024-01-18 15:54:24 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] ibmc-vpcblock-snapshot [2024-01-18T15:54:24.761Z] 2024-01-18 15:54:24 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T15:54:25.686Z] 2024/01/18 15:54:25 DoRetryWithTimeout - Error: {backup status for [tp-backup-pdcfvjlwkc] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T15:54:57.706Z] 2024/01/18 15:54:55 DoRetryWithTimeout - Error: {backup status for [tp-backup-pdcfvjlwkc] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T15:55:29.732Z] 2024/01/18 15:55:25 DoRetryWithTimeout - Error: {backup status for [tp-backup-pdcfvjlwkc] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T15:55:56.232Z] 2024-01-18 15:55:55 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.CreateBackup:#313] - Backup [tp-backup-pdcfvjlwkc] created successfully [2024-01-18T15:55:56.232Z] 2024-01-18 15:55:55 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating backup [tp-backup-pdcfvjlwkc] in org [default] [2024-01-18T15:55:56.232Z] 2024-01-18 15:55:55 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-pdcfvjlwkc] [2024-01-18T15:55:56.232Z] 2024-01-18 15:55:55 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-79915-01-18-15h52m42s] in backup [tp-backup-pdcfvjlwkc] [2024-01-18T15:55:56.233Z] 2024-01-18 15:55:55 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-79915-01-18-15h52m42s] in backup [tp-backup-pdcfvjlwkc] [2024-01-18T15:55:56.233Z] 2024-01-18 15:55:55 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T15:55:56.233Z] 2024-01-18 15:55:55 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T15:55:56.233Z] 2024-01-18 15:55:55 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T15:55:56.233Z] 2024-01-18 15:55:55 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-79915-01-18-15h52m42s] are [[mysql-data]] [2024-01-18T15:55:56.233Z] 2024-01-18 15:55:55 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T15:55:56.233Z] 2024-01-18 15:55:55 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating CR cleanup [2024-01-18T15:55:56.233Z] 2024-01-18 15:55:55 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T15:55:56.233Z] 2024-01-18 15:55:55 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] All CRs in [mysql-ibm-pxb-0-79915-01-18-15h52m42s] are [[]] [2024-01-18T15:55:56.233Z] INFO[2024-01-18 15:55:55] Verifying : Description : Creation and Validation of backup [tp-backup-pdcfvjlwkc] [2024-01-18T15:55:56.233Z] INFO[2024-01-18 15:55:55] Actual:true, Expected: true [2024-01-18T15:55:56.233Z] 2024-01-18 15:55:55 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T15:55:56.233Z] 2024-01-18 15:55:55 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Volume snapshot classes found in the cluster - [2024-01-18T15:55:56.233Z] 2024-01-18 15:55:55 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] ibmc-vpcblock-snapshot [2024-01-18T15:55:56.233Z] 2024-01-18 15:55:55 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T15:55:56.492Z] 2024/01/18 15:55:56 DoRetryWithTimeout - Error: {backup status for [encryption-tp-backup-cwllygmmpl] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T15:56:28.539Z] 2024/01/18 15:56:26 DoRetryWithTimeout - Error: {backup status for [encryption-tp-backup-cwllygmmpl] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T15:57:00.607Z] 2024/01/18 15:56:56 DoRetryWithTimeout - Error: {backup status for [encryption-tp-backup-cwllygmmpl] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T15:57:27.122Z] 2024-01-18 15:57:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.CreateBackup:#313] - Backup [encryption-tp-backup-cwllygmmpl] created successfully [2024-01-18T15:57:27.122Z] 2024-01-18 15:57:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating backup [encryption-tp-backup-cwllygmmpl] in org [default] [2024-01-18T15:57:27.122Z] 2024-01-18 15:57:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [encryption-tp-backup-cwllygmmpl] [2024-01-18T15:57:27.122Z] 2024-01-18 15:57:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-79915-01-18-15h52m42s] in backup [encryption-tp-backup-cwllygmmpl] [2024-01-18T15:57:27.122Z] 2024-01-18 15:57:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-79915-01-18-15h52m42s] in backup [encryption-tp-backup-cwllygmmpl] [2024-01-18T15:57:27.122Z] 2024-01-18 15:57:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T15:57:27.122Z] 2024-01-18 15:57:26 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T15:57:27.122Z] 2024-01-18 15:57:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T15:57:27.122Z] 2024-01-18 15:57:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-79915-01-18-15h52m42s] are [[mysql-data]] [2024-01-18T15:57:27.122Z] 2024-01-18 15:57:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T15:57:27.122Z] 2024-01-18 15:57:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating CR cleanup [2024-01-18T15:57:27.122Z] 2024-01-18 15:57:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T15:57:27.122Z] 2024-01-18 15:57:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] All CRs in [mysql-ibm-pxb-0-79915-01-18-15h52m42s] are [[]] [2024-01-18T15:57:27.122Z] INFO[2024-01-18 15:57:26] Verifying : Description : Creation and Validation of backup [encryption-tp-backup-cwllygmmpl] [2024-01-18T15:57:27.122Z] INFO[2024-01-18 15:57:26] Actual:true, Expected: true [2024-01-18T15:57:27.122Z] 2024-01-18 15:57:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Restoring encrypted and non-encrypted backups [2024-01-18T15:57:27.691Z] 2024/01/18 15:57:27 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-pdcfvjlwkc-1705593446] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T15:58:00.890Z] 2024-01-18 15:57:57 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.CreateRestore:#936] - Restore [tp-restore-tp-backup-pdcfvjlwkc-1705593446] created successfully [2024-01-18T15:58:00.891Z] 2024-01-18 15:57:57 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Switching cluster context to cluster [destination-cluster] [2024-01-18T15:58:00.891Z] 2024-01-18 15:57:57 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Switching context to [/tmp/destination-config] [2024-01-18T15:58:00.891Z] 2024-01-18 15:57:57 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:58:00.891Z] 2024-01-18 15:57:57 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:58:00.891Z] 2024-01-18 15:57:57 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:58:00.891Z] 2024-01-18 15:57:57 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:58:00.891Z] 2024-01-18 15:57:57 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:58:00.891Z] 2024-01-18 15:57:58 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:58:00.891Z] 2024-01-18 15:57:58 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:58:00.891Z] 2024-01-18 15:57:58 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:58:00.891Z] 2024-01-18 15:57:59 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:58:00.891Z] 2024-01-18 15:57:59 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T15:58:00.891Z] 2024-01-18 15:57:59 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T15:58:00.891Z] 2024-01-18 15:57:59 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T15:58:00.891Z] 2024-01-18 15:57:59 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T15:58:00.891Z] 2024-01-18 15:57:59 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Switched context to [/tmp/destination-config] [2024-01-18T15:58:00.892Z] 2024-01-18 15:57:59 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-79915-01-18-15h52m42s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-18T15:58:00.892Z] 2024-01-18 15:57:59 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T15:58:00.892Z] 2024-01-18 15:57:59 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T15:58:00.892Z] 2024-01-18 15:57:59 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T15:58:00.892Z] 2024-01-18 15:57:59 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating restore [tp-restore-tp-backup-pdcfvjlwkc-1705593446] in org [default] [2024-01-18T15:58:00.892Z] 2024-01-18 15:57:59 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-restore-tp-backup-pdcfvjlwkc-1705593446] [2024-01-18T15:58:00.892Z] 2024-01-18 15:57:59 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-restore-tp-backup-pdcfvjlwkc-1705593446] [2024-01-18T15:58:00.892Z] 2024-01-18 15:57:59 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-79915-01-18-15h52m42s] in restore [tp-restore-tp-backup-pdcfvjlwkc-1705593446] [2024-01-18T15:58:00.892Z] 2024-01-18 15:57:59 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-79915-01-18-15h52m42s]) was found in restore [tp-restore-tp-backup-pdcfvjlwkc-1705593446], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T15:58:00.892Z] 2024-01-18 15:57:59 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-79915-01-18-15h52m42s]) found in the restore [tp-restore-tp-backup-pdcfvjlwkc-1705593446] was also present on the cluster/namespace [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T15:58:00.892Z] 2024-01-18 15:57:59 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-79915-01-18-15h52m42s]) was found in restore [tp-restore-tp-backup-pdcfvjlwkc-1705593446], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T15:58:00.892Z] 2024-01-18 15:58:00 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-79915-01-18-15h52m42s]) found in the restore [tp-restore-tp-backup-pdcfvjlwkc-1705593446] was also present on the cluster/namespace [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T15:58:00.893Z] 2024-01-18 15:58:00 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-79915-01-18-15h52m42s]) was found in restore [tp-restore-tp-backup-pdcfvjlwkc-1705593446], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T15:58:00.893Z] 2024-01-18 15:58:00 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-79915-01-18-15h52m42s]) found in the restore [tp-restore-tp-backup-pdcfvjlwkc-1705593446] was also present on the cluster/namespace [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T15:58:00.893Z] 2024-01-18 15:58:00 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-79915-01-18-15h52m42s]) was found in restore [tp-restore-tp-backup-pdcfvjlwkc-1705593446], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T15:58:00.893Z] 2024-01-18 15:58:00 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-79915-01-18-15h52m42s]) found in the restore [tp-restore-tp-backup-pdcfvjlwkc-1705593446] was also present on the cluster/namespace [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T15:58:00.893Z] 2024-01-18 15:58:00 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-79915-01-18-15h52m42s] in restore [tp-restore-tp-backup-pdcfvjlwkc-1705593446] [2024-01-18T15:58:00.893Z] 2024-01-18 15:58:00 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-79915-01-18-15h52m42s] in restore [tp-restore-tp-backup-pdcfvjlwkc-1705593446] [2024-01-18T15:58:00.893Z] 2024-01-18 15:58:00 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T15:58:00.893Z] 2024-01-18 15:58:00 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T15:58:00.893Z] 2024-01-18 15:58:00 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-79915-01-18-15h52m42s] are [[mysql-data]] [2024-01-18T15:58:00.893Z] 2024-01-18 15:58:00 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2240] - in restore [tp-restore-tp-backup-pdcfvjlwkc-1705593446], the status of the restored volume [pvc-ef082755-325f-4883-80ab-19465d4921fc] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-18T15:58:00.893Z] 2024-01-18 15:58:00 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validate applications in restored namespace [mysql-ibm-pxb-0-79915-01-18-15h52m42s] due to restore [tp-restore-tp-backup-pdcfvjlwkc-1705593446] [2024-01-18T15:58:00.893Z] 2024-01-18 15:58:00 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating mysql-ibm app [2024-01-18T15:58:00.894Z] 2024-01-18 15:58:00 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] wait for mysql-ibm app to start running [2024-01-18T15:58:00.894Z] 2024-01-18 15:58:00 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T15:58:00.894Z] 2024/01/18 15:58:00 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:58:00.894Z] pod name:mysql-7d766cd65c-99jqc namespace:mysql-ibm-pxb-0-79915-01-18-15h52m42s running:false ready:false node:10.240.64.7 [2024-01-18T15:58:00.894Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:58:10.839Z] 2024/01/18 15:58:10 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:58:10.839Z] pod name:mysql-7d766cd65c-99jqc namespace:mysql-ibm-pxb-0-79915-01-18-15h52m42s running:false ready:false node:10.240.64.7 [2024-01-18T15:58:10.839Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:58:20.778Z] 2024/01/18 15:58:20 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:58:20.778Z] pod name:mysql-7d766cd65c-99jqc namespace:mysql-ibm-pxb-0-79915-01-18-15h52m42s running:false ready:false node:10.240.64.7 [2024-01-18T15:58:20.778Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:58:30.715Z] 2024/01/18 15:58:30 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:58:30.715Z] pod name:mysql-7d766cd65c-99jqc namespace:mysql-ibm-pxb-0-79915-01-18-15h52m42s running:true ready:false node:10.240.64.7 [2024-01-18T15:58:30.715Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:58:42.883Z] 2024/01/18 15:58:40 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:58:42.883Z] pod name:mysql-7d766cd65c-99jqc namespace:mysql-ibm-pxb-0-79915-01-18-15h52m42s running:true ready:false node:10.240.64.7 [2024-01-18T15:58:42.883Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:58:51.004Z] 2024/01/18 15:58:50 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:58:51.004Z] pod name:mysql-7d766cd65c-99jqc namespace:mysql-ibm-pxb-0-79915-01-18-15h52m42s running:true ready:false node:10.240.64.7 [2024-01-18T15:58:51.004Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:59:00.944Z] 2024/01/18 15:59:00 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:59:00.944Z] pod name:mysql-7d766cd65c-99jqc namespace:mysql-ibm-pxb-0-79915-01-18-15h52m42s running:true ready:false node:10.240.64.7 [2024-01-18T15:59:00.944Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:59:13.107Z] 2024/01/18 15:59:10 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:59:13.107Z] pod name:mysql-7d766cd65c-99jqc namespace:mysql-ibm-pxb-0-79915-01-18-15h52m42s running:true ready:false node:10.240.64.7 [2024-01-18T15:59:13.107Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:59:21.203Z] 2024/01/18 15:59:20 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:59:21.203Z] pod name:mysql-7d766cd65c-99jqc namespace:mysql-ibm-pxb-0-79915-01-18-15h52m42s running:true ready:false node:10.240.64.7 [2024-01-18T15:59:21.203Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:59:31.142Z] 2024/01/18 15:59:31 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T15:59:31.142Z] pod name:mysql-7d766cd65c-99jqc namespace:mysql-ibm-pxb-0-79915-01-18-15h52m42s running:true ready:false node:10.240.64.7 [2024-01-18T15:59:31.142Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T15:59:43.327Z] 2024-01-18 15:59:41 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T15:59:43.327Z] 2024-01-18 15:59:41 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T15:59:43.327Z] 2024-01-18 15:59:41 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating CR cleanup [2024-01-18T15:59:43.327Z] 2024-01-18 15:59:41 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T15:59:43.327Z] 2024-01-18 15:59:41 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] All CRs in [mysql-ibm-pxb-0-79915-01-18-15h52m42s] are [[]] [2024-01-18T15:59:43.327Z] 2024-01-18 15:59:41 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Switching cluster context back to cluster path [/tmp/source-config] [2024-01-18T15:59:43.327Z] 2024-01-18 15:59:41 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Switching context to [/tmp/source-config] [2024-01-18T15:59:43.327Z] 2024-01-18 15:59:41 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:59:43.327Z] 2024-01-18 15:59:41 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:59:43.328Z] 2024-01-18 15:59:41 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T15:59:43.328Z] 2024-01-18 15:59:41 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:59:43.328Z] 2024-01-18 15:59:41 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:59:43.328Z] 2024-01-18 15:59:41 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T15:59:43.328Z] 2024-01-18 15:59:42 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:59:43.328Z] 2024-01-18 15:59:42 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:59:43.328Z] 2024-01-18 15:59:43 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T15:59:43.328Z] 2024-01-18 15:59:43 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T15:59:43.587Z] 2024-01-18 15:59:43 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T15:59:43.587Z] 2024-01-18 15:59:43 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T15:59:43.587Z] 2024-01-18 15:59:43 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T15:59:43.587Z] 2024-01-18 15:59:43 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Switched context to [/tmp/source-config] [2024-01-18T16:04:05.080Z] INFO[2024-01-18 16:04:01] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T16:04:05.080Z] INFO[2024-01-18 16:04:01] sched stats: tick interval: 1801 < 1.2s (100%) [2024-01-18T16:04:05.080Z] INFO[2024-01-18 16:04:01] sched stats: tick process durations: 1800 < 500ms (100%) [2024-01-18T16:04:51.717Z] 2024/01/18 16:04:43 DoRetryWithTimeout - Error: {restore status for [tp-restore-encryption-tp-backup-cwllygmmpl] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T16:05:18.214Z] 2024-01-18 16:05:13 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.CreateRestore:#936] - Restore [tp-restore-encryption-tp-backup-cwllygmmpl] created successfully [2024-01-18T16:05:18.214Z] 2024-01-18 16:05:13 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Switching cluster context to cluster [destination-cluster] [2024-01-18T16:05:18.214Z] 2024-01-18 16:05:13 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Switching context to [/tmp/destination-config] [2024-01-18T16:05:18.214Z] 2024-01-18 16:05:14 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:05:18.214Z] 2024-01-18 16:05:14 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:05:18.214Z] 2024-01-18 16:05:14 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:05:18.214Z] 2024-01-18 16:05:14 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:05:18.214Z] 2024-01-18 16:05:14 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:05:18.215Z] 2024-01-18 16:05:14 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:05:18.215Z] 2024-01-18 16:05:14 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:05:18.215Z] 2024-01-18 16:05:15 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:05:18.215Z] 2024-01-18 16:05:15 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:05:18.215Z] 2024-01-18 16:05:15 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:05:18.215Z] 2024-01-18 16:05:15 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:05:18.215Z] 2024-01-18 16:05:15 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:05:18.215Z] 2024-01-18 16:05:15 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:05:18.226Z] 2024-01-18 16:05:15 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Switched context to [/tmp/destination-config] [2024-01-18T16:05:18.226Z] 2024-01-18 16:05:15 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-79915-01-18-15h52m42s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-18T16:05:18.226Z] 2024-01-18 16:05:15 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T16:05:18.226Z] 2024-01-18 16:05:15 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T16:05:18.226Z] 2024-01-18 16:05:15 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T16:05:18.226Z] 2024-01-18 16:05:15 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating restore [tp-restore-encryption-tp-backup-cwllygmmpl] in org [default] [2024-01-18T16:05:18.226Z] 2024-01-18 16:05:15 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-restore-encryption-tp-backup-cwllygmmpl] [2024-01-18T16:05:18.226Z] 2024-01-18 16:05:15 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-restore-encryption-tp-backup-cwllygmmpl] [2024-01-18T16:05:18.226Z] 2024-01-18 16:05:15 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-79915-01-18-15h52m42s] in restore [tp-restore-encryption-tp-backup-cwllygmmpl] [2024-01-18T16:05:18.227Z] 2024-01-18 16:05:15 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-79915-01-18-15h52m42s]) was found in restore [tp-restore-encryption-tp-backup-cwllygmmpl], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T16:05:18.227Z] 2024-01-18 16:05:16 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-79915-01-18-15h52m42s]) found in the restore [tp-restore-encryption-tp-backup-cwllygmmpl] was also present on the cluster/namespace [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T16:05:18.227Z] 2024-01-18 16:05:16 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-79915-01-18-15h52m42s]) was found in restore [tp-restore-encryption-tp-backup-cwllygmmpl], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T16:05:18.228Z] 2024-01-18 16:05:16 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-79915-01-18-15h52m42s]) found in the restore [tp-restore-encryption-tp-backup-cwllygmmpl] was also present on the cluster/namespace [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T16:05:18.228Z] 2024-01-18 16:05:16 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-79915-01-18-15h52m42s]) was found in restore [tp-restore-encryption-tp-backup-cwllygmmpl], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T16:05:18.228Z] 2024-01-18 16:05:16 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-79915-01-18-15h52m42s]) found in the restore [tp-restore-encryption-tp-backup-cwllygmmpl] was also present on the cluster/namespace [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T16:05:18.228Z] 2024-01-18 16:05:16 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-79915-01-18-15h52m42s]) was found in restore [tp-restore-encryption-tp-backup-cwllygmmpl], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T16:05:18.228Z] 2024-01-18 16:05:16 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-79915-01-18-15h52m42s]) found in the restore [tp-restore-encryption-tp-backup-cwllygmmpl] was also present on the cluster/namespace [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T16:05:18.228Z] 2024-01-18 16:05:16 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-79915-01-18-15h52m42s] in restore [tp-restore-encryption-tp-backup-cwllygmmpl] [2024-01-18T16:05:18.228Z] 2024-01-18 16:05:16 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-79915-01-18-15h52m42s] in restore [tp-restore-encryption-tp-backup-cwllygmmpl] [2024-01-18T16:05:18.228Z] 2024-01-18 16:05:16 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:05:18.228Z] 2024-01-18 16:05:16 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:05:18.228Z] 2024-01-18 16:05:16 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-79915-01-18-15h52m42s] are [[mysql-data]] [2024-01-18T16:05:18.228Z] 2024-01-18 16:05:16 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2243] - in restore [tp-restore-encryption-tp-backup-cwllygmmpl], the status of the restored volume [pvc-ef082755-325f-4883-80ab-19465d4921fc] was not Success. It was [Retained] with reason [Skipped from volume restore as policy is set to Retain and pvc already exists] [2024-01-18T16:05:18.228Z] 2024-01-18 16:05:16 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validate applications in restored namespace [mysql-ibm-pxb-0-79915-01-18-15h52m42s] due to restore [tp-restore-encryption-tp-backup-cwllygmmpl] [2024-01-18T16:05:18.228Z] 2024-01-18 16:05:16 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating mysql-ibm app [2024-01-18T16:05:18.228Z] 2024-01-18 16:05:16 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] wait for mysql-ibm app to start running [2024-01-18T16:05:18.228Z] 2024-01-18 16:05:16 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T16:05:18.228Z] 2024-01-18 16:05:16 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T16:05:18.228Z] 2024-01-18 16:05:16 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T16:05:18.228Z] 2024-01-18 16:05:17 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating CR cleanup [2024-01-18T16:05:18.229Z] 2024-01-18 16:05:17 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:05:18.229Z] 2024-01-18 16:05:17 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] All CRs in [mysql-ibm-pxb-0-79915-01-18-15h52m42s] are [[]] [2024-01-18T16:05:18.229Z] 2024-01-18 16:05:17 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Switching cluster context back to cluster path [/tmp/source-config] [2024-01-18T16:05:18.229Z] 2024-01-18 16:05:17 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Switching context to [/tmp/source-config] [2024-01-18T16:05:18.229Z] 2024-01-18 16:05:17 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:05:18.229Z] 2024-01-18 16:05:17 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:05:18.229Z] 2024-01-18 16:05:17 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:05:18.229Z] 2024-01-18 16:05:17 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:05:18.229Z] 2024-01-18 16:05:17 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:05:18.229Z] 2024-01-18 16:05:17 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:05:18.485Z] 2024-01-18 16:05:18 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:05:18.741Z] 2024-01-18 16:05:18 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:05:19.301Z] 2024-01-18 16:05:19 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:05:19.556Z] 2024-01-18 16:05:19 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:05:19.556Z] 2024-01-18 16:05:19 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:05:19.556Z] 2024-01-18 16:05:19 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:05:19.556Z] 2024-01-18 16:05:19 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:05:19.556Z] 2024-01-18 16:05:19 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Switched context to [/tmp/source-config] [2024-01-18T16:05:19.556Z] 2024-01-18 16:05:19 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Taking new encrypted backup [2024-01-18T16:05:20.922Z] 2024-01-18 16:05:20 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T16:05:21.177Z] 2024-01-18 16:05:20 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Volume snapshot classes found in the cluster - [2024-01-18T16:05:21.177Z] 2024-01-18 16:05:20 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] ibmc-vpcblock-snapshot [2024-01-18T16:05:21.177Z] 2024-01-18 16:05:20 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T16:05:21.738Z] 2024/01/18 16:05:21 DoRetryWithTimeout - Error: {backup status for [new-encrypted-tp-backup-aenvlrjeih] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T16:05:53.764Z] 2024/01/18 16:05:51 DoRetryWithTimeout - Error: {backup status for [new-encrypted-tp-backup-aenvlrjeih] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:06:25.816Z] 2024/01/18 16:06:21 DoRetryWithTimeout - Error: {backup status for [new-encrypted-tp-backup-aenvlrjeih] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:06:52.541Z] 2024-01-18 16:06:51 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.CreateBackup:#313] - Backup [new-encrypted-tp-backup-aenvlrjeih] created successfully [2024-01-18T16:06:52.541Z] 2024-01-18 16:06:51 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating backup [new-encrypted-tp-backup-aenvlrjeih] in org [default] [2024-01-18T16:06:52.541Z] 2024-01-18 16:06:51 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [new-encrypted-tp-backup-aenvlrjeih] [2024-01-18T16:06:52.541Z] 2024-01-18 16:06:51 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-79915-01-18-15h52m42s] in backup [new-encrypted-tp-backup-aenvlrjeih] [2024-01-18T16:06:52.541Z] 2024-01-18 16:06:51 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-79915-01-18-15h52m42s] in backup [new-encrypted-tp-backup-aenvlrjeih] [2024-01-18T16:06:52.541Z] 2024-01-18 16:06:51 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T16:06:52.541Z] 2024-01-18 16:06:51 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:06:52.541Z] 2024-01-18 16:06:51 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:06:52.542Z] 2024-01-18 16:06:51 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-79915-01-18-15h52m42s] are [[mysql-data]] [2024-01-18T16:06:52.542Z] 2024-01-18 16:06:51 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T16:06:52.542Z] 2024-01-18 16:06:51 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating CR cleanup [2024-01-18T16:06:52.542Z] 2024-01-18 16:06:51 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:06:52.542Z] 2024-01-18 16:06:52 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] All CRs in [mysql-ibm-pxb-0-79915-01-18-15h52m42s] are [[]] [2024-01-18T16:06:52.542Z] INFO[2024-01-18 16:06:52] Verifying : Description : Creation and Validation of new encrypted backup [new-encrypted-tp-backup-aenvlrjeih] [2024-01-18T16:06:52.542Z] INFO[2024-01-18 16:06:52] Actual:true, Expected: true [2024-01-18T16:06:52.542Z] 2024-01-18 16:06:52 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validate the encrypted backup location after taking backup [2024-01-18T16:06:53.470Z] 2024-01-18 16:06:53 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Restore the encrypted backups after validating the encrypted backup location [2024-01-18T16:06:53.725Z] 2024/01/18 16:06:53 DoRetryWithTimeout - Error: {restore status for [tp-restore-new-encrypted-tp-backup-aenvlrjeih-encrypted] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T16:07:25.760Z] 2024-01-18 16:07:23 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.CreateRestore:#936] - Restore [tp-restore-new-encrypted-tp-backup-aenvlrjeih-encrypted] created successfully [2024-01-18T16:07:25.760Z] 2024-01-18 16:07:23 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Switching cluster context to cluster [destination-cluster] [2024-01-18T16:07:25.760Z] 2024-01-18 16:07:23 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Switching context to [/tmp/destination-config] [2024-01-18T16:07:25.760Z] 2024-01-18 16:07:23 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:07:25.760Z] 2024-01-18 16:07:23 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:07:25.760Z] 2024-01-18 16:07:23 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:07:25.760Z] 2024-01-18 16:07:23 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:07:25.760Z] 2024-01-18 16:07:23 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:07:25.760Z] 2024-01-18 16:07:24 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:07:25.761Z] 2024-01-18 16:07:24 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:07:25.761Z] 2024-01-18 16:07:24 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:07:25.761Z] 2024-01-18 16:07:25 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:07:25.761Z] 2024-01-18 16:07:25 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:07:25.761Z] 2024-01-18 16:07:25 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:07:25.761Z] 2024-01-18 16:07:25 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:07:25.761Z] 2024-01-18 16:07:25 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:07:25.761Z] 2024-01-18 16:07:25 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Switched context to [/tmp/destination-config] [2024-01-18T16:07:25.761Z] 2024-01-18 16:07:25 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-79915-01-18-15h52m42s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-18T16:07:25.761Z] 2024-01-18 16:07:25 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T16:07:25.761Z] 2024-01-18 16:07:25 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T16:07:25.761Z] 2024-01-18 16:07:25 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T16:07:25.761Z] 2024-01-18 16:07:25 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating restore [tp-restore-new-encrypted-tp-backup-aenvlrjeih-encrypted] in org [default] [2024-01-18T16:07:25.761Z] 2024-01-18 16:07:25 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-restore-new-encrypted-tp-backup-aenvlrjeih-encrypted] [2024-01-18T16:07:25.761Z] 2024-01-18 16:07:25 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-restore-new-encrypted-tp-backup-aenvlrjeih-encrypted] [2024-01-18T16:07:25.761Z] 2024-01-18 16:07:25 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-79915-01-18-15h52m42s] in restore [tp-restore-new-encrypted-tp-backup-aenvlrjeih-encrypted] [2024-01-18T16:07:25.761Z] 2024-01-18 16:07:25 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-79915-01-18-15h52m42s]) was found in restore [tp-restore-new-encrypted-tp-backup-aenvlrjeih-encrypted], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T16:07:26.020Z] 2024-01-18 16:07:25 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-79915-01-18-15h52m42s]) found in the restore [tp-restore-new-encrypted-tp-backup-aenvlrjeih-encrypted] was also present on the cluster/namespace [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T16:07:26.020Z] 2024-01-18 16:07:25 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-79915-01-18-15h52m42s]) was found in restore [tp-restore-new-encrypted-tp-backup-aenvlrjeih-encrypted], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T16:07:26.276Z] 2024-01-18 16:07:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-79915-01-18-15h52m42s]) found in the restore [tp-restore-new-encrypted-tp-backup-aenvlrjeih-encrypted] was also present on the cluster/namespace [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T16:07:26.277Z] 2024-01-18 16:07:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-79915-01-18-15h52m42s]) was found in restore [tp-restore-new-encrypted-tp-backup-aenvlrjeih-encrypted], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T16:07:26.277Z] 2024-01-18 16:07:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-79915-01-18-15h52m42s]) found in the restore [tp-restore-new-encrypted-tp-backup-aenvlrjeih-encrypted] was also present on the cluster/namespace [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T16:07:26.277Z] 2024-01-18 16:07:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-79915-01-18-15h52m42s]) was found in restore [tp-restore-new-encrypted-tp-backup-aenvlrjeih-encrypted], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T16:07:26.277Z] 2024-01-18 16:07:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-79915-01-18-15h52m42s]) found in the restore [tp-restore-new-encrypted-tp-backup-aenvlrjeih-encrypted] was also present on the cluster/namespace [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T16:07:26.277Z] 2024-01-18 16:07:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-79915-01-18-15h52m42s] in restore [tp-restore-new-encrypted-tp-backup-aenvlrjeih-encrypted] [2024-01-18T16:07:26.277Z] 2024-01-18 16:07:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-79915-01-18-15h52m42s] in restore [tp-restore-new-encrypted-tp-backup-aenvlrjeih-encrypted] [2024-01-18T16:07:26.534Z] 2024-01-18 16:07:26 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:07:26.534Z] 2024-01-18 16:07:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:07:26.534Z] 2024-01-18 16:07:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-79915-01-18-15h52m42s] are [[mysql-data]] [2024-01-18T16:07:26.534Z] 2024-01-18 16:07:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.ValidateRestore:#2243] - in restore [tp-restore-new-encrypted-tp-backup-aenvlrjeih-encrypted], the status of the restored volume [pvc-ef082755-325f-4883-80ab-19465d4921fc] was not Success. It was [Retained] with reason [Skipped from volume restore as policy is set to Retain and pvc already exists] [2024-01-18T16:07:26.534Z] 2024-01-18 16:07:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validate applications in restored namespace [mysql-ibm-pxb-0-79915-01-18-15h52m42s] due to restore [tp-restore-new-encrypted-tp-backup-aenvlrjeih-encrypted] [2024-01-18T16:07:26.534Z] 2024-01-18 16:07:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating mysql-ibm app [2024-01-18T16:07:26.534Z] 2024-01-18 16:07:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] wait for mysql-ibm app to start running [2024-01-18T16:07:26.534Z] 2024-01-18 16:07:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T16:07:26.790Z] 2024-01-18 16:07:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T16:07:26.790Z] 2024-01-18 16:07:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T16:07:27.047Z] 2024-01-18 16:07:26 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Validating CR cleanup [2024-01-18T16:07:27.304Z] 2024-01-18 16:07:27 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:07:27.304Z] 2024-01-18 16:07:27 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] All CRs in [mysql-ibm-pxb-0-79915-01-18-15h52m42s] are [[]] [2024-01-18T16:07:27.304Z] 2024-01-18 16:07:27 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Switching cluster context back to cluster path [/tmp/source-config] [2024-01-18T16:07:27.304Z] 2024-01-18 16:07:27 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Switching context to [/tmp/source-config] [2024-01-18T16:07:27.304Z] 2024-01-18 16:07:27 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:07:27.304Z] 2024-01-18 16:07:27 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:07:27.561Z] 2024-01-18 16:07:27 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:07:27.561Z] 2024-01-18 16:07:27 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:07:27.561Z] 2024-01-18 16:07:27 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:07:27.816Z] 2024-01-18 16:07:27 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:07:28.073Z] 2024-01-18 16:07:28 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:07:28.637Z] 2024-01-18 16:07:28 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:07:28.892Z] 2024-01-18 16:07:28 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:07:29.148Z] 2024-01-18 16:07:28 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:07:29.403Z] 2024-01-18 16:07:29 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:07:29.403Z] 2024-01-18 16:07:29 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:07:29.403Z] 2024-01-18 16:07:29 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:07:29.403Z] 2024-01-18 16:07:29 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Switched context to [/tmp/source-config] [2024-01-18T16:07:29.403Z] 2024-01-18 16:07:29 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Deleting Restores, Backups and Backup locations, cloud account [2024-01-18T16:07:30.771Z] INFO[2024-01-18 16:07:30] Verifying : Description : Getting the backup driver [2024-01-18T16:07:30.771Z] INFO[2024-01-18 16:07:30] Actual:true, Expected: true [2024-01-18T16:07:30.771Z] INFO[2024-01-18 16:07:30] Verifying : Description : Deleting Restore tp-restore-tp-backup-pdcfvjlwkc-1705593446 [2024-01-18T16:07:30.771Z] INFO[2024-01-18 16:07:30] Actual:true, Expected: true [2024-01-18T16:07:30.771Z] INFO[2024-01-18 16:07:30] Verifying : Description : Getting the backup driver [2024-01-18T16:07:30.771Z] INFO[2024-01-18 16:07:30] Actual:true, Expected: true [2024-01-18T16:07:30.771Z] INFO[2024-01-18 16:07:30] Verifying : Description : Deleting Restore tp-restore-encryption-tp-backup-cwllygmmpl [2024-01-18T16:07:30.771Z] INFO[2024-01-18 16:07:30] Actual:true, Expected: true [2024-01-18T16:07:30.771Z] INFO[2024-01-18 16:07:30] Verifying : Description : Getting the backup driver [2024-01-18T16:07:30.771Z] INFO[2024-01-18 16:07:30] Actual:true, Expected: true [2024-01-18T16:07:30.771Z] INFO[2024-01-18 16:07:30] Verifying : Description : Deleting Restore tp-restore-new-encrypted-tp-backup-aenvlrjeih-encrypted [2024-01-18T16:07:30.771Z] INFO[2024-01-18 16:07:30] Actual:true, Expected: true [2024-01-18T16:07:32.138Z] INFO[2024-01-18 16:07:32] Verifying : Description : Getting backup UID for backup tp-backup-pdcfvjlwkc [2024-01-18T16:07:32.138Z] INFO[2024-01-18 16:07:32] Actual:true, Expected: true [2024-01-18T16:07:32.138Z] INFO[2024-01-18 16:07:32] Verifying : Description : Deleting backup tp-backup-pdcfvjlwkc [2024-01-18T16:07:32.138Z] INFO[2024-01-18 16:07:32] Actual:true, Expected: true [2024-01-18T16:07:32.138Z] INFO[2024-01-18 16:07:32] Verifying : Description : Getting backup UID for backup encryption-tp-backup-cwllygmmpl [2024-01-18T16:07:32.138Z] INFO[2024-01-18 16:07:32] Actual:true, Expected: true [2024-01-18T16:07:32.396Z] INFO[2024-01-18 16:07:32] Verifying : Description : Deleting backup encryption-tp-backup-cwllygmmpl [2024-01-18T16:07:32.396Z] INFO[2024-01-18 16:07:32] Actual:true, Expected: true [2024-01-18T16:07:32.396Z] INFO[2024-01-18 16:07:32] Verifying : Description : Getting backup UID for backup new-encrypted-tp-backup-aenvlrjeih [2024-01-18T16:07:32.396Z] INFO[2024-01-18 16:07:32] Actual:true, Expected: true [2024-01-18T16:07:32.396Z] INFO[2024-01-18 16:07:32] Verifying : Description : Deleting backup new-encrypted-tp-backup-aenvlrjeih [2024-01-18T16:07:32.396Z] INFO[2024-01-18 16:07:32] Actual:true, Expected: true [2024-01-18T16:07:32.396Z] 2024-01-18 16:07:32 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Cleaning backup locations in map [map[209b20ea-941c-4f85-b2c0-bf7411e30606:encryption-location-aws-1705593163 83cab118-6743-4021-be63-e32ce482ee5a:location-aws]], cloud credential [cred-aws-1705593163], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:07:32.397Z] INFO[2024-01-18 16:07:32] Verifying : Description : Verifying deletion of backup location [location-aws] [2024-01-18T16:07:32.397Z] INFO[2024-01-18 16:07:32] Actual:true, Expected: true [2024-01-18T16:07:32.652Z] 2024-01-18 16:07:32 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.IsBackupLocationPresent:#2442] - Backup location [location-aws] is present [2024-01-18T16:07:32.652Z] 2024/01/18 16:07:32 DoRetryWithTimeout - Error: {backup location location-aws is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T16:08:04.692Z] 2024-01-18 16:08:02 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.IsBackupLocationPresent:#2442] - Backup location [location-aws] is present [2024-01-18T16:08:04.692Z] 2024/01/18 16:08:02 DoRetryWithTimeout - Error: {backup location location-aws is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T16:08:36.777Z] 2024-01-18 16:08:32 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu encryption-location-aws-1705593163] [2024-01-18T16:08:36.777Z] INFO[2024-01-18 16:08:32] Verifying : Description : Verifying backup location deletion status location-aws [2024-01-18T16:08:36.777Z] INFO[2024-01-18 16:08:32] Actual:true, Expected: true [2024-01-18T16:08:36.777Z] INFO[2024-01-18 16:08:32] Verifying : Description : Verifying deletion of backup location [encryption-location-aws-1705593163] [2024-01-18T16:08:36.777Z] INFO[2024-01-18 16:08:32] Actual:true, Expected: true [2024-01-18T16:08:36.777Z] 2024-01-18 16:08:32 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.IsBackupLocationPresent:#2442] - Backup location [encryption-location-aws-1705593163] is present [2024-01-18T16:08:36.777Z] 2024/01/18 16:08:32 DoRetryWithTimeout - Error: {backup location encryption-location-aws-1705593163 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T16:09:03.328Z] 2024-01-18 16:09:02 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.IsBackupLocationPresent:#2442] - Backup location [encryption-location-aws-1705593163] is present [2024-01-18T16:09:03.328Z] 2024/01/18 16:09:02 DoRetryWithTimeout - Error: {backup location encryption-location-aws-1705593163 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T16:09:35.367Z] 2024-01-18 16:09:32 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu] [2024-01-18T16:09:35.367Z] INFO[2024-01-18 16:09:32] Verifying : Description : Verifying backup location deletion status encryption-location-aws-1705593163 [2024-01-18T16:09:35.367Z] INFO[2024-01-18 16:09:32] Actual:true, Expected: true [2024-01-18T16:09:35.367Z] 2024-01-18 16:09:32 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cred-aws-1705593163] is present [2024-01-18T16:09:35.367Z] INFO[2024-01-18 16:09:32] Verifying : Description : Verifying if cloud cred [cred-aws-1705593163] is present [2024-01-18T16:09:35.367Z] INFO[2024-01-18 16:09:32] Actual:true, Expected: true [2024-01-18T16:09:35.367Z] INFO[2024-01-18 16:09:33] Verifying : Description : Verifying deletion of cloud cred [cred-aws-1705593163] [2024-01-18T16:09:35.367Z] INFO[2024-01-18 16:09:33] Actual:true, Expected: true [2024-01-18T16:09:35.368Z] INFO[2024-01-18 16:09:33] Verifying : Description : Deleting cloud cred cred-aws-1705593163 [2024-01-18T16:09:35.368Z] INFO[2024-01-18 16:09:33] Actual:true, Expected: true [2024-01-18T16:09:35.368Z] INFO[2024-01-18 16:09:33] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:09:35.368Z] INFO[2024-01-18 16:09:33] Actual:true, Expected: true [2024-01-18T16:09:35.368Z] INFO[2024-01-18 16:09:33] Verifying : Description : Deleting cluster source-cluster [2024-01-18T16:09:35.368Z] INFO[2024-01-18 16:09:33] Actual:true, Expected: true [2024-01-18T16:09:35.368Z] INFO[2024-01-18 16:09:33] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-odnat] [2024-01-18T16:09:35.368Z] INFO[2024-01-18 16:09:33] Actual:true, Expected: true [2024-01-18T16:09:35.368Z] INFO[2024-01-18 16:09:33] Verifying : Description : Deleting cluster destination-cluster [2024-01-18T16:09:35.368Z] INFO[2024-01-18 16:09:33] Actual:true, Expected: true [2024-01-18T16:09:35.368Z] INFO[2024-01-18 16:09:34] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-sipgz] [2024-01-18T16:09:35.368Z] INFO[2024-01-18 16:09:34] Actual:true, Expected: true [2024-01-18T16:09:35.368Z] INFO[2024-01-18 16:09:35] --------Test End------ [2024-01-18T16:09:35.368Z] INFO[2024-01-18 16:09:35] #Test: [2024-01-18T16:09:35.368Z] INFO[2024-01-18 16:09:35] #Description: [2024-01-18T16:09:35.368Z] INFO[2024-01-18 16:09:35] ------------------------ [2024-01-18T16:09:35.368Z] 2024-01-18 16:09:35 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T16:09:35.368Z] 2024-01-18 16:09:35 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T16:09:35.368Z] 2024-01-18 16:09:35 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Switching context to [/tmp/destination-config] [2024-01-18T16:09:35.368Z] 2024-01-18 16:09:35 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:09:35.625Z] 2024-01-18 16:09:35 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:09:35.625Z] 2024-01-18 16:09:35 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:09:35.625Z] 2024-01-18 16:09:35 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:09:35.625Z] 2024-01-18 16:09:35 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:09:35.882Z] 2024-01-18 16:09:35 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:09:36.139Z] 2024-01-18 16:09:36 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:09:36.701Z] 2024-01-18 16:09:36 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:09:36.957Z] 2024-01-18 16:09:36 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:09:37.213Z] 2024-01-18 16:09:37 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:09:37.472Z] 2024-01-18 16:09:37 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:09:37.472Z] 2024-01-18 16:09:37 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:09:37.472Z] 2024-01-18 16:09:37 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:09:37.472Z] 2024-01-18 16:09:37 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Switched context to [/tmp/destination-config] [2024-01-18T16:09:37.728Z] 2024-01-18 16:09:37 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-79915-01-18-15h52m42s] [2024-01-18T16:09:37.728Z] 2024-01-18 16:09:37 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {RestoreEncryptedAndNonEncryptedBackups} Restore encrypted and non encrypted backups [2024-01-18T16:09:37.728Z] 2024-01-18 16:09:37 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-18T16:09:37.728Z] 2024/01/18 16:09:37 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:09:43.023Z] 2024/01/18 16:09:42 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:09:48.270Z] 2024/01/18 16:09:47 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:09:53.559Z] 2024/01/18 16:09:52 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:09:57.738Z] 2024/01/18 16:09:57 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:10:02.997Z] 2024/01/18 16:10:02 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:10:08.287Z] 2024/01/18 16:10:07 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:10:13.533Z] 2024/01/18 16:10:12 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:10:17.708Z] 2024/01/18 16:10:17 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.0.4. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.4 10.240.0.4]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T16:10:17.708Z] {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T16:10:17.708Z] {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T16:10:17.708Z] {dial tcp 10.240.0.4:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-18T16:10:22.969Z] 2024-01-18 16:10:22 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.4. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.4 10.240.0.4]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T16:10:22.969Z] {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T16:10:22.969Z] {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T16:10:22.969Z] {dial tcp 10.240.0.4:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T16:10:22.970Z] 2024-01-18 16:10:22 +0000:[ERROR] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.4. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.4 10.240.0.4]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T16:10:22.970Z] {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T16:10:22.970Z] {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T16:10:22.970Z] {dial tcp 10.240.0.4:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T16:10:22.970Z] 2024-01-18 16:10:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T16:10:22.970Z] 2024-01-18 16:10:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Switching context to [/tmp/source-config] [2024-01-18T16:10:22.970Z] 2024-01-18 16:10:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:10:22.970Z] 2024-01-18 16:10:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:10:22.970Z] 2024-01-18 16:10:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:10:22.970Z] 2024-01-18 16:10:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:10:22.970Z] 2024-01-18 16:10:22 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:10:23.227Z] 2024-01-18 16:10:23 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:10:23.790Z] 2024-01-18 16:10:23 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:10:24.046Z] 2024-01-18 16:10:23 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:10:24.608Z] 2024-01-18 16:10:24 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:10:24.608Z] 2024-01-18 16:10:24 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:10:24.865Z] 2024-01-18 16:10:24 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:10:24.865Z] 2024-01-18 16:10:24 +0000:[WARNING] [{RestoreEncryptedAndNonEncryptedBackups}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:10:24.866Z] 2024-01-18 16:10:24 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:10:24.866Z] 2024-01-18 16:10:24 +0000:[INFO] [{RestoreEncryptedAndNonEncryptedBackups}] Switched context to [/tmp/source-config] [2024-01-18T16:10:24.866Z] [2024-01-18T16:10:24.866Z] ------------------------------ [2024-01-18T16:10:24.866Z] • [SLOW TEST:1061.849 seconds] [2024-01-18T16:10:24.866Z] {RestoreEncryptedAndNonEncryptedBackups} [2024-01-18T16:10:24.866Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_portworx_test.go:481 [2024-01-18T16:10:24.866Z] Restore encrypted and non encrypted backups [2024-01-18T16:10:24.866Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_portworx_test.go:520 [2024-01-18T16:10:24.866Z] ------------------------------ [2024-01-18T16:10:24.866Z] 2024-01-18 16:10:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Log Dir: /testresults//ShareBackupWithUsersAndGroups.log [2024-01-18T16:10:24.866Z] INFO[2024-01-18 16:10:24] --------Test Start------ [2024-01-18T16:10:24.866Z] INFO[2024-01-18 16:10:24] #Test: ShareBackupWithUsersAndGroups [2024-01-18T16:10:24.866Z] INFO[2024-01-18 16:10:24] #Description: Share large number of backups with multiple users and groups with View only, Restore and Full Access [2024-01-18T16:10:24.866Z] INFO[2024-01-18 16:10:24] ------------------------ [2024-01-18T16:10:24.866Z] 2024-01-18 16:10:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Deploy applications [2024-01-18T16:10:24.866Z] 2024-01-18 16:10:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T16:10:24.866Z] 2024-01-18 16:10:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T16:10:25.124Z] 2024-01-18 16:10:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T16:10:25.124Z] 2024-01-18 16:10:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T16:10:25.380Z] 2024-01-18 16:10:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T16:10:25.640Z] 2024-01-18 16:10:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T16:10:25.641Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T16:10:25.641Z] 2024-01-18 16:10:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T16:10:25.642Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T16:10:25.642Z] 2024-01-18 16:10:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T16:10:25.642Z] 2024-01-18 16:10:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validate applications [2024-01-18T16:10:25.642Z] 2024-01-18 16:10:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validate applications [2024-01-18T16:10:25.642Z] 2024-01-18 16:10:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating mysql-ibm app [2024-01-18T16:10:25.642Z] 2024-01-18 16:10:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating mysql-ibm app's volumes [2024-01-18T16:10:25.642Z] 2024-01-18 16:10:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:10:25.899Z] 2024/01/18 16:10:25 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T16:10:35.842Z] 2024/01/18 16:10:35 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T16:10:45.791Z] 2024-01-18 16:10:45 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-82934-01-18-16h10m24s [2024-01-18T16:10:46.048Z] 2024-01-18 16:10:45 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T16:10:46.048Z] 2024-01-18 16:10:45 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T16:10:46.048Z] 2024-01-18 16:10:45 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] wait for mysql-ibm app to start running [2024-01-18T16:10:46.048Z] 2024-01-18 16:10:45 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T16:10:46.048Z] 2024/01/18 16:10:45 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:10:46.048Z] pod name:mysql-7d766cd65c-5sjj5 namespace:mysql-ibm-pxb-0-82934-01-18-16h10m24s running:false ready:false node:10.240.64.9 [2024-01-18T16:10:46.048Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:10:58.216Z] 2024/01/18 16:10:56 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:10:58.216Z] pod name:mysql-7d766cd65c-5sjj5 namespace:mysql-ibm-pxb-0-82934-01-18-16h10m24s running:true ready:false node:10.240.64.9 [2024-01-18T16:10:58.216Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:11:06.330Z] 2024/01/18 16:11:06 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:11:06.330Z] pod name:mysql-7d766cd65c-5sjj5 namespace:mysql-ibm-pxb-0-82934-01-18-16h10m24s running:true ready:false node:10.240.64.9 [2024-01-18T16:11:06.330Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:11:16.280Z] 2024/01/18 16:11:16 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:11:16.280Z] pod name:mysql-7d766cd65c-5sjj5 namespace:mysql-ibm-pxb-0-82934-01-18-16h10m24s running:true ready:false node:10.240.64.9 [2024-01-18T16:11:16.280Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:11:28.493Z] 2024/01/18 16:11:26 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:11:28.493Z] pod name:mysql-7d766cd65c-5sjj5 namespace:mysql-ibm-pxb-0-82934-01-18-16h10m24s running:true ready:false node:10.240.64.9 [2024-01-18T16:11:28.493Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:11:36.594Z] 2024/01/18 16:11:36 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:11:36.594Z] pod name:mysql-7d766cd65c-5sjj5 namespace:mysql-ibm-pxb-0-82934-01-18-16h10m24s running:true ready:false node:10.240.64.9 [2024-01-18T16:11:36.594Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:11:46.544Z] 2024/01/18 16:11:46 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:11:46.544Z] pod name:mysql-7d766cd65c-5sjj5 namespace:mysql-ibm-pxb-0-82934-01-18-16h10m24s running:true ready:false node:10.240.64.9 [2024-01-18T16:11:46.544Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:11:58.715Z] 2024/01/18 16:11:56 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:11:58.715Z] pod name:mysql-7d766cd65c-5sjj5 namespace:mysql-ibm-pxb-0-82934-01-18-16h10m24s running:true ready:false node:10.240.64.9 [2024-01-18T16:11:58.715Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:12:06.861Z] 2024-01-18 16:12:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T16:12:06.861Z] 2024-01-18 16:12:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T16:12:06.861Z] 2024-01-18 16:12:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] validate if mysql-ibm app's volumes are setup [2024-01-18T16:12:06.861Z] 2024-01-18 16:12:06 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:12:06.862Z] 2024-01-18 16:12:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:12:06.862Z] 2024-01-18 16:12:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T16:12:06.862Z] 2024-01-18 16:12:06 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T16:12:07.117Z] 2024-01-18 16:12:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Creating 30 users [2024-01-18T16:12:08.044Z] I0118 16:12:07.993631 14 request.go:682] Waited for 1.063244561s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/services [2024-01-18T16:12:20.212Z] I0118 16:12:18.194060 14 request.go:682] Waited for 5.983537376s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/services [2024-01-18T16:12:24.384Z] 2024-01-18 16:12:23 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser30] creation - [2024-01-18T16:12:24.384Z] 2024-01-18 16:12:23 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser30] created [2024-01-18T16:12:24.384Z] 2024-01-18 16:12:23 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser16] creation - [2024-01-18T16:12:24.384Z] 2024-01-18 16:12:23 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser16] created [2024-01-18T16:12:24.384Z] 2024-01-18 16:12:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser15] creation - [2024-01-18T16:12:24.384Z] 2024-01-18 16:12:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser15] created [2024-01-18T16:12:24.384Z] 2024-01-18 16:12:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser19] creation - [2024-01-18T16:12:24.384Z] 2024-01-18 16:12:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser19] created [2024-01-18T16:12:24.642Z] 2024-01-18 16:12:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser3] creation - [2024-01-18T16:12:24.642Z] 2024-01-18 16:12:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser3] created [2024-01-18T16:12:24.897Z] 2024-01-18 16:12:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser17] creation - [2024-01-18T16:12:24.897Z] 2024-01-18 16:12:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser17] created [2024-01-18T16:12:25.154Z] 2024-01-18 16:12:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser18] creation - [2024-01-18T16:12:25.154Z] 2024-01-18 16:12:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser18] created [2024-01-18T16:12:25.154Z] 2024-01-18 16:12:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser4] creation - [2024-01-18T16:12:25.154Z] 2024-01-18 16:12:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser4] created [2024-01-18T16:12:25.409Z] 2024-01-18 16:12:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser20] creation - [2024-01-18T16:12:25.409Z] 2024-01-18 16:12:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser20] created [2024-01-18T16:12:25.664Z] 2024-01-18 16:12:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser1] creation - [2024-01-18T16:12:25.664Z] 2024-01-18 16:12:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser1] created [2024-01-18T16:12:25.921Z] 2024-01-18 16:12:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser5] creation - [2024-01-18T16:12:25.921Z] 2024-01-18 16:12:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser5] created [2024-01-18T16:12:26.176Z] 2024-01-18 16:12:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser6] creation - [2024-01-18T16:12:26.177Z] 2024-01-18 16:12:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser6] created [2024-01-18T16:12:26.177Z] 2024-01-18 16:12:26 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser11] creation - [2024-01-18T16:12:26.177Z] 2024-01-18 16:12:26 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser11] created [2024-01-18T16:12:26.432Z] 2024-01-18 16:12:26 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser7] creation - [2024-01-18T16:12:26.432Z] 2024-01-18 16:12:26 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser7] created [2024-01-18T16:12:26.687Z] 2024-01-18 16:12:26 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser8] creation - [2024-01-18T16:12:26.687Z] 2024-01-18 16:12:26 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser8] created [2024-01-18T16:12:26.943Z] 2024-01-18 16:12:26 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser12] creation - [2024-01-18T16:12:26.943Z] 2024-01-18 16:12:26 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser12] created [2024-01-18T16:12:27.201Z] 2024-01-18 16:12:26 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser9] creation - [2024-01-18T16:12:27.201Z] 2024-01-18 16:12:26 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser9] created [2024-01-18T16:12:27.201Z] 2024-01-18 16:12:27 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser13] creation - [2024-01-18T16:12:27.201Z] 2024-01-18 16:12:27 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser13] created [2024-01-18T16:12:27.457Z] 2024-01-18 16:12:27 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser10] creation - [2024-01-18T16:12:27.457Z] 2024-01-18 16:12:27 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser10] created [2024-01-18T16:12:27.714Z] 2024-01-18 16:12:27 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser14] creation - [2024-01-18T16:12:27.714Z] 2024-01-18 16:12:27 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser14] created [2024-01-18T16:12:27.970Z] 2024-01-18 16:12:27 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser2] creation - [2024-01-18T16:12:27.970Z] 2024-01-18 16:12:27 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser2] created [2024-01-18T16:12:28.225Z] 2024-01-18 16:12:27 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser25] creation - [2024-01-18T16:12:28.225Z] 2024-01-18 16:12:27 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser25] created [2024-01-18T16:12:28.225Z] 2024-01-18 16:12:28 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser23] creation - [2024-01-18T16:12:28.225Z] 2024-01-18 16:12:28 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser23] created [2024-01-18T16:12:28.483Z] 2024-01-18 16:12:28 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser24] creation - [2024-01-18T16:12:28.483Z] 2024-01-18 16:12:28 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser24] created [2024-01-18T16:12:28.483Z] I0118 16:12:28.393943 14 request.go:682] Waited for 5.96748743s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-18T16:12:28.738Z] 2024-01-18 16:12:28 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser22] creation - [2024-01-18T16:12:28.738Z] 2024-01-18 16:12:28 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser22] created [2024-01-18T16:12:28.994Z] 2024-01-18 16:12:28 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser27] creation - [2024-01-18T16:12:28.994Z] 2024-01-18 16:12:28 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser27] created [2024-01-18T16:12:28.994Z] 2024-01-18 16:12:28 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser26] creation - [2024-01-18T16:12:28.994Z] 2024-01-18 16:12:28 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser26] created [2024-01-18T16:12:29.250Z] 2024-01-18 16:12:29 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser28] creation - [2024-01-18T16:12:29.250Z] 2024-01-18 16:12:29 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser28] created [2024-01-18T16:12:29.506Z] 2024-01-18 16:12:29 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser29] creation - [2024-01-18T16:12:29.506Z] 2024-01-18 16:12:29 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser29] created [2024-01-18T16:12:29.761Z] 2024-01-18 16:12:29 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser21] creation - [2024-01-18T16:12:29.761Z] 2024-01-18 16:12:29 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddUser:#733] - User [testuser21] created [2024-01-18T16:12:29.761Z] 2024-01-18 16:12:29 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Creating 3 groups [2024-01-18T16:12:31.651Z] 2024-01-18 16:12:31 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroup:#989] - Creating group response - [2024-01-18T16:12:31.651Z] 2024-01-18 16:12:31 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroup:#994] - Group [testGroup3] created [2024-01-18T16:12:31.907Z] 2024-01-18 16:12:31 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroup:#989] - Creating group response - [2024-01-18T16:12:31.907Z] 2024-01-18 16:12:31 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroup:#994] - Group [testGroup2] created [2024-01-18T16:12:31.907Z] 2024-01-18 16:12:31 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroup:#989] - Creating group response - [2024-01-18T16:12:31.907Z] 2024-01-18 16:12:31 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroup:#994] - Group [testGroup1] created [2024-01-18T16:12:31.907Z] 2024-01-18 16:12:31 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Adding users to groups [2024-01-18T16:12:39.990Z] I0118 16:12:38.593859 14 request.go:682] Waited for 5.970016154s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-18T16:12:49.952Z] I0118 16:12:48.793791 14 request.go:682] Waited for 5.943839804s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/services [2024-01-18T16:12:50.208Z] 2024-01-18 16:12:50 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 4d075c71-08af-4c19-b984-43e56fb3163b [2024-01-18T16:12:50.464Z] 2024-01-18 16:12:50 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup3 - 18ad0d59-cf6b-43c8-b03f-131d76391cb7 [2024-01-18T16:12:50.464Z] 2024-01-18 16:12:50 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup3 - 18ad0d59-cf6b-43c8-b03f-131d76391cb7 [2024-01-18T16:12:50.722Z] 2024-01-18 16:12:50 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup2 - a0d64e30-6000-4fb3-9987-7d187337cd0a [2024-01-18T16:12:50.979Z] 2024-01-18 16:12:50 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup3 - 18ad0d59-cf6b-43c8-b03f-131d76391cb7 [2024-01-18T16:12:51.234Z] 2024-01-18 16:12:51 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup2 - a0d64e30-6000-4fb3-9987-7d187337cd0a [2024-01-18T16:12:51.492Z] 2024-01-18 16:12:51 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup2 - a0d64e30-6000-4fb3-9987-7d187337cd0a [2024-01-18T16:12:51.492Z] 2024-01-18 16:12:51 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup3 - 18ad0d59-cf6b-43c8-b03f-131d76391cb7 [2024-01-18T16:12:51.749Z] 2024-01-18 16:12:51 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup3 - 18ad0d59-cf6b-43c8-b03f-131d76391cb7 [2024-01-18T16:12:52.004Z] 2024-01-18 16:12:51 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup3 - 18ad0d59-cf6b-43c8-b03f-131d76391cb7 [2024-01-18T16:12:52.260Z] 2024-01-18 16:12:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup2 - a0d64e30-6000-4fb3-9987-7d187337cd0a [2024-01-18T16:12:52.260Z] 2024-01-18 16:12:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup3 - 18ad0d59-cf6b-43c8-b03f-131d76391cb7 [2024-01-18T16:12:52.515Z] 2024-01-18 16:12:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup3 - 18ad0d59-cf6b-43c8-b03f-131d76391cb7 [2024-01-18T16:12:52.773Z] 2024-01-18 16:12:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 4d075c71-08af-4c19-b984-43e56fb3163b [2024-01-18T16:12:53.028Z] 2024-01-18 16:12:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup3 - 18ad0d59-cf6b-43c8-b03f-131d76391cb7 [2024-01-18T16:12:53.284Z] 2024-01-18 16:12:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 4d075c71-08af-4c19-b984-43e56fb3163b [2024-01-18T16:12:53.284Z] 2024-01-18 16:12:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup3 - 18ad0d59-cf6b-43c8-b03f-131d76391cb7 [2024-01-18T16:12:53.540Z] 2024-01-18 16:12:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 4d075c71-08af-4c19-b984-43e56fb3163b [2024-01-18T16:12:53.796Z] 2024-01-18 16:12:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 4d075c71-08af-4c19-b984-43e56fb3163b [2024-01-18T16:12:54.052Z] 2024-01-18 16:12:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup2 - a0d64e30-6000-4fb3-9987-7d187337cd0a [2024-01-18T16:12:54.052Z] 2024-01-18 16:12:54 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 4d075c71-08af-4c19-b984-43e56fb3163b [2024-01-18T16:12:54.310Z] 2024-01-18 16:12:54 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 4d075c71-08af-4c19-b984-43e56fb3163b [2024-01-18T16:12:54.569Z] 2024-01-18 16:12:54 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup2 - a0d64e30-6000-4fb3-9987-7d187337cd0a [2024-01-18T16:12:54.825Z] 2024-01-18 16:12:54 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 4d075c71-08af-4c19-b984-43e56fb3163b [2024-01-18T16:12:55.083Z] 2024-01-18 16:12:54 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 4d075c71-08af-4c19-b984-43e56fb3163b [2024-01-18T16:12:55.083Z] 2024-01-18 16:12:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 4d075c71-08af-4c19-b984-43e56fb3163b [2024-01-18T16:12:55.338Z] 2024-01-18 16:12:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup2 - a0d64e30-6000-4fb3-9987-7d187337cd0a [2024-01-18T16:12:55.594Z] 2024-01-18 16:12:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup2 - a0d64e30-6000-4fb3-9987-7d187337cd0a [2024-01-18T16:12:55.849Z] 2024-01-18 16:12:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup2 - a0d64e30-6000-4fb3-9987-7d187337cd0a [2024-01-18T16:12:55.849Z] 2024-01-18 16:12:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup2 - a0d64e30-6000-4fb3-9987-7d187337cd0a [2024-01-18T16:12:59.154Z] I0118 16:12:58.993164 14 request.go:682] Waited for 5.978898828s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/services [2024-01-18T16:13:09.102Z] I0118 16:13:08.993375 14 request.go:682] Waited for 5.943937661s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/services [2024-01-18T16:13:14.362Z] 2024-01-18 16:13:14 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser21 - b2397937-b93d-418f-9e5c-76f038287674 [2024-01-18T16:13:14.362Z] 2024-01-18 16:13:14 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser18 - 0fe37ee7-2c57-4ea1-bfda-965b6552d578 [2024-01-18T16:13:14.618Z] 2024-01-18 16:13:14 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser30 - 6b3e77b4-0c35-4128-9d2a-d0c87fe2213b [2024-01-18T16:13:14.874Z] 2024-01-18 16:13:14 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser13 - 57f3811d-4de9-4ec4-80e5-030c5264840d [2024-01-18T16:13:14.874Z] 2024-01-18 16:13:14 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser16 - 3cf6916f-6377-4b5a-8b15-fdc5abd42949 [2024-01-18T16:13:15.130Z] 2024-01-18 16:13:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser9 - c56347b4-2d1a-4976-a0be-517b2b77c3e3 [2024-01-18T16:13:15.386Z] 2024-01-18 16:13:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser10 - 58716ecb-6f47-4df3-b258-706298644350 [2024-01-18T16:13:15.641Z] 2024-01-18 16:13:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser15 - a0fb1c7b-46f4-4246-907c-18f6a827a996 [2024-01-18T16:13:15.897Z] 2024-01-18 16:13:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser19 - 64796d3f-b41d-4387-ba7d-0a313701c792 [2024-01-18T16:13:15.897Z] 2024-01-18 16:13:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser4 - 53771e25-ac89-41a9-b6e9-ab0f47eacfdf [2024-01-18T16:13:16.153Z] 2024-01-18 16:13:16 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser14 - 3528b287-f84c-478a-a4c3-d292bc741866 [2024-01-18T16:13:16.411Z] 2024-01-18 16:13:16 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser3 - fb198252-1d32-4b0c-9bde-af15603b57db [2024-01-18T16:13:16.667Z] 2024-01-18 16:13:16 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser20 - 301e807b-65ff-4e2d-8e6b-bf2a3c5d3c76 [2024-01-18T16:13:16.667Z] 2024-01-18 16:13:16 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2 - b2f8cd7d-526e-4ffd-9096-134d8d8560d9 [2024-01-18T16:13:16.922Z] 2024-01-18 16:13:16 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser17 - 8faa871e-75a8-4888-83ec-19f531408614 [2024-01-18T16:13:17.179Z] 2024-01-18 16:13:17 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser27 - 270817d8-75b8-454c-9d46-9d1797f4beae [2024-01-18T16:13:17.435Z] 2024-01-18 16:13:17 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1 - adc3e6f5-ef9f-40b8-84e5-01f4eb60e534 [2024-01-18T16:13:17.691Z] 2024-01-18 16:13:17 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser25 - 428618c8-0ac2-4dcb-9e13-c142c2b56a51 [2024-01-18T16:13:17.691Z] 2024-01-18 16:13:17 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser23 - a21f3c2e-a4cc-40ce-9890-cd3b41dc5e1b [2024-01-18T16:13:17.946Z] 2024-01-18 16:13:17 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser5 - 270f338e-2a21-4ce9-8c7a-809633d068e2 [2024-01-18T16:13:18.202Z] 2024-01-18 16:13:18 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser24 - c0116ca8-4fbd-4aa2-a698-74594a386929 [2024-01-18T16:13:18.459Z] 2024-01-18 16:13:18 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser26 - c5d1d55f-df19-4d58-b3a9-f55c8cde08e2 [2024-01-18T16:13:18.459Z] 2024-01-18 16:13:18 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser6 - 9ebb8fd7-6ba5-4fc4-b3be-86479b298a43 [2024-01-18T16:13:18.716Z] 2024-01-18 16:13:18 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser22 - b4ac58d6-e50e-4ab2-acef-60a5f199c08f [2024-01-18T16:13:18.971Z] 2024-01-18 16:13:18 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser28 - 3b5b4e76-ab60-4e61-be30-8274bd814f56 [2024-01-18T16:13:19.227Z] I0118 16:13:18.993856 14 request.go:682] Waited for 5.967954946s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-18T16:13:19.227Z] 2024-01-18 16:13:19 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser29 - 388a33f8-f8bb-4b86-86ff-6bfe389be493 [2024-01-18T16:13:19.482Z] 2024-01-18 16:13:19 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser8 - 1d1b04e2-4083-4be8-b015-f1d0cb4dce6c [2024-01-18T16:13:19.483Z] 2024-01-18 16:13:19 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser12 - 29f788b4-c927-44ed-9ed0-3be78b78209b [2024-01-18T16:13:19.738Z] 2024-01-18 16:13:19 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser7 - 62348be6-6d86-43ab-878e-cfe73ecde896 [2024-01-18T16:13:19.994Z] 2024-01-18 16:13:19 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser11 - 284064c8-085f-42e4-824c-2ac98310a559 [2024-01-18T16:13:29.934Z] I0118 16:13:29.193940 14 request.go:682] Waited for 5.966540045s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-18T16:13:39.873Z] 2024-01-18 16:13:38 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:39.873Z] 2024-01-18 16:13:38 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser21] to Group [testGroup1] [2024-01-18T16:13:39.873Z] 2024-01-18 16:13:38 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:39.873Z] 2024-01-18 16:13:38 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser18] to Group [testGroup3] [2024-01-18T16:13:39.873Z] 2024-01-18 16:13:38 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:39.873Z] 2024-01-18 16:13:38 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser30] to Group [testGroup3] [2024-01-18T16:13:39.873Z] 2024-01-18 16:13:38 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:39.873Z] 2024-01-18 16:13:38 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser13] to Group [testGroup2] [2024-01-18T16:13:39.873Z] 2024-01-18 16:13:38 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:39.873Z] 2024-01-18 16:13:38 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser16] to Group [testGroup3] [2024-01-18T16:13:39.873Z] 2024-01-18 16:13:39 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:39.873Z] 2024-01-18 16:13:39 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser9] to Group [testGroup2] [2024-01-18T16:13:39.873Z] I0118 16:13:39.194223 14 request.go:682] Waited for 5.97068743s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-18T16:13:39.874Z] 2024-01-18 16:13:39 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:39.874Z] 2024-01-18 16:13:39 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser10] to Group [testGroup2] [2024-01-18T16:13:39.874Z] 2024-01-18 16:13:39 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:39.874Z] 2024-01-18 16:13:39 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser15] to Group [testGroup3] [2024-01-18T16:13:39.874Z] 2024-01-18 16:13:39 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:39.874Z] 2024-01-18 16:13:39 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser19] to Group [testGroup3] [2024-01-18T16:13:40.131Z] 2024-01-18 16:13:39 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:40.131Z] 2024-01-18 16:13:39 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser4] to Group [testGroup3] [2024-01-18T16:13:40.131Z] 2024-01-18 16:13:40 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:40.131Z] 2024-01-18 16:13:40 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser14] to Group [testGroup2] [2024-01-18T16:13:40.387Z] 2024-01-18 16:13:40 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:40.387Z] 2024-01-18 16:13:40 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser3] to Group [testGroup3] [2024-01-18T16:13:40.644Z] 2024-01-18 16:13:40 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:40.644Z] 2024-01-18 16:13:40 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser20] to Group [testGroup3] [2024-01-18T16:13:40.899Z] 2024-01-18 16:13:40 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:40.899Z] 2024-01-18 16:13:40 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser2] to Group [testGroup1] [2024-01-18T16:13:40.899Z] 2024-01-18 16:13:40 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:40.899Z] 2024-01-18 16:13:40 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser17] to Group [testGroup3] [2024-01-18T16:13:41.155Z] 2024-01-18 16:13:41 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:41.155Z] 2024-01-18 16:13:41 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser27] to Group [testGroup1] [2024-01-18T16:13:41.411Z] 2024-01-18 16:13:41 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:41.411Z] 2024-01-18 16:13:41 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser1] to Group [testGroup3] [2024-01-18T16:13:41.666Z] 2024-01-18 16:13:41 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:41.666Z] 2024-01-18 16:13:41 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser25] to Group [testGroup1] [2024-01-18T16:13:41.922Z] 2024-01-18 16:13:41 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:41.922Z] 2024-01-18 16:13:41 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser23] to Group [testGroup1] [2024-01-18T16:13:41.922Z] 2024-01-18 16:13:41 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:41.922Z] 2024-01-18 16:13:41 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser5] to Group [testGroup2] [2024-01-18T16:13:42.178Z] 2024-01-18 16:13:42 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:42.178Z] 2024-01-18 16:13:42 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser24] to Group [testGroup1] [2024-01-18T16:13:42.435Z] 2024-01-18 16:13:42 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:42.435Z] 2024-01-18 16:13:42 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser26] to Group [testGroup1] [2024-01-18T16:13:42.691Z] 2024-01-18 16:13:42 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:42.691Z] 2024-01-18 16:13:42 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser6] to Group [testGroup2] [2024-01-18T16:13:42.950Z] 2024-01-18 16:13:42 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:42.950Z] 2024-01-18 16:13:42 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser22] to Group [testGroup1] [2024-01-18T16:13:42.950Z] 2024-01-18 16:13:42 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:42.950Z] 2024-01-18 16:13:42 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser28] to Group [testGroup1] [2024-01-18T16:13:43.205Z] 2024-01-18 16:13:43 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:43.205Z] 2024-01-18 16:13:43 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser29] to Group [testGroup1] [2024-01-18T16:13:43.463Z] 2024-01-18 16:13:43 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:43.463Z] 2024-01-18 16:13:43 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser8] to Group [testGroup2] [2024-01-18T16:13:43.718Z] 2024-01-18 16:13:43 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:43.719Z] 2024-01-18 16:13:43 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser12] to Group [testGroup2] [2024-01-18T16:13:43.976Z] 2024-01-18 16:13:43 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:43.976Z] 2024-01-18 16:13:43 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser7] to Group [testGroup2] [2024-01-18T16:13:43.976Z] 2024-01-18 16:13:43 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T16:13:43.976Z] 2024-01-18 16:13:43 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AddGroupToUser:#1119] - Added User [testuser11] to Group [testGroup2] [2024-01-18T16:13:45.342Z] 2024-01-18 16:13:45 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup3 - 18ad0d59-cf6b-43c8-b03f-131d76391cb7 [2024-01-18T16:13:45.600Z] 2024-01-18 16:13:45 +0000:[DEBUG] [{ShareBackupWithUsersAndGroups}] [backup.GetMembersOfGroup:#961] - list of members : [testuser1 testuser15 testuser16 testuser17 testuser18 testuser19 testuser20 testuser3 testuser30 testuser4] [2024-01-18T16:13:45.600Z] 2024-01-18 16:13:45 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.glob..func83.2.4:#540] - Group [testGroup3] contains the following users: [2024-01-18T16:13:45.600Z] [testuser1 testuser15 testuser16 testuser17 testuser18 testuser19 testuser20 testuser3 testuser30 testuser4] [2024-01-18T16:13:46.966Z] 2024-01-18 16:13:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup2 - a0d64e30-6000-4fb3-9987-7d187337cd0a [2024-01-18T16:13:47.227Z] 2024-01-18 16:13:47 +0000:[DEBUG] [{ShareBackupWithUsersAndGroups}] [backup.GetMembersOfGroup:#961] - list of members : [testuser10 testuser11 testuser12 testuser13 testuser14 testuser5 testuser6 testuser7 testuser8 testuser9] [2024-01-18T16:13:47.227Z] 2024-01-18 16:13:47 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.glob..func83.2.4:#540] - Group [testGroup2] contains the following users: [2024-01-18T16:13:47.227Z] [testuser10 testuser11 testuser12 testuser13 testuser14 testuser5 testuser6 testuser7 testuser8 testuser9] [2024-01-18T16:13:48.595Z] 2024-01-18 16:13:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 4d075c71-08af-4c19-b984-43e56fb3163b [2024-01-18T16:13:48.872Z] 2024-01-18 16:13:48 +0000:[DEBUG] [{ShareBackupWithUsersAndGroups}] [backup.GetMembersOfGroup:#961] - list of members : [testuser2 testuser21 testuser22 testuser23 testuser24 testuser25 testuser26 testuser27 testuser28 testuser29] [2024-01-18T16:13:48.873Z] 2024-01-18 16:13:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.glob..func83.2.4:#540] - Group [testGroup1] contains the following users: [2024-01-18T16:13:48.873Z] [testuser2 testuser21 testuser22 testuser23 testuser24 testuser25 testuser26 testuser27 testuser28 testuser29] [2024-01-18T16:13:48.873Z] 2024-01-18 16:13:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Creating cloud credentials and backup location [2024-01-18T16:13:50.240Z] 2024-01-18 16:13:50 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-1705594430] for org [default] with [aws] as provider [2024-01-18T16:13:50.240Z] 2024-01-18 16:13:50 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-18T16:13:50.240Z] 2024-01-18 16:13:50 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-1705594430] is present [2024-01-18T16:13:50.240Z] INFO[2024-01-18 16:13:50] Verifying : Description : Verifying creation of cloud credential named [autogenerated-cred-1705594430] for org [default] with [aws] as provider [2024-01-18T16:13:50.240Z] INFO[2024-01-18 16:13:50] Actual:true, Expected: true [2024-01-18T16:13:50.240Z] 2024-01-18 16:13:50 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Created Cloud Credentials with name - autogenerated-cred-1705594430 [2024-01-18T16:13:53.506Z] INFO[2024-01-18 16:13:53] Verifying : Description : Creating backup location autogenerated-backup-location-1705594430 [2024-01-18T16:13:53.506Z] INFO[2024-01-18 16:13:53] Actual:true, Expected: true [2024-01-18T16:13:53.506Z] 2024-01-18 16:13:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Created Backup Location with name - autogenerated-backup-location-1705594430 [2024-01-18T16:13:53.506Z] 2024-01-18 16:13:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Registering Source and Destination clusters and verifying the status [2024-01-18T16:13:53.761Z] INFO[2024-01-18 16:13:53] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T16:13:53.761Z] INFO[2024-01-18 16:13:53] Actual:true, Expected: true [2024-01-18T16:13:53.761Z] 2024-01-18 16:13:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T16:13:53.761Z] 2024-01-18 16:13:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T16:13:53.761Z] 2024-01-18 16:13:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T16:13:53.761Z] 2024-01-18 16:13:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T16:13:53.761Z] 2024-01-18 16:13:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Create cluster [source-cluster] in org [default] [2024-01-18T16:13:53.761Z] 2024-01-18 16:13:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T16:13:53.761Z] 2024-01-18 16:13:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T16:13:53.761Z] 2024-01-18 16:13:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Create cluster [destination-cluster] in org [default] [2024-01-18T16:13:53.762Z] 2024-01-18 16:13:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T16:13:53.762Z] 2024-01-18 16:13:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T16:13:53.762Z] 2024-01-18 16:13:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-pgvpv] for IBM [2024-01-18T16:13:53.762Z] 2024-01-18 16:13:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-pgvpv] for org [default] with [ibm] as provider [2024-01-18T16:13:53.762Z] 2024-01-18 16:13:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:13:53.762Z] 2024-01-18 16:13:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:13:54.019Z] 2024-01-18 16:13:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-pgvpv] is present [2024-01-18T16:13:54.019Z] 2024-01-18 16:13:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Create cluster [source-cluster] in org [default] [2024-01-18T16:13:54.585Z] 2024-01-18 16:13:54 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-jncin] for IBM [2024-01-18T16:13:54.585Z] 2024-01-18 16:13:54 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-jncin] for org [default] with [ibm] as provider [2024-01-18T16:13:54.585Z] 2024-01-18 16:13:54 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:13:54.585Z] 2024-01-18 16:13:54 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:13:54.585Z] 2024-01-18 16:13:54 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-jncin] is present [2024-01-18T16:13:54.585Z] 2024-01-18 16:13:54 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Create cluster [destination-cluster] in org [default] [2024-01-18T16:13:55.149Z] INFO[2024-01-18 16:13:55] Verifying : Description : Creating source and destination cluster [2024-01-18T16:13:55.149Z] INFO[2024-01-18 16:13:55] Actual:true, Expected: true [2024-01-18T16:13:55.149Z] INFO[2024-01-18 16:13:55] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-18T16:13:55.149Z] INFO[2024-01-18 16:13:55] Actual:Online, Expected: Online [2024-01-18T16:13:55.149Z] INFO[2024-01-18 16:13:55] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-18T16:13:55.149Z] INFO[2024-01-18 16:13:55] Actual:true, Expected: true [2024-01-18T16:13:55.149Z] 2024-01-18 16:13:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Taking backup of applications [2024-01-18T16:14:03.231Z] INFO[2024-01-18 16:14:02] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T16:14:03.231Z] INFO[2024-01-18 16:14:02] sched stats: tick interval: 2402 < 1.2s (100%) [2024-01-18T16:14:03.231Z] INFO[2024-01-18 16:14:02] sched stats: tick process durations: 2401 < 500ms (100%) [2024-01-18T16:14:05.745Z] 2024-01-18 16:14:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T16:14:05.745Z] 2024-01-18 16:14:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Volume snapshot classes found in the cluster - [2024-01-18T16:14:05.745Z] 2024-01-18 16:14:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] ibmc-vpcblock-snapshot [2024-01-18T16:14:05.746Z] 2024-01-18 16:14:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T16:14:06.001Z] 2024/01/18 16:14:05 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594445] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T16:14:15.941Z] 2024-01-18 16:14:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T16:14:15.941Z] 2024-01-18 16:14:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Volume snapshot classes found in the cluster - [2024-01-18T16:14:15.941Z] 2024-01-18 16:14:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] ibmc-vpcblock-snapshot [2024-01-18T16:14:15.941Z] 2024-01-18 16:14:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T16:14:15.941Z] 2024/01/18 16:14:15 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594455] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T16:14:25.885Z] 2024-01-18 16:14:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T16:14:25.885Z] 2024-01-18 16:14:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Volume snapshot classes found in the cluster - [2024-01-18T16:14:25.885Z] 2024-01-18 16:14:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] ibmc-vpcblock-snapshot [2024-01-18T16:14:25.885Z] 2024-01-18 16:14:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T16:14:25.885Z] 2024/01/18 16:14:25 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594465] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T16:14:35.822Z] 2024-01-18 16:14:35 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T16:14:35.822Z] 2024-01-18 16:14:35 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Volume snapshot classes found in the cluster - [2024-01-18T16:14:35.822Z] 2024-01-18 16:14:35 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] ibmc-vpcblock-snapshot [2024-01-18T16:14:35.822Z] 2024-01-18 16:14:35 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T16:14:35.822Z] 2024/01/18 16:14:35 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594475] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T16:14:35.822Z] 2024/01/18 16:14:35 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594445] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:14:45.766Z] 2024-01-18 16:14:45 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T16:14:45.766Z] 2024-01-18 16:14:45 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Volume snapshot classes found in the cluster - [2024-01-18T16:14:45.766Z] 2024-01-18 16:14:45 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] ibmc-vpcblock-snapshot [2024-01-18T16:14:45.766Z] 2024-01-18 16:14:45 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T16:14:45.766Z] 2024/01/18 16:14:45 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594485] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T16:14:45.766Z] 2024/01/18 16:14:45 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594455] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:14:55.726Z] 2024-01-18 16:14:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T16:14:55.726Z] 2024-01-18 16:14:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Volume snapshot classes found in the cluster - [2024-01-18T16:14:55.726Z] 2024-01-18 16:14:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] ibmc-vpcblock-snapshot [2024-01-18T16:14:55.726Z] 2024-01-18 16:14:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T16:14:55.982Z] 2024/01/18 16:14:55 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594465] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:14:55.982Z] 2024/01/18 16:14:55 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594495] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T16:15:05.925Z] 2024-01-18 16:15:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T16:15:05.925Z] 2024-01-18 16:15:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Volume snapshot classes found in the cluster - [2024-01-18T16:15:05.925Z] 2024-01-18 16:15:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] ibmc-vpcblock-snapshot [2024-01-18T16:15:05.925Z] 2024-01-18 16:15:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T16:15:05.925Z] 2024/01/18 16:15:05 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594475] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:15:05.925Z] 2024/01/18 16:15:05 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594505] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T16:15:05.925Z] 2024/01/18 16:15:05 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594445] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:15:15.875Z] 2024-01-18 16:15:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T16:15:15.875Z] 2024-01-18 16:15:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Volume snapshot classes found in the cluster - [2024-01-18T16:15:15.875Z] 2024-01-18 16:15:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] ibmc-vpcblock-snapshot [2024-01-18T16:15:15.875Z] 2024-01-18 16:15:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T16:15:16.154Z] 2024/01/18 16:15:15 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594485] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:15:16.154Z] 2024/01/18 16:15:15 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594455] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:15:16.154Z] 2024/01/18 16:15:16 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594515] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T16:15:26.098Z] 2024-01-18 16:15:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T16:15:26.098Z] 2024-01-18 16:15:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Volume snapshot classes found in the cluster - [2024-01-18T16:15:26.098Z] 2024-01-18 16:15:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] ibmc-vpcblock-snapshot [2024-01-18T16:15:26.098Z] 2024-01-18 16:15:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T16:15:26.098Z] 2024/01/18 16:15:25 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594525] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T16:15:26.098Z] 2024/01/18 16:15:25 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594465] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:15:26.099Z] 2024/01/18 16:15:25 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594495] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:15:36.057Z] 2024/01/18 16:15:35 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594475] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:15:36.057Z] 2024/01/18 16:15:35 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594505] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:15:36.058Z] 2024/01/18 16:15:35 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594445] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:15:46.039Z] 2024/01/18 16:15:45 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594485] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:15:46.039Z] 2024/01/18 16:15:45 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594455] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:15:46.296Z] 2024/01/18 16:15:46 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594515] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:15:56.321Z] 2024/01/18 16:15:55 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594525] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:15:56.321Z] 2024/01/18 16:15:55 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594465] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:15:56.321Z] 2024/01/18 16:15:55 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594495] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:16:06.264Z] 2024/01/18 16:16:05 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594475] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:16:06.264Z] 2024/01/18 16:16:05 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594505] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:16:06.264Z] 2024/01/18 16:16:05 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594445] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:16:17.675Z] 2024/01/18 16:16:15 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594485] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:16:17.675Z] 2024/01/18 16:16:15 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594455] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:16:17.675Z] 2024/01/18 16:16:16 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594515] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:16:27.614Z] 2024/01/18 16:16:25 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594525] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:16:27.614Z] 2024/01/18 16:16:25 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594465] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:16:27.614Z] 2024/01/18 16:16:25 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594495] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:16:37.556Z] 2024/01/18 16:16:35 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594505] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:16:37.556Z] 2024/01/18 16:16:35 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594475] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:16:37.556Z] 2024/01/18 16:16:35 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594445] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:16:47.490Z] 2024/01/18 16:16:45 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594485] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:16:47.490Z] 2024/01/18 16:16:45 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594455] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:16:47.490Z] 2024/01/18 16:16:46 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594515] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:16:57.472Z] 2024/01/18 16:16:55 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594525] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:16:57.472Z] 2024/01/18 16:16:55 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594465] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:16:57.472Z] 2024/01/18 16:16:55 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594495] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:17:07.413Z] 2024/01/18 16:17:05 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594505] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:17:07.413Z] 2024/01/18 16:17:05 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594475] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:17:07.413Z] 2024/01/18 16:17:05 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594445] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:17:17.360Z] 2024/01/18 16:17:15 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594485] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:17:17.360Z] 2024/01/18 16:17:15 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594455] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:17:17.360Z] 2024/01/18 16:17:16 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594515] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:17:27.298Z] 2024/01/18 16:17:25 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594525] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:17:27.298Z] 2024/01/18 16:17:25 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594465] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:17:27.298Z] 2024/01/18 16:17:25 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594495] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:17:37.239Z] 2024/01/18 16:17:35 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594505] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:17:37.239Z] 2024/01/18 16:17:35 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594475] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:17:37.239Z] 2024/01/18 16:17:35 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594445] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:17:47.174Z] 2024/01/18 16:17:45 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594485] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:17:47.174Z] 2024/01/18 16:17:45 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594455] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:17:47.174Z] 2024/01/18 16:17:46 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594515] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:17:57.112Z] 2024/01/18 16:17:55 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594525] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:17:57.112Z] 2024/01/18 16:17:55 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594465] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:17:57.112Z] 2024/01/18 16:17:55 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594495] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:18:07.054Z] 2024/01/18 16:18:05 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594475] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:18:07.054Z] 2024/01/18 16:18:05 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594505] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:18:07.054Z] 2024/01/18 16:18:05 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594445] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:18:17.002Z] 2024/01/18 16:18:15 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594485] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:18:17.002Z] 2024/01/18 16:18:16 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594455] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:18:17.002Z] 2024/01/18 16:18:16 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594515] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:18:26.945Z] 2024/01/18 16:18:25 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594525] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:18:26.945Z] 2024/01/18 16:18:25 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594465] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:18:26.945Z] 2024/01/18 16:18:25 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594495] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:18:36.940Z] 2024/01/18 16:18:35 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594505] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:18:36.941Z] 2024/01/18 16:18:35 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594475] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:18:36.941Z] 2024/01/18 16:18:35 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594445] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:18:46.879Z] 2024/01/18 16:18:45 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594485] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:18:46.879Z] 2024/01/18 16:18:46 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594455] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:18:46.879Z] 2024/01/18 16:18:46 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594515] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:18:56.818Z] 2024/01/18 16:18:55 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594525] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:18:56.818Z] 2024/01/18 16:18:55 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594465] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:18:56.819Z] 2024/01/18 16:18:55 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594495] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:19:06.801Z] 2024/01/18 16:19:05 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594475] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:19:06.801Z] 2024/01/18 16:19:05 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594505] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:19:06.801Z] 2024-01-18 16:19:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.CreateBackup:#313] - Backup [tp-backup-1705594445] created successfully [2024-01-18T16:19:06.801Z] 2024-01-18 16:19:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating backup [tp-backup-1705594445] in org [default] [2024-01-18T16:19:06.801Z] 2024-01-18 16:19:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705594445] [2024-01-18T16:19:06.801Z] 2024-01-18 16:19:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in backup [tp-backup-1705594445] [2024-01-18T16:19:06.801Z] 2024-01-18 16:19:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in backup [tp-backup-1705594445] [2024-01-18T16:19:06.801Z] 2024-01-18 16:19:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:19:06.801Z] 2024-01-18 16:19:06 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:19:06.801Z] 2024-01-18 16:19:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:19:06.801Z] 2024-01-18 16:19:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[mysql-data]] [2024-01-18T16:19:06.801Z] 2024-01-18 16:19:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T16:19:06.801Z] 2024-01-18 16:19:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating CR cleanup [2024-01-18T16:19:06.801Z] 2024-01-18 16:19:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:19:06.801Z] 2024-01-18 16:19:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] All CRs in [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[tp-backup-1705594445-dc70861 tp-backup-1705594455-bf53460 tp-backup-1705594465-8b7a3ee tp-backup-1705594475-05a42cd tp-backup-1705594485-4b3e9f4 tp-backup-1705594495-30274b5 tp-backup-1705594505-9338f4a tp-backup-1705594515-d02d739 tp-backup-1705594525-fd4216b]] [2024-01-18T16:19:06.801Z] 2024-01-18 16:19:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] CR found for [[tp-backup-1705594445-dc70861 tp-backup-1705594455-bf53460 tp-backup-1705594465-8b7a3ee tp-backup-1705594475-05a42cd tp-backup-1705594485-4b3e9f4 tp-backup-1705594495-30274b5 tp-backup-1705594505-9338f4a tp-backup-1705594515-d02d739 tp-backup-1705594525-fd4216b]] under [mysql-ibm-pxb-0-82934-01-18-16h10m24s] namespace [2024-01-18T16:19:06.801Z] 2024/01/18 16:19:06 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-1705594445]}, Next try in [5s], timeout [5m0s] [2024-01-18T16:19:12.047Z] 2024-01-18 16:19:11 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] All CRs in [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[tp-backup-1705594445-dc70861 tp-backup-1705594455-bf53460 tp-backup-1705594465-8b7a3ee tp-backup-1705594475-05a42cd tp-backup-1705594485-4b3e9f4 tp-backup-1705594495-30274b5 tp-backup-1705594505-9338f4a tp-backup-1705594515-d02d739 tp-backup-1705594525-fd4216b]] [2024-01-18T16:19:12.047Z] 2024-01-18 16:19:11 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] CR found for [[tp-backup-1705594445-dc70861 tp-backup-1705594455-bf53460 tp-backup-1705594465-8b7a3ee tp-backup-1705594475-05a42cd tp-backup-1705594485-4b3e9f4 tp-backup-1705594495-30274b5 tp-backup-1705594505-9338f4a tp-backup-1705594515-d02d739 tp-backup-1705594525-fd4216b]] under [mysql-ibm-pxb-0-82934-01-18-16h10m24s] namespace [2024-01-18T16:19:12.047Z] 2024/01/18 16:19:11 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-1705594445]}, Next try in [5s], timeout [5m0s] [2024-01-18T16:19:16.213Z] 2024/01/18 16:19:15 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594485] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:19:16.213Z] 2024/01/18 16:19:16 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594455] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:19:16.213Z] 2024/01/18 16:19:16 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594515] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:19:16.470Z] 2024-01-18 16:19:16 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] All CRs in [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[tp-backup-1705594455-bf53460 tp-backup-1705594465-8b7a3ee tp-backup-1705594475-05a42cd tp-backup-1705594485-4b3e9f4 tp-backup-1705594495-30274b5 tp-backup-1705594505-9338f4a tp-backup-1705594515-d02d739 tp-backup-1705594525-fd4216b]] [2024-01-18T16:19:16.470Z] INFO[2024-01-18 16:19:16] Verifying : Description : Creation and Validation of backup [tp-backup-1705594445] [2024-01-18T16:19:16.470Z] INFO[2024-01-18 16:19:16] Actual:true, Expected: true [2024-01-18T16:19:26.453Z] 2024/01/18 16:19:25 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594525] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:19:26.453Z] 2024/01/18 16:19:25 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594465] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:19:26.453Z] 2024/01/18 16:19:25 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594495] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:19:36.398Z] 2024/01/18 16:19:35 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594475] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:19:36.398Z] 2024/01/18 16:19:35 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594505] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:19:46.363Z] 2024/01/18 16:19:46 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594485] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:19:46.363Z] 2024-01-18 16:19:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.CreateBackup:#313] - Backup [tp-backup-1705594455] created successfully [2024-01-18T16:19:46.364Z] 2024-01-18 16:19:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating backup [tp-backup-1705594455] in org [default] [2024-01-18T16:19:46.364Z] 2024-01-18 16:19:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705594455] [2024-01-18T16:19:46.364Z] 2024-01-18 16:19:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in backup [tp-backup-1705594455] [2024-01-18T16:19:46.364Z] 2024-01-18 16:19:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in backup [tp-backup-1705594455] [2024-01-18T16:19:46.364Z] 2024-01-18 16:19:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:19:46.364Z] 2024-01-18 16:19:46 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:19:46.364Z] 2024-01-18 16:19:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:19:46.364Z] 2024-01-18 16:19:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[mysql-data]] [2024-01-18T16:19:46.364Z] 2024/01/18 16:19:46 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594515] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:19:46.364Z] 2024-01-18 16:19:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T16:19:46.364Z] 2024-01-18 16:19:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating CR cleanup [2024-01-18T16:19:46.364Z] 2024-01-18 16:19:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:19:46.620Z] 2024-01-18 16:19:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] All CRs in [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[tp-backup-1705594465-8b7a3ee tp-backup-1705594475-05a42cd tp-backup-1705594485-4b3e9f4 tp-backup-1705594495-30274b5 tp-backup-1705594505-9338f4a tp-backup-1705594515-d02d739 tp-backup-1705594525-fd4216b]] [2024-01-18T16:19:46.620Z] INFO[2024-01-18 16:19:46] Verifying : Description : Creation and Validation of backup [tp-backup-1705594455] [2024-01-18T16:19:46.620Z] INFO[2024-01-18 16:19:46] Actual:true, Expected: true [2024-01-18T16:19:56.565Z] 2024/01/18 16:19:55 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594525] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:19:56.565Z] 2024-01-18 16:19:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.CreateBackup:#313] - Backup [tp-backup-1705594465] created successfully [2024-01-18T16:19:56.565Z] 2024-01-18 16:19:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating backup [tp-backup-1705594465] in org [default] [2024-01-18T16:19:56.565Z] 2024-01-18 16:19:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705594465] [2024-01-18T16:19:56.565Z] 2024/01/18 16:19:55 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594495] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:19:56.565Z] 2024-01-18 16:19:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in backup [tp-backup-1705594465] [2024-01-18T16:19:56.565Z] 2024-01-18 16:19:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in backup [tp-backup-1705594465] [2024-01-18T16:19:56.565Z] 2024-01-18 16:19:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:19:56.565Z] 2024-01-18 16:19:55 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:19:56.565Z] 2024-01-18 16:19:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:19:56.565Z] 2024-01-18 16:19:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[mysql-data]] [2024-01-18T16:19:56.565Z] 2024-01-18 16:19:56 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T16:19:56.565Z] 2024-01-18 16:19:56 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating CR cleanup [2024-01-18T16:19:56.565Z] 2024-01-18 16:19:56 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:19:56.565Z] 2024-01-18 16:19:56 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] All CRs in [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[tp-backup-1705594475-05a42cd tp-backup-1705594485-4b3e9f4 tp-backup-1705594495-30274b5 tp-backup-1705594505-9338f4a tp-backup-1705594515-d02d739 tp-backup-1705594525-fd4216b]] [2024-01-18T16:19:56.565Z] INFO[2024-01-18 16:19:56] Verifying : Description : Creation and Validation of backup [tp-backup-1705594465] [2024-01-18T16:19:56.565Z] INFO[2024-01-18 16:19:56] Actual:true, Expected: true [2024-01-18T16:20:06.509Z] 2024-01-18 16:20:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.CreateBackup:#313] - Backup [tp-backup-1705594475] created successfully [2024-01-18T16:20:06.509Z] 2024-01-18 16:20:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating backup [tp-backup-1705594475] in org [default] [2024-01-18T16:20:06.509Z] 2024-01-18 16:20:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705594475] [2024-01-18T16:20:06.509Z] 2024/01/18 16:20:05 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594505] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:20:06.509Z] 2024-01-18 16:20:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in backup [tp-backup-1705594475] [2024-01-18T16:20:06.509Z] 2024-01-18 16:20:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in backup [tp-backup-1705594475] [2024-01-18T16:20:06.509Z] 2024-01-18 16:20:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:20:06.509Z] 2024-01-18 16:20:05 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:20:06.509Z] 2024-01-18 16:20:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:20:06.509Z] 2024-01-18 16:20:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[mysql-data]] [2024-01-18T16:20:06.509Z] 2024-01-18 16:20:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T16:20:06.509Z] 2024-01-18 16:20:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating CR cleanup [2024-01-18T16:20:06.509Z] 2024-01-18 16:20:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:20:06.510Z] 2024-01-18 16:20:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] All CRs in [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[tp-backup-1705594485-4b3e9f4 tp-backup-1705594495-30274b5 tp-backup-1705594505-9338f4a tp-backup-1705594515-d02d739 tp-backup-1705594525-fd4216b]] [2024-01-18T16:20:06.510Z] INFO[2024-01-18 16:20:06] Verifying : Description : Creation and Validation of backup [tp-backup-1705594475] [2024-01-18T16:20:06.510Z] INFO[2024-01-18 16:20:06] Actual:true, Expected: true [2024-01-18T16:20:16.454Z] 2024/01/18 16:20:16 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594485] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:20:16.454Z] 2024/01/18 16:20:16 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594515] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:20:26.394Z] 2024/01/18 16:20:25 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594525] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:20:26.394Z] 2024/01/18 16:20:25 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705594495] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:20:36.333Z] 2024-01-18 16:20:35 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.CreateBackup:#313] - Backup [tp-backup-1705594505] created successfully [2024-01-18T16:20:36.333Z] 2024-01-18 16:20:35 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating backup [tp-backup-1705594505] in org [default] [2024-01-18T16:20:36.333Z] 2024-01-18 16:20:35 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705594505] [2024-01-18T16:20:36.333Z] 2024-01-18 16:20:35 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in backup [tp-backup-1705594505] [2024-01-18T16:20:36.333Z] 2024-01-18 16:20:35 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in backup [tp-backup-1705594505] [2024-01-18T16:20:36.333Z] 2024-01-18 16:20:35 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:20:36.333Z] 2024-01-18 16:20:36 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:20:36.333Z] 2024-01-18 16:20:36 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:20:36.333Z] 2024-01-18 16:20:36 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[mysql-data]] [2024-01-18T16:20:36.333Z] 2024-01-18 16:20:36 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T16:20:36.334Z] 2024-01-18 16:20:36 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating CR cleanup [2024-01-18T16:20:36.334Z] 2024-01-18 16:20:36 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:20:36.334Z] 2024-01-18 16:20:36 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] All CRs in [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[tp-backup-1705594495-30274b5 tp-backup-1705594515-d02d739 tp-backup-1705594525-fd4216b]] [2024-01-18T16:20:36.334Z] INFO[2024-01-18 16:20:36] Verifying : Description : Creation and Validation of backup [tp-backup-1705594505] [2024-01-18T16:20:36.334Z] INFO[2024-01-18 16:20:36] Actual:true, Expected: true [2024-01-18T16:20:46.311Z] 2024-01-18 16:20:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.CreateBackup:#313] - Backup [tp-backup-1705594485] created successfully [2024-01-18T16:20:46.311Z] 2024-01-18 16:20:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating backup [tp-backup-1705594485] in org [default] [2024-01-18T16:20:46.311Z] 2024-01-18 16:20:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705594485] [2024-01-18T16:20:46.311Z] 2024-01-18 16:20:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in backup [tp-backup-1705594485] [2024-01-18T16:20:46.311Z] 2024-01-18 16:20:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in backup [tp-backup-1705594485] [2024-01-18T16:20:46.311Z] 2024-01-18 16:20:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:20:46.311Z] 2024-01-18 16:20:46 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:20:46.311Z] 2024-01-18 16:20:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:20:46.311Z] 2024-01-18 16:20:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[mysql-data]] [2024-01-18T16:20:46.311Z] 2024-01-18 16:20:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T16:20:46.311Z] 2024-01-18 16:20:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating CR cleanup [2024-01-18T16:20:46.311Z] 2024-01-18 16:20:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.CreateBackup:#313] - Backup [tp-backup-1705594515] created successfully [2024-01-18T16:20:46.311Z] 2024-01-18 16:20:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating backup [tp-backup-1705594515] in org [default] [2024-01-18T16:20:46.311Z] 2024-01-18 16:20:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705594515] [2024-01-18T16:20:46.312Z] 2024-01-18 16:20:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in backup [tp-backup-1705594515] [2024-01-18T16:20:46.312Z] 2024-01-18 16:20:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in backup [tp-backup-1705594515] [2024-01-18T16:20:46.312Z] 2024-01-18 16:20:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:20:46.312Z] 2024-01-18 16:20:46 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:20:46.312Z] 2024-01-18 16:20:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:20:46.312Z] 2024-01-18 16:20:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[mysql-data]] [2024-01-18T16:20:46.312Z] 2024-01-18 16:20:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T16:20:46.312Z] 2024-01-18 16:20:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating CR cleanup [2024-01-18T16:20:46.568Z] 2024-01-18 16:20:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:20:46.569Z] 2024-01-18 16:20:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] All CRs in [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[tp-backup-1705594515-d02d739]] [2024-01-18T16:20:46.569Z] INFO[2024-01-18 16:20:46] Verifying : Description : Creation and Validation of backup [tp-backup-1705594485] [2024-01-18T16:20:46.569Z] INFO[2024-01-18 16:20:46] Actual:true, Expected: true [2024-01-18T16:20:46.569Z] 2024-01-18 16:20:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:20:46.569Z] 2024-01-18 16:20:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] All CRs in [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[tp-backup-1705594515-d02d739]] [2024-01-18T16:20:46.569Z] 2024-01-18 16:20:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] CR found for [[tp-backup-1705594515-d02d739]] under [mysql-ibm-pxb-0-82934-01-18-16h10m24s] namespace [2024-01-18T16:20:46.569Z] 2024/01/18 16:20:46 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-1705594515]}, Next try in [5s], timeout [5m0s] [2024-01-18T16:20:51.818Z] 2024-01-18 16:20:51 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] All CRs in [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[]] [2024-01-18T16:20:51.818Z] INFO[2024-01-18 16:20:51] Verifying : Description : Creation and Validation of backup [tp-backup-1705594515] [2024-01-18T16:20:51.818Z] INFO[2024-01-18 16:20:51] Actual:true, Expected: true [2024-01-18T16:20:55.986Z] 2024-01-18 16:20:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.CreateBackup:#313] - Backup [tp-backup-1705594525] created successfully [2024-01-18T16:20:55.986Z] 2024-01-18 16:20:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating backup [tp-backup-1705594525] in org [default] [2024-01-18T16:20:55.987Z] 2024-01-18 16:20:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705594525] [2024-01-18T16:20:56.242Z] 2024-01-18 16:20:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in backup [tp-backup-1705594525] [2024-01-18T16:20:56.242Z] 2024-01-18 16:20:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in backup [tp-backup-1705594525] [2024-01-18T16:20:56.242Z] 2024-01-18 16:20:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:20:56.242Z] 2024-01-18 16:20:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.CreateBackup:#313] - Backup [tp-backup-1705594495] created successfully [2024-01-18T16:20:56.242Z] 2024-01-18 16:20:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating backup [tp-backup-1705594495] in org [default] [2024-01-18T16:20:56.242Z] 2024-01-18 16:20:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705594495] [2024-01-18T16:20:56.242Z] 2024-01-18 16:20:56 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:20:56.242Z] 2024-01-18 16:20:56 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:20:56.242Z] 2024-01-18 16:20:56 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[mysql-data]] [2024-01-18T16:20:56.243Z] 2024-01-18 16:20:56 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T16:20:56.243Z] 2024-01-18 16:20:56 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating CR cleanup [2024-01-18T16:20:56.243Z] 2024-01-18 16:20:56 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in backup [tp-backup-1705594495] [2024-01-18T16:20:56.243Z] 2024-01-18 16:20:56 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in backup [tp-backup-1705594495] [2024-01-18T16:20:56.243Z] 2024-01-18 16:20:56 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:20:56.243Z] 2024-01-18 16:20:56 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:20:56.243Z] 2024-01-18 16:20:56 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:20:56.243Z] 2024-01-18 16:20:56 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[mysql-data]] [2024-01-18T16:20:56.243Z] 2024-01-18 16:20:56 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T16:20:56.243Z] 2024-01-18 16:20:56 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating CR cleanup [2024-01-18T16:20:56.243Z] 2024-01-18 16:20:56 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:20:56.243Z] 2024-01-18 16:20:56 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:20:56.243Z] 2024-01-18 16:20:56 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] All CRs in [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[]] [2024-01-18T16:20:56.243Z] INFO[2024-01-18 16:20:56] Verifying : Description : Creation and Validation of backup [tp-backup-1705594525] [2024-01-18T16:20:56.243Z] INFO[2024-01-18 16:20:56] Actual:true, Expected: true [2024-01-18T16:20:56.499Z] 2024-01-18 16:20:56 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] All CRs in [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[]] [2024-01-18T16:20:56.499Z] INFO[2024-01-18 16:20:56] Verifying : Description : Creation and Validation of backup [tp-backup-1705594495] [2024-01-18T16:20:56.499Z] INFO[2024-01-18 16:20:56] Actual:true, Expected: true [2024-01-18T16:20:56.499Z] 2024-01-18 16:20:56 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.glob..func83.2.7:#600] - List of backups - [tp-backup-1705594445 tp-backup-1705594455 tp-backup-1705594465 tp-backup-1705594475 tp-backup-1705594485 tp-backup-1705594495 tp-backup-1705594505 tp-backup-1705594515 tp-backup-1705594525] [2024-01-18T16:20:56.499Z] 2024-01-18 16:20:56 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Sharing backups with groups [2024-01-18T16:20:56.754Z] 2024-01-18 16:20:56 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705594445 - dc70861c-2f4c-44ef-8761-4f1bbd8434a0 [2024-01-18T16:20:57.683Z] 2024-01-18 16:20:57 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup3 - 18ad0d59-cf6b-43c8-b03f-131d76391cb7 [2024-01-18T16:20:57.683Z] 2024-01-18 16:20:57 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705594455 - bf53460b-f447-4a14-b6bd-1ac622aa0188 [2024-01-18T16:20:58.243Z] 2024-01-18 16:20:58 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup3 - 18ad0d59-cf6b-43c8-b03f-131d76391cb7 [2024-01-18T16:20:58.502Z] 2024-01-18 16:20:58 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705594465 - 8b7a3eef-2e35-4c3c-b8e7-ab5c5095de84 [2024-01-18T16:20:59.063Z] 2024-01-18 16:20:58 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup3 - 18ad0d59-cf6b-43c8-b03f-131d76391cb7 [2024-01-18T16:20:59.319Z] 2024-01-18 16:20:59 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705594475 - 05a42cdd-d08b-4ae8-8f7e-7a345ec1efd9 [2024-01-18T16:20:59.881Z] 2024-01-18 16:20:59 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup2 - a0d64e30-6000-4fb3-9987-7d187337cd0a [2024-01-18T16:21:00.137Z] 2024-01-18 16:20:59 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705594485 - 4b3e9f44-a756-45e2-9601-0731c8fd58ba [2024-01-18T16:21:00.698Z] 2024-01-18 16:21:00 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup2 - a0d64e30-6000-4fb3-9987-7d187337cd0a [2024-01-18T16:21:00.954Z] 2024-01-18 16:21:00 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705594495 - 30274b56-3153-4840-9dea-862721845513 [2024-01-18T16:21:01.516Z] 2024-01-18 16:21:01 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup2 - a0d64e30-6000-4fb3-9987-7d187337cd0a [2024-01-18T16:21:01.772Z] 2024-01-18 16:21:01 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705594505 - 9338f4aa-a717-4d3a-89f9-10ab7f2260b5 [2024-01-18T16:21:02.346Z] 2024-01-18 16:21:02 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 4d075c71-08af-4c19-b984-43e56fb3163b [2024-01-18T16:21:02.602Z] 2024-01-18 16:21:02 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705594515 - d02d7393-2ebe-4ba9-99a4-11a55dfb16f7 [2024-01-18T16:21:03.166Z] 2024-01-18 16:21:02 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 4d075c71-08af-4c19-b984-43e56fb3163b [2024-01-18T16:21:03.166Z] 2024-01-18 16:21:03 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705594525 - fd4216b3-c15d-4cc4-8829-1f47fe7f3230 [2024-01-18T16:21:04.098Z] 2024-01-18 16:21:03 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 4d075c71-08af-4c19-b984-43e56fb3163b [2024-01-18T16:21:04.098Z] 2024-01-18 16:21:03 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Share Backup with Full access to a user of View Only access group and Validate [2024-01-18T16:21:05.462Z] 2024-01-18 16:21:04 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup3 - 18ad0d59-cf6b-43c8-b03f-131d76391cb7 [2024-01-18T16:21:05.719Z] 2024-01-18 16:21:05 +0000:[DEBUG] [{ShareBackupWithUsersAndGroups}] [backup.GetMembersOfGroup:#961] - list of members : [testuser1 testuser15 testuser16 testuser17 testuser18 testuser19 testuser20 testuser3 testuser30 testuser4] [2024-01-18T16:21:05.719Z] 2024-01-18 16:21:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.glob..func83.2.9:#633] - Sharing backup with user - testuser4 [2024-01-18T16:21:07.083Z] 2024-01-18 16:21:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705594445 - dc70861c-2f4c-44ef-8761-4f1bbd8434a0 [2024-01-18T16:21:08.012Z] 2024-01-18 16:21:07 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser4 - 53771e25-ac89-41a9-b6e9-ab0f47eacfdf [2024-01-18T16:21:08.268Z] 2024-01-18 16:21:08 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Registering Source and Destination clusters from user context [2024-01-18T16:21:08.268Z] INFO[2024-01-18 16:21:08] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T16:21:08.268Z] INFO[2024-01-18 16:21:08] Actual:true, Expected: true [2024-01-18T16:21:08.268Z] 2024-01-18 16:21:08 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T16:21:08.268Z] 2024-01-18 16:21:08 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T16:21:08.268Z] 2024-01-18 16:21:08 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T16:21:08.268Z] 2024-01-18 16:21:08 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T16:21:08.268Z] 2024-01-18 16:21:08 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Create cluster [source-cluster] in org [default] [2024-01-18T16:21:08.268Z] 2024-01-18 16:21:08 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T16:21:08.268Z] 2024-01-18 16:21:08 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T16:21:08.268Z] 2024-01-18 16:21:08 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Create cluster [destination-cluster] in org [default] [2024-01-18T16:21:08.268Z] 2024-01-18 16:21:08 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T16:21:08.269Z] 2024-01-18 16:21:08 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T16:21:08.269Z] 2024-01-18 16:21:08 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-tiuda] for IBM [2024-01-18T16:21:08.269Z] 2024-01-18 16:21:08 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-tiuda] for org [default] with [ibm] as provider [2024-01-18T16:21:08.269Z] 2024-01-18 16:21:08 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:21:08.269Z] 2024-01-18 16:21:08 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:21:08.269Z] 2024-01-18 16:21:08 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-tiuda] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName4 LastName4 with roles: [px-backup-app.user]] [2024-01-18T16:21:08.269Z] 2024-01-18 16:21:08 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName4 LastName4 with roles: [px-backup-app.user] [2024-01-18T16:21:09.636Z] 2024-01-18 16:21:09 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-tiuda from admin context and sharing with all the users [2024-01-18T16:21:09.636Z] 2024-01-18 16:21:09 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-tiuda] for org [default] with [ibm] as provider [2024-01-18T16:21:09.636Z] 2024-01-18 16:21:09 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:21:09.636Z] 2024-01-18 16:21:09 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:21:09.892Z] 2024-01-18 16:21:09 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-tiuda] is present [2024-01-18T16:21:09.892Z] 2024-01-18 16:21:09 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Create cluster [source-cluster] in org [default] [2024-01-18T16:21:10.456Z] 2024-01-18 16:21:10 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-xhyew] for IBM [2024-01-18T16:21:10.456Z] 2024-01-18 16:21:10 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-xhyew] for org [default] with [ibm] as provider [2024-01-18T16:21:10.456Z] 2024-01-18 16:21:10 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:21:10.456Z] 2024-01-18 16:21:10 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:21:10.457Z] 2024-01-18 16:21:10 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-xhyew] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName4 LastName4 with roles: [px-backup-app.user]] [2024-01-18T16:21:10.457Z] 2024-01-18 16:21:10 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName4 LastName4 with roles: [px-backup-app.user] [2024-01-18T16:21:11.383Z] 2024-01-18 16:21:10 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-xhyew from admin context and sharing with all the users [2024-01-18T16:21:11.383Z] 2024-01-18 16:21:10 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-xhyew] for org [default] with [ibm] as provider [2024-01-18T16:21:11.383Z] 2024-01-18 16:21:10 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:21:11.384Z] 2024-01-18 16:21:10 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:21:11.384Z] 2024-01-18 16:21:11 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-xhyew] is present [2024-01-18T16:21:11.384Z] 2024-01-18 16:21:11 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Create cluster [destination-cluster] in org [default] [2024-01-18T16:21:11.946Z] INFO[2024-01-18 16:21:11] Verifying : Description : Creating source and destination cluster [2024-01-18T16:21:11.946Z] INFO[2024-01-18 16:21:11] Actual:true, Expected: true [2024-01-18T16:21:12.507Z] 2024/01/18 16:21:12 DoRetryWithTimeout - Error: {restore status for [tp-restore-1705594871] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T16:21:44.554Z] 2024-01-18 16:21:42 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.CreateRestore:#936] - Restore [tp-restore-1705594871] created successfully [2024-01-18T16:21:44.554Z] 2024-01-18 16:21:42 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Switching cluster context to cluster [destination-cluster] [2024-01-18T16:21:44.554Z] 2024-01-18 16:21:42 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Switching context to [/tmp/destination-config] [2024-01-18T16:21:44.554Z] 2024-01-18 16:21:42 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:21:44.554Z] 2024-01-18 16:21:42 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:21:44.554Z] 2024-01-18 16:21:42 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:21:44.554Z] 2024-01-18 16:21:42 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:21:44.554Z] 2024-01-18 16:21:42 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:21:44.554Z] 2024-01-18 16:21:42 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:21:44.554Z] 2024-01-18 16:21:43 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:21:44.554Z] 2024-01-18 16:21:43 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:21:44.554Z] 2024-01-18 16:21:43 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:21:44.554Z] 2024-01-18 16:21:44 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:21:44.555Z] 2024-01-18 16:21:44 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:21:44.555Z] 2024-01-18 16:21:44 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:21:44.555Z] 2024-01-18 16:21:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:21:44.555Z] 2024-01-18 16:21:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Switched context to [/tmp/destination-config] [2024-01-18T16:21:44.555Z] 2024-01-18 16:21:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-82934-01-18-16h10m24s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-18T16:21:44.555Z] 2024-01-18 16:21:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:21:44.555Z] 2024-01-18 16:21:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T16:21:44.555Z] 2024-01-18 16:21:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T16:21:44.555Z] 2024-01-18 16:21:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating restore [tp-restore-1705594871] in org [default] [2024-01-18T16:21:44.555Z] 2024-01-18 16:21:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-restore-1705594871] [2024-01-18T16:21:44.555Z] 2024-01-18 16:21:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-restore-1705594871] [2024-01-18T16:21:44.555Z] 2024-01-18 16:21:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in restore [tp-restore-1705594871] [2024-01-18T16:21:44.555Z] 2024-01-18 16:21:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-82934-01-18-16h10m24s]) was found in restore [tp-restore-1705594871], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:21:44.814Z] 2024-01-18 16:21:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-82934-01-18-16h10m24s]) found in the restore [tp-restore-1705594871] was also present on the cluster/namespace [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:21:44.814Z] 2024-01-18 16:21:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-82934-01-18-16h10m24s]) was found in restore [tp-restore-1705594871], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:21:44.814Z] 2024-01-18 16:21:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-82934-01-18-16h10m24s]) found in the restore [tp-restore-1705594871] was also present on the cluster/namespace [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:21:44.814Z] 2024-01-18 16:21:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-82934-01-18-16h10m24s]) was found in restore [tp-restore-1705594871], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:21:44.814Z] 2024-01-18 16:21:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-82934-01-18-16h10m24s]) found in the restore [tp-restore-1705594871] was also present on the cluster/namespace [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:21:44.814Z] 2024-01-18 16:21:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-82934-01-18-16h10m24s]) was found in restore [tp-restore-1705594871], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:21:44.814Z] 2024-01-18 16:21:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-82934-01-18-16h10m24s]) found in the restore [tp-restore-1705594871] was also present on the cluster/namespace [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:21:44.814Z] 2024-01-18 16:21:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in restore [tp-restore-1705594871] [2024-01-18T16:21:44.814Z] 2024-01-18 16:21:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in restore [tp-restore-1705594871] [2024-01-18T16:21:45.070Z] 2024-01-18 16:21:44 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:21:45.070Z] 2024-01-18 16:21:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:21:45.070Z] 2024-01-18 16:21:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[mysql-data]] [2024-01-18T16:21:45.070Z] 2024-01-18 16:21:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2240] - in restore [tp-restore-1705594871], the status of the restored volume [pvc-5e1058d4-0546-40e4-bd6d-cf85fd428f46] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-18T16:21:45.070Z] 2024-01-18 16:21:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validate applications in restored namespace [mysql-ibm-pxb-0-82934-01-18-16h10m24s] due to restore [tp-restore-1705594871] [2024-01-18T16:21:45.070Z] 2024-01-18 16:21:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating mysql-ibm app [2024-01-18T16:21:45.070Z] 2024-01-18 16:21:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] wait for mysql-ibm app to start running [2024-01-18T16:21:45.326Z] 2024-01-18 16:21:45 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T16:21:45.326Z] 2024/01/18 16:21:45 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:21:45.326Z] pod name:mysql-7d766cd65c-k87zx namespace:mysql-ibm-pxb-0-82934-01-18-16h10m24s running:false ready:false node:10.240.64.7 [2024-01-18T16:21:45.327Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:21:57.498Z] 2024/01/18 16:21:55 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:21:57.498Z] pod name:mysql-7d766cd65c-k87zx namespace:mysql-ibm-pxb-0-82934-01-18-16h10m24s running:false ready:false node:10.240.64.7 [2024-01-18T16:21:57.498Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:22:05.594Z] 2024/01/18 16:22:05 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:22:05.594Z] pod name:mysql-7d766cd65c-k87zx namespace:mysql-ibm-pxb-0-82934-01-18-16h10m24s running:false ready:false node:10.240.64.7 [2024-01-18T16:22:05.594Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:22:15.539Z] 2024/01/18 16:22:15 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:22:15.539Z] pod name:mysql-7d766cd65c-k87zx namespace:mysql-ibm-pxb-0-82934-01-18-16h10m24s running:true ready:false node:10.240.64.7 [2024-01-18T16:22:15.539Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:22:27.729Z] 2024/01/18 16:22:25 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:22:27.729Z] pod name:mysql-7d766cd65c-k87zx namespace:mysql-ibm-pxb-0-82934-01-18-16h10m24s running:true ready:false node:10.240.64.7 [2024-01-18T16:22:27.729Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:22:35.868Z] 2024/01/18 16:22:35 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:22:35.868Z] pod name:mysql-7d766cd65c-k87zx namespace:mysql-ibm-pxb-0-82934-01-18-16h10m24s running:true ready:false node:10.240.64.7 [2024-01-18T16:22:35.868Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:22:47.374Z] 2024/01/18 16:22:45 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:22:47.374Z] pod name:mysql-7d766cd65c-k87zx namespace:mysql-ibm-pxb-0-82934-01-18-16h10m24s running:true ready:false node:10.240.64.7 [2024-01-18T16:22:47.374Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:22:57.310Z] 2024/01/18 16:22:55 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:22:57.310Z] pod name:mysql-7d766cd65c-k87zx namespace:mysql-ibm-pxb-0-82934-01-18-16h10m24s running:true ready:false node:10.240.64.7 [2024-01-18T16:22:57.310Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:23:07.250Z] 2024/01/18 16:23:05 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:23:07.250Z] pod name:mysql-7d766cd65c-k87zx namespace:mysql-ibm-pxb-0-82934-01-18-16h10m24s running:true ready:false node:10.240.64.7 [2024-01-18T16:23:07.250Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:23:17.191Z] 2024-01-18 16:23:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T16:23:17.191Z] 2024-01-18 16:23:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T16:23:17.191Z] 2024-01-18 16:23:16 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating CR cleanup [2024-01-18T16:23:17.191Z] 2024-01-18 16:23:16 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:23:17.191Z] 2024-01-18 16:23:16 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] All CRs in [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[]] [2024-01-18T16:23:17.191Z] 2024-01-18 16:23:16 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Switching cluster context back to cluster path [/tmp/source-config] [2024-01-18T16:23:17.191Z] 2024-01-18 16:23:16 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Switching context to [/tmp/source-config] [2024-01-18T16:23:17.191Z] 2024-01-18 16:23:16 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:23:17.191Z] 2024-01-18 16:23:16 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:23:17.191Z] 2024-01-18 16:23:16 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:23:17.191Z] 2024-01-18 16:23:16 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:23:17.191Z] 2024-01-18 16:23:16 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:23:17.191Z] 2024-01-18 16:23:16 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:23:17.191Z] 2024-01-18 16:23:17 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:23:17.448Z] 2024-01-18 16:23:17 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:23:18.012Z] 2024-01-18 16:23:17 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:23:18.268Z] 2024-01-18 16:23:18 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:23:18.268Z] 2024-01-18 16:23:18 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:23:18.268Z] 2024-01-18 16:23:18 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:23:18.269Z] 2024-01-18 16:23:18 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:23:18.269Z] 2024-01-18 16:23:18 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Switched context to [/tmp/source-config] [2024-01-18T16:23:18.269Z] 2024-01-18 16:23:18 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Restoring of backup [tp-backup-1705594445] was successful with name - [tp-restore-1705594871] [2024-01-18T16:23:18.269Z] 2024-01-18 16:23:18 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.glob..func83.2.9:#662] - About to delete restore - tp-restore-1705594871 [2024-01-18T16:23:18.269Z] INFO[2024-01-18 16:23:18] Verifying : Description : Getting the backup driver [2024-01-18T16:23:18.269Z] INFO[2024-01-18 16:23:18] Actual:true, Expected: true [2024-01-18T16:23:18.533Z] INFO[2024-01-18 16:23:18] Verifying : Description : Deleting Restore tp-restore-1705594871 [2024-01-18T16:23:18.533Z] INFO[2024-01-18 16:23:18] Actual:true, Expected: true [2024-01-18T16:23:18.533Z] INFO[2024-01-18 16:23:18] Verifying : Description : Verifying backup tp-backup-1705594445 deletion [2024-01-18T16:23:18.533Z] INFO[2024-01-18 16:23:18] Actual:, Expected: [2024-01-18T16:23:18.533Z] 2024-01-18 16:23:18 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Share Backup with View Only access to a user of Full access group and Validate [2024-01-18T16:23:19.465Z] 2024-01-18 16:23:19 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 4d075c71-08af-4c19-b984-43e56fb3163b [2024-01-18T16:23:20.025Z] 2024-01-18 16:23:19 +0000:[DEBUG] [{ShareBackupWithUsersAndGroups}] [backup.GetMembersOfGroup:#961] - list of members : [testuser2 testuser21 testuser22 testuser23 testuser24 testuser25 testuser26 testuser27 testuser28 testuser29] [2024-01-18T16:23:20.025Z] 2024-01-18 16:23:19 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.glob..func83.2.10:#682] - Sharing backup with user - testuser21 [2024-01-18T16:23:21.387Z] 2024-01-18 16:23:21 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705594505 - 9338f4aa-a717-4d3a-89f9-10ab7f2260b5 [2024-01-18T16:23:22.313Z] 2024-01-18 16:23:22 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser21 - b2397937-b93d-418f-9e5c-76f038287674 [2024-01-18T16:23:22.569Z] 2024-01-18 16:23:22 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Registering Source and Destination clusters from user context [2024-01-18T16:23:22.569Z] INFO[2024-01-18 16:23:22] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T16:23:22.569Z] INFO[2024-01-18 16:23:22] Actual:true, Expected: true [2024-01-18T16:23:22.569Z] 2024-01-18 16:23:22 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T16:23:22.829Z] 2024-01-18 16:23:22 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T16:23:22.829Z] 2024-01-18 16:23:22 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T16:23:22.829Z] 2024-01-18 16:23:22 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T16:23:22.829Z] 2024-01-18 16:23:22 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Create cluster [source-cluster] in org [default] [2024-01-18T16:23:22.829Z] 2024-01-18 16:23:22 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T16:23:22.829Z] 2024-01-18 16:23:22 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T16:23:22.829Z] 2024-01-18 16:23:22 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Create cluster [destination-cluster] in org [default] [2024-01-18T16:23:22.829Z] 2024-01-18 16:23:22 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T16:23:22.830Z] 2024-01-18 16:23:22 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T16:23:22.830Z] 2024-01-18 16:23:22 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-iesgm] for IBM [2024-01-18T16:23:22.830Z] 2024-01-18 16:23:22 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-iesgm] for org [default] with [ibm] as provider [2024-01-18T16:23:22.830Z] 2024-01-18 16:23:22 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:23:22.830Z] 2024-01-18 16:23:22 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:23:22.830Z] 2024-01-18 16:23:22 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-iesgm] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName21 LastName21 with roles: [px-backup-app.user]] [2024-01-18T16:23:22.830Z] 2024-01-18 16:23:22 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName21 LastName21 with roles: [px-backup-app.user] [2024-01-18T16:23:24.194Z] 2024-01-18 16:23:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-iesgm from admin context and sharing with all the users [2024-01-18T16:23:24.194Z] 2024-01-18 16:23:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-iesgm] for org [default] with [ibm] as provider [2024-01-18T16:23:24.194Z] 2024-01-18 16:23:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:23:24.194Z] 2024-01-18 16:23:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:23:24.194Z] 2024-01-18 16:23:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-iesgm] is present [2024-01-18T16:23:24.450Z] 2024-01-18 16:23:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Create cluster [source-cluster] in org [default] [2024-01-18T16:23:25.010Z] 2024-01-18 16:23:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-kboet] for IBM [2024-01-18T16:23:25.010Z] 2024-01-18 16:23:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-kboet] for org [default] with [ibm] as provider [2024-01-18T16:23:25.010Z] 2024-01-18 16:23:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:23:25.010Z] 2024-01-18 16:23:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:23:25.010Z] 2024-01-18 16:23:24 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-kboet] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName21 LastName21 with roles: [px-backup-app.user]] [2024-01-18T16:23:25.010Z] 2024-01-18 16:23:24 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName21 LastName21 with roles: [px-backup-app.user] [2024-01-18T16:23:25.573Z] 2024-01-18 16:23:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-kboet from admin context and sharing with all the users [2024-01-18T16:23:25.573Z] 2024-01-18 16:23:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-kboet] for org [default] with [ibm] as provider [2024-01-18T16:23:25.573Z] 2024-01-18 16:23:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:23:25.573Z] 2024-01-18 16:23:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:23:25.573Z] 2024-01-18 16:23:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-kboet] is present [2024-01-18T16:23:25.852Z] 2024-01-18 16:23:25 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Create cluster [destination-cluster] in org [default] [2024-01-18T16:23:26.413Z] INFO[2024-01-18 16:23:26] Verifying : Description : Creating source and destination cluster [2024-01-18T16:23:26.413Z] INFO[2024-01-18 16:23:26] Actual:true, Expected: true [2024-01-18T16:23:26.413Z] 2024-01-18 16:23:26 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.glob..func83.2.10:#710] - Error message - rpc error: code = Unknown desc = user [b2397937-b93d-418f-9e5c-76f038287674] doesn't have permission to delete backup [tp-backup-1705594505] [2024-01-18T16:23:26.413Z] INFO[2024-01-18 16:23:26] Verifying : Description : Verifying backup deletion is not possible [2024-01-18T16:23:26.413Z] INFO[2024-01-18 16:23:26] Actual:true, Expected: true [2024-01-18T16:23:26.413Z] 2024-01-18 16:23:26 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Share Backup with Restore access to a user of View Only access group and Validate [2024-01-18T16:23:26.668Z] 2024-01-18 16:23:26 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup3 - 18ad0d59-cf6b-43c8-b03f-131d76391cb7 [2024-01-18T16:23:27.229Z] 2024-01-18 16:23:27 +0000:[DEBUG] [{ShareBackupWithUsersAndGroups}] [backup.GetMembersOfGroup:#961] - list of members : [testuser1 testuser15 testuser16 testuser17 testuser18 testuser19 testuser20 testuser3 testuser30 testuser4] [2024-01-18T16:23:27.229Z] 2024-01-18 16:23:27 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.glob..func83.2.11:#719] - Sharing backup with user - testuser4 [2024-01-18T16:23:28.595Z] 2024-01-18 16:23:28 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705594455 - bf53460b-f447-4a14-b6bd-1ac622aa0188 [2024-01-18T16:23:29.521Z] 2024-01-18 16:23:29 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser4 - 53771e25-ac89-41a9-b6e9-ab0f47eacfdf [2024-01-18T16:23:29.777Z] 2024-01-18 16:23:29 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Registering Source and Destination clusters from user context [2024-01-18T16:23:29.777Z] INFO[2024-01-18 16:23:29] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T16:23:29.777Z] INFO[2024-01-18 16:23:29] Actual:true, Expected: true [2024-01-18T16:23:29.777Z] 2024-01-18 16:23:29 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T16:23:30.032Z] 2024-01-18 16:23:29 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T16:23:30.032Z] 2024-01-18 16:23:29 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T16:23:30.033Z] 2024-01-18 16:23:29 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T16:23:30.033Z] 2024-01-18 16:23:29 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Create cluster [source-cluster] in org [default] [2024-01-18T16:23:30.033Z] 2024-01-18 16:23:29 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T16:23:30.033Z] 2024-01-18 16:23:29 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T16:23:30.033Z] 2024-01-18 16:23:29 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Create cluster [destination-cluster] in org [default] [2024-01-18T16:23:30.033Z] 2024-01-18 16:23:29 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T16:23:30.033Z] 2024-01-18 16:23:29 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T16:23:30.033Z] 2024-01-18 16:23:29 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-kvjhs] for IBM [2024-01-18T16:23:30.033Z] 2024-01-18 16:23:29 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-kvjhs] for org [default] with [ibm] as provider [2024-01-18T16:23:30.033Z] 2024-01-18 16:23:29 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:23:30.033Z] 2024-01-18 16:23:29 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:23:30.034Z] 2024-01-18 16:23:29 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-kvjhs] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName4 LastName4 with roles: [px-backup-app.user]] [2024-01-18T16:23:30.034Z] 2024-01-18 16:23:29 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName4 LastName4 with roles: [px-backup-app.user] [2024-01-18T16:23:31.400Z] 2024-01-18 16:23:31 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-kvjhs from admin context and sharing with all the users [2024-01-18T16:23:31.400Z] 2024-01-18 16:23:31 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-kvjhs] for org [default] with [ibm] as provider [2024-01-18T16:23:31.400Z] 2024-01-18 16:23:31 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:23:31.401Z] 2024-01-18 16:23:31 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:23:31.401Z] 2024-01-18 16:23:31 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-kvjhs] is present [2024-01-18T16:23:31.657Z] 2024-01-18 16:23:31 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Create cluster [source-cluster] in org [default] [2024-01-18T16:23:31.657Z] 2024-01-18 16:23:31 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-vhryz] for IBM [2024-01-18T16:23:31.657Z] 2024-01-18 16:23:31 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-vhryz] for org [default] with [ibm] as provider [2024-01-18T16:23:31.657Z] 2024-01-18 16:23:31 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:23:31.657Z] 2024-01-18 16:23:31 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:23:31.657Z] 2024-01-18 16:23:31 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-vhryz] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName4 LastName4 with roles: [px-backup-app.user]] [2024-01-18T16:23:31.658Z] 2024-01-18 16:23:31 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName4 LastName4 with roles: [px-backup-app.user] [2024-01-18T16:23:33.028Z] 2024-01-18 16:23:32 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-vhryz from admin context and sharing with all the users [2024-01-18T16:23:33.028Z] 2024-01-18 16:23:32 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-vhryz] for org [default] with [ibm] as provider [2024-01-18T16:23:33.028Z] 2024-01-18 16:23:32 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:23:33.028Z] 2024-01-18 16:23:32 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:23:33.029Z] 2024-01-18 16:23:32 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-vhryz] is present [2024-01-18T16:23:33.029Z] 2024-01-18 16:23:32 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Create cluster [destination-cluster] in org [default] [2024-01-18T16:23:33.029Z] INFO[2024-01-18 16:23:32] Verifying : Description : Creating source and destination cluster [2024-01-18T16:23:33.029Z] INFO[2024-01-18 16:23:32] Actual:true, Expected: true [2024-01-18T16:23:33.602Z] 2024/01/18 16:23:33 DoRetryWithTimeout - Error: {restore status for [tp-restore-1705595012] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T16:24:05.625Z] 2024-01-18 16:24:03 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.CreateRestore:#936] - Restore [tp-restore-1705595012] created successfully [2024-01-18T16:24:05.625Z] 2024-01-18 16:24:03 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Switching cluster context to cluster [destination-cluster] [2024-01-18T16:24:05.625Z] 2024-01-18 16:24:03 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Switching context to [/tmp/destination-config] [2024-01-18T16:24:05.625Z] 2024-01-18 16:24:03 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:24:05.625Z] 2024-01-18 16:24:03 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:24:05.625Z] 2024-01-18 16:24:03 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:24:05.625Z] 2024-01-18 16:24:03 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:24:05.626Z] 2024-01-18 16:24:03 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:24:05.626Z] INFO[2024-01-18 16:24:03] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T16:24:05.626Z] INFO[2024-01-18 16:24:03] sched stats: tick interval: 3003 < 1.2s (100%) [2024-01-18T16:24:05.626Z] INFO[2024-01-18 16:24:03] sched stats: tick process durations: 3002 < 500ms (100%) [2024-01-18T16:24:05.626Z] 2024-01-18 16:24:03 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:24:05.626Z] 2024-01-18 16:24:04 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:24:05.626Z] 2024-01-18 16:24:04 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:24:05.626Z] 2024-01-18 16:24:04 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:24:05.626Z] 2024-01-18 16:24:05 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:24:05.626Z] 2024-01-18 16:24:05 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:24:05.626Z] 2024-01-18 16:24:05 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:24:05.626Z] 2024-01-18 16:24:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:24:05.626Z] 2024-01-18 16:24:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Switched context to [/tmp/destination-config] [2024-01-18T16:24:05.626Z] 2024-01-18 16:24:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-82934-01-18-16h10m24s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-18T16:24:05.626Z] 2024-01-18 16:24:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:24:05.626Z] 2024-01-18 16:24:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T16:24:05.626Z] 2024-01-18 16:24:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T16:24:05.626Z] 2024-01-18 16:24:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating restore [tp-restore-1705595012] in org [default] [2024-01-18T16:24:05.626Z] 2024-01-18 16:24:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-restore-1705595012] [2024-01-18T16:24:05.626Z] 2024-01-18 16:24:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-restore-1705595012] [2024-01-18T16:24:05.626Z] 2024-01-18 16:24:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in restore [tp-restore-1705595012] [2024-01-18T16:24:05.627Z] 2024-01-18 16:24:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-82934-01-18-16h10m24s]) was found in restore [tp-restore-1705595012], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:24:05.627Z] 2024-01-18 16:24:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-82934-01-18-16h10m24s]) found in the restore [tp-restore-1705595012] was also present on the cluster/namespace [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:24:05.627Z] 2024-01-18 16:24:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-82934-01-18-16h10m24s]) was found in restore [tp-restore-1705595012], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:24:05.883Z] 2024-01-18 16:24:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-82934-01-18-16h10m24s]) found in the restore [tp-restore-1705595012] was also present on the cluster/namespace [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:24:05.883Z] 2024-01-18 16:24:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-82934-01-18-16h10m24s]) was found in restore [tp-restore-1705595012], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:24:05.883Z] 2024-01-18 16:24:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-82934-01-18-16h10m24s]) found in the restore [tp-restore-1705595012] was also present on the cluster/namespace [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:24:05.883Z] 2024-01-18 16:24:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-82934-01-18-16h10m24s]) was found in restore [tp-restore-1705595012], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:24:05.883Z] 2024-01-18 16:24:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-82934-01-18-16h10m24s]) found in the restore [tp-restore-1705595012] was also present on the cluster/namespace [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:24:05.883Z] 2024-01-18 16:24:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in restore [tp-restore-1705595012] [2024-01-18T16:24:05.883Z] 2024-01-18 16:24:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in restore [tp-restore-1705595012] [2024-01-18T16:24:06.139Z] 2024-01-18 16:24:05 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:24:06.139Z] 2024-01-18 16:24:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:24:06.139Z] 2024-01-18 16:24:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[mysql-data]] [2024-01-18T16:24:06.139Z] 2024-01-18 16:24:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2243] - in restore [tp-restore-1705595012], the status of the restored volume [pvc-5e1058d4-0546-40e4-bd6d-cf85fd428f46] was not Success. It was [Retained] with reason [Skipped from volume restore as policy is set to Retain and pvc already exists] [2024-01-18T16:24:06.139Z] 2024-01-18 16:24:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validate applications in restored namespace [mysql-ibm-pxb-0-82934-01-18-16h10m24s] due to restore [tp-restore-1705595012] [2024-01-18T16:24:06.140Z] 2024-01-18 16:24:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating mysql-ibm app [2024-01-18T16:24:06.140Z] 2024-01-18 16:24:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] wait for mysql-ibm app to start running [2024-01-18T16:24:06.398Z] 2024-01-18 16:24:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T16:24:06.398Z] 2024-01-18 16:24:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T16:24:06.398Z] 2024-01-18 16:24:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T16:24:06.654Z] 2024-01-18 16:24:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating CR cleanup [2024-01-18T16:24:06.910Z] 2024-01-18 16:24:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:24:06.910Z] 2024-01-18 16:24:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] All CRs in [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[]] [2024-01-18T16:24:06.910Z] 2024-01-18 16:24:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Switching cluster context back to cluster path [/tmp/source-config] [2024-01-18T16:24:06.910Z] 2024-01-18 16:24:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Switching context to [/tmp/source-config] [2024-01-18T16:24:07.165Z] 2024-01-18 16:24:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:24:07.165Z] 2024-01-18 16:24:06 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:24:07.166Z] 2024-01-18 16:24:07 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:24:07.166Z] 2024-01-18 16:24:07 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:24:07.421Z] 2024-01-18 16:24:07 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:24:07.421Z] 2024-01-18 16:24:07 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:24:07.982Z] 2024-01-18 16:24:07 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:24:08.238Z] 2024-01-18 16:24:08 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:24:08.495Z] 2024-01-18 16:24:08 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:24:08.751Z] 2024-01-18 16:24:08 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:24:09.007Z] 2024-01-18 16:24:08 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:24:09.008Z] 2024-01-18 16:24:08 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:24:09.008Z] 2024-01-18 16:24:08 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:24:09.008Z] 2024-01-18 16:24:08 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Switched context to [/tmp/source-config] [2024-01-18T16:24:09.008Z] 2024-01-18 16:24:08 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Restoring of backup [tp-backup-1705594455] was successful with name - [tp-restore-1705595012] [2024-01-18T16:24:09.008Z] 2024-01-18 16:24:08 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.glob..func83.2.11:#748] - About to delete restore - tp-restore-1705595012 [2024-01-18T16:24:09.008Z] INFO[2024-01-18 16:24:08] Verifying : Description : Getting the backup driver [2024-01-18T16:24:09.008Z] INFO[2024-01-18 16:24:08] Actual:true, Expected: true [2024-01-18T16:24:09.008Z] INFO[2024-01-18 16:24:08] Verifying : Description : Deleting Restore tp-restore-1705595012 [2024-01-18T16:24:09.008Z] INFO[2024-01-18 16:24:08] Actual:true, Expected: true [2024-01-18T16:24:09.268Z] INFO[2024-01-18 16:24:08] Verifying : Description : Verifying backup deletion is not possible [2024-01-18T16:24:09.268Z] INFO[2024-01-18 16:24:08] Actual:true, Expected: true [2024-01-18T16:24:09.268Z] 2024-01-18 16:24:08 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validate Restore access for a user of Restore group [2024-01-18T16:24:10.195Z] 2024-01-18 16:24:10 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup2 - a0d64e30-6000-4fb3-9987-7d187337cd0a [2024-01-18T16:24:10.756Z] 2024-01-18 16:24:10 +0000:[DEBUG] [{ShareBackupWithUsersAndGroups}] [backup.GetMembersOfGroup:#961] - list of members : [testuser10 testuser11 testuser12 testuser13 testuser14 testuser5 testuser6 testuser7 testuser8 testuser9] [2024-01-18T16:24:10.757Z] 2024-01-18 16:24:10 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.glob..func83.2.12:#767] - Sharing backup with user - testuser11 [2024-01-18T16:24:12.650Z] 2024-01-18 16:24:12 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Registering Source and Destination clusters from user context [2024-01-18T16:24:12.650Z] INFO[2024-01-18 16:24:12] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T16:24:12.650Z] INFO[2024-01-18 16:24:12] Actual:true, Expected: true [2024-01-18T16:24:12.650Z] 2024-01-18 16:24:12 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T16:24:12.650Z] 2024-01-18 16:24:12 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T16:24:12.650Z] 2024-01-18 16:24:12 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T16:24:12.650Z] 2024-01-18 16:24:12 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T16:24:12.650Z] 2024-01-18 16:24:12 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Create cluster [source-cluster] in org [default] [2024-01-18T16:24:12.650Z] 2024-01-18 16:24:12 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T16:24:12.650Z] 2024-01-18 16:24:12 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T16:24:12.650Z] 2024-01-18 16:24:12 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Create cluster [destination-cluster] in org [default] [2024-01-18T16:24:12.650Z] 2024-01-18 16:24:12 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T16:24:12.650Z] 2024-01-18 16:24:12 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T16:24:12.650Z] 2024-01-18 16:24:12 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-jeget] for IBM [2024-01-18T16:24:12.650Z] 2024-01-18 16:24:12 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-jeget] for org [default] with [ibm] as provider [2024-01-18T16:24:12.650Z] 2024-01-18 16:24:12 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:24:12.651Z] 2024-01-18 16:24:12 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:24:12.651Z] 2024-01-18 16:24:12 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-jeget] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName11 LastName11 with roles: [px-backup-app.user]] [2024-01-18T16:24:12.651Z] 2024-01-18 16:24:12 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName11 LastName11 with roles: [px-backup-app.user] [2024-01-18T16:24:14.021Z] 2024-01-18 16:24:13 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-jeget from admin context and sharing with all the users [2024-01-18T16:24:14.021Z] 2024-01-18 16:24:13 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-jeget] for org [default] with [ibm] as provider [2024-01-18T16:24:14.021Z] 2024-01-18 16:24:13 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:24:14.021Z] 2024-01-18 16:24:13 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:24:14.021Z] 2024-01-18 16:24:13 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-jeget] is present [2024-01-18T16:24:14.278Z] 2024-01-18 16:24:14 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Create cluster [source-cluster] in org [default] [2024-01-18T16:24:14.840Z] 2024-01-18 16:24:14 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-kweyt] for IBM [2024-01-18T16:24:14.840Z] 2024-01-18 16:24:14 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-kweyt] for org [default] with [ibm] as provider [2024-01-18T16:24:14.840Z] 2024-01-18 16:24:14 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:24:14.841Z] 2024-01-18 16:24:14 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:24:14.841Z] 2024-01-18 16:24:14 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-kweyt] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName11 LastName11 with roles: [px-backup-app.user]] [2024-01-18T16:24:14.841Z] 2024-01-18 16:24:14 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName11 LastName11 with roles: [px-backup-app.user] [2024-01-18T16:24:15.402Z] 2024-01-18 16:24:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-kweyt from admin context and sharing with all the users [2024-01-18T16:24:15.402Z] 2024-01-18 16:24:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-kweyt] for org [default] with [ibm] as provider [2024-01-18T16:24:15.402Z] 2024-01-18 16:24:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:24:15.402Z] 2024-01-18 16:24:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:24:15.402Z] 2024-01-18 16:24:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-kweyt] is present [2024-01-18T16:24:15.659Z] 2024-01-18 16:24:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Create cluster [destination-cluster] in org [default] [2024-01-18T16:24:16.220Z] INFO[2024-01-18 16:24:15] Verifying : Description : Creating source and destination cluster [2024-01-18T16:24:16.220Z] INFO[2024-01-18 16:24:15] Actual:true, Expected: true [2024-01-18T16:24:16.477Z] 2024/01/18 16:24:16 DoRetryWithTimeout - Error: {restore status for [tp-restore-1705595055] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T16:24:48.563Z] 2024-01-18 16:24:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.CreateRestore:#936] - Restore [tp-restore-1705595055] created successfully [2024-01-18T16:24:48.563Z] 2024-01-18 16:24:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Switching cluster context to cluster [destination-cluster] [2024-01-18T16:24:48.563Z] 2024-01-18 16:24:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Switching context to [/tmp/destination-config] [2024-01-18T16:24:48.563Z] 2024-01-18 16:24:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:24:48.563Z] 2024-01-18 16:24:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:24:48.563Z] 2024-01-18 16:24:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:24:48.563Z] 2024-01-18 16:24:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:24:48.563Z] 2024-01-18 16:24:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:24:48.563Z] 2024-01-18 16:24:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:24:48.563Z] 2024-01-18 16:24:47 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:24:48.563Z] 2024-01-18 16:24:47 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:24:48.563Z] 2024-01-18 16:24:47 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:24:48.563Z] 2024-01-18 16:24:48 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:24:48.563Z] 2024-01-18 16:24:48 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:24:48.563Z] 2024-01-18 16:24:48 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:24:48.563Z] 2024-01-18 16:24:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:24:48.563Z] 2024-01-18 16:24:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Switched context to [/tmp/destination-config] [2024-01-18T16:24:48.563Z] 2024-01-18 16:24:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-82934-01-18-16h10m24s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-18T16:24:48.563Z] 2024-01-18 16:24:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:24:48.563Z] 2024-01-18 16:24:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T16:24:48.564Z] 2024-01-18 16:24:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T16:24:48.564Z] 2024-01-18 16:24:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating restore [tp-restore-1705595055] in org [default] [2024-01-18T16:24:48.564Z] 2024-01-18 16:24:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-restore-1705595055] [2024-01-18T16:24:48.564Z] 2024-01-18 16:24:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-restore-1705595055] [2024-01-18T16:24:48.564Z] 2024-01-18 16:24:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in restore [tp-restore-1705595055] [2024-01-18T16:24:48.564Z] 2024-01-18 16:24:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-82934-01-18-16h10m24s]) was found in restore [tp-restore-1705595055], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:24:48.824Z] 2024-01-18 16:24:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-82934-01-18-16h10m24s]) found in the restore [tp-restore-1705595055] was also present on the cluster/namespace [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:24:48.824Z] 2024-01-18 16:24:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-82934-01-18-16h10m24s]) was found in restore [tp-restore-1705595055], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:24:48.824Z] 2024-01-18 16:24:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-82934-01-18-16h10m24s]) found in the restore [tp-restore-1705595055] was also present on the cluster/namespace [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:24:48.824Z] 2024-01-18 16:24:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-82934-01-18-16h10m24s]) was found in restore [tp-restore-1705595055], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:24:49.080Z] 2024-01-18 16:24:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-82934-01-18-16h10m24s]) found in the restore [tp-restore-1705595055] was also present on the cluster/namespace [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:24:49.080Z] 2024-01-18 16:24:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-82934-01-18-16h10m24s]) was found in restore [tp-restore-1705595055], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:24:49.080Z] 2024-01-18 16:24:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-82934-01-18-16h10m24s]) found in the restore [tp-restore-1705595055] was also present on the cluster/namespace [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:24:49.080Z] 2024-01-18 16:24:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in restore [tp-restore-1705595055] [2024-01-18T16:24:49.080Z] 2024-01-18 16:24:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-82934-01-18-16h10m24s] in restore [tp-restore-1705595055] [2024-01-18T16:24:49.080Z] 2024-01-18 16:24:48 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:24:49.080Z] 2024-01-18 16:24:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:24:49.080Z] 2024-01-18 16:24:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[mysql-data]] [2024-01-18T16:24:49.080Z] 2024-01-18 16:24:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.ValidateRestore:#2243] - in restore [tp-restore-1705595055], the status of the restored volume [pvc-5e1058d4-0546-40e4-bd6d-cf85fd428f46] was not Success. It was [Retained] with reason [Skipped from volume restore as policy is set to Retain and pvc already exists] [2024-01-18T16:24:49.080Z] 2024-01-18 16:24:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validate applications in restored namespace [mysql-ibm-pxb-0-82934-01-18-16h10m24s] due to restore [tp-restore-1705595055] [2024-01-18T16:24:49.080Z] 2024-01-18 16:24:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating mysql-ibm app [2024-01-18T16:24:49.080Z] 2024-01-18 16:24:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] wait for mysql-ibm app to start running [2024-01-18T16:24:49.336Z] 2024-01-18 16:24:49 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T16:24:49.336Z] 2024-01-18 16:24:49 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T16:24:49.592Z] 2024-01-18 16:24:49 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T16:24:49.848Z] 2024-01-18 16:24:49 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validating CR cleanup [2024-01-18T16:24:49.848Z] 2024-01-18 16:24:49 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:24:50.104Z] 2024-01-18 16:24:49 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] All CRs in [mysql-ibm-pxb-0-82934-01-18-16h10m24s] are [[]] [2024-01-18T16:24:50.104Z] 2024-01-18 16:24:49 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Switching cluster context back to cluster path [/tmp/source-config] [2024-01-18T16:24:50.104Z] 2024-01-18 16:24:49 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Switching context to [/tmp/source-config] [2024-01-18T16:24:50.105Z] 2024-01-18 16:24:49 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:24:50.105Z] 2024-01-18 16:24:49 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:24:50.105Z] 2024-01-18 16:24:50 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:24:50.105Z] 2024-01-18 16:24:50 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:24:50.373Z] 2024-01-18 16:24:50 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:24:50.373Z] 2024-01-18 16:24:50 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:24:50.943Z] 2024-01-18 16:24:50 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:24:51.200Z] 2024-01-18 16:24:51 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:24:51.761Z] 2024-01-18 16:24:51 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:24:51.761Z] 2024-01-18 16:24:51 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:24:52.017Z] 2024-01-18 16:24:51 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:24:52.017Z] 2024-01-18 16:24:51 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:24:52.017Z] 2024-01-18 16:24:51 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:24:52.017Z] 2024-01-18 16:24:51 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Switched context to [/tmp/source-config] [2024-01-18T16:24:52.017Z] 2024-01-18 16:24:51 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Restoring of backup [tp-backup-1705594475] was successful with name - [tp-restore-1705595055] [2024-01-18T16:24:52.017Z] 2024-01-18 16:24:51 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.glob..func83.2.12:#792] - About to delete restore - tp-restore-1705595055 [2024-01-18T16:24:52.017Z] INFO[2024-01-18 16:24:51] Verifying : Description : Getting the backup driver [2024-01-18T16:24:52.017Z] INFO[2024-01-18 16:24:51] Actual:true, Expected: true [2024-01-18T16:24:52.017Z] INFO[2024-01-18 16:24:51] Verifying : Description : Deleting Restore tp-restore-1705595055 [2024-01-18T16:24:52.017Z] INFO[2024-01-18 16:24:51] Actual:true, Expected: true [2024-01-18T16:24:52.017Z] 2024-01-18 16:24:51 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Deleting Restore [tp-restore-1705595055] was successful [2024-01-18T16:24:52.272Z] INFO[2024-01-18 16:24:52] Verifying : Description : Verifying backup deletion is not possible [2024-01-18T16:24:52.272Z] INFO[2024-01-18 16:24:52] Actual:true, Expected: true [2024-01-18T16:24:52.272Z] 2024-01-18 16:24:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Validate that user with View Only access cannot restore or delete the backup [2024-01-18T16:24:52.272Z] 2024-01-18 16:24:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.glob..func83.2.13:#810] - Sharing backup with user - testuser4 [2024-01-18T16:24:52.541Z] 2024-01-18 16:24:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Registering Source and Destination clusters from user context [2024-01-18T16:24:52.541Z] INFO[2024-01-18 16:24:52] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T16:24:52.541Z] INFO[2024-01-18 16:24:52] Actual:true, Expected: true [2024-01-18T16:24:52.541Z] 2024-01-18 16:24:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T16:24:52.541Z] 2024-01-18 16:24:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T16:24:52.541Z] 2024-01-18 16:24:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T16:24:52.542Z] 2024-01-18 16:24:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T16:24:52.542Z] 2024-01-18 16:24:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Create cluster [source-cluster] in org [default] [2024-01-18T16:24:52.542Z] 2024-01-18 16:24:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T16:24:52.542Z] 2024-01-18 16:24:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T16:24:52.542Z] 2024-01-18 16:24:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Create cluster [destination-cluster] in org [default] [2024-01-18T16:24:52.542Z] 2024-01-18 16:24:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T16:24:52.542Z] 2024-01-18 16:24:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T16:24:52.542Z] 2024-01-18 16:24:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-tgism] for IBM [2024-01-18T16:24:52.542Z] 2024-01-18 16:24:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-tgism] for org [default] with [ibm] as provider [2024-01-18T16:24:52.542Z] 2024-01-18 16:24:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:24:52.542Z] 2024-01-18 16:24:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:24:52.542Z] 2024-01-18 16:24:52 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-tgism] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName4 LastName4 with roles: [px-backup-app.user]] [2024-01-18T16:24:52.542Z] 2024-01-18 16:24:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName4 LastName4 with roles: [px-backup-app.user] [2024-01-18T16:24:54.430Z] 2024-01-18 16:24:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-tgism from admin context and sharing with all the users [2024-01-18T16:24:54.430Z] 2024-01-18 16:24:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-tgism] for org [default] with [ibm] as provider [2024-01-18T16:24:54.430Z] 2024-01-18 16:24:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:24:54.430Z] 2024-01-18 16:24:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:24:54.430Z] 2024-01-18 16:24:54 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-tgism] is present [2024-01-18T16:24:54.430Z] 2024-01-18 16:24:54 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Create cluster [source-cluster] in org [default] [2024-01-18T16:24:54.430Z] 2024-01-18 16:24:54 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-teqcf] for IBM [2024-01-18T16:24:54.430Z] 2024-01-18 16:24:54 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-teqcf] for org [default] with [ibm] as provider [2024-01-18T16:24:54.430Z] 2024-01-18 16:24:54 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:24:54.430Z] 2024-01-18 16:24:54 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:24:54.430Z] 2024-01-18 16:24:54 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-teqcf] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName4 LastName4 with roles: [px-backup-app.user]] [2024-01-18T16:24:54.430Z] 2024-01-18 16:24:54 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName4 LastName4 with roles: [px-backup-app.user] [2024-01-18T16:24:55.357Z] 2024-01-18 16:24:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-teqcf from admin context and sharing with all the users [2024-01-18T16:24:55.357Z] 2024-01-18 16:24:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-teqcf] for org [default] with [ibm] as provider [2024-01-18T16:24:55.357Z] 2024-01-18 16:24:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:24:55.357Z] 2024-01-18 16:24:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:24:55.612Z] 2024-01-18 16:24:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-teqcf] is present [2024-01-18T16:24:55.612Z] 2024-01-18 16:24:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Create cluster [destination-cluster] in org [default] [2024-01-18T16:24:55.612Z] INFO[2024-01-18 16:24:55] Verifying : Description : Creating source and destination cluster [2024-01-18T16:24:55.612Z] INFO[2024-01-18 16:24:55] Actual:true, Expected: true [2024-01-18T16:24:55.868Z] 2024-01-18 16:24:55 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.glob..func83.2.13:#826] - Error while trying to restore - rpc error: code = Unknown desc = user [53771e25-ac89-41a9-b6e9-ab0f47eacfdf] doesn't have permission to restore backup [tp-restore-1705595095] [2024-01-18T16:24:55.868Z] INFO[2024-01-18 16:24:55] Verifying : Description : Verifying backup restore is not possible [2024-01-18T16:24:55.868Z] INFO[2024-01-18 16:24:55] Actual:true, Expected: true [2024-01-18T16:24:56.797Z] INFO[2024-01-18 16:24:56] Verifying : Description : Verifying backup deletion is not possible [2024-01-18T16:24:56.797Z] INFO[2024-01-18 16:24:56] Actual:true, Expected: true [2024-01-18T16:24:56.797Z] 2024-01-18 16:24:56 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Deleting the deployed apps after the testcase [2024-01-18T16:24:56.797Z] 2024-01-18 16:24:56 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Destroying apps [2024-01-18T16:24:56.797Z] 2024-01-18 16:24:56 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T16:24:57.359Z] 2024-01-18 16:24:57 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T16:24:57.359Z] 2024-01-18 16:24:57 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] start destroying mysql-ibm app [2024-01-18T16:24:57.359Z] 2024-01-18 16:24:57 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T16:24:57.617Z] 2024-01-18 16:24:57 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T16:24:57.617Z] 2024-01-18 16:24:57 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T16:24:57.617Z] INFO[2024-01-18 16:24:57] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T16:24:57.617Z] INFO[2024-01-18 16:24:57] Actual:true, Expected: true [2024-01-18T16:24:57.617Z] 2024-01-18 16:24:57 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.glob..func83.3:#851] - Generating user context [2024-01-18T16:25:09.784Z] 2024-01-18 16:25:09 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.glob..func83.3:#857] - Deleting registered clusters for non-admin context [2024-01-18T16:25:09.784Z] 2024-01-18 16:25:09 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:09.784Z] INFO[2024-01-18 16:25:09] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:09.784Z] INFO[2024-01-18 16:25:09] Actual:true, Expected: true [2024-01-18T16:25:09.784Z] 2024-01-18 16:25:09 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:09.784Z] INFO[2024-01-18 16:25:09] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:09.784Z] INFO[2024-01-18 16:25:09] Actual:true, Expected: true [2024-01-18T16:25:09.784Z] 2024-01-18 16:25:09 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:09.784Z] INFO[2024-01-18 16:25:09] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:09.784Z] INFO[2024-01-18 16:25:09] Actual:true, Expected: true [2024-01-18T16:25:09.784Z] 2024-01-18 16:25:09 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:09.784Z] INFO[2024-01-18 16:25:09] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:09.784Z] INFO[2024-01-18 16:25:09] Actual:true, Expected: true [2024-01-18T16:25:09.785Z] 2024-01-18 16:25:09 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:09.785Z] INFO[2024-01-18 16:25:09] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:09.785Z] INFO[2024-01-18 16:25:09] Actual:true, Expected: true [2024-01-18T16:25:09.785Z] 2024-01-18 16:25:09 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:09.785Z] INFO[2024-01-18 16:25:09] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:09.785Z] INFO[2024-01-18 16:25:09] Actual:true, Expected: true [2024-01-18T16:25:09.785Z] 2024-01-18 16:25:09 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:09.785Z] INFO[2024-01-18 16:25:09] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:09.785Z] INFO[2024-01-18 16:25:09] Actual:true, Expected: true [2024-01-18T16:25:09.785Z] 2024-01-18 16:25:09 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:09.785Z] INFO[2024-01-18 16:25:09] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:09.785Z] INFO[2024-01-18 16:25:09] Actual:true, Expected: true [2024-01-18T16:25:09.785Z] INFO[2024-01-18 16:25:09] Verifying : Description : Deleting cluster source-cluster [2024-01-18T16:25:09.785Z] INFO[2024-01-18 16:25:09] Actual:true, Expected: true [2024-01-18T16:25:11.149Z] INFO[2024-01-18 16:25:10] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-tiuda] [2024-01-18T16:25:11.149Z] INFO[2024-01-18 16:25:10] Actual:true, Expected: true [2024-01-18T16:25:11.149Z] INFO[2024-01-18 16:25:11] Verifying : Description : Deleting cluster destination-cluster [2024-01-18T16:25:11.149Z] INFO[2024-01-18 16:25:11] Actual:true, Expected: true [2024-01-18T16:25:12.513Z] INFO[2024-01-18 16:25:12] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-xhyew] [2024-01-18T16:25:12.513Z] INFO[2024-01-18 16:25:12] Actual:true, Expected: true [2024-01-18T16:25:12.513Z] 2024-01-18 16:25:12 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:12.513Z] INFO[2024-01-18 16:25:12] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:12.513Z] INFO[2024-01-18 16:25:12] Actual:true, Expected: true [2024-01-18T16:25:12.513Z] 2024-01-18 16:25:12 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:12.513Z] INFO[2024-01-18 16:25:12] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:12.513Z] INFO[2024-01-18 16:25:12] Actual:true, Expected: true [2024-01-18T16:25:12.513Z] 2024-01-18 16:25:12 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:12.513Z] INFO[2024-01-18 16:25:12] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:12.513Z] INFO[2024-01-18 16:25:12] Actual:true, Expected: true [2024-01-18T16:25:12.513Z] 2024-01-18 16:25:12 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:12.513Z] INFO[2024-01-18 16:25:12] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:12.513Z] INFO[2024-01-18 16:25:12] Actual:true, Expected: true [2024-01-18T16:25:12.513Z] 2024-01-18 16:25:12 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:12.513Z] INFO[2024-01-18 16:25:12] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:12.513Z] INFO[2024-01-18 16:25:12] Actual:true, Expected: true [2024-01-18T16:25:12.769Z] INFO[2024-01-18 16:25:12] Verifying : Description : Deleting cluster source-cluster [2024-01-18T16:25:12.769Z] INFO[2024-01-18 16:25:12] Actual:true, Expected: true [2024-01-18T16:25:13.695Z] INFO[2024-01-18 16:25:13] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-jeget] [2024-01-18T16:25:13.695Z] INFO[2024-01-18 16:25:13] Actual:true, Expected: true [2024-01-18T16:25:13.953Z] INFO[2024-01-18 16:25:13] Verifying : Description : Deleting cluster destination-cluster [2024-01-18T16:25:13.953Z] INFO[2024-01-18 16:25:13] Actual:true, Expected: true [2024-01-18T16:25:15.319Z] INFO[2024-01-18 16:25:14] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-kweyt] [2024-01-18T16:25:15.319Z] INFO[2024-01-18 16:25:14] Actual:true, Expected: true [2024-01-18T16:25:15.319Z] 2024-01-18 16:25:14 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:15.319Z] INFO[2024-01-18 16:25:15] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:15.319Z] INFO[2024-01-18 16:25:15] Actual:true, Expected: true [2024-01-18T16:25:15.319Z] 2024-01-18 16:25:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:15.319Z] INFO[2024-01-18 16:25:15] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:15.319Z] INFO[2024-01-18 16:25:15] Actual:true, Expected: true [2024-01-18T16:25:15.319Z] 2024-01-18 16:25:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:15.319Z] INFO[2024-01-18 16:25:15] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:15.319Z] INFO[2024-01-18 16:25:15] Actual:true, Expected: true [2024-01-18T16:25:15.319Z] 2024-01-18 16:25:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:15.319Z] INFO[2024-01-18 16:25:15] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:15.319Z] INFO[2024-01-18 16:25:15] Actual:true, Expected: true [2024-01-18T16:25:15.319Z] 2024-01-18 16:25:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:15.319Z] INFO[2024-01-18 16:25:15] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:15.319Z] INFO[2024-01-18 16:25:15] Actual:true, Expected: true [2024-01-18T16:25:15.319Z] 2024-01-18 16:25:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:15.319Z] INFO[2024-01-18 16:25:15] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:15.319Z] INFO[2024-01-18 16:25:15] Actual:true, Expected: true [2024-01-18T16:25:15.319Z] 2024-01-18 16:25:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:15.319Z] INFO[2024-01-18 16:25:15] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:15.319Z] INFO[2024-01-18 16:25:15] Actual:true, Expected: true [2024-01-18T16:25:15.320Z] 2024-01-18 16:25:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:15.320Z] INFO[2024-01-18 16:25:15] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:15.320Z] INFO[2024-01-18 16:25:15] Actual:true, Expected: true [2024-01-18T16:25:15.320Z] 2024-01-18 16:25:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:15.320Z] INFO[2024-01-18 16:25:15] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:15.320Z] INFO[2024-01-18 16:25:15] Actual:true, Expected: true [2024-01-18T16:25:15.320Z] 2024-01-18 16:25:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:15.320Z] INFO[2024-01-18 16:25:15] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:15.320Z] INFO[2024-01-18 16:25:15] Actual:true, Expected: true [2024-01-18T16:25:15.320Z] 2024-01-18 16:25:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:15.320Z] INFO[2024-01-18 16:25:15] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:15.320Z] INFO[2024-01-18 16:25:15] Actual:true, Expected: true [2024-01-18T16:25:15.320Z] 2024-01-18 16:25:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:15.320Z] INFO[2024-01-18 16:25:15] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:15.320Z] INFO[2024-01-18 16:25:15] Actual:true, Expected: true [2024-01-18T16:25:15.320Z] 2024-01-18 16:25:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:15.320Z] INFO[2024-01-18 16:25:15] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:15.320Z] INFO[2024-01-18 16:25:15] Actual:true, Expected: true [2024-01-18T16:25:15.320Z] 2024-01-18 16:25:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:15.320Z] INFO[2024-01-18 16:25:15] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:15.320Z] INFO[2024-01-18 16:25:15] Actual:true, Expected: true [2024-01-18T16:25:15.320Z] 2024-01-18 16:25:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:15.320Z] INFO[2024-01-18 16:25:15] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:15.320Z] INFO[2024-01-18 16:25:15] Actual:true, Expected: true [2024-01-18T16:25:15.320Z] 2024-01-18 16:25:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:15.320Z] INFO[2024-01-18 16:25:15] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:15.320Z] INFO[2024-01-18 16:25:15] Actual:true, Expected: true [2024-01-18T16:25:15.320Z] 2024-01-18 16:25:15 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:25:15.321Z] INFO[2024-01-18 16:25:15] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:25:15.321Z] INFO[2024-01-18 16:25:15] Actual:true, Expected: true [2024-01-18T16:25:15.576Z] INFO[2024-01-18 16:25:15] Verifying : Description : Deleting cluster source-cluster [2024-01-18T16:25:15.576Z] INFO[2024-01-18 16:25:15] Actual:true, Expected: true [2024-01-18T16:25:16.502Z] INFO[2024-01-18 16:25:16] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-iesgm] [2024-01-18T16:25:16.502Z] INFO[2024-01-18 16:25:16] Actual:true, Expected: true [2024-01-18T16:25:16.759Z] INFO[2024-01-18 16:25:16] Verifying : Description : Deleting cluster destination-cluster [2024-01-18T16:25:16.759Z] INFO[2024-01-18 16:25:16] Actual:true, Expected: true [2024-01-18T16:25:18.125Z] INFO[2024-01-18 16:25:17] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-kboet] [2024-01-18T16:25:18.125Z] INFO[2024-01-18 16:25:17] Actual:true, Expected: true [2024-01-18T16:25:18.125Z] 2024-01-18 16:25:17 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.glob..func83.3:#863] - Cleaning up users [2024-01-18T16:25:19.054Z] I0118 16:25:18.862569 14 request.go:682] Waited for 1.071308269s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/services [2024-01-18T16:25:28.996Z] I0118 16:25:28.863136 14 request.go:682] Waited for 5.959533997s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-18T16:25:41.161Z] I0118 16:25:39.062529 14 request.go:682] Waited for 5.957832021s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-18T16:25:49.242Z] 2024-01-18 16:25:47 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser21 - b2397937-b93d-418f-9e5c-76f038287674 [2024-01-18T16:25:49.242Z] 2024-01-18 16:25:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser8 - 1d1b04e2-4083-4be8-b015-f1d0cb4dce6c [2024-01-18T16:25:49.242Z] 2024-01-18 16:25:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser12 - 29f788b4-c927-44ed-9ed0-3be78b78209b [2024-01-18T16:25:49.242Z] 2024-01-18 16:25:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser4 - 53771e25-ac89-41a9-b6e9-ab0f47eacfdf [2024-01-18T16:25:49.242Z] 2024-01-18 16:25:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser9 - c56347b4-2d1a-4976-a0be-517b2b77c3e3 [2024-01-18T16:25:49.242Z] 2024-01-18 16:25:48 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser20 - 301e807b-65ff-4e2d-8e6b-bf2a3c5d3c76 [2024-01-18T16:25:49.242Z] I0118 16:25:49.063368 14 request.go:682] Waited for 5.947855257s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-18T16:25:49.242Z] 2024-01-18 16:25:49 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser13 - 57f3811d-4de9-4ec4-80e5-030c5264840d [2024-01-18T16:25:49.498Z] 2024-01-18 16:25:49 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1 - adc3e6f5-ef9f-40b8-84e5-01f4eb60e534 [2024-01-18T16:25:49.754Z] 2024-01-18 16:25:49 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser10 - 58716ecb-6f47-4df3-b258-706298644350 [2024-01-18T16:25:49.754Z] 2024-01-18 16:25:49 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser5 - 270f338e-2a21-4ce9-8c7a-809633d068e2 [2024-01-18T16:25:50.010Z] 2024-01-18 16:25:49 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser18 - 0fe37ee7-2c57-4ea1-bfda-965b6552d578 [2024-01-18T16:25:50.266Z] 2024-01-18 16:25:50 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser6 - 9ebb8fd7-6ba5-4fc4-b3be-86479b298a43 [2024-01-18T16:25:50.522Z] 2024-01-18 16:25:50 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser14 - 3528b287-f84c-478a-a4c3-d292bc741866 [2024-01-18T16:25:50.778Z] 2024-01-18 16:25:50 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser11 - 284064c8-085f-42e4-824c-2ac98310a559 [2024-01-18T16:25:50.778Z] 2024-01-18 16:25:50 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2 - b2f8cd7d-526e-4ffd-9096-134d8d8560d9 [2024-01-18T16:25:51.039Z] 2024-01-18 16:25:50 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser30 - 6b3e77b4-0c35-4128-9d2a-d0c87fe2213b [2024-01-18T16:25:51.338Z] 2024-01-18 16:25:51 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser7 - 62348be6-6d86-43ab-878e-cfe73ecde896 [2024-01-18T16:25:51.338Z] 2024-01-18 16:25:51 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser25 - 428618c8-0ac2-4dcb-9e13-c142c2b56a51 [2024-01-18T16:25:51.594Z] 2024-01-18 16:25:51 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser16 - 3cf6916f-6377-4b5a-8b15-fdc5abd42949 [2024-01-18T16:25:51.850Z] 2024-01-18 16:25:51 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser27 - 270817d8-75b8-454c-9d46-9d1797f4beae [2024-01-18T16:25:52.106Z] 2024-01-18 16:25:51 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser15 - a0fb1c7b-46f4-4246-907c-18f6a827a996 [2024-01-18T16:25:52.362Z] 2024-01-18 16:25:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser19 - 64796d3f-b41d-4387-ba7d-0a313701c792 [2024-01-18T16:25:52.362Z] 2024-01-18 16:25:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser24 - c0116ca8-4fbd-4aa2-a698-74594a386929 [2024-01-18T16:25:52.619Z] 2024-01-18 16:25:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser3 - fb198252-1d32-4b0c-9bde-af15603b57db [2024-01-18T16:25:52.874Z] 2024-01-18 16:25:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser22 - b4ac58d6-e50e-4ab2-acef-60a5f199c08f [2024-01-18T16:25:53.131Z] 2024-01-18 16:25:52 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser17 - 8faa871e-75a8-4888-83ec-19f531408614 [2024-01-18T16:25:53.131Z] 2024-01-18 16:25:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser29 - 388a33f8-f8bb-4b86-86ff-6bfe389be493 [2024-01-18T16:25:53.386Z] 2024-01-18 16:25:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser28 - 3b5b4e76-ab60-4e61-be30-8274bd814f56 [2024-01-18T16:25:53.642Z] 2024-01-18 16:25:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser26 - c5d1d55f-df19-4d58-b3a9-f55c8cde08e2 [2024-01-18T16:25:53.898Z] 2024-01-18 16:25:53 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser23 - a21f3c2e-a4cc-40ce-9890-cd3b41dc5e1b [2024-01-18T16:26:00.431Z] I0118 16:25:59.262699 14 request.go:682] Waited for 5.967536869s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/services [2024-01-18T16:26:00.431Z] 2024-01-18 16:26:00 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser21] deletion - [2024-01-18T16:26:00.431Z] 2024-01-18 16:26:00 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser21 [2024-01-18T16:26:00.431Z] 2024-01-18 16:26:00 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser8] deletion - [2024-01-18T16:26:00.431Z] 2024-01-18 16:26:00 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser8 [2024-01-18T16:26:00.432Z] 2024-01-18 16:26:00 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser12] deletion - [2024-01-18T16:26:00.432Z] 2024-01-18 16:26:00 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser12 [2024-01-18T16:26:00.690Z] 2024-01-18 16:26:00 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser4] deletion - [2024-01-18T16:26:00.690Z] 2024-01-18 16:26:00 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser4 [2024-01-18T16:26:00.949Z] 2024-01-18 16:26:00 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser9] deletion - [2024-01-18T16:26:00.949Z] 2024-01-18 16:26:00 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser9 [2024-01-18T16:26:01.205Z] 2024-01-18 16:26:01 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser20] deletion - [2024-01-18T16:26:01.205Z] 2024-01-18 16:26:01 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser20 [2024-01-18T16:26:01.462Z] 2024-01-18 16:26:01 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser13] deletion - [2024-01-18T16:26:01.462Z] 2024-01-18 16:26:01 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser13 [2024-01-18T16:26:01.718Z] 2024-01-18 16:26:01 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser1] deletion - [2024-01-18T16:26:01.718Z] 2024-01-18 16:26:01 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser1 [2024-01-18T16:26:01.718Z] 2024-01-18 16:26:01 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser10] deletion - [2024-01-18T16:26:01.718Z] 2024-01-18 16:26:01 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser10 [2024-01-18T16:26:01.974Z] 2024-01-18 16:26:01 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser5] deletion - [2024-01-18T16:26:01.974Z] 2024-01-18 16:26:01 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser5 [2024-01-18T16:26:02.231Z] 2024-01-18 16:26:01 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser18] deletion - [2024-01-18T16:26:02.231Z] 2024-01-18 16:26:01 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser18 [2024-01-18T16:26:02.487Z] 2024-01-18 16:26:02 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser6] deletion - [2024-01-18T16:26:02.487Z] 2024-01-18 16:26:02 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser6 [2024-01-18T16:26:02.487Z] 2024-01-18 16:26:02 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser14] deletion - [2024-01-18T16:26:02.487Z] 2024-01-18 16:26:02 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser14 [2024-01-18T16:26:02.742Z] 2024-01-18 16:26:02 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser11] deletion - [2024-01-18T16:26:02.742Z] 2024-01-18 16:26:02 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser11 [2024-01-18T16:26:02.998Z] 2024-01-18 16:26:02 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser2] deletion - [2024-01-18T16:26:02.998Z] 2024-01-18 16:26:02 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser2 [2024-01-18T16:26:03.255Z] 2024-01-18 16:26:03 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser30] deletion - [2024-01-18T16:26:03.255Z] 2024-01-18 16:26:03 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser30 [2024-01-18T16:26:03.255Z] 2024-01-18 16:26:03 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser7] deletion - [2024-01-18T16:26:03.256Z] 2024-01-18 16:26:03 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser7 [2024-01-18T16:26:03.511Z] 2024-01-18 16:26:03 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser25] deletion - [2024-01-18T16:26:03.512Z] 2024-01-18 16:26:03 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser25 [2024-01-18T16:26:03.767Z] 2024-01-18 16:26:03 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser16] deletion - [2024-01-18T16:26:03.767Z] 2024-01-18 16:26:03 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser16 [2024-01-18T16:26:04.023Z] 2024-01-18 16:26:03 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser27] deletion - [2024-01-18T16:26:04.023Z] 2024-01-18 16:26:03 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser27 [2024-01-18T16:26:04.278Z] 2024-01-18 16:26:03 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser15] deletion - [2024-01-18T16:26:04.278Z] 2024-01-18 16:26:03 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser15 [2024-01-18T16:26:04.278Z] 2024-01-18 16:26:04 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser19] deletion - [2024-01-18T16:26:04.278Z] 2024-01-18 16:26:04 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser19 [2024-01-18T16:26:04.580Z] 2024-01-18 16:26:04 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser24] deletion - [2024-01-18T16:26:04.580Z] 2024-01-18 16:26:04 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser24 [2024-01-18T16:26:04.836Z] 2024-01-18 16:26:04 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser3] deletion - [2024-01-18T16:26:04.836Z] 2024-01-18 16:26:04 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser3 [2024-01-18T16:26:04.836Z] 2024-01-18 16:26:04 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser22] deletion - [2024-01-18T16:26:04.836Z] 2024-01-18 16:26:04 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser22 [2024-01-18T16:26:05.095Z] 2024-01-18 16:26:04 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser17] deletion - [2024-01-18T16:26:05.095Z] 2024-01-18 16:26:04 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser17 [2024-01-18T16:26:05.350Z] 2024-01-18 16:26:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser29] deletion - [2024-01-18T16:26:05.350Z] 2024-01-18 16:26:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser29 [2024-01-18T16:26:05.606Z] 2024-01-18 16:26:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser28] deletion - [2024-01-18T16:26:05.606Z] 2024-01-18 16:26:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser28 [2024-01-18T16:26:05.862Z] 2024-01-18 16:26:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser26] deletion - [2024-01-18T16:26:05.862Z] 2024-01-18 16:26:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser26 [2024-01-18T16:26:05.862Z] 2024-01-18 16:26:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser23] deletion - [2024-01-18T16:26:05.862Z] 2024-01-18 16:26:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser23 [2024-01-18T16:26:05.862Z] 2024-01-18 16:26:05 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.glob..func83.3:#875] - Cleaning up groups [2024-01-18T16:26:09.125Z] 2024-01-18 16:26:08 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 4d075c71-08af-4c19-b984-43e56fb3163b [2024-01-18T16:26:09.125Z] 2024-01-18 16:26:09 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup3 - 18ad0d59-cf6b-43c8-b03f-131d76391cb7 [2024-01-18T16:26:09.381Z] 2024-01-18 16:26:09 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup2 - a0d64e30-6000-4fb3-9987-7d187337cd0a [2024-01-18T16:26:10.316Z] 2024-01-18 16:26:10 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteGroup:#1018] - Deleted Group - testGroup1 [2024-01-18T16:26:10.879Z] 2024-01-18 16:26:10 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteGroup:#1018] - Deleted Group - testGroup3 [2024-01-18T16:26:10.879Z] 2024-01-18 16:26:10 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.DeleteGroup:#1018] - Deleted Group - testGroup2 [2024-01-18T16:26:12.294Z] 2024-01-18 16:26:11 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Cleaning backup locations in map [map[be530265-bfe7-4ca2-92f7-7e246e709c59:autogenerated-backup-location-1705594430]], cloud credential [autogenerated-cred-1705594430], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:26:12.294Z] INFO[2024-01-18 16:26:11] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-1705594430] [2024-01-18T16:26:12.294Z] INFO[2024-01-18 16:26:11] Actual:true, Expected: true [2024-01-18T16:26:12.294Z] 2024-01-18 16:26:12 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705594430] is present [2024-01-18T16:26:12.294Z] 2024/01/18 16:26:12 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705594430 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T16:26:44.328Z] 2024-01-18 16:26:42 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705594430] is present [2024-01-18T16:26:44.328Z] 2024/01/18 16:26:42 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705594430 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T16:27:16.358Z] 2024-01-18 16:27:12 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705594430] is present [2024-01-18T16:27:16.358Z] 2024/01/18 16:27:12 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705594430 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T16:27:42.854Z] 2024-01-18 16:27:42 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu] [2024-01-18T16:27:42.854Z] INFO[2024-01-18 16:27:42] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-1705594430 [2024-01-18T16:27:42.854Z] INFO[2024-01-18 16:27:42] Actual:true, Expected: true [2024-01-18T16:27:42.854Z] 2024-01-18 16:27:42 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-1705594430] is present [2024-01-18T16:27:42.854Z] INFO[2024-01-18 16:27:42] Verifying : Description : Verifying if cloud cred [autogenerated-cred-1705594430] is present [2024-01-18T16:27:42.854Z] INFO[2024-01-18 16:27:42] Actual:true, Expected: true [2024-01-18T16:27:42.854Z] INFO[2024-01-18 16:27:42] Verifying : Description : Verifying deletion of cloud cred [autogenerated-cred-1705594430] [2024-01-18T16:27:42.854Z] INFO[2024-01-18 16:27:42] Actual:true, Expected: true [2024-01-18T16:27:42.855Z] INFO[2024-01-18 16:27:42] Verifying : Description : Deleting cloud cred autogenerated-cred-1705594430 [2024-01-18T16:27:42.855Z] INFO[2024-01-18 16:27:42] Actual:true, Expected: true [2024-01-18T16:27:42.855Z] INFO[2024-01-18 16:27:42] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:27:42.855Z] INFO[2024-01-18 16:27:42] Actual:true, Expected: true [2024-01-18T16:27:42.855Z] INFO[2024-01-18 16:27:42] Verifying : Description : Deleting cluster source-cluster [2024-01-18T16:27:42.855Z] INFO[2024-01-18 16:27:42] Actual:true, Expected: true [2024-01-18T16:27:43.415Z] INFO[2024-01-18 16:27:43] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-pgvpv] [2024-01-18T16:27:43.415Z] INFO[2024-01-18 16:27:43] Actual:true, Expected: true [2024-01-18T16:27:43.415Z] INFO[2024-01-18 16:27:43] Verifying : Description : Deleting cluster destination-cluster [2024-01-18T16:27:43.415Z] INFO[2024-01-18 16:27:43] Actual:true, Expected: true [2024-01-18T16:27:43.672Z] INFO[2024-01-18 16:27:43] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-jncin] [2024-01-18T16:27:43.672Z] INFO[2024-01-18 16:27:43] Actual:true, Expected: true [2024-01-18T16:27:43.672Z] INFO[2024-01-18 16:27:43] --------Test End------ [2024-01-18T16:27:43.672Z] INFO[2024-01-18 16:27:43] #Test: [2024-01-18T16:27:43.672Z] INFO[2024-01-18 16:27:43] #Description: [2024-01-18T16:27:43.672Z] INFO[2024-01-18 16:27:43] ------------------------ [2024-01-18T16:27:43.927Z] 2024-01-18 16:27:43 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:27:44.182Z] 2024-01-18 16:27:43 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T16:27:44.182Z] 2024-01-18 16:27:43 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Switching context to [/tmp/destination-config] [2024-01-18T16:27:44.182Z] 2024-01-18 16:27:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:27:44.182Z] 2024-01-18 16:27:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:27:44.438Z] 2024-01-18 16:27:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:27:44.438Z] 2024-01-18 16:27:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:27:44.438Z] 2024-01-18 16:27:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:27:44.438Z] 2024-01-18 16:27:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:27:44.999Z] 2024-01-18 16:27:44 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:27:45.260Z] 2024-01-18 16:27:45 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:27:45.821Z] 2024-01-18 16:27:45 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:27:45.821Z] 2024-01-18 16:27:45 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:27:46.076Z] 2024-01-18 16:27:46 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:27:46.076Z] 2024-01-18 16:27:46 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:27:46.076Z] 2024-01-18 16:27:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:27:46.076Z] 2024-01-18 16:27:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Switched context to [/tmp/destination-config] [2024-01-18T16:27:46.331Z] 2024-01-18 16:27:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-82934-01-18-16h10m24s] [2024-01-18T16:27:46.587Z] 2024-01-18 16:27:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {ShareBackupWithUsersAndGroups} Share large number of backups [2024-01-18T16:27:46.587Z] 2024-01-18 16:27:46 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-18T16:27:46.587Z] 2024/01/18 16:27:46 DoRetryWithTimeout - Error: {dial tcp 10.240.128.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:27:51.830Z] 2024/01/18 16:27:51 DoRetryWithTimeout - Error: {dial tcp 10.240.128.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:27:57.074Z] 2024/01/18 16:27:56 DoRetryWithTimeout - Error: {dial tcp 10.240.128.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:28:02.321Z] 2024/01/18 16:28:01 DoRetryWithTimeout - Error: {dial tcp 10.240.128.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:28:06.485Z] 2024/01/18 16:28:06 DoRetryWithTimeout - Error: {dial tcp 10.240.128.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:28:11.729Z] 2024/01/18 16:28:11 DoRetryWithTimeout - Error: {dial tcp 10.240.128.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:28:16.967Z] 2024/01/18 16:28:16 DoRetryWithTimeout - Error: {dial tcp 10.240.128.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:28:22.209Z] 2024/01/18 16:28:21 DoRetryWithTimeout - Error: {dial tcp 10.240.128.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:28:27.447Z] 2024/01/18 16:28:26 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.128.4. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.4 10.240.128.4]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T16:28:27.447Z] {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T16:28:27.447Z] {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T16:28:27.447Z] {dial tcp 10.240.128.4:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-18T16:28:31.608Z] 2024-01-18 16:28:31 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.4. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.4 10.240.128.4]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T16:28:31.608Z] {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T16:28:31.608Z] {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T16:28:31.608Z] {dial tcp 10.240.128.4:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T16:28:31.608Z] 2024-01-18 16:28:31 +0000:[ERROR] [{ShareBackupWithUsersAndGroups}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.4. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.4 10.240.128.4]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T16:28:31.608Z] {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T16:28:31.608Z] {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T16:28:31.608Z] {dial tcp 10.240.128.4:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T16:28:31.608Z] 2024-01-18 16:28:31 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T16:28:31.608Z] 2024-01-18 16:28:31 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Switching context to [/tmp/source-config] [2024-01-18T16:28:31.608Z] 2024-01-18 16:28:31 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:28:31.608Z] 2024-01-18 16:28:31 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:28:31.864Z] 2024-01-18 16:28:31 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:28:31.864Z] 2024-01-18 16:28:31 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:28:31.864Z] 2024-01-18 16:28:31 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:28:32.121Z] 2024-01-18 16:28:31 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:28:32.379Z] 2024-01-18 16:28:32 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:28:32.946Z] 2024-01-18 16:28:32 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:28:33.203Z] 2024-01-18 16:28:33 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:28:33.459Z] 2024-01-18 16:28:33 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:28:33.715Z] 2024-01-18 16:28:33 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:28:33.715Z] 2024-01-18 16:28:33 +0000:[WARNING] [{ShareBackupWithUsersAndGroups}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:28:33.715Z] 2024-01-18 16:28:33 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:28:33.715Z] 2024-01-18 16:28:33 +0000:[INFO] [{ShareBackupWithUsersAndGroups}] Switched context to [/tmp/source-config] [2024-01-18T16:28:33.715Z] • [SLOW TEST:1088.752 seconds] [2024-01-18T16:28:33.715Z] {ShareBackupWithUsersAndGroups} [2024-01-18T16:28:33.715Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:436 [2024-01-18T16:28:33.715Z] Share large number of backups [2024-01-18T16:28:33.715Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:477 [2024-01-18T16:28:33.715Z] ------------------------------ [2024-01-18T16:28:33.715Z] 2024-01-18 16:28:33 +0000:[INFO] [{BasicBackupCreation}] Log Dir: /testresults//BasicBackupCreation.log [2024-01-18T16:28:33.715Z] INFO[2024-01-18 16:28:33] --------Test Start------ [2024-01-18T16:28:33.715Z] INFO[2024-01-18 16:28:33] #Test: BasicBackupCreation [2024-01-18T16:28:33.715Z] INFO[2024-01-18 16:28:33] #Description: Deploying backup [2024-01-18T16:28:33.715Z] INFO[2024-01-18 16:28:33] ------------------------ [2024-01-18T16:28:33.716Z] 2024-01-18 16:28:33 +0000:[INFO] [{BasicBackupCreation}] scheduling applications [2024-01-18T16:28:33.716Z] 2024-01-18 16:28:33 +0000:[INFO] [{BasicBackupCreation}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T16:28:33.716Z] 2024-01-18 16:28:33 +0000:[INFO] [{BasicBackupCreation}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T16:28:33.716Z] 2024-01-18 16:28:33 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T16:28:33.971Z] 2024-01-18 16:28:33 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T16:28:33.971Z] 2024-01-18 16:28:33 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T16:28:34.230Z] 2024-01-18 16:28:34 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T16:28:34.231Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T16:28:34.231Z] 2024-01-18 16:28:34 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T16:28:34.231Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T16:28:34.231Z] 2024-01-18 16:28:34 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T16:28:34.231Z] 2024-01-18 16:28:34 +0000:[INFO] [{BasicBackupCreation}] Validating applications [2024-01-18T16:28:34.231Z] 2024-01-18 16:28:34 +0000:[INFO] [{BasicBackupCreation}] Validate applications [2024-01-18T16:28:34.231Z] 2024-01-18 16:28:34 +0000:[INFO] [{BasicBackupCreation}] Validating mysql-ibm app [2024-01-18T16:28:34.231Z] 2024-01-18 16:28:34 +0000:[INFO] [{BasicBackupCreation}] Validating mysql-ibm app's volumes [2024-01-18T16:28:34.487Z] 2024-01-18 16:28:34 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:28:34.487Z] 2024/01/18 16:28:34 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T16:28:46.655Z] 2024/01/18 16:28:44 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T16:28:54.737Z] 2024-01-18 16:28:54 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-84755-01-18-16h28m33s [2024-01-18T16:28:54.738Z] 2024-01-18 16:28:54 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T16:28:54.993Z] 2024-01-18 16:28:54 +0000:[WARNING] [{BasicBackupCreation}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T16:28:54.993Z] 2024-01-18 16:28:54 +0000:[INFO] [{BasicBackupCreation}] wait for mysql-ibm app to start running [2024-01-18T16:28:54.993Z] 2024-01-18 16:28:54 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T16:28:54.993Z] 2024/01/18 16:28:54 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:28:54.993Z] pod name:mysql-7d766cd65c-4d8lc namespace:mysql-ibm-pxb-0-84755-01-18-16h28m33s running:false ready:false node:10.240.64.9 [2024-01-18T16:28:54.993Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:29:07.156Z] 2024/01/18 16:29:05 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:29:07.156Z] pod name:mysql-7d766cd65c-4d8lc namespace:mysql-ibm-pxb-0-84755-01-18-16h28m33s running:true ready:false node:10.240.64.9 [2024-01-18T16:29:07.156Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:29:15.237Z] 2024/01/18 16:29:15 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:29:15.237Z] pod name:mysql-7d766cd65c-4d8lc namespace:mysql-ibm-pxb-0-84755-01-18-16h28m33s running:true ready:false node:10.240.64.9 [2024-01-18T16:29:15.237Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:29:27.402Z] 2024/01/18 16:29:25 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:29:27.402Z] pod name:mysql-7d766cd65c-4d8lc namespace:mysql-ibm-pxb-0-84755-01-18-16h28m33s running:true ready:false node:10.240.64.9 [2024-01-18T16:29:27.402Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:29:35.484Z] 2024/01/18 16:29:35 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:29:35.484Z] pod name:mysql-7d766cd65c-4d8lc namespace:mysql-ibm-pxb-0-84755-01-18-16h28m33s running:true ready:false node:10.240.64.9 [2024-01-18T16:29:35.484Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:29:47.653Z] 2024/01/18 16:29:45 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:29:47.653Z] pod name:mysql-7d766cd65c-4d8lc namespace:mysql-ibm-pxb-0-84755-01-18-16h28m33s running:true ready:false node:10.240.64.9 [2024-01-18T16:29:47.653Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:29:55.736Z] 2024/01/18 16:29:55 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:29:55.736Z] pod name:mysql-7d766cd65c-4d8lc namespace:mysql-ibm-pxb-0-84755-01-18-16h28m33s running:true ready:false node:10.240.64.9 [2024-01-18T16:29:55.737Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:30:07.899Z] 2024-01-18 16:30:05 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T16:30:07.899Z] 2024-01-18 16:30:05 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T16:30:07.899Z] 2024-01-18 16:30:05 +0000:[INFO] [{BasicBackupCreation}] validate if mysql-ibm app's volumes are setup [2024-01-18T16:30:07.899Z] 2024-01-18 16:30:06 +0000:[WARNING] [{BasicBackupCreation}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:30:07.899Z] 2024-01-18 16:30:06 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:30:07.899Z] 2024-01-18 16:30:06 +0000:[INFO] [{BasicBackupCreation}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T16:30:07.899Z] 2024-01-18 16:30:06 +0000:[WARNING] [{BasicBackupCreation}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T16:30:07.899Z] 2024-01-18 16:30:06 +0000:[INFO] [{BasicBackupCreation}] Registering cluster for backup [2024-01-18T16:30:07.899Z] INFO[2024-01-18 16:30:06] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T16:30:07.899Z] INFO[2024-01-18 16:30:06] Actual:true, Expected: true [2024-01-18T16:30:07.899Z] 2024-01-18 16:30:06 +0000:[INFO] [{BasicBackupCreation}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T16:30:07.899Z] 2024-01-18 16:30:06 +0000:[INFO] [{BasicBackupCreation}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T16:30:07.899Z] 2024-01-18 16:30:06 +0000:[INFO] [{BasicBackupCreation}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T16:30:07.900Z] 2024-01-18 16:30:06 +0000:[INFO] [{BasicBackupCreation}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T16:30:07.900Z] 2024-01-18 16:30:06 +0000:[INFO] [{BasicBackupCreation}] Create cluster [source-cluster] in org [default] [2024-01-18T16:30:07.900Z] 2024-01-18 16:30:06 +0000:[INFO] [{BasicBackupCreation}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T16:30:07.900Z] 2024-01-18 16:30:06 +0000:[INFO] [{BasicBackupCreation}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T16:30:07.900Z] 2024-01-18 16:30:06 +0000:[INFO] [{BasicBackupCreation}] Create cluster [destination-cluster] in org [default] [2024-01-18T16:30:07.900Z] 2024-01-18 16:30:06 +0000:[INFO] [{BasicBackupCreation}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T16:30:07.900Z] 2024-01-18 16:30:06 +0000:[INFO] [{BasicBackupCreation}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T16:30:07.900Z] 2024-01-18 16:30:06 +0000:[INFO] [{BasicBackupCreation}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-hssdu] for IBM [2024-01-18T16:30:07.900Z] 2024-01-18 16:30:06 +0000:[INFO] [{BasicBackupCreation}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-hssdu] for org [default] with [ibm] as provider [2024-01-18T16:30:07.900Z] 2024-01-18 16:30:06 +0000:[INFO] [{BasicBackupCreation}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:30:07.900Z] 2024-01-18 16:30:06 +0000:[INFO] [{BasicBackupCreation}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:30:07.900Z] 2024-01-18 16:30:06 +0000:[INFO] [{BasicBackupCreation}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-hssdu] is present [2024-01-18T16:30:07.900Z] 2024-01-18 16:30:06 +0000:[INFO] [{BasicBackupCreation}] Create cluster [source-cluster] in org [default] [2024-01-18T16:30:07.900Z] 2024-01-18 16:30:07 +0000:[INFO] [{BasicBackupCreation}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-arbek] for IBM [2024-01-18T16:30:07.900Z] 2024-01-18 16:30:07 +0000:[INFO] [{BasicBackupCreation}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-arbek] for org [default] with [ibm] as provider [2024-01-18T16:30:07.900Z] 2024-01-18 16:30:07 +0000:[INFO] [{BasicBackupCreation}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:30:07.900Z] 2024-01-18 16:30:07 +0000:[INFO] [{BasicBackupCreation}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:30:07.900Z] 2024-01-18 16:30:07 +0000:[INFO] [{BasicBackupCreation}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-arbek] is present [2024-01-18T16:30:07.900Z] 2024-01-18 16:30:07 +0000:[INFO] [{BasicBackupCreation}] Create cluster [destination-cluster] in org [default] [2024-01-18T16:30:07.900Z] INFO[2024-01-18 16:30:07] Verifying : Description : Creating source and destination cluster [2024-01-18T16:30:07.900Z] INFO[2024-01-18 16:30:07] Actual:true, Expected: true [2024-01-18T16:30:07.900Z] INFO[2024-01-18 16:30:07] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-18T16:30:07.900Z] INFO[2024-01-18 16:30:07] Actual:Online, Expected: Online [2024-01-18T16:30:07.900Z] INFO[2024-01-18 16:30:07] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-18T16:30:07.900Z] INFO[2024-01-18 16:30:07] Actual:true, Expected: true [2024-01-18T16:30:07.900Z] INFO[2024-01-18 16:30:07] Verifying : Description : Verifying if [destination-cluster] cluster is online [2024-01-18T16:30:07.900Z] INFO[2024-01-18 16:30:07] Actual:Online, Expected: Online [2024-01-18T16:30:07.900Z] 2024-01-18 16:30:07 +0000:[INFO] [{BasicBackupCreation}] Creating rules for backup [2024-01-18T16:30:07.900Z] 2024-01-18 16:30:07 +0000:[INFO] [{BasicBackupCreation}] Creating pre rule for deployed apps [2024-01-18T16:30:07.901Z] 2024-01-18 16:30:07 +0000:[INFO] [{BasicBackupCreation}] [portworx.(*portworx).CreateRuleForBackup:#2137] - Pre rule not required for this application [2024-01-18T16:30:07.901Z] 2024-01-18 16:30:07 +0000:[INFO] [{BasicBackupCreation}] [portworx.(*portworx).CreateRuleForBackup:#2159] - Rules not required for the apps [2024-01-18T16:30:07.901Z] INFO[2024-01-18 16:30:07] Verifying : Description : Verifying pre rule for backup [2024-01-18T16:30:07.901Z] INFO[2024-01-18 16:30:07] Actual:true, Expected: true [2024-01-18T16:30:07.901Z] 2024-01-18 16:30:07 +0000:[INFO] [{BasicBackupCreation}] Creating post rule for deployed apps [2024-01-18T16:30:07.901Z] 2024-01-18 16:30:07 +0000:[INFO] [{BasicBackupCreation}] [portworx.(*portworx).CreateRuleForBackup:#2154] - Post rule not required for this application [2024-01-18T16:30:07.901Z] 2024-01-18 16:30:07 +0000:[INFO] [{BasicBackupCreation}] [portworx.(*portworx).CreateRuleForBackup:#2159] - Rules not required for the apps [2024-01-18T16:30:07.901Z] INFO[2024-01-18 16:30:07] Verifying : Description : Verifying Post rule for backup [2024-01-18T16:30:07.901Z] INFO[2024-01-18 16:30:07] Actual:true, Expected: true [2024-01-18T16:30:07.901Z] 2024-01-18 16:30:07 +0000:[INFO] [{BasicBackupCreation}] Creating backup location and cloud setting [2024-01-18T16:30:07.901Z] 2024-01-18 16:30:07 +0000:[INFO] [{BasicBackupCreation}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [cred-aws-1705595407] for org [default] with [aws] as provider [2024-01-18T16:30:07.901Z] 2024-01-18 16:30:07 +0000:[INFO] [{BasicBackupCreation}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-18T16:30:08.158Z] 2024-01-18 16:30:07 +0000:[INFO] [{BasicBackupCreation}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cred-aws-1705595407] is present [2024-01-18T16:30:08.158Z] INFO[2024-01-18 16:30:07] Verifying : Description : Verifying creation of cloud credential named [cred-aws-1705595407] for org [default] with [aws] as provider [2024-01-18T16:30:08.158Z] INFO[2024-01-18 16:30:07] Actual:true, Expected: true [2024-01-18T16:30:11.422Z] INFO[2024-01-18 16:30:11] Verifying : Description : Creating backup location [2024-01-18T16:30:11.422Z] INFO[2024-01-18 16:30:11] Actual:true, Expected: true [2024-01-18T16:30:11.422Z] 2024-01-18 16:30:11 +0000:[INFO] [{BasicBackupCreation}] Creating backup schedule policies [2024-01-18T16:30:11.422Z] 2024-01-18 16:30:11 +0000:[INFO] [{BasicBackupCreation}] Creating backup interval schedule policy [2024-01-18T16:30:11.422Z] 2024-01-18 16:30:11 +0000:[INFO] [{BasicBackupCreation}] Create Backup Schedule Policy [2024-01-18T16:30:11.680Z] INFO[2024-01-18 16:30:11] Verifying : Description : Creating interval schedule policy [2024-01-18T16:30:11.680Z] INFO[2024-01-18 16:30:11] Actual:true, Expected: true [2024-01-18T16:30:11.680Z] 2024-01-18 16:30:11 +0000:[INFO] [{BasicBackupCreation}] Creating backup daily schedule policy [2024-01-18T16:30:11.680Z] 2024-01-18 16:30:11 +0000:[INFO] [{BasicBackupCreation}] Create Backup Schedule Policy [2024-01-18T16:30:12.240Z] INFO[2024-01-18 16:30:12] Verifying : Description : Creating daily schedule policy [2024-01-18T16:30:12.240Z] INFO[2024-01-18 16:30:12] Actual:true, Expected: true [2024-01-18T16:30:12.240Z] 2024-01-18 16:30:12 +0000:[INFO] [{BasicBackupCreation}] Creating backup weekly schedule policy [2024-01-18T16:30:12.240Z] 2024-01-18 16:30:12 +0000:[INFO] [{BasicBackupCreation}] Create Backup Schedule Policy [2024-01-18T16:30:13.606Z] INFO[2024-01-18 16:30:13] Verifying : Description : Creating weekly schedule policy [2024-01-18T16:30:13.606Z] INFO[2024-01-18 16:30:13] Actual:true, Expected: true [2024-01-18T16:30:13.606Z] 2024-01-18 16:30:13 +0000:[INFO] [{BasicBackupCreation}] Creating backup monthly schedule policy [2024-01-18T16:30:13.606Z] 2024-01-18 16:30:13 +0000:[INFO] [{BasicBackupCreation}] Create Backup Schedule Policy [2024-01-18T16:30:14.969Z] INFO[2024-01-18 16:30:14] Verifying : Description : Creating monthly schedule policy [2024-01-18T16:30:14.969Z] INFO[2024-01-18 16:30:14] Actual:true, Expected: true [2024-01-18T16:30:14.969Z] 2024-01-18 16:30:14 +0000:[INFO] [{BasicBackupCreation}] taking backup of applications [2024-01-18T16:30:16.336Z] 2024-01-18 16:30:16 +0000:[INFO] [{BasicBackupCreation}] creating backup [autogenerated-backup-mysql-ibm-pxb-0-84755-01-18-16h28m33s-1705595416] in source cluster [source-cluster] (e397a896-add0-4ab3-bbdb-123860145e93), organization [default], of namespace [mysql-ibm-pxb-0-84755-01-18-16h28m33s], in backup location [aws-global-aws-px-backup-system-test-4cac76dc-bl-1705595407] [2024-01-18T16:30:16.336Z] 2024-01-18 16:30:16 +0000:[INFO] [{BasicBackupCreation}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T16:30:16.336Z] 2024-01-18 16:30:16 +0000:[INFO] [{BasicBackupCreation}] Volume snapshot classes found in the cluster - [2024-01-18T16:30:16.336Z] 2024-01-18 16:30:16 +0000:[INFO] [{BasicBackupCreation}] ibmc-vpcblock-snapshot [2024-01-18T16:30:16.336Z] 2024-01-18 16:30:16 +0000:[INFO] [{BasicBackupCreation}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T16:30:16.594Z] 2024/01/18 16:30:16 DoRetryWithTimeout - Error: {backup status for [autogenerated-backup-mysql-ibm-pxb-0-84755-01-18-16h28m33s-1705595416] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T16:30:48.635Z] 2024/01/18 16:30:46 DoRetryWithTimeout - Error: {backup status for [autogenerated-backup-mysql-ibm-pxb-0-84755-01-18-16h28m33s-1705595416] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:31:20.701Z] 2024-01-18 16:31:16 +0000:[INFO] [{BasicBackupCreation}] [backup.CreateBackup:#313] - Backup [autogenerated-backup-mysql-ibm-pxb-0-84755-01-18-16h28m33s-1705595416] created successfully [2024-01-18T16:31:20.701Z] 2024-01-18 16:31:16 +0000:[INFO] [{BasicBackupCreation}] Validating backup [autogenerated-backup-mysql-ibm-pxb-0-84755-01-18-16h28m33s-1705595416] in org [default] [2024-01-18T16:31:20.701Z] 2024-01-18 16:31:16 +0000:[INFO] [{BasicBackupCreation}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [autogenerated-backup-mysql-ibm-pxb-0-84755-01-18-16h28m33s-1705595416] [2024-01-18T16:31:20.701Z] 2024-01-18 16:31:16 +0000:[INFO] [{BasicBackupCreation}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84755-01-18-16h28m33s] in backup [autogenerated-backup-mysql-ibm-pxb-0-84755-01-18-16h28m33s-1705595416] [2024-01-18T16:31:20.701Z] 2024-01-18 16:31:16 +0000:[INFO] [{BasicBackupCreation}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84755-01-18-16h28m33s] in backup [autogenerated-backup-mysql-ibm-pxb-0-84755-01-18-16h28m33s-1705595416] [2024-01-18T16:31:20.701Z] 2024-01-18 16:31:16 +0000:[INFO] [{BasicBackupCreation}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84755-01-18-16h28m33s] [2024-01-18T16:31:20.701Z] 2024-01-18 16:31:16 +0000:[WARNING] [{BasicBackupCreation}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:31:20.701Z] 2024-01-18 16:31:16 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:31:20.701Z] 2024-01-18 16:31:16 +0000:[INFO] [{BasicBackupCreation}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-84755-01-18-16h28m33s] are [[mysql-data]] [2024-01-18T16:31:20.701Z] 2024-01-18 16:31:16 +0000:[INFO] [{BasicBackupCreation}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T16:31:20.701Z] 2024-01-18 16:31:16 +0000:[INFO] [{BasicBackupCreation}] Validating CR cleanup [2024-01-18T16:31:20.701Z] 2024-01-18 16:31:16 +0000:[INFO] [{BasicBackupCreation}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:31:20.701Z] 2024-01-18 16:31:16 +0000:[INFO] [{BasicBackupCreation}] All CRs in [mysql-ibm-pxb-0-84755-01-18-16h28m33s] are [[autogenerated-backup-mysql-ibm-pxb-0-84755-01-18-16h28m33s-1705595416-5c7720b]] [2024-01-18T16:31:20.701Z] 2024-01-18 16:31:16 +0000:[INFO] [{BasicBackupCreation}] CR found for [[autogenerated-backup-mysql-ibm-pxb-0-84755-01-18-16h28m33s-1705595416-5c7720b]] under [mysql-ibm-pxb-0-84755-01-18-16h28m33s] namespace [2024-01-18T16:31:20.701Z] 2024/01/18 16:31:16 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [autogenerated-backup-mysql-ibm-pxb-0-84755-01-18-16h28m33s-1705595416]}, Next try in [5s], timeout [5m0s] [2024-01-18T16:31:22.072Z] 2024-01-18 16:31:21 +0000:[INFO] [{BasicBackupCreation}] All CRs in [mysql-ibm-pxb-0-84755-01-18-16h28m33s] are [[]] [2024-01-18T16:31:22.072Z] INFO[2024-01-18 16:31:21] Verifying : Description : Creation and Validation of backup [autogenerated-backup-mysql-ibm-pxb-0-84755-01-18-16h28m33s-1705595416] [2024-01-18T16:31:22.072Z] INFO[2024-01-18 16:31:21] Actual:true, Expected: true [2024-01-18T16:31:22.072Z] 2024-01-18 16:31:21 +0000:[INFO] [{BasicBackupCreation}] Restoring the backed up namespaces [2024-01-18T16:31:22.328Z] 2024-01-18 16:31:22 +0000:[INFO] [{BasicBackupCreation}] Restoring [mysql-ibm-pxb-0-84755-01-18-16h28m33s] namespace from the [autogenerated-backup-mysql-ibm-pxb-0-84755-01-18-16h28m33s-1705595416] backup [2024-01-18T16:31:22.892Z] 2024/01/18 16:31:22 DoRetryWithTimeout - Error: {restore status for [test-restore-mysql-ibm-pxb-0-84755-01-18-16h28m33s-ucnl] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T16:31:54.909Z] 2024-01-18 16:31:52 +0000:[INFO] [{BasicBackupCreation}] [backup.CreateRestore:#936] - Restore [test-restore-mysql-ibm-pxb-0-84755-01-18-16h28m33s-ucnl] created successfully [2024-01-18T16:31:54.909Z] 2024-01-18 16:31:52 +0000:[INFO] [{BasicBackupCreation}] Switching cluster context to cluster [destination-cluster] [2024-01-18T16:31:54.909Z] 2024-01-18 16:31:52 +0000:[INFO] [{BasicBackupCreation}] Switching context to [/tmp/destination-config] [2024-01-18T16:31:54.909Z] 2024-01-18 16:31:52 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:31:54.909Z] 2024-01-18 16:31:52 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:31:54.909Z] 2024-01-18 16:31:52 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:31:54.909Z] 2024-01-18 16:31:52 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:31:54.909Z] 2024-01-18 16:31:52 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:31:54.909Z] 2024-01-18 16:31:53 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:31:54.910Z] 2024-01-18 16:31:53 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:31:54.910Z] 2024-01-18 16:31:53 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:31:54.910Z] 2024-01-18 16:31:54 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:31:54.910Z] 2024-01-18 16:31:54 +0000:[WARNING] [{BasicBackupCreation}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:31:54.910Z] 2024-01-18 16:31:54 +0000:[WARNING] [{BasicBackupCreation}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:31:54.910Z] 2024-01-18 16:31:54 +0000:[WARNING] [{BasicBackupCreation}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:31:54.910Z] 2024-01-18 16:31:54 +0000:[INFO] [{BasicBackupCreation}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:31:54.910Z] 2024-01-18 16:31:54 +0000:[INFO] [{BasicBackupCreation}] Switched context to [/tmp/destination-config] [2024-01-18T16:31:54.910Z] 2024-01-18 16:31:54 +0000:[INFO] [{BasicBackupCreation}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-84755-01-18-16h28m33s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-18T16:31:54.910Z] 2024-01-18 16:31:54 +0000:[INFO] [{BasicBackupCreation}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-84755-01-18-16h28m33s] [2024-01-18T16:31:54.910Z] 2024-01-18 16:31:54 +0000:[INFO] [{BasicBackupCreation}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T16:31:54.910Z] 2024-01-18 16:31:54 +0000:[INFO] [{BasicBackupCreation}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T16:31:54.910Z] 2024-01-18 16:31:54 +0000:[INFO] [{BasicBackupCreation}] Validating restore [test-restore-mysql-ibm-pxb-0-84755-01-18-16h28m33s-ucnl] in org [default] [2024-01-18T16:31:54.910Z] 2024-01-18 16:31:54 +0000:[INFO] [{BasicBackupCreation}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [test-restore-mysql-ibm-pxb-0-84755-01-18-16h28m33s-ucnl] [2024-01-18T16:31:54.910Z] 2024-01-18 16:31:54 +0000:[INFO] [{BasicBackupCreation}] [backup.ValidateRestore:#2110] - Checking status of restore [test-restore-mysql-ibm-pxb-0-84755-01-18-16h28m33s-ucnl] [2024-01-18T16:31:54.910Z] 2024-01-18 16:31:54 +0000:[INFO] [{BasicBackupCreation}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-84755-01-18-16h28m33s] in restore [test-restore-mysql-ibm-pxb-0-84755-01-18-16h28m33s-ucnl] [2024-01-18T16:31:54.910Z] 2024-01-18 16:31:54 +0000:[INFO] [{BasicBackupCreation}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-84755-01-18-16h28m33s]) was found in restore [test-restore-mysql-ibm-pxb-0-84755-01-18-16h28m33s-ucnl], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-84755-01-18-16h28m33s] [2024-01-18T16:31:54.911Z] 2024-01-18 16:31:54 +0000:[INFO] [{BasicBackupCreation}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-84755-01-18-16h28m33s]) found in the restore [test-restore-mysql-ibm-pxb-0-84755-01-18-16h28m33s-ucnl] was also present on the cluster/namespace [mysql-ibm-pxb-0-84755-01-18-16h28m33s] [2024-01-18T16:31:54.911Z] 2024-01-18 16:31:54 +0000:[INFO] [{BasicBackupCreation}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-84755-01-18-16h28m33s]) was found in restore [test-restore-mysql-ibm-pxb-0-84755-01-18-16h28m33s-ucnl], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-84755-01-18-16h28m33s] [2024-01-18T16:31:55.172Z] 2024-01-18 16:31:55 +0000:[INFO] [{BasicBackupCreation}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-84755-01-18-16h28m33s]) found in the restore [test-restore-mysql-ibm-pxb-0-84755-01-18-16h28m33s-ucnl] was also present on the cluster/namespace [mysql-ibm-pxb-0-84755-01-18-16h28m33s] [2024-01-18T16:31:55.172Z] 2024-01-18 16:31:55 +0000:[INFO] [{BasicBackupCreation}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-84755-01-18-16h28m33s]) was found in restore [test-restore-mysql-ibm-pxb-0-84755-01-18-16h28m33s-ucnl], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-84755-01-18-16h28m33s] [2024-01-18T16:31:55.172Z] 2024-01-18 16:31:55 +0000:[INFO] [{BasicBackupCreation}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-84755-01-18-16h28m33s]) found in the restore [test-restore-mysql-ibm-pxb-0-84755-01-18-16h28m33s-ucnl] was also present on the cluster/namespace [mysql-ibm-pxb-0-84755-01-18-16h28m33s] [2024-01-18T16:31:55.172Z] 2024-01-18 16:31:55 +0000:[INFO] [{BasicBackupCreation}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-84755-01-18-16h28m33s]) was found in restore [test-restore-mysql-ibm-pxb-0-84755-01-18-16h28m33s-ucnl], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-84755-01-18-16h28m33s] [2024-01-18T16:31:55.172Z] 2024-01-18 16:31:55 +0000:[INFO] [{BasicBackupCreation}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-84755-01-18-16h28m33s]) found in the restore [test-restore-mysql-ibm-pxb-0-84755-01-18-16h28m33s-ucnl] was also present on the cluster/namespace [mysql-ibm-pxb-0-84755-01-18-16h28m33s] [2024-01-18T16:31:55.172Z] 2024-01-18 16:31:55 +0000:[INFO] [{BasicBackupCreation}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-84755-01-18-16h28m33s] in restore [test-restore-mysql-ibm-pxb-0-84755-01-18-16h28m33s-ucnl] [2024-01-18T16:31:55.172Z] 2024-01-18 16:31:55 +0000:[INFO] [{BasicBackupCreation}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-84755-01-18-16h28m33s] in restore [test-restore-mysql-ibm-pxb-0-84755-01-18-16h28m33s-ucnl] [2024-01-18T16:31:55.428Z] 2024-01-18 16:31:55 +0000:[WARNING] [{BasicBackupCreation}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:31:55.429Z] 2024-01-18 16:31:55 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:31:55.429Z] 2024-01-18 16:31:55 +0000:[INFO] [{BasicBackupCreation}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-84755-01-18-16h28m33s] are [[mysql-data]] [2024-01-18T16:31:55.429Z] 2024-01-18 16:31:55 +0000:[INFO] [{BasicBackupCreation}] [backup.ValidateRestore:#2240] - in restore [test-restore-mysql-ibm-pxb-0-84755-01-18-16h28m33s-ucnl], the status of the restored volume [pvc-98ebc8d4-a32a-41ee-b76f-f9e354dd25b7] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-18T16:31:55.429Z] 2024-01-18 16:31:55 +0000:[INFO] [{BasicBackupCreation}] Validate applications in restored namespace [mysql-ibm-pxb-0-84755-01-18-16h28m33s] due to restore [test-restore-mysql-ibm-pxb-0-84755-01-18-16h28m33s-ucnl] [2024-01-18T16:31:55.429Z] 2024-01-18 16:31:55 +0000:[INFO] [{BasicBackupCreation}] Validating mysql-ibm app [2024-01-18T16:31:55.429Z] 2024-01-18 16:31:55 +0000:[INFO] [{BasicBackupCreation}] wait for mysql-ibm app to start running [2024-01-18T16:31:55.684Z] 2024-01-18 16:31:55 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T16:31:55.684Z] 2024/01/18 16:31:55 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:31:55.684Z] pod name:mysql-7d766cd65c-942r2 namespace:mysql-ibm-pxb-0-84755-01-18-16h28m33s running:false ready:false node:10.240.64.7 [2024-01-18T16:31:55.684Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:32:05.615Z] 2024/01/18 16:32:05 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:32:05.615Z] pod name:mysql-7d766cd65c-942r2 namespace:mysql-ibm-pxb-0-84755-01-18-16h28m33s running:false ready:false node:10.240.64.7 [2024-01-18T16:32:05.615Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:32:17.781Z] 2024/01/18 16:32:15 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:32:17.781Z] pod name:mysql-7d766cd65c-942r2 namespace:mysql-ibm-pxb-0-84755-01-18-16h28m33s running:true ready:false node:10.240.64.7 [2024-01-18T16:32:17.781Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:32:25.871Z] 2024/01/18 16:32:25 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:32:25.871Z] pod name:mysql-7d766cd65c-942r2 namespace:mysql-ibm-pxb-0-84755-01-18-16h28m33s running:true ready:false node:10.240.64.7 [2024-01-18T16:32:25.871Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:32:35.805Z] 2024/01/18 16:32:35 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:32:35.805Z] pod name:mysql-7d766cd65c-942r2 namespace:mysql-ibm-pxb-0-84755-01-18-16h28m33s running:true ready:false node:10.240.64.7 [2024-01-18T16:32:35.805Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:32:47.972Z] 2024/01/18 16:32:45 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:32:47.972Z] pod name:mysql-7d766cd65c-942r2 namespace:mysql-ibm-pxb-0-84755-01-18-16h28m33s running:true ready:false node:10.240.64.7 [2024-01-18T16:32:47.972Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:32:56.057Z] 2024/01/18 16:32:55 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:32:56.057Z] pod name:mysql-7d766cd65c-942r2 namespace:mysql-ibm-pxb-0-84755-01-18-16h28m33s running:true ready:false node:10.240.64.7 [2024-01-18T16:32:56.057Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:33:08.226Z] 2024/01/18 16:33:05 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:33:08.226Z] pod name:mysql-7d766cd65c-942r2 namespace:mysql-ibm-pxb-0-84755-01-18-16h28m33s running:true ready:false node:10.240.64.7 [2024-01-18T16:33:08.226Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:33:16.315Z] 2024/01/18 16:33:16 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:33:16.315Z] pod name:mysql-7d766cd65c-942r2 namespace:mysql-ibm-pxb-0-84755-01-18-16h28m33s running:true ready:false node:10.240.64.7 [2024-01-18T16:33:16.315Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:33:26.253Z] 2024-01-18 16:33:26 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T16:33:26.253Z] 2024-01-18 16:33:26 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T16:33:26.509Z] 2024-01-18 16:33:26 +0000:[INFO] [{BasicBackupCreation}] Validating CR cleanup [2024-01-18T16:33:26.509Z] 2024-01-18 16:33:26 +0000:[INFO] [{BasicBackupCreation}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:33:26.509Z] 2024-01-18 16:33:26 +0000:[INFO] [{BasicBackupCreation}] All CRs in [mysql-ibm-pxb-0-84755-01-18-16h28m33s] are [[]] [2024-01-18T16:33:26.509Z] 2024-01-18 16:33:26 +0000:[INFO] [{BasicBackupCreation}] Switching cluster context back to cluster path [/tmp/source-config] [2024-01-18T16:33:26.509Z] 2024-01-18 16:33:26 +0000:[INFO] [{BasicBackupCreation}] Switching context to [/tmp/source-config] [2024-01-18T16:33:26.766Z] 2024-01-18 16:33:26 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:33:26.766Z] 2024-01-18 16:33:26 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:33:26.766Z] 2024-01-18 16:33:26 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:33:26.766Z] 2024-01-18 16:33:26 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:33:27.022Z] 2024-01-18 16:33:26 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:33:27.022Z] 2024-01-18 16:33:26 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:33:27.596Z] 2024-01-18 16:33:27 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:33:27.852Z] 2024-01-18 16:33:27 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:33:28.416Z] 2024-01-18 16:33:28 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:33:28.416Z] 2024-01-18 16:33:28 +0000:[WARNING] [{BasicBackupCreation}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:33:28.673Z] 2024-01-18 16:33:28 +0000:[WARNING] [{BasicBackupCreation}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:33:28.673Z] 2024-01-18 16:33:28 +0000:[WARNING] [{BasicBackupCreation}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:33:28.673Z] 2024-01-18 16:33:28 +0000:[INFO] [{BasicBackupCreation}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:33:28.673Z] 2024-01-18 16:33:28 +0000:[INFO] [{BasicBackupCreation}] Switched context to [/tmp/source-config] [2024-01-18T16:33:28.673Z] INFO[2024-01-18 16:33:28] Verifying : Description : Creation and Validation of restore [test-restore-mysql-ibm-pxb-0-84755-01-18-16h28m33s-ucnl] [2024-01-18T16:33:28.673Z] INFO[2024-01-18 16:33:28] Actual:true, Expected: true [2024-01-18T16:33:28.673Z] 2024-01-18 16:33:28 +0000:[INFO] [{BasicBackupCreation}] switching to default context [2024-01-18T16:33:28.673Z] 2024-01-18 16:33:28 +0000:[INFO] [{BasicBackupCreation}] Switching context to [default] [2024-01-18T16:33:28.673Z] 2024-01-18 16:33:28 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:33:28.673Z] 2024-01-18 16:33:28 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:33:28.673Z] 2024-01-18 16:33:28 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:33:28.928Z] 2024-01-18 16:33:28 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:33:28.928Z] 2024-01-18 16:33:28 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:33:29.184Z] 2024-01-18 16:33:28 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:33:29.440Z] 2024-01-18 16:33:29 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:33:30.001Z] 2024-01-18 16:33:29 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:33:30.256Z] 2024-01-18 16:33:30 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:33:30.511Z] 2024-01-18 16:33:30 +0000:[WARNING] [{BasicBackupCreation}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:33:30.781Z] 2024-01-18 16:33:30 +0000:[WARNING] [{BasicBackupCreation}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:33:30.781Z] 2024-01-18 16:33:30 +0000:[WARNING] [{BasicBackupCreation}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:33:30.781Z] 2024-01-18 16:33:30 +0000:[INFO] [{BasicBackupCreation}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:33:30.781Z] 2024-01-18 16:33:30 +0000:[INFO] [{BasicBackupCreation}] Switched context to [default] [2024-01-18T16:33:34.053Z] INFO[2024-01-18 16:33:33] Verifying : Description : Deleting backup schedule policies [2024-01-18T16:33:34.053Z] INFO[2024-01-18 16:33:33] Actual:true, Expected: true [2024-01-18T16:33:34.053Z] 2024-01-18 16:33:33 +0000:[INFO] [{BasicBackupCreation}] [backup.glob..func78.3:#294] - Destroying scheduled apps on source cluster [2024-01-18T16:33:34.053Z] 2024-01-18 16:33:33 +0000:[INFO] [{BasicBackupCreation}] Destroying apps [2024-01-18T16:33:34.053Z] 2024-01-18 16:33:33 +0000:[INFO] [{BasicBackupCreation}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T16:33:34.053Z] 2024-01-18 16:33:33 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T16:33:34.053Z] 2024-01-18 16:33:33 +0000:[INFO] [{BasicBackupCreation}] start destroying mysql-ibm app [2024-01-18T16:33:34.053Z] 2024-01-18 16:33:33 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T16:33:34.053Z] 2024-01-18 16:33:34 +0000:[INFO] [{BasicBackupCreation}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T16:33:34.053Z] 2024-01-18 16:33:34 +0000:[WARNING] [{BasicBackupCreation}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T16:33:34.053Z] INFO[2024-01-18 16:33:34] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T16:33:34.053Z] INFO[2024-01-18 16:33:34] Actual:true, Expected: true [2024-01-18T16:33:34.053Z] 2024-01-18 16:33:34 +0000:[INFO] [{BasicBackupCreation}] switching to destination context [2024-01-18T16:33:34.053Z] 2024-01-18 16:33:34 +0000:[INFO] [{BasicBackupCreation}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T16:33:34.053Z] 2024-01-18 16:33:34 +0000:[INFO] [{BasicBackupCreation}] Switching context to [/tmp/destination-config] [2024-01-18T16:33:34.336Z] 2024-01-18 16:33:34 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:33:34.336Z] 2024-01-18 16:33:34 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:33:34.336Z] 2024-01-18 16:33:34 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:33:34.336Z] 2024-01-18 16:33:34 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:33:34.336Z] 2024-01-18 16:33:34 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:33:34.592Z] 2024-01-18 16:33:34 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:33:35.153Z] 2024-01-18 16:33:34 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:33:35.409Z] 2024-01-18 16:33:35 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:33:35.666Z] 2024-01-18 16:33:35 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:33:35.922Z] 2024-01-18 16:33:35 +0000:[WARNING] [{BasicBackupCreation}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:33:36.179Z] 2024-01-18 16:33:36 +0000:[WARNING] [{BasicBackupCreation}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:33:36.179Z] 2024-01-18 16:33:36 +0000:[WARNING] [{BasicBackupCreation}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:33:36.179Z] 2024-01-18 16:33:36 +0000:[INFO] [{BasicBackupCreation}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:33:36.179Z] 2024-01-18 16:33:36 +0000:[INFO] [{BasicBackupCreation}] Switched context to [/tmp/destination-config] [2024-01-18T16:33:36.179Z] 2024-01-18 16:33:36 +0000:[INFO] [{BasicBackupCreation}] Destroying restored apps on destination clusters [2024-01-18T16:33:36.179Z] 2024-01-18 16:33:36 +0000:[INFO] [{BasicBackupCreation}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-84755-01-18-16h28m33s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-18T16:33:36.179Z] 2024-01-18 16:33:36 +0000:[INFO] [{BasicBackupCreation}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-84755-01-18-16h28m33s] [2024-01-18T16:33:36.179Z] 2024-01-18 16:33:36 +0000:[INFO] [{BasicBackupCreation}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T16:33:36.179Z] 2024-01-18 16:33:36 +0000:[INFO] [{BasicBackupCreation}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T16:33:36.179Z] 2024-01-18 16:33:36 +0000:[INFO] [{BasicBackupCreation}] Destroying apps [2024-01-18T16:33:36.180Z] 2024-01-18 16:33:36 +0000:[INFO] [{BasicBackupCreation}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T16:33:36.740Z] 2024-01-18 16:33:36 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T16:33:36.740Z] 2024-01-18 16:33:36 +0000:[INFO] [{BasicBackupCreation}] start destroying mysql-ibm app [2024-01-18T16:33:36.740Z] 2024-01-18 16:33:36 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T16:33:36.996Z] 2024-01-18 16:33:36 +0000:[INFO] [{BasicBackupCreation}] switching to default context [2024-01-18T16:33:36.996Z] 2024-01-18 16:33:36 +0000:[INFO] [{BasicBackupCreation}] Switching context to [default] [2024-01-18T16:33:36.996Z] 2024-01-18 16:33:36 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:33:36.996Z] 2024-01-18 16:33:36 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:33:36.996Z] 2024-01-18 16:33:36 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:33:36.996Z] 2024-01-18 16:33:36 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:33:36.996Z] 2024-01-18 16:33:36 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:33:37.251Z] 2024-01-18 16:33:37 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:33:37.813Z] 2024-01-18 16:33:37 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:33:38.068Z] 2024-01-18 16:33:37 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:33:38.629Z] 2024-01-18 16:33:38 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:33:38.629Z] 2024-01-18 16:33:38 +0000:[WARNING] [{BasicBackupCreation}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:33:38.884Z] 2024-01-18 16:33:38 +0000:[WARNING] [{BasicBackupCreation}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:33:38.884Z] 2024-01-18 16:33:38 +0000:[WARNING] [{BasicBackupCreation}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:33:38.884Z] 2024-01-18 16:33:38 +0000:[INFO] [{BasicBackupCreation}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:33:38.884Z] 2024-01-18 16:33:38 +0000:[INFO] [{BasicBackupCreation}] Switched context to [default] [2024-01-18T16:33:38.885Z] 2024-01-18 16:33:38 +0000:[INFO] [{BasicBackupCreation}] [backup.glob..func78.3:#318] - Deleting backed up namespaces [2024-01-18T16:33:39.139Z] INFO[2024-01-18 16:33:38] Verifying : Description : Verifying [autogenerated-backup-mysql-ibm-pxb-0-84755-01-18-16h28m33s-1705595416] backup deletion is successful [2024-01-18T16:33:39.139Z] INFO[2024-01-18 16:33:38] Actual:, Expected: [2024-01-18T16:33:39.140Z] 2024-01-18 16:33:38 +0000:[INFO] [{BasicBackupCreation}] [backup.glob..func78.3:#326] - Deleting restored namespaces [2024-01-18T16:33:39.140Z] INFO[2024-01-18 16:33:38] Verifying : Description : Getting the backup driver [2024-01-18T16:33:39.140Z] INFO[2024-01-18 16:33:38] Actual:true, Expected: true [2024-01-18T16:33:39.140Z] INFO[2024-01-18 16:33:38] Verifying : Description : Deleting Restore [test-restore-mysql-ibm-pxb-0-84755-01-18-16h28m33s-ucnl] [2024-01-18T16:33:39.140Z] INFO[2024-01-18 16:33:38] Actual:true, Expected: true [2024-01-18T16:33:39.140Z] 2024-01-18 16:33:38 +0000:[INFO] [{BasicBackupCreation}] Cleaning backup locations in map [map[a2aa10a8-b1b2-4bff-b61f-293608a00690:aws-global-aws-px-backup-system-test-4cac76dc-bl-1705595407]], cloud credential [cred-aws-1705595407], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T16:33:39.140Z] INFO[2024-01-18 16:33:39] Verifying : Description : Verifying deletion of backup location [aws-global-aws-px-backup-system-test-4cac76dc-bl-1705595407] [2024-01-18T16:33:39.140Z] INFO[2024-01-18 16:33:39] Actual:true, Expected: true [2024-01-18T16:33:39.140Z] 2024-01-18 16:33:39 +0000:[INFO] [{BasicBackupCreation}] [backup.IsBackupLocationPresent:#2442] - Backup location [aws-global-aws-px-backup-system-test-4cac76dc-bl-1705595407] is present [2024-01-18T16:33:39.395Z] 2024/01/18 16:33:39 DoRetryWithTimeout - Error: {backup location aws-global-aws-px-backup-system-test-4cac76dc-bl-1705595407 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T16:34:05.888Z] INFO[2024-01-18 16:34:04] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T16:34:05.888Z] INFO[2024-01-18 16:34:04] sched stats: tick interval: 3604 < 1.2s (100%) [2024-01-18T16:34:05.888Z] INFO[2024-01-18 16:34:04] sched stats: tick process durations: 3603 < 500ms (100%) [2024-01-18T16:34:10.073Z] 2024-01-18 16:34:09 +0000:[INFO] [{BasicBackupCreation}] [backup.IsBackupLocationPresent:#2442] - Backup location [aws-global-aws-px-backup-system-test-4cac76dc-bl-1705595407] is present [2024-01-18T16:34:10.074Z] 2024/01/18 16:34:09 DoRetryWithTimeout - Error: {backup location aws-global-aws-px-backup-system-test-4cac76dc-bl-1705595407 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T16:34:42.149Z] 2024-01-18 16:34:39 +0000:[INFO] [{BasicBackupCreation}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu] [2024-01-18T16:34:42.149Z] INFO[2024-01-18 16:34:39] Verifying : Description : Verifying backup location deletion status aws-global-aws-px-backup-system-test-4cac76dc-bl-1705595407 [2024-01-18T16:34:42.149Z] INFO[2024-01-18 16:34:39] Actual:true, Expected: true [2024-01-18T16:34:42.149Z] 2024-01-18 16:34:39 +0000:[INFO] [{BasicBackupCreation}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cred-aws-1705595407] is present [2024-01-18T16:34:42.149Z] INFO[2024-01-18 16:34:39] Verifying : Description : Verifying if cloud cred [cred-aws-1705595407] is present [2024-01-18T16:34:42.149Z] INFO[2024-01-18 16:34:39] Actual:true, Expected: true [2024-01-18T16:34:42.149Z] INFO[2024-01-18 16:34:39] Verifying : Description : Verifying deletion of cloud cred [cred-aws-1705595407] [2024-01-18T16:34:42.149Z] INFO[2024-01-18 16:34:39] Actual:true, Expected: true [2024-01-18T16:34:42.149Z] INFO[2024-01-18 16:34:39] Verifying : Description : Deleting cloud cred cred-aws-1705595407 [2024-01-18T16:34:42.150Z] INFO[2024-01-18 16:34:39] Actual:true, Expected: true [2024-01-18T16:34:42.150Z] INFO[2024-01-18 16:34:39] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T16:34:42.150Z] INFO[2024-01-18 16:34:39] Actual:true, Expected: true [2024-01-18T16:34:42.150Z] INFO[2024-01-18 16:34:39] Verifying : Description : Deleting cluster source-cluster [2024-01-18T16:34:42.150Z] INFO[2024-01-18 16:34:39] Actual:true, Expected: true [2024-01-18T16:34:42.150Z] INFO[2024-01-18 16:34:39] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-hssdu] [2024-01-18T16:34:42.150Z] INFO[2024-01-18 16:34:39] Actual:true, Expected: true [2024-01-18T16:34:42.150Z] INFO[2024-01-18 16:34:40] Verifying : Description : Deleting cluster destination-cluster [2024-01-18T16:34:42.150Z] INFO[2024-01-18 16:34:40] Actual:true, Expected: true [2024-01-18T16:34:42.150Z] INFO[2024-01-18 16:34:40] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-arbek] [2024-01-18T16:34:42.150Z] INFO[2024-01-18 16:34:40] Actual:true, Expected: true [2024-01-18T16:34:42.150Z] 2024-01-18 16:34:40 +0000:[INFO] [{BasicBackupCreation}] switching to default context [2024-01-18T16:34:42.150Z] 2024-01-18 16:34:40 +0000:[INFO] [{BasicBackupCreation}] Switching context: The context is already [default] [2024-01-18T16:34:42.150Z] INFO[2024-01-18 16:34:40] --------Test End------ [2024-01-18T16:34:42.150Z] INFO[2024-01-18 16:34:40] #Test: [2024-01-18T16:34:42.150Z] INFO[2024-01-18 16:34:40] #Description: [2024-01-18T16:34:42.150Z] INFO[2024-01-18 16:34:40] ------------------------ [2024-01-18T16:34:42.150Z] 2024-01-18 16:34:40 +0000:[INFO] [{BasicBackupCreation}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-84755-01-18-16h28m33s] [2024-01-18T16:34:42.150Z] 2024-01-18 16:34:40 +0000:[INFO] [{BasicBackupCreation}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T16:34:42.150Z] 2024-01-18 16:34:40 +0000:[INFO] [{BasicBackupCreation}] Switching context to [/tmp/destination-config] [2024-01-18T16:34:42.150Z] 2024-01-18 16:34:40 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:34:42.150Z] 2024-01-18 16:34:40 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:34:42.150Z] 2024-01-18 16:34:40 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:34:42.150Z] 2024-01-18 16:34:41 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:34:42.150Z] 2024-01-18 16:34:41 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:34:42.150Z] 2024-01-18 16:34:41 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:34:42.150Z] 2024-01-18 16:34:41 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:34:42.150Z] 2024-01-18 16:34:42 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:34:42.715Z] 2024-01-18 16:34:42 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:34:42.970Z] 2024-01-18 16:34:42 +0000:[WARNING] [{BasicBackupCreation}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:34:42.970Z] 2024-01-18 16:34:42 +0000:[WARNING] [{BasicBackupCreation}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:34:42.970Z] 2024-01-18 16:34:42 +0000:[WARNING] [{BasicBackupCreation}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:34:42.971Z] 2024-01-18 16:34:42 +0000:[INFO] [{BasicBackupCreation}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:34:42.971Z] 2024-01-18 16:34:42 +0000:[INFO] [{BasicBackupCreation}] Switched context to [/tmp/destination-config] [2024-01-18T16:34:43.226Z] 2024-01-18 16:34:43 +0000:[INFO] [{BasicBackupCreation}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-84755-01-18-16h28m33s] [2024-01-18T16:34:43.482Z] 2024-01-18 16:34:43 +0000:[INFO] [{BasicBackupCreation}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {BasicBackupCreation} Basic Backup Creation [2024-01-18T16:34:43.482Z] 2024-01-18 16:34:43 +0000:[INFO] [{BasicBackupCreation}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-18T16:34:43.482Z] 2024/01/18 16:34:43 DoRetryWithTimeout - Error: {dial tcp 10.240.0.44:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:34:48.724Z] 2024/01/18 16:34:48 DoRetryWithTimeout - Error: {dial tcp 10.240.0.44:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:34:53.969Z] 2024/01/18 16:34:53 DoRetryWithTimeout - Error: {dial tcp 10.240.0.44:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:34:59.210Z] 2024/01/18 16:34:58 DoRetryWithTimeout - Error: {dial tcp 10.240.0.44:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:35:03.374Z] 2024/01/18 16:35:03 DoRetryWithTimeout - Error: {dial tcp 10.240.0.44:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:35:08.629Z] 2024/01/18 16:35:08 DoRetryWithTimeout - Error: {dial tcp 10.240.0.44:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:35:13.879Z] 2024/01/18 16:35:13 DoRetryWithTimeout - Error: {dial tcp 10.240.0.44:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:35:19.122Z] 2024/01/18 16:35:18 DoRetryWithTimeout - Error: {dial tcp 10.240.0.44:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:35:24.364Z] 2024/01/18 16:35:23 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.0.44. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.44 10.240.0.44]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.44:22: connect: connection refused} [2024-01-18T16:35:24.364Z] {dial tcp 10.240.0.44:22: connect: connection refused} [2024-01-18T16:35:24.364Z] {dial tcp 10.240.0.44:22: connect: connection refused} [2024-01-18T16:35:24.364Z] {dial tcp 10.240.0.44:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-18T16:35:28.532Z] 2024-01-18 16:35:28 +0000:[WARNING] [{BasicBackupCreation}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.44. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.44 10.240.0.44]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.44:22: connect: connection refused} [2024-01-18T16:35:28.532Z] {dial tcp 10.240.0.44:22: connect: connection refused} [2024-01-18T16:35:28.532Z] {dial tcp 10.240.0.44:22: connect: connection refused} [2024-01-18T16:35:28.532Z] {dial tcp 10.240.0.44:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T16:35:28.532Z] 2024-01-18 16:35:28 +0000:[ERROR] [{BasicBackupCreation}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.44. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.44 10.240.0.44]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.44:22: connect: connection refused} [2024-01-18T16:35:28.532Z] {dial tcp 10.240.0.44:22: connect: connection refused} [2024-01-18T16:35:28.532Z] {dial tcp 10.240.0.44:22: connect: connection refused} [2024-01-18T16:35:28.532Z] {dial tcp 10.240.0.44:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T16:35:28.532Z] 2024-01-18 16:35:28 +0000:[INFO] [{BasicBackupCreation}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T16:35:28.532Z] 2024-01-18 16:35:28 +0000:[INFO] [{BasicBackupCreation}] Switching context to [/tmp/source-config] [2024-01-18T16:35:28.532Z] 2024-01-18 16:35:28 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:35:28.788Z] 2024-01-18 16:35:28 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:35:28.789Z] 2024-01-18 16:35:28 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:35:28.789Z] 2024-01-18 16:35:28 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:35:28.789Z] 2024-01-18 16:35:28 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:35:29.045Z] 2024-01-18 16:35:28 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:35:29.301Z] 2024-01-18 16:35:29 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:35:29.862Z] 2024-01-18 16:35:29 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:35:30.117Z] 2024-01-18 16:35:29 +0000:[INFO] [{BasicBackupCreation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:35:30.373Z] 2024-01-18 16:35:30 +0000:[WARNING] [{BasicBackupCreation}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:35:30.628Z] 2024-01-18 16:35:30 +0000:[WARNING] [{BasicBackupCreation}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:35:30.628Z] 2024-01-18 16:35:30 +0000:[WARNING] [{BasicBackupCreation}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:35:30.628Z] 2024-01-18 16:35:30 +0000:[INFO] [{BasicBackupCreation}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:35:30.628Z] 2024-01-18 16:35:30 +0000:[INFO] [{BasicBackupCreation}] Switched context to [/tmp/source-config] [2024-01-18T16:35:30.628Z] •SS2024-01-18 16:35:30 +0000:[INFO] [{CreateMultipleUsersAndGroups}] Log Dir: /testresults//CreateMultipleUsersAndGroups.log [2024-01-18T16:35:30.628Z] INFO[2024-01-18 16:35:30] --------Test Start------ [2024-01-18T16:35:30.628Z] INFO[2024-01-18 16:35:30] #Test: CreateMultipleUsersAndGroups [2024-01-18T16:35:30.628Z] INFO[2024-01-18 16:35:30] #Description: Creation of multiple users and groups [2024-01-18T16:35:30.628Z] INFO[2024-01-18 16:35:30] ------------------------ [2024-01-18T16:35:30.628Z] 2024-01-18 16:35:30 +0000:[INFO] [{CreateMultipleUsersAndGroups}] Creating 10 groups [2024-01-18T16:35:33.141Z] 2024-01-18 16:35:32 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddGroup:#989] - Creating group response - [2024-01-18T16:35:33.141Z] 2024-01-18 16:35:32 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddGroup:#994] - Group [testGroup1705595730] created [2024-01-18T16:35:35.046Z] 2024-01-18 16:35:34 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddGroup:#989] - Creating group response - [2024-01-18T16:35:35.046Z] 2024-01-18 16:35:34 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddGroup:#994] - Group [testGroup1705595732] created [2024-01-18T16:35:36.941Z] 2024-01-18 16:35:36 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddGroup:#989] - Creating group response - [2024-01-18T16:35:36.941Z] 2024-01-18 16:35:36 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddGroup:#994] - Group [testGroup1705595734] created [2024-01-18T16:35:38.832Z] 2024-01-18 16:35:38 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddGroup:#989] - Creating group response - [2024-01-18T16:35:38.832Z] 2024-01-18 16:35:38 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddGroup:#994] - Group [testGroup1705595736] created [2024-01-18T16:35:40.722Z] 2024-01-18 16:35:40 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddGroup:#989] - Creating group response - [2024-01-18T16:35:40.722Z] 2024-01-18 16:35:40 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddGroup:#994] - Group [testGroup1705595738] created [2024-01-18T16:35:42.609Z] 2024-01-18 16:35:42 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddGroup:#989] - Creating group response - [2024-01-18T16:35:42.609Z] 2024-01-18 16:35:42 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddGroup:#994] - Group [testGroup1705595740] created [2024-01-18T16:35:45.128Z] 2024-01-18 16:35:44 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddGroup:#989] - Creating group response - [2024-01-18T16:35:45.128Z] 2024-01-18 16:35:44 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddGroup:#994] - Group [testGroup1705595742] created [2024-01-18T16:35:47.020Z] 2024-01-18 16:35:46 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddGroup:#989] - Creating group response - [2024-01-18T16:35:47.020Z] 2024-01-18 16:35:46 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddGroup:#994] - Group [testGroup1705595744] created [2024-01-18T16:35:49.214Z] 2024-01-18 16:35:48 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddGroup:#989] - Creating group response - [2024-01-18T16:35:49.214Z] 2024-01-18 16:35:48 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddGroup:#994] - Group [testGroup1705595746] created [2024-01-18T16:35:51.106Z] 2024-01-18 16:35:50 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddGroup:#989] - Creating group response - [2024-01-18T16:35:51.106Z] 2024-01-18 16:35:50 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddGroup:#994] - Group [testGroup1705595748] created [2024-01-18T16:35:51.106Z] 2024-01-18 16:35:50 +0000:[INFO] [{CreateMultipleUsersAndGroups}] Creating 20 users [2024-01-18T16:35:52.996Z] 2024-01-18 16:35:52 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser1705595750] creation - [2024-01-18T16:35:52.996Z] 2024-01-18 16:35:52 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#733] - User [testuser1705595750] created [2024-01-18T16:35:54.881Z] 2024-01-18 16:35:54 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser1705595752] creation - [2024-01-18T16:35:54.881Z] 2024-01-18 16:35:54 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#733] - User [testuser1705595752] created [2024-01-18T16:35:57.396Z] 2024-01-18 16:35:56 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser1705595754] creation - [2024-01-18T16:35:57.396Z] 2024-01-18 16:35:56 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#733] - User [testuser1705595754] created [2024-01-18T16:35:59.286Z] 2024-01-18 16:35:58 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser1705595756] creation - [2024-01-18T16:35:59.286Z] 2024-01-18 16:35:58 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#733] - User [testuser1705595756] created [2024-01-18T16:36:01.174Z] 2024-01-18 16:36:00 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser1705595758] creation - [2024-01-18T16:36:01.174Z] 2024-01-18 16:36:00 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#733] - User [testuser1705595758] created [2024-01-18T16:36:03.066Z] 2024-01-18 16:36:02 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser1705595760] creation - [2024-01-18T16:36:03.066Z] 2024-01-18 16:36:02 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#733] - User [testuser1705595760] created [2024-01-18T16:36:04.952Z] 2024-01-18 16:36:04 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser1705595762] creation - [2024-01-18T16:36:04.952Z] 2024-01-18 16:36:04 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#733] - User [testuser1705595762] created [2024-01-18T16:36:06.845Z] 2024-01-18 16:36:06 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser1705595764] creation - [2024-01-18T16:36:06.845Z] 2024-01-18 16:36:06 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#733] - User [testuser1705595764] created [2024-01-18T16:36:09.358Z] 2024-01-18 16:36:08 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser1705595766] creation - [2024-01-18T16:36:09.358Z] 2024-01-18 16:36:08 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#733] - User [testuser1705595766] created [2024-01-18T16:36:11.244Z] 2024-01-18 16:36:10 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser1705595768] creation - [2024-01-18T16:36:11.244Z] 2024-01-18 16:36:10 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#733] - User [testuser1705595768] created [2024-01-18T16:36:13.132Z] 2024-01-18 16:36:12 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser1705595770] creation - [2024-01-18T16:36:13.132Z] 2024-01-18 16:36:12 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#733] - User [testuser1705595770] created [2024-01-18T16:36:15.019Z] 2024-01-18 16:36:14 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser1705595772] creation - [2024-01-18T16:36:15.019Z] 2024-01-18 16:36:14 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#733] - User [testuser1705595772] created [2024-01-18T16:36:16.910Z] 2024-01-18 16:36:16 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser1705595774] creation - [2024-01-18T16:36:16.910Z] 2024-01-18 16:36:16 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#733] - User [testuser1705595774] created [2024-01-18T16:36:19.424Z] 2024-01-18 16:36:18 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser1705595776] creation - [2024-01-18T16:36:19.424Z] 2024-01-18 16:36:18 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#733] - User [testuser1705595776] created [2024-01-18T16:36:21.313Z] 2024-01-18 16:36:20 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser1705595778] creation - [2024-01-18T16:36:21.313Z] 2024-01-18 16:36:20 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#733] - User [testuser1705595778] created [2024-01-18T16:36:23.204Z] 2024-01-18 16:36:22 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser1705595780] creation - [2024-01-18T16:36:23.204Z] 2024-01-18 16:36:22 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#733] - User [testuser1705595780] created [2024-01-18T16:36:25.092Z] 2024-01-18 16:36:24 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser1705595782] creation - [2024-01-18T16:36:25.092Z] 2024-01-18 16:36:24 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#733] - User [testuser1705595782] created [2024-01-18T16:36:26.983Z] 2024-01-18 16:36:26 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser1705595784] creation - [2024-01-18T16:36:26.983Z] 2024-01-18 16:36:26 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#733] - User [testuser1705595784] created [2024-01-18T16:36:28.873Z] 2024-01-18 16:36:28 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser1705595786] creation - [2024-01-18T16:36:28.873Z] 2024-01-18 16:36:28 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#733] - User [testuser1705595786] created [2024-01-18T16:36:31.388Z] 2024-01-18 16:36:30 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#728] - Response for user [testuser1705595788] creation - [2024-01-18T16:36:31.388Z] 2024-01-18 16:36:30 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.AddUser:#733] - User [testuser1705595788] created [2024-01-18T16:36:31.388Z] 2024-01-18 16:36:31 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.glob..func80.2.3:#103] - Validating created groups [px-admin-group testGroup1705595730 testGroup1705595732 testGroup1705595734 testGroup1705595736 testGroup1705595738 testGroup1705595740 testGroup1705595742 testGroup1705595744 testGroup1705595746 testGroup1705595748] [2024-01-18T16:36:31.388Z] INFO[2024-01-18 16:36:31] Verifying : Description : Group [testGroup1705595730 testGroup1705595732 testGroup1705595734 testGroup1705595736 testGroup1705595738 testGroup1705595740 testGroup1705595742 testGroup1705595744 testGroup1705595746 testGroup1705595748] Creation Verification [2024-01-18T16:36:31.388Z] INFO[2024-01-18 16:36:31] Actual:, Expected: [2024-01-18T16:36:31.644Z] 2024-01-18 16:36:31 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.glob..func80.2.4:#124] - Validating created users [admin px-central-admin testuser1-1705568457 testuser1-1705569021 testuser1-1705569023 testuser1-1705569027 testuser1-1705569626 testuser1-1705570505 testuser1-1705570507 testuser1-1705570511 testuser1-1705576411 testuser1-1705576804 testuser1-1705592708 testuser1705595750 testuser1705595752 testuser1705595754 testuser1705595756 testuser1705595758 testuser1705595760 testuser1705595762 testuser1705595764 testuser1705595766 testuser1705595768 testuser1705595770 testuser1705595772 testuser1705595774 testuser1705595776 testuser1705595778 testuser1705595780 testuser1705595782 testuser1705595784 testuser1705595786 testuser1705595788] [2024-01-18T16:36:31.644Z] INFO[2024-01-18 16:36:31] Verifying : Description : Users [testuser1705595750 testuser1705595752 testuser1705595754 testuser1705595756 testuser1705595758 testuser1705595760 testuser1705595762 testuser1705595764 testuser1705595766 testuser1705595768 testuser1705595770 testuser1705595772 testuser1705595774 testuser1705595776 testuser1705595778 testuser1705595780 testuser1705595782 testuser1705595784 testuser1705595786 testuser1705595788] creation verification [2024-01-18T16:36:31.644Z] INFO[2024-01-18 16:36:31] Actual:, Expected: [2024-01-18T16:36:31.644Z] 2024-01-18 16:36:31 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.glob..func80.3:#132] - Cleanup started [2024-01-18T16:36:33.008Z] I0118 16:36:32.593319 14 request.go:682] Waited for 1.071691485s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/services [2024-01-18T16:36:42.951Z] 2024-01-18 16:36:41 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1705595738 - 1b0f8dbd-9804-4469-8f7a-d684ca218567 [2024-01-18T16:36:42.951Z] 2024-01-18 16:36:41 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1705595748 - f2cd0b3f-8c50-4189-b59a-7ac2bb058dca [2024-01-18T16:36:42.951Z] 2024-01-18 16:36:41 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1705595730 - 22db45b9-04d9-4949-8afd-b73b436a00e0 [2024-01-18T16:36:42.951Z] 2024-01-18 16:36:41 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1705595732 - 0b45989b-9a02-4e02-b673-9c769b1723e5 [2024-01-18T16:36:42.951Z] 2024-01-18 16:36:42 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1705595734 - 0833db7b-3ed6-4ca3-9be1-33575e608f32 [2024-01-18T16:36:42.951Z] 2024-01-18 16:36:42 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1705595736 - b9a2707b-1a64-4b43-9632-5f8975de7b8a [2024-01-18T16:36:42.951Z] 2024-01-18 16:36:42 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1705595744 - f1dfbfac-6a23-4eca-8cb4-ce8c2d10a019 [2024-01-18T16:36:42.951Z] 2024-01-18 16:36:42 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1705595740 - 436811a1-57f6-463b-a40d-476c4f5d2259 [2024-01-18T16:36:42.951Z] I0118 16:36:42.793137 14 request.go:682] Waited for 1.95931939s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-18T16:36:42.951Z] 2024-01-18 16:36:42 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1705595742 - 35ba569b-2daa-4334-ac23-7b5e5c130642 [2024-01-18T16:36:43.207Z] 2024-01-18 16:36:43 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1705595746 - eeaa1b76-26d5-4162-8455-5933064c8efc [2024-01-18T16:36:45.722Z] 2024-01-18 16:36:45 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteGroup:#1018] - Deleted Group - testGroup1705595738 [2024-01-18T16:36:45.722Z] 2024-01-18 16:36:45 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleGroups.func1:#1035] - Deleted Group - testGroup1705595738 [2024-01-18T16:36:45.979Z] 2024-01-18 16:36:45 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteGroup:#1018] - Deleted Group - testGroup1705595730 [2024-01-18T16:36:45.979Z] 2024-01-18 16:36:45 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleGroups.func1:#1035] - Deleted Group - testGroup1705595730 [2024-01-18T16:36:45.979Z] 2024-01-18 16:36:45 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteGroup:#1018] - Deleted Group - testGroup1705595748 [2024-01-18T16:36:45.979Z] 2024-01-18 16:36:45 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleGroups.func1:#1035] - Deleted Group - testGroup1705595748 [2024-01-18T16:36:46.235Z] 2024-01-18 16:36:46 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteGroup:#1018] - Deleted Group - testGroup1705595732 [2024-01-18T16:36:46.235Z] 2024-01-18 16:36:46 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleGroups.func1:#1035] - Deleted Group - testGroup1705595732 [2024-01-18T16:36:46.490Z] 2024-01-18 16:36:46 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteGroup:#1018] - Deleted Group - testGroup1705595734 [2024-01-18T16:36:46.491Z] 2024-01-18 16:36:46 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleGroups.func1:#1035] - Deleted Group - testGroup1705595734 [2024-01-18T16:36:46.747Z] 2024-01-18 16:36:46 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteGroup:#1018] - Deleted Group - testGroup1705595736 [2024-01-18T16:36:46.747Z] 2024-01-18 16:36:46 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleGroups.func1:#1035] - Deleted Group - testGroup1705595736 [2024-01-18T16:36:46.747Z] 2024-01-18 16:36:46 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteGroup:#1018] - Deleted Group - testGroup1705595744 [2024-01-18T16:36:46.747Z] 2024-01-18 16:36:46 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleGroups.func1:#1035] - Deleted Group - testGroup1705595744 [2024-01-18T16:36:47.311Z] 2024-01-18 16:36:47 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteGroup:#1018] - Deleted Group - testGroup1705595740 [2024-01-18T16:36:47.311Z] 2024-01-18 16:36:47 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleGroups.func1:#1035] - Deleted Group - testGroup1705595740 [2024-01-18T16:36:47.311Z] 2024-01-18 16:36:47 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteGroup:#1018] - Deleted Group - testGroup1705595742 [2024-01-18T16:36:47.311Z] 2024-01-18 16:36:47 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleGroups.func1:#1035] - Deleted Group - testGroup1705595742 [2024-01-18T16:36:47.311Z] 2024-01-18 16:36:47 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteGroup:#1018] - Deleted Group - testGroup1705595746 [2024-01-18T16:36:47.311Z] 2024-01-18 16:36:47 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleGroups.func1:#1035] - Deleted Group - testGroup1705595746 [2024-01-18T16:36:47.311Z] INFO[2024-01-18 16:36:47] Verifying : Description : Delete Groups [testGroup1705595730 testGroup1705595732 testGroup1705595734 testGroup1705595736 testGroup1705595738 testGroup1705595740 testGroup1705595742 testGroup1705595744 testGroup1705595746 testGroup1705595748] [2024-01-18T16:36:47.311Z] INFO[2024-01-18 16:36:47] Actual:true, Expected: true [2024-01-18T16:36:53.856Z] I0118 16:36:52.793346 14 request.go:682] Waited for 3.968801781s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-18T16:37:03.797Z] I0118 16:37:02.793332 14 request.go:682] Waited for 3.968928225s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-18T16:37:07.963Z] 2024-01-18 16:37:07 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1705595788 - 59145423-854d-4def-90c7-24d6290e61bd [2024-01-18T16:37:07.963Z] 2024-01-18 16:37:07 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1705595750 - 8b9123d5-5893-4a0a-a732-752ce04b20bd [2024-01-18T16:37:07.963Z] 2024-01-18 16:37:07 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1705595752 - 4e1730ab-85a6-4dd3-b09e-39b4cb799741 [2024-01-18T16:37:07.963Z] 2024-01-18 16:37:07 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1705595754 - 985d0cf8-9fd5-414b-add6-b706959c6649 [2024-01-18T16:37:08.219Z] 2024-01-18 16:37:08 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1705595756 - 615126eb-32cf-4c8f-ba99-739da18cd23a [2024-01-18T16:37:08.475Z] 2024-01-18 16:37:08 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1705595758 - 2b45713f-b7ad-47d9-b3f1-c4cfee90675d [2024-01-18T16:37:08.475Z] 2024-01-18 16:37:08 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1705595760 - 2f70a779-b931-4851-a78e-8b13d8bb1e77 [2024-01-18T16:37:08.730Z] 2024-01-18 16:37:08 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1705595762 - 6393920f-db1e-419a-9e7f-c8a866c10a24 [2024-01-18T16:37:08.988Z] 2024-01-18 16:37:08 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1705595764 - f896a168-5e6b-49cf-9185-12cd91f8fde0 [2024-01-18T16:37:09.251Z] 2024-01-18 16:37:09 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1705595776 - 4bbe1b22-cdf4-4be1-bcc1-7e8ba516f2a6 [2024-01-18T16:37:09.251Z] 2024-01-18 16:37:09 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1705595778 - 66c991f0-aeec-4654-b471-f078f338cad0 [2024-01-18T16:37:09.508Z] 2024-01-18 16:37:09 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1705595766 - 468df9bf-94dd-4d6a-a053-24ef2daef544 [2024-01-18T16:37:09.764Z] 2024-01-18 16:37:09 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1705595784 - 44966bec-e741-4228-92cf-536b40f1e08e [2024-01-18T16:37:10.020Z] 2024-01-18 16:37:09 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1705595768 - ce94fa6f-9dad-4ba0-99bf-249cd1ee029b [2024-01-18T16:37:10.276Z] 2024-01-18 16:37:10 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1705595786 - 6ca7c391-0d9c-4237-9b17-8dbc4f275db2 [2024-01-18T16:37:10.276Z] 2024-01-18 16:37:10 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1705595770 - d450c0d8-4afc-4532-93a9-785678ae7332 [2024-01-18T16:37:10.533Z] 2024-01-18 16:37:10 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1705595772 - e37493c6-edf8-45ca-ac8d-fe87f179306c [2024-01-18T16:37:10.790Z] 2024-01-18 16:37:10 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1705595774 - a98334c5-a93b-4c75-bb42-dbdee67cfa80 [2024-01-18T16:37:11.046Z] 2024-01-18 16:37:10 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1705595780 - 50520ec3-fb0a-4a77-910c-7201e57b1c3e [2024-01-18T16:37:11.325Z] 2024-01-18 16:37:11 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1705595782 - 4143594f-180b-4a84-802a-25ed7593fe5f [2024-01-18T16:37:13.224Z] I0118 16:37:12.793387 14 request.go:682] Waited for 3.970741614s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/services [2024-01-18T16:37:15.744Z] 2024-01-18 16:37:15 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser1705595788] deletion - [2024-01-18T16:37:15.744Z] 2024-01-18 16:37:15 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser1705595788 [2024-01-18T16:37:15.744Z] 2024-01-18 16:37:15 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleUsers.func1:#1063] - Deleted User - testuser1705595788 [2024-01-18T16:37:15.744Z] 2024-01-18 16:37:15 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser1705595750] deletion - [2024-01-18T16:37:15.744Z] 2024-01-18 16:37:15 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser1705595750 [2024-01-18T16:37:15.744Z] 2024-01-18 16:37:15 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleUsers.func1:#1063] - Deleted User - testuser1705595750 [2024-01-18T16:37:16.000Z] 2024-01-18 16:37:15 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser1705595752] deletion - [2024-01-18T16:37:16.000Z] 2024-01-18 16:37:15 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser1705595752 [2024-01-18T16:37:16.000Z] 2024-01-18 16:37:15 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleUsers.func1:#1063] - Deleted User - testuser1705595752 [2024-01-18T16:37:16.000Z] 2024-01-18 16:37:15 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser1705595754] deletion - [2024-01-18T16:37:16.000Z] 2024-01-18 16:37:15 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser1705595754 [2024-01-18T16:37:16.000Z] 2024-01-18 16:37:15 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleUsers.func1:#1063] - Deleted User - testuser1705595754 [2024-01-18T16:37:16.255Z] 2024-01-18 16:37:16 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser1705595756] deletion - [2024-01-18T16:37:16.255Z] 2024-01-18 16:37:16 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser1705595756 [2024-01-18T16:37:16.255Z] 2024-01-18 16:37:16 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleUsers.func1:#1063] - Deleted User - testuser1705595756 [2024-01-18T16:37:16.511Z] 2024-01-18 16:37:16 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser1705595758] deletion - [2024-01-18T16:37:16.511Z] 2024-01-18 16:37:16 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser1705595758 [2024-01-18T16:37:16.511Z] 2024-01-18 16:37:16 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleUsers.func1:#1063] - Deleted User - testuser1705595758 [2024-01-18T16:37:16.766Z] 2024-01-18 16:37:16 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser1705595760] deletion - [2024-01-18T16:37:16.766Z] 2024-01-18 16:37:16 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser1705595760 [2024-01-18T16:37:16.766Z] 2024-01-18 16:37:16 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleUsers.func1:#1063] - Deleted User - testuser1705595760 [2024-01-18T16:37:16.766Z] 2024-01-18 16:37:16 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser1705595762] deletion - [2024-01-18T16:37:16.766Z] 2024-01-18 16:37:16 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser1705595762 [2024-01-18T16:37:16.766Z] 2024-01-18 16:37:16 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleUsers.func1:#1063] - Deleted User - testuser1705595762 [2024-01-18T16:37:17.022Z] 2024-01-18 16:37:16 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser1705595764] deletion - [2024-01-18T16:37:17.022Z] 2024-01-18 16:37:16 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser1705595764 [2024-01-18T16:37:17.022Z] 2024-01-18 16:37:16 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleUsers.func1:#1063] - Deleted User - testuser1705595764 [2024-01-18T16:37:17.277Z] 2024-01-18 16:37:17 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser1705595776] deletion - [2024-01-18T16:37:17.277Z] 2024-01-18 16:37:17 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser1705595776 [2024-01-18T16:37:17.278Z] 2024-01-18 16:37:17 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleUsers.func1:#1063] - Deleted User - testuser1705595776 [2024-01-18T16:37:17.535Z] 2024-01-18 16:37:17 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser1705595778] deletion - [2024-01-18T16:37:17.535Z] 2024-01-18 16:37:17 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser1705595778 [2024-01-18T16:37:17.535Z] 2024-01-18 16:37:17 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleUsers.func1:#1063] - Deleted User - testuser1705595778 [2024-01-18T16:37:17.791Z] 2024-01-18 16:37:17 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser1705595766] deletion - [2024-01-18T16:37:17.791Z] 2024-01-18 16:37:17 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser1705595766 [2024-01-18T16:37:17.791Z] 2024-01-18 16:37:17 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleUsers.func1:#1063] - Deleted User - testuser1705595766 [2024-01-18T16:37:17.791Z] 2024-01-18 16:37:17 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser1705595784] deletion - [2024-01-18T16:37:17.791Z] 2024-01-18 16:37:17 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser1705595784 [2024-01-18T16:37:17.791Z] 2024-01-18 16:37:17 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleUsers.func1:#1063] - Deleted User - testuser1705595784 [2024-01-18T16:37:18.047Z] 2024-01-18 16:37:17 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser1705595768] deletion - [2024-01-18T16:37:18.047Z] 2024-01-18 16:37:17 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser1705595768 [2024-01-18T16:37:18.047Z] 2024-01-18 16:37:17 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleUsers.func1:#1063] - Deleted User - testuser1705595768 [2024-01-18T16:37:18.302Z] 2024-01-18 16:37:18 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser1705595786] deletion - [2024-01-18T16:37:18.302Z] 2024-01-18 16:37:18 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser1705595786 [2024-01-18T16:37:18.302Z] 2024-01-18 16:37:18 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleUsers.func1:#1063] - Deleted User - testuser1705595786 [2024-01-18T16:37:18.558Z] 2024-01-18 16:37:18 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser1705595770] deletion - [2024-01-18T16:37:18.558Z] 2024-01-18 16:37:18 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser1705595770 [2024-01-18T16:37:18.558Z] 2024-01-18 16:37:18 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleUsers.func1:#1063] - Deleted User - testuser1705595770 [2024-01-18T16:37:18.558Z] 2024-01-18 16:37:18 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser1705595772] deletion - [2024-01-18T16:37:18.558Z] 2024-01-18 16:37:18 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser1705595772 [2024-01-18T16:37:18.558Z] 2024-01-18 16:37:18 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleUsers.func1:#1063] - Deleted User - testuser1705595772 [2024-01-18T16:37:18.813Z] 2024-01-18 16:37:18 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser1705595774] deletion - [2024-01-18T16:37:18.813Z] 2024-01-18 16:37:18 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser1705595774 [2024-01-18T16:37:18.814Z] 2024-01-18 16:37:18 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleUsers.func1:#1063] - Deleted User - testuser1705595774 [2024-01-18T16:37:19.070Z] 2024-01-18 16:37:18 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser1705595780] deletion - [2024-01-18T16:37:19.070Z] 2024-01-18 16:37:18 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser1705595780 [2024-01-18T16:37:19.071Z] 2024-01-18 16:37:18 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleUsers.func1:#1063] - Deleted User - testuser1705595780 [2024-01-18T16:37:19.326Z] 2024-01-18 16:37:19 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#755] - Response for user [testuser1705595782] deletion - [2024-01-18T16:37:19.326Z] 2024-01-18 16:37:19 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteUser:#759] - Deleted User - testuser1705595782 [2024-01-18T16:37:19.326Z] 2024-01-18 16:37:19 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.DeleteMultipleUsers.func1:#1063] - Deleted User - testuser1705595782 [2024-01-18T16:37:19.326Z] INFO[2024-01-18 16:37:19] Verifying : Description : Delete users [testuser1705595750 testuser1705595752 testuser1705595754 testuser1705595756 testuser1705595758 testuser1705595760 testuser1705595762 testuser1705595764 testuser1705595766 testuser1705595768 testuser1705595770 testuser1705595772 testuser1705595774 testuser1705595776 testuser1705595778 testuser1705595780 testuser1705595782 testuser1705595784 testuser1705595786 testuser1705595788] [2024-01-18T16:37:19.326Z] INFO[2024-01-18 16:37:19] Actual:true, Expected: true [2024-01-18T16:37:19.326Z] 2024-01-18 16:37:19 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [backup.glob..func80.3:#137] - Cleanup done [2024-01-18T16:37:19.326Z] INFO[2024-01-18 16:37:19] --------Test End------ [2024-01-18T16:37:19.327Z] INFO[2024-01-18 16:37:19] #Test: [2024-01-18T16:37:19.327Z] INFO[2024-01-18 16:37:19] #Description: [2024-01-18T16:37:19.327Z] INFO[2024-01-18 16:37:19] ------------------------ [2024-01-18T16:37:19.327Z] 2024-01-18 16:37:19 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T16:37:19.327Z] 2024-01-18 16:37:19 +0000:[INFO] [{CreateMultipleUsersAndGroups}] Switching context to [/tmp/destination-config] [2024-01-18T16:37:19.583Z] 2024-01-18 16:37:19 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:37:19.583Z] 2024-01-18 16:37:19 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:37:19.583Z] 2024-01-18 16:37:19 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:37:19.583Z] 2024-01-18 16:37:19 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:37:19.583Z] 2024-01-18 16:37:19 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:37:19.840Z] 2024-01-18 16:37:19 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:37:20.401Z] 2024-01-18 16:37:20 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:37:20.657Z] 2024-01-18 16:37:20 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:37:21.220Z] 2024-01-18 16:37:20 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:37:21.220Z] 2024-01-18 16:37:21 +0000:[WARNING] [{CreateMultipleUsersAndGroups}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:37:21.476Z] 2024-01-18 16:37:21 +0000:[WARNING] [{CreateMultipleUsersAndGroups}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:37:21.476Z] 2024-01-18 16:37:21 +0000:[WARNING] [{CreateMultipleUsersAndGroups}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:37:21.476Z] 2024-01-18 16:37:21 +0000:[INFO] [{CreateMultipleUsersAndGroups}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:37:21.477Z] 2024-01-18 16:37:21 +0000:[INFO] [{CreateMultipleUsersAndGroups}] Switched context to [/tmp/destination-config] [2024-01-18T16:37:21.732Z] 2024-01-18 16:37:21 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {CreateMultipleUsersAndGroups} Create multiple users and Group [2024-01-18T16:37:21.732Z] 2024-01-18 16:37:21 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-18T16:37:21.732Z] 2024/01/18 16:37:21 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:37:26.975Z] 2024/01/18 16:37:26 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:37:32.220Z] 2024/01/18 16:37:31 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:37:37.471Z] 2024/01/18 16:37:36 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:37:41.637Z] 2024/01/18 16:37:41 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:37:46.882Z] 2024/01/18 16:37:46 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:37:52.129Z] 2024/01/18 16:37:51 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:37:57.374Z] 2024/01/18 16:37:56 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T16:38:01.566Z] 2024/01/18 16:38:01 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.0.4. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.4 10.240.0.4]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T16:38:01.566Z] {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T16:38:01.566Z] {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T16:38:01.566Z] {dial tcp 10.240.0.4:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-18T16:38:06.812Z] 2024-01-18 16:38:06 +0000:[WARNING] [{CreateMultipleUsersAndGroups}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.4. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.4 10.240.0.4]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T16:38:06.812Z] {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T16:38:06.812Z] {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T16:38:06.812Z] {dial tcp 10.240.0.4:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T16:38:06.812Z] 2024-01-18 16:38:06 +0000:[ERROR] [{CreateMultipleUsersAndGroups}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.4. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.4 10.240.0.4]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T16:38:06.812Z] {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T16:38:06.812Z] {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T16:38:06.812Z] {dial tcp 10.240.0.4:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T16:38:06.812Z] 2024-01-18 16:38:06 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T16:38:06.812Z] 2024-01-18 16:38:06 +0000:[INFO] [{CreateMultipleUsersAndGroups}] Switching context to [/tmp/source-config] [2024-01-18T16:38:06.813Z] 2024-01-18 16:38:06 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:38:06.813Z] 2024-01-18 16:38:06 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:38:06.813Z] 2024-01-18 16:38:06 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:38:06.813Z] 2024-01-18 16:38:06 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:38:07.068Z] 2024-01-18 16:38:06 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:38:07.068Z] 2024-01-18 16:38:06 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:38:07.629Z] 2024-01-18 16:38:07 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:38:07.885Z] 2024-01-18 16:38:07 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:38:08.448Z] 2024-01-18 16:38:08 +0000:[INFO] [{CreateMultipleUsersAndGroups}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:38:08.448Z] 2024-01-18 16:38:08 +0000:[WARNING] [{CreateMultipleUsersAndGroups}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:38:08.704Z] 2024-01-18 16:38:08 +0000:[WARNING] [{CreateMultipleUsersAndGroups}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:38:08.704Z] 2024-01-18 16:38:08 +0000:[WARNING] [{CreateMultipleUsersAndGroups}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:38:08.704Z] 2024-01-18 16:38:08 +0000:[INFO] [{CreateMultipleUsersAndGroups}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:38:08.704Z] 2024-01-18 16:38:08 +0000:[INFO] [{CreateMultipleUsersAndGroups}] Switched context to [/tmp/source-config] [2024-01-18T16:38:08.704Z] •2024-01-18 16:38:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Log Dir: /testresults//VerifyRBACForInfraAdmin.log [2024-01-18T16:38:08.704Z] INFO[2024-01-18 16:38:08] --------Test Start------ [2024-01-18T16:38:08.704Z] INFO[2024-01-18 16:38:08] #Test: VerifyRBACForInfraAdmin [2024-01-18T16:38:08.704Z] INFO[2024-01-18 16:38:08] #Description: Validates the RBAC operation for infra-admin user. [2024-01-18T16:38:08.704Z] INFO[2024-01-18 16:38:08] ------------------------ [2024-01-18T16:38:08.704Z] 2024-01-18 16:38:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] scheduling applications [2024-01-18T16:38:08.704Z] 2024-01-18 16:38:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T16:38:08.704Z] 2024-01-18 16:38:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T16:38:08.959Z] 2024-01-18 16:38:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T16:38:08.959Z] 2024-01-18 16:38:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T16:38:09.215Z] 2024-01-18 16:38:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T16:38:09.471Z] 2024-01-18 16:38:09 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T16:38:09.471Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T16:38:09.471Z] 2024-01-18 16:38:09 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T16:38:09.472Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T16:38:09.472Z] 2024-01-18 16:38:09 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T16:38:09.472Z] 2024-01-18 16:38:09 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating applications [2024-01-18T16:38:09.472Z] 2024-01-18 16:38:09 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validate applications [2024-01-18T16:38:09.472Z] 2024-01-18 16:38:09 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating mysql-ibm app [2024-01-18T16:38:09.472Z] 2024-01-18 16:38:09 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating mysql-ibm app's volumes [2024-01-18T16:38:09.472Z] 2024-01-18 16:38:09 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:38:09.731Z] 2024/01/18 16:38:09 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T16:38:19.673Z] 2024/01/18 16:38:19 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T16:38:31.845Z] 2024-01-18 16:38:29 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-87886-01-18-16h38m08s [2024-01-18T16:38:31.845Z] 2024-01-18 16:38:29 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T16:38:31.845Z] 2024-01-18 16:38:29 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T16:38:31.845Z] 2024-01-18 16:38:29 +0000:[INFO] [{VerifyRBACForInfraAdmin}] wait for mysql-ibm app to start running [2024-01-18T16:38:31.845Z] 2024-01-18 16:38:29 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T16:38:31.845Z] 2024/01/18 16:38:29 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:38:31.845Z] pod name:mysql-7d766cd65c-vkj7k namespace:mysql-ibm-pxb-0-87886-01-18-16h38m08s running:false ready:false node:10.240.64.9 [2024-01-18T16:38:31.845Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:38:39.930Z] 2024/01/18 16:38:39 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:38:39.930Z] pod name:mysql-7d766cd65c-vkj7k namespace:mysql-ibm-pxb-0-87886-01-18-16h38m08s running:true ready:false node:10.240.64.9 [2024-01-18T16:38:39.930Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:38:52.096Z] 2024/01/18 16:38:49 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:38:52.096Z] pod name:mysql-7d766cd65c-vkj7k namespace:mysql-ibm-pxb-0-87886-01-18-16h38m08s running:true ready:false node:10.240.64.9 [2024-01-18T16:38:52.097Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:39:00.183Z] 2024/01/18 16:39:00 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:39:00.183Z] pod name:mysql-7d766cd65c-vkj7k namespace:mysql-ibm-pxb-0-87886-01-18-16h38m08s running:true ready:false node:10.240.64.9 [2024-01-18T16:39:00.183Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:39:12.361Z] 2024/01/18 16:39:10 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:39:12.361Z] pod name:mysql-7d766cd65c-vkj7k namespace:mysql-ibm-pxb-0-87886-01-18-16h38m08s running:true ready:false node:10.240.64.9 [2024-01-18T16:39:12.361Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:39:20.452Z] 2024/01/18 16:39:20 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:39:20.452Z] pod name:mysql-7d766cd65c-vkj7k namespace:mysql-ibm-pxb-0-87886-01-18-16h38m08s running:true ready:false node:10.240.64.9 [2024-01-18T16:39:20.452Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:39:30.392Z] 2024/01/18 16:39:30 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:39:30.392Z] pod name:mysql-7d766cd65c-vkj7k namespace:mysql-ibm-pxb-0-87886-01-18-16h38m08s running:true ready:false node:10.240.64.9 [2024-01-18T16:39:30.392Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:39:42.569Z] 2024-01-18 16:39:40 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T16:39:42.569Z] 2024-01-18 16:39:40 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T16:39:42.569Z] 2024-01-18 16:39:40 +0000:[INFO] [{VerifyRBACForInfraAdmin}] validate if mysql-ibm app's volumes are setup [2024-01-18T16:39:42.570Z] 2024-01-18 16:39:40 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:39:42.570Z] 2024-01-18 16:39:40 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:39:42.570Z] 2024-01-18 16:39:40 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T16:39:42.570Z] 2024-01-18 16:39:40 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T16:39:42.570Z] 2024-01-18 16:39:40 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Adding labels to namespaces [2024-01-18T16:39:42.570Z] INFO[2024-01-18 16:39:40] Verifying : Description : Adding labels [map[0-0235ca01-2486-49e1-ba1d-6858b4c6d492:0235ca01-2486-49e1-ba1d-6858b4c6d492 1-0235ca01-2486-49e1-ba1d-6858b4c6d492:0235ca01-2486-49e1-ba1d-6858b4c6d492]] to namespaces [[mysql-ibm-pxb-0-87886-01-18-16h38m08s]] for infra admin validation [2024-01-18T16:39:42.570Z] INFO[2024-01-18 16:39:40] Actual:true, Expected: true [2024-01-18T16:39:42.570Z] 2024-01-18 16:39:40 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Generating namespace label string from label map for namespaces [2024-01-18T16:39:42.570Z] 2024-01-18 16:39:40 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.glob..func43.2.3:#101] - label string for namespaces 0-0235ca01-2486-49e1-ba1d-6858b4c6d492=0235ca01-2486-49e1-ba1d-6858b4c6d492,1-0235ca01-2486-49e1-ba1d-6858b4c6d492=0235ca01-2486-49e1-ba1d-6858b4c6d492 [2024-01-18T16:39:42.570Z] 2024-01-18 16:39:40 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Creating a user with px-backup-infra.admin role [2024-01-18T16:39:42.570Z] 2024-01-18 16:39:40 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Creating 1 users [2024-01-18T16:39:42.570Z] 2024-01-18 16:39:40 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.AddUser:#728] - Response for user [testuser1-1705595980] creation - [2024-01-18T16:39:42.570Z] 2024-01-18 16:39:40 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.AddUser:#733] - User [testuser1-1705595980] created [2024-01-18T16:39:42.570Z] INFO[2024-01-18 16:39:40] Verifying : Description : Creating user - testuser1-1705595980 [2024-01-18T16:39:42.570Z] INFO[2024-01-18 16:39:40] Actual:true, Expected: true [2024-01-18T16:39:42.570Z] 2024-01-18 16:39:41 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705595980 - cf87c088-7fb7-4cc7-a18d-95f00d2a9974 [2024-01-18T16:39:43.133Z] 2024-01-18 16:39:42 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Verify Infra-Admin User has permission to create cloud credential and backup location [2024-01-18T16:39:43.390Z] 2024-01-18 16:39:43 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-lvxlx] for org [default] with [aws] as provider [2024-01-18T16:39:43.390Z] 2024-01-18 16:39:43 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-18T16:39:43.646Z] 2024-01-18 16:39:43 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-lvxlx] is present [2024-01-18T16:39:43.646Z] INFO[2024-01-18 16:39:43] Verifying : Description : Verifying creation of create cloud credential autogenerated-cred-lvxlx using provider aws for the user [2024-01-18T16:39:43.646Z] INFO[2024-01-18 16:39:43] Actual:true, Expected: true [2024-01-18T16:39:46.165Z] INFO[2024-01-18 16:39:45] Verifying : Description : Verifying creation of create backup location autogenerated-backup-location-mpzou using provider aws for the user [2024-01-18T16:39:46.165Z] INFO[2024-01-18 16:39:45] Actual:true, Expected: true [2024-01-18T16:39:46.165Z] 2024-01-18 16:39:45 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Verify Infra-Admin User has permission to create a schedule policy [2024-01-18T16:39:46.165Z] INFO[2024-01-18 16:39:45] Verifying : Description : Verifying creation of periodic schedule policy of interval [15] minutes named [periodic-oyizj] [2024-01-18T16:39:46.165Z] INFO[2024-01-18 16:39:45] Actual:true, Expected: true [2024-01-18T16:39:46.165Z] 2024-01-18 16:39:45 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Verify Infra-Admin User has permission to create pre and post exec rules for applications [2024-01-18T16:39:46.422Z] 2024-01-18 16:39:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.CreateRuleForBackupWithMultipleApplications:#4309] - Validate pre-rules for backup [2024-01-18T16:39:46.422Z] 2024-01-18 16:39:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.CreateRuleForBackupWithMultipleApplications:#4322] - Validate post-rules for backup [2024-01-18T16:39:46.422Z] INFO[2024-01-18 16:39:46] Verifying : Description : Verifying creation of pre and post exec rules for applications from px-admin [2024-01-18T16:39:46.422Z] INFO[2024-01-18 16:39:46] Actual:true, Expected: true [2024-01-18T16:39:46.422Z] 2024-01-18 16:39:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.glob..func43.2.7:#149] - Pre backup rule [pre-rule-jdrjo] uid: [34193f4f-e6a8-4c7d-aaa3-73ab6e791009] [2024-01-18T16:39:46.422Z] 2024-01-18 16:39:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.glob..func43.2.7:#154] - Post backup rule [post-rule-lenid] uid: [0012c705-ceb6-4e94-bc66-982dec7933df] [2024-01-18T16:39:46.422Z] 2024-01-18 16:39:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Creating source and destination cluster from the infra admin user testuser1-1705595980 [2024-01-18T16:39:46.422Z] INFO[2024-01-18 16:39:46] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T16:39:46.422Z] INFO[2024-01-18 16:39:46] Actual:true, Expected: true [2024-01-18T16:39:46.422Z] 2024-01-18 16:39:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T16:39:46.422Z] 2024-01-18 16:39:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T16:39:46.422Z] 2024-01-18 16:39:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T16:39:46.422Z] 2024-01-18 16:39:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T16:39:46.422Z] 2024-01-18 16:39:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Create cluster [source-cluster] in org [default] [2024-01-18T16:39:46.422Z] 2024-01-18 16:39:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T16:39:46.422Z] 2024-01-18 16:39:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T16:39:46.423Z] 2024-01-18 16:39:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-18T16:39:46.423Z] 2024-01-18 16:39:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T16:39:46.423Z] 2024-01-18 16:39:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T16:39:46.423Z] 2024-01-18 16:39:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-htmnt] for IBM [2024-01-18T16:39:46.423Z] 2024-01-18 16:39:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-htmnt] for org [default] with [ibm] as provider [2024-01-18T16:39:46.679Z] 2024-01-18 16:39:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:39:46.679Z] 2024-01-18 16:39:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:39:46.679Z] 2024-01-18 16:39:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-htmnt] is present [2024-01-18T16:39:46.679Z] 2024-01-18 16:39:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Create cluster [source-cluster] in org [default] [2024-01-18T16:39:47.242Z] 2024-01-18 16:39:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-btcfu] for IBM [2024-01-18T16:39:47.242Z] 2024-01-18 16:39:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-btcfu] for org [default] with [ibm] as provider [2024-01-18T16:39:47.242Z] 2024-01-18 16:39:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:39:47.242Z] 2024-01-18 16:39:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:39:47.242Z] 2024-01-18 16:39:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-btcfu] is present [2024-01-18T16:39:47.242Z] 2024-01-18 16:39:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-18T16:39:47.804Z] INFO[2024-01-18 16:39:47] Verifying : Description : Verifying creation of source [source-cluster] and destination [destination-cluster] clusters with App-User ctx [2024-01-18T16:39:47.804Z] INFO[2024-01-18 16:39:47] Actual:true, Expected: true [2024-01-18T16:39:47.805Z] INFO[2024-01-18 16:39:47] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-18T16:39:47.805Z] INFO[2024-01-18 16:39:47] Actual:Online, Expected: Online [2024-01-18T16:39:47.805Z] INFO[2024-01-18 16:39:47] Verifying : Description : Verifying if [destination-cluster] cluster is online [2024-01-18T16:39:47.805Z] INFO[2024-01-18 16:39:47] Actual:Online, Expected: Online [2024-01-18T16:39:47.805Z] 2024-01-18 16:39:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.glob..func43.2.8:#171] - Cluster [source-cluster] uid: [94b0ebdb-99a8-495e-a458-a6793487c91d] [2024-01-18T16:39:47.805Z] 2024-01-18 16:39:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validate taking manual backup of applications with namespace label [2024-01-18T16:39:47.805Z] 2024-01-18 16:39:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.glob..func43.2.9:#178] - Cluster [source-cluster] uid: [94b0ebdb-99a8-495e-a458-a6793487c91d] [2024-01-18T16:39:47.805Z] 2024-01-18 16:39:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T16:39:47.805Z] 2024-01-18 16:39:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Volume snapshot classes found in the cluster - [2024-01-18T16:39:47.805Z] 2024-01-18 16:39:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] ibmc-vpcblock-snapshot [2024-01-18T16:39:47.805Z] 2024-01-18 16:39:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T16:39:48.733Z] 2024/01/18 16:39:48 DoRetryWithTimeout - Error: {backup status for [infraadmin-nvtjmmgolieqpmr] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T16:40:20.813Z] 2024/01/18 16:40:18 DoRetryWithTimeout - Error: {backup status for [infraadmin-nvtjmmgolieqpmr] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:40:52.852Z] 2024/01/18 16:40:48 DoRetryWithTimeout - Error: {backup status for [infraadmin-nvtjmmgolieqpmr] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:41:19.352Z] 2024-01-18 16:41:18 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.CreateBackupWithNamespaceLabel:#3030] - Successfully created backup [infraadmin-nvtjmmgolieqpmr] with namespace label [0-0235ca01-2486-49e1-ba1d-6858b4c6d492=0235ca01-2486-49e1-ba1d-6858b4c6d492,1-0235ca01-2486-49e1-ba1d-6858b4c6d492=0235ca01-2486-49e1-ba1d-6858b4c6d492] [2024-01-18T16:41:19.352Z] 2024-01-18 16:41:18 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating backup [infraadmin-nvtjmmgolieqpmr] in org [default] [2024-01-18T16:41:19.352Z] 2024-01-18 16:41:18 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [infraadmin-nvtjmmgolieqpmr] [2024-01-18T16:41:19.352Z] 2024-01-18 16:41:18 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in backup [infraadmin-nvtjmmgolieqpmr] [2024-01-18T16:41:19.352Z] 2024-01-18 16:41:18 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in backup [infraadmin-nvtjmmgolieqpmr] [2024-01-18T16:41:19.352Z] 2024-01-18 16:41:18 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T16:41:19.352Z] 2024-01-18 16:41:18 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:41:19.352Z] 2024-01-18 16:41:18 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:41:19.352Z] 2024-01-18 16:41:18 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[mysql-data]] [2024-01-18T16:41:19.352Z] 2024-01-18 16:41:18 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T16:41:19.352Z] 2024-01-18 16:41:18 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating CR cleanup [2024-01-18T16:41:19.352Z] 2024-01-18 16:41:18 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:41:19.353Z] 2024-01-18 16:41:18 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[]] [2024-01-18T16:41:19.353Z] INFO[2024-01-18 16:41:18] Verifying : Description : Verifying backup [infraadmin-nvtjmmgolieqpmr] creation with labels [0-0235ca01-2486-49e1-ba1d-6858b4c6d492=0235ca01-2486-49e1-ba1d-6858b4c6d492,1-0235ca01-2486-49e1-ba1d-6858b4c6d492=0235ca01-2486-49e1-ba1d-6858b4c6d492] [2024-01-18T16:41:19.353Z] INFO[2024-01-18 16:41:18] Actual:true, Expected: true [2024-01-18T16:41:19.353Z] 2024-01-18 16:41:18 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup infraadmin-nvtjmmgolieqpmr [2024-01-18T16:41:19.353Z] 2024-01-18 16:41:18 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T16:41:19.353Z] 2024-01-18 16:41:18 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are 0-0235ca01-2486-49e1-ba1d-6858b4c6d492=0235ca01-2486-49e1-ba1d-6858b4c6d492,1-0235ca01-2486-49e1-ba1d-6858b4c6d492=0235ca01-2486-49e1-ba1d-6858b4c6d492 [2024-01-18T16:41:19.353Z] INFO[2024-01-18 16:41:18] Verifying : Description : Verifying if the labeled namespaces [[mysql-ibm-pxb-0-87886-01-18-16h38m08s]] are backed up and check for labels [0-0235ca01-2486-49e1-ba1d-6858b4c6d492=0235ca01-2486-49e1-ba1d-6858b4c6d492,1-0235ca01-2486-49e1-ba1d-6858b4c6d492=0235ca01-2486-49e1-ba1d-6858b4c6d492] applied to backups [infraadmin-nvtjmmgolieqpmr] [2024-01-18T16:41:19.353Z] INFO[2024-01-18 16:41:18] Actual:true, Expected: true [2024-01-18T16:41:19.353Z] 2024-01-18 16:41:18 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validate restoring manual backup of applications with namespace label [2024-01-18T16:41:19.918Z] 2024/01/18 16:41:19 DoRetryWithTimeout - Error: {restore status for [tp-restore-infraadmin-nvtjmmgolieqpmr] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T16:41:51.946Z] 2024-01-18 16:41:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.CreateRestore:#936] - Restore [tp-restore-infraadmin-nvtjmmgolieqpmr] created successfully [2024-01-18T16:41:51.946Z] 2024-01-18 16:41:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switching cluster context to cluster [destination-cluster] [2024-01-18T16:41:51.946Z] 2024-01-18 16:41:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switching context to [/tmp/destination-config] [2024-01-18T16:41:51.946Z] 2024-01-18 16:41:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:41:51.946Z] 2024-01-18 16:41:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:41:51.946Z] 2024-01-18 16:41:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:41:51.946Z] 2024-01-18 16:41:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:41:51.946Z] 2024-01-18 16:41:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:41:51.946Z] 2024-01-18 16:41:50 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:41:51.946Z] 2024-01-18 16:41:50 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:41:51.947Z] 2024-01-18 16:41:50 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:41:51.947Z] 2024-01-18 16:41:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:41:51.947Z] 2024-01-18 16:41:51 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:41:51.947Z] 2024-01-18 16:41:51 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:41:51.947Z] 2024-01-18 16:41:51 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:41:51.947Z] 2024-01-18 16:41:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:41:51.947Z] 2024-01-18 16:41:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switched context to [/tmp/destination-config] [2024-01-18T16:41:51.947Z] 2024-01-18 16:41:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-87886-01-18-16h38m08s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-18T16:41:51.947Z] 2024-01-18 16:41:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T16:41:51.947Z] 2024-01-18 16:41:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T16:41:51.947Z] 2024-01-18 16:41:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T16:41:51.947Z] 2024-01-18 16:41:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating restore [tp-restore-infraadmin-nvtjmmgolieqpmr] in org [default] [2024-01-18T16:41:51.947Z] 2024-01-18 16:41:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-restore-infraadmin-nvtjmmgolieqpmr] [2024-01-18T16:41:51.947Z] 2024-01-18 16:41:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-restore-infraadmin-nvtjmmgolieqpmr] [2024-01-18T16:41:51.947Z] 2024-01-18 16:41:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in restore [tp-restore-infraadmin-nvtjmmgolieqpmr] [2024-01-18T16:41:51.947Z] 2024-01-18 16:41:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-87886-01-18-16h38m08s]) was found in restore [tp-restore-infraadmin-nvtjmmgolieqpmr], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T16:41:51.947Z] 2024-01-18 16:41:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-87886-01-18-16h38m08s]) found in the restore [tp-restore-infraadmin-nvtjmmgolieqpmr] was also present on the cluster/namespace [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T16:41:51.947Z] 2024-01-18 16:41:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-87886-01-18-16h38m08s]) was found in restore [tp-restore-infraadmin-nvtjmmgolieqpmr], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T16:41:52.203Z] 2024-01-18 16:41:52 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-87886-01-18-16h38m08s]) found in the restore [tp-restore-infraadmin-nvtjmmgolieqpmr] was also present on the cluster/namespace [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T16:41:52.203Z] 2024-01-18 16:41:52 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-87886-01-18-16h38m08s]) was found in restore [tp-restore-infraadmin-nvtjmmgolieqpmr], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T16:41:52.203Z] 2024-01-18 16:41:52 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-87886-01-18-16h38m08s]) found in the restore [tp-restore-infraadmin-nvtjmmgolieqpmr] was also present on the cluster/namespace [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T16:41:52.204Z] 2024-01-18 16:41:52 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-87886-01-18-16h38m08s]) was found in restore [tp-restore-infraadmin-nvtjmmgolieqpmr], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T16:41:52.204Z] 2024-01-18 16:41:52 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-87886-01-18-16h38m08s]) found in the restore [tp-restore-infraadmin-nvtjmmgolieqpmr] was also present on the cluster/namespace [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T16:41:52.204Z] 2024-01-18 16:41:52 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in restore [tp-restore-infraadmin-nvtjmmgolieqpmr] [2024-01-18T16:41:52.204Z] 2024-01-18 16:41:52 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in restore [tp-restore-infraadmin-nvtjmmgolieqpmr] [2024-01-18T16:41:52.460Z] 2024-01-18 16:41:52 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:41:52.460Z] 2024-01-18 16:41:52 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:41:52.460Z] 2024-01-18 16:41:52 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[mysql-data]] [2024-01-18T16:41:52.460Z] 2024-01-18 16:41:52 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2240] - in restore [tp-restore-infraadmin-nvtjmmgolieqpmr], the status of the restored volume [pvc-08811e3e-a792-4522-bacc-c14fe6aeb7de] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-18T16:41:52.460Z] 2024-01-18 16:41:52 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validate applications in restored namespace [mysql-ibm-pxb-0-87886-01-18-16h38m08s] due to restore [tp-restore-infraadmin-nvtjmmgolieqpmr] [2024-01-18T16:41:52.460Z] 2024-01-18 16:41:52 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating mysql-ibm app [2024-01-18T16:41:52.460Z] 2024-01-18 16:41:52 +0000:[INFO] [{VerifyRBACForInfraAdmin}] wait for mysql-ibm app to start running [2024-01-18T16:41:52.718Z] 2024-01-18 16:41:52 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T16:41:52.718Z] 2024/01/18 16:41:52 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:41:52.718Z] pod name:mysql-7d766cd65c-9smbv namespace:mysql-ibm-pxb-0-87886-01-18-16h38m08s running:false ready:false node:10.240.64.7 [2024-01-18T16:41:52.718Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:42:02.677Z] 2024/01/18 16:42:02 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:42:02.678Z] pod name:mysql-7d766cd65c-9smbv namespace:mysql-ibm-pxb-0-87886-01-18-16h38m08s running:false ready:false node:10.240.64.7 [2024-01-18T16:42:02.678Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:42:14.852Z] 2024/01/18 16:42:12 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:42:14.852Z] pod name:mysql-7d766cd65c-9smbv namespace:mysql-ibm-pxb-0-87886-01-18-16h38m08s running:true ready:false node:10.240.64.7 [2024-01-18T16:42:14.852Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:42:22.947Z] 2024/01/18 16:42:22 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:42:22.947Z] pod name:mysql-7d766cd65c-9smbv namespace:mysql-ibm-pxb-0-87886-01-18-16h38m08s running:true ready:false node:10.240.64.7 [2024-01-18T16:42:22.947Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:42:32.912Z] 2024/01/18 16:42:32 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:42:32.912Z] pod name:mysql-7d766cd65c-9smbv namespace:mysql-ibm-pxb-0-87886-01-18-16h38m08s running:true ready:false node:10.240.64.7 [2024-01-18T16:42:32.912Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:42:45.118Z] 2024/01/18 16:42:42 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:42:45.118Z] pod name:mysql-7d766cd65c-9smbv namespace:mysql-ibm-pxb-0-87886-01-18-16h38m08s running:true ready:false node:10.240.64.7 [2024-01-18T16:42:45.118Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:42:53.218Z] 2024/01/18 16:42:52 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:42:53.218Z] pod name:mysql-7d766cd65c-9smbv namespace:mysql-ibm-pxb-0-87886-01-18-16h38m08s running:true ready:false node:10.240.64.7 [2024-01-18T16:42:53.218Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:43:03.189Z] 2024/01/18 16:43:03 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:43:03.189Z] pod name:mysql-7d766cd65c-9smbv namespace:mysql-ibm-pxb-0-87886-01-18-16h38m08s running:true ready:false node:10.240.64.7 [2024-01-18T16:43:03.189Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:43:15.372Z] 2024-01-18 16:43:13 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T16:43:15.372Z] 2024-01-18 16:43:13 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T16:43:15.372Z] 2024-01-18 16:43:13 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating CR cleanup [2024-01-18T16:43:15.372Z] 2024-01-18 16:43:13 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:43:15.373Z] 2024-01-18 16:43:13 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[]] [2024-01-18T16:43:15.373Z] 2024-01-18 16:43:13 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switching cluster context back to cluster path [/tmp/source-config] [2024-01-18T16:43:15.373Z] 2024-01-18 16:43:13 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switching context to [/tmp/source-config] [2024-01-18T16:43:15.373Z] 2024-01-18 16:43:13 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:43:15.373Z] 2024-01-18 16:43:13 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:43:15.373Z] 2024-01-18 16:43:13 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:43:15.373Z] 2024-01-18 16:43:13 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:43:15.373Z] 2024-01-18 16:43:13 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:43:15.373Z] 2024-01-18 16:43:13 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:43:15.373Z] 2024-01-18 16:43:14 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:43:15.373Z] 2024-01-18 16:43:14 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:43:15.373Z] 2024-01-18 16:43:15 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:43:15.373Z] 2024-01-18 16:43:15 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:43:15.631Z] 2024-01-18 16:43:15 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:43:15.631Z] 2024-01-18 16:43:15 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:43:15.631Z] 2024-01-18 16:43:15 +0000:[INFO] [{VerifyRBACForInfraAdmin}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:43:15.631Z] 2024-01-18 16:43:15 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switched context to [/tmp/source-config] [2024-01-18T16:43:15.631Z] INFO[2024-01-18 16:43:15] Verifying : Description : Verifying restoration of backup infraadmin-nvtjmmgolieqpmr [2024-01-18T16:43:15.631Z] INFO[2024-01-18 16:43:15] Actual:true, Expected: true [2024-01-18T16:43:15.632Z] 2024-01-18 16:43:15 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validate creating scheduled backup with namespace label [2024-01-18T16:43:16.193Z] 2024-01-18 16:43:15 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-18T16:43:16.194Z] 2024-01-18 16:43:15 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Volume snapshot classes found in the cluster - [2024-01-18T16:43:16.194Z] 2024-01-18 16:43:15 +0000:[INFO] [{VerifyRBACForInfraAdmin}] ibmc-vpcblock-snapshot [2024-01-18T16:43:16.194Z] 2024-01-18 16:43:15 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T16:44:12.373Z] INFO[2024-01-18 16:44:04] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T16:44:12.373Z] INFO[2024-01-18 16:44:04] sched stats: tick interval: 4204 < 1.2s (100%) [2024-01-18T16:44:12.373Z] INFO[2024-01-18 16:44:04] sched stats: tick process durations: 4203 < 500ms (100%) [2024-01-18T16:44:17.617Z] 2024-01-18 16:44:16 +0000:[INFO] [{VerifyRBACForInfraAdmin}] first schedule backup for schedule name [tp-backup-cqoe] is [tp-backup-cqoe-480a0df-interval-2024-01-18-164316] [2024-01-18T16:44:17.617Z] 2024/01/18 16:44:16 DoRetryWithTimeout - Error: {backup status for [tp-backup-cqoe-480a0df-interval-2024-01-18-164316] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:44:49.644Z] 2024-01-18 16:44:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating backup [tp-backup-cqoe-480a0df-interval-2024-01-18-164316] in org [default] [2024-01-18T16:44:49.644Z] 2024-01-18 16:44:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-cqoe-480a0df-interval-2024-01-18-164316] [2024-01-18T16:44:49.644Z] 2024-01-18 16:44:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in backup [tp-backup-cqoe-480a0df-interval-2024-01-18-164316] [2024-01-18T16:44:49.644Z] 2024-01-18 16:44:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in backup [tp-backup-cqoe-480a0df-interval-2024-01-18-164316] [2024-01-18T16:44:49.644Z] 2024-01-18 16:44:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T16:44:49.644Z] 2024-01-18 16:44:46 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:44:49.644Z] 2024-01-18 16:44:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:44:49.644Z] 2024-01-18 16:44:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[mysql-data]] [2024-01-18T16:44:49.644Z] 2024-01-18 16:44:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T16:44:49.645Z] 2024-01-18 16:44:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating CR cleanup [2024-01-18T16:44:49.645Z] 2024-01-18 16:44:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:44:49.645Z] 2024-01-18 16:44:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[]] [2024-01-18T16:44:49.645Z] INFO[2024-01-18 16:44:47] Verifying : Description : Verification of creating first schedule backup tp-backup-cqoe with labels [0-0235ca01-2486-49e1-ba1d-6858b4c6d492=0235ca01-2486-49e1-ba1d-6858b4c6d492,1-0235ca01-2486-49e1-ba1d-6858b4c6d492=0235ca01-2486-49e1-ba1d-6858b4c6d492] [2024-01-18T16:44:49.645Z] INFO[2024-01-18 16:44:47] Actual:true, Expected: true [2024-01-18T16:44:49.645Z] INFO[2024-01-18 16:44:47] Verifying : Description : Suspending Backup Schedule [tp-backup-cqoe] for user [testuser1-1705595980] [2024-01-18T16:44:49.645Z] INFO[2024-01-18 16:44:47] Actual:true, Expected: true [2024-01-18T16:44:49.645Z] 2024-01-18 16:44:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup tp-backup-cqoe-480a0df-interval-2024-01-18-164316 [2024-01-18T16:44:49.645Z] 2024-01-18 16:44:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T16:44:49.645Z] 2024-01-18 16:44:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are 0-0235ca01-2486-49e1-ba1d-6858b4c6d492=0235ca01-2486-49e1-ba1d-6858b4c6d492,1-0235ca01-2486-49e1-ba1d-6858b4c6d492=0235ca01-2486-49e1-ba1d-6858b4c6d492 [2024-01-18T16:44:49.645Z] INFO[2024-01-18 16:44:47] Verifying : Description : Verifying if the labeled namespaces [[mysql-ibm-pxb-0-87886-01-18-16h38m08s]] are backed up and check for labels [0-0235ca01-2486-49e1-ba1d-6858b4c6d492=0235ca01-2486-49e1-ba1d-6858b4c6d492,1-0235ca01-2486-49e1-ba1d-6858b4c6d492=0235ca01-2486-49e1-ba1d-6858b4c6d492] applied to backups [tp-backup-cqoe-480a0df-interval-2024-01-18-164316] [2024-01-18T16:44:49.645Z] INFO[2024-01-18 16:44:47] Actual:true, Expected: true [2024-01-18T16:44:49.645Z] 2024-01-18 16:44:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validate restoring the scheduled backup with namespace label [2024-01-18T16:44:49.645Z] INFO[2024-01-18 16:44:47] Verifying : Description : Fetching namespaces [mysql-ibm-pxb-0-87886-01-18-16h38m08s] from backup tp-backup-cqoe-480a0df-interval-2024-01-18-164316 [2024-01-18T16:44:49.645Z] INFO[2024-01-18 16:44:47] Actual:true, Expected: true [2024-01-18T16:44:49.645Z] 2024/01/18 16:44:47 DoRetryWithTimeout - Error: {restore status for [customrestore-iarazclpet] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T16:45:21.673Z] 2024-01-18 16:45:17 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.CreateRestore:#936] - Restore [customrestore-iarazclpet] created successfully [2024-01-18T16:45:21.673Z] 2024-01-18 16:45:17 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switching cluster context to cluster [destination-cluster] [2024-01-18T16:45:21.673Z] 2024-01-18 16:45:17 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switching context to [/tmp/destination-config] [2024-01-18T16:45:21.673Z] 2024-01-18 16:45:17 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:45:21.673Z] 2024-01-18 16:45:17 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:45:21.673Z] 2024-01-18 16:45:17 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:45:21.673Z] 2024-01-18 16:45:17 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:45:21.673Z] 2024-01-18 16:45:18 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:45:21.674Z] 2024-01-18 16:45:18 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:45:21.674Z] 2024-01-18 16:45:18 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:45:21.674Z] 2024-01-18 16:45:19 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:45:21.674Z] 2024-01-18 16:45:19 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:45:21.674Z] 2024-01-18 16:45:19 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:45:21.674Z] 2024-01-18 16:45:19 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:45:21.674Z] 2024-01-18 16:45:19 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:45:21.674Z] 2024-01-18 16:45:19 +0000:[INFO] [{VerifyRBACForInfraAdmin}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:45:21.674Z] 2024-01-18 16:45:19 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switched context to [/tmp/destination-config] [2024-01-18T16:45:21.674Z] 2024-01-18 16:45:19 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-87886-01-18-16h38m08s] with namespace mapping [map[mysql-ibm-pxb-0-87886-01-18-16h38m08s:tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig]] and storage Class Mapping [map[]] [2024-01-18T16:45:21.674Z] 2024-01-18 16:45:19 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T16:45:21.674Z] 2024-01-18 16:45:19 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T16:45:21.674Z] 2024-01-18 16:45:19 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T16:45:21.674Z] 2024-01-18 16:45:19 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating restore [customrestore-iarazclpet] in org [default] [2024-01-18T16:45:21.674Z] 2024-01-18 16:45:19 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [customrestore-iarazclpet] [2024-01-18T16:45:21.674Z] 2024-01-18 16:45:19 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2110] - Checking status of restore [customrestore-iarazclpet] [2024-01-18T16:45:21.675Z] 2024-01-18 16:45:19 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating specs for the namespace (restoredAppContext) [tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig] in restore [customrestore-iarazclpet] [2024-01-18T16:45:21.675Z] 2024-01-18 16:45:19 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig]) was found in restore [customrestore-iarazclpet], as expected by presence in expectedRestoredAppContext [tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig] [2024-01-18T16:45:21.675Z] 2024-01-18 16:45:20 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig]) found in the restore [customrestore-iarazclpet] was also present on the cluster/namespace [tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig] [2024-01-18T16:45:21.675Z] 2024-01-18 16:45:20 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig]) was found in restore [customrestore-iarazclpet], as expected by presence in expectedRestoredAppContext [tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig] [2024-01-18T16:45:21.675Z] 2024-01-18 16:45:20 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig]) found in the restore [customrestore-iarazclpet] was also present on the cluster/namespace [tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig] [2024-01-18T16:45:21.675Z] 2024-01-18 16:45:20 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig]) was found in restore [customrestore-iarazclpet], as expected by presence in expectedRestoredAppContext [tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig] [2024-01-18T16:45:21.675Z] 2024-01-18 16:45:20 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig]) found in the restore [customrestore-iarazclpet] was also present on the cluster/namespace [tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig] [2024-01-18T16:45:21.676Z] 2024-01-18 16:45:20 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig]) was found in restore [customrestore-iarazclpet], as expected by presence in expectedRestoredAppContext [tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig] [2024-01-18T16:45:21.676Z] 2024-01-18 16:45:20 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig]) found in the restore [customrestore-iarazclpet] was also present on the cluster/namespace [tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig] [2024-01-18T16:45:21.676Z] 2024-01-18 16:45:20 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating Restored Volumes for the namespace (restoredAppContext) [tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig] in restore [customrestore-iarazclpet] [2024-01-18T16:45:21.676Z] 2024-01-18 16:45:20 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig] in restore [customrestore-iarazclpet] [2024-01-18T16:45:21.676Z] 2024-01-18 16:45:20 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:45:21.676Z] 2024-01-18 16:45:20 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:45:21.676Z] 2024-01-18 16:45:20 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig] are [[mysql-data]] [2024-01-18T16:45:21.676Z] 2024-01-18 16:45:20 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2240] - in restore [customrestore-iarazclpet], the status of the restored volume [pvc-72c57770-1e12-4da4-91b7-967895866f13] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-18T16:45:21.676Z] 2024-01-18 16:45:20 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validate applications in restored namespace [tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig] due to restore [customrestore-iarazclpet] [2024-01-18T16:45:21.676Z] 2024-01-18 16:45:20 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating mysql-ibm app [2024-01-18T16:45:21.676Z] 2024-01-18 16:45:20 +0000:[INFO] [{VerifyRBACForInfraAdmin}] wait for mysql-ibm app to start running [2024-01-18T16:45:21.676Z] 2024-01-18 16:45:20 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T16:45:21.676Z] 2024/01/18 16:45:20 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:45:21.676Z] pod name:mysql-7d766cd65c-s8hzd namespace:tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig running:false ready:false node:10.240.64.7 [2024-01-18T16:45:21.676Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:45:31.618Z] 2024/01/18 16:45:30 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:45:31.619Z] pod name:mysql-7d766cd65c-s8hzd namespace:tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig running:false ready:false node:10.240.64.7 [2024-01-18T16:45:31.619Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:45:41.560Z] 2024/01/18 16:45:40 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:45:41.560Z] pod name:mysql-7d766cd65c-s8hzd namespace:tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig running:false ready:false node:10.240.64.7 [2024-01-18T16:45:41.560Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:45:51.512Z] 2024/01/18 16:45:50 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:45:51.513Z] pod name:mysql-7d766cd65c-s8hzd namespace:tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig running:false ready:false node:10.240.64.7 [2024-01-18T16:45:51.513Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:46:01.458Z] 2024/01/18 16:46:00 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:46:01.458Z] pod name:mysql-7d766cd65c-s8hzd namespace:tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig running:true ready:false node:10.240.64.7 [2024-01-18T16:46:01.458Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:46:11.413Z] 2024/01/18 16:46:11 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:46:11.413Z] pod name:mysql-7d766cd65c-s8hzd namespace:tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig running:true ready:false node:10.240.64.7 [2024-01-18T16:46:11.413Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:46:21.361Z] 2024/01/18 16:46:21 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:46:21.361Z] pod name:mysql-7d766cd65c-s8hzd namespace:tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig running:true ready:false node:10.240.64.7 [2024-01-18T16:46:21.361Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:46:31.305Z] 2024/01/18 16:46:31 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:46:31.305Z] pod name:mysql-7d766cd65c-s8hzd namespace:tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig running:true ready:false node:10.240.64.7 [2024-01-18T16:46:31.305Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:46:43.471Z] 2024/01/18 16:46:41 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:46:43.471Z] pod name:mysql-7d766cd65c-s8hzd namespace:tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig running:true ready:false node:10.240.64.7 [2024-01-18T16:46:43.471Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:46:51.563Z] 2024/01/18 16:46:51 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:46:51.563Z] pod name:mysql-7d766cd65c-s8hzd namespace:tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig running:true ready:false node:10.240.64.7 [2024-01-18T16:46:51.563Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:47:01.500Z] 2024/01/18 16:47:01 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T16:47:01.500Z] pod name:mysql-7d766cd65c-s8hzd namespace:tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig running:true ready:false node:10.240.64.7 [2024-01-18T16:47:01.500Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T16:47:11.442Z] 2024-01-18 16:47:11 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T16:47:11.698Z] 2024-01-18 16:47:11 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T16:47:11.698Z] 2024-01-18 16:47:11 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating CR cleanup [2024-01-18T16:47:11.954Z] 2024-01-18 16:47:11 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:47:11.954Z] 2024-01-18 16:47:11 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig] are [[]] [2024-01-18T16:47:11.954Z] 2024-01-18 16:47:11 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switching cluster context back to cluster path [/tmp/source-config] [2024-01-18T16:47:11.954Z] 2024-01-18 16:47:11 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switching context to [/tmp/source-config] [2024-01-18T16:47:12.211Z] 2024-01-18 16:47:11 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:47:12.211Z] 2024-01-18 16:47:12 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:47:12.211Z] 2024-01-18 16:47:12 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T16:47:12.211Z] 2024-01-18 16:47:12 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:47:12.467Z] 2024-01-18 16:47:12 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:47:12.467Z] 2024-01-18 16:47:12 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T16:47:12.725Z] 2024-01-18 16:47:12 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:47:13.288Z] 2024-01-18 16:47:12 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:47:13.544Z] 2024-01-18 16:47:13 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T16:47:13.799Z] 2024-01-18 16:47:13 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T16:47:13.799Z] 2024-01-18 16:47:13 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T16:47:13.799Z] 2024-01-18 16:47:13 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T16:47:13.799Z] 2024-01-18 16:47:13 +0000:[INFO] [{VerifyRBACForInfraAdmin}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T16:47:13.799Z] 2024-01-18 16:47:13 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switched context to [/tmp/source-config] [2024-01-18T16:47:13.799Z] INFO[2024-01-18 16:47:13] Verifying : Description : Verifying multiple backup restore [customrestore-iarazclpet] in custom namespace [map[mysql-ibm-pxb-0-87886-01-18-16h38m08s:tp-backup-cqoe-480a0df-interval-2024-01-18-164316-hig]] [2024-01-18T16:47:13.799Z] INFO[2024-01-18 16:47:13] Actual:true, Expected: true [2024-01-18T16:47:13.799Z] 2024-01-18 16:47:13 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validate deleting namespace labelled backups and restores from the infra admin user testuser1-1705595980 user context [2024-01-18T16:47:14.362Z] 2024/01/18 16:47:14 DoRetryWithTimeout - Error: {backupObject [infraadmin-nvtjmmgolieqpmr] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-18T16:47:14.362Z] 2024/01/18 16:47:14 DoRetryWithTimeout - Error: {backupObject [tp-backup-cqoe-480a0df-interval-2024-01-18-164316] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-18T16:47:46.470Z] INFO[2024-01-18 16:47:44] Verifying : Description : Getting the backup driver [2024-01-18T16:47:46.470Z] INFO[2024-01-18 16:47:44] Actual:true, Expected: true [2024-01-18T16:47:46.470Z] INFO[2024-01-18 16:47:44] Verifying : Description : Verifying the deletion of the restore named [tp-restore-infraadmin-nvtjmmgolieqpmr] [2024-01-18T16:47:46.470Z] INFO[2024-01-18 16:47:44] Actual:true, Expected: true [2024-01-18T16:47:46.470Z] INFO[2024-01-18 16:47:44] Verifying : Description : Getting the backup driver [2024-01-18T16:47:46.470Z] INFO[2024-01-18 16:47:44] Actual:true, Expected: true [2024-01-18T16:47:46.470Z] INFO[2024-01-18 16:47:44] Verifying : Description : Verifying the deletion of the restore named [customrestore-iarazclpet] [2024-01-18T16:47:46.470Z] INFO[2024-01-18 16:47:44] Actual:true, Expected: true [2024-01-18T16:47:46.470Z] 2024-01-18 16:47:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Delete Infra Admin testuser1-1705595980 backup schedule [2024-01-18T16:47:46.470Z] 2024/01/18 16:47:44 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-cqoe] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T16:47:56.419Z] 2024/01/18 16:47:54 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-cqoe] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T16:48:06.371Z] 2024/01/18 16:48:04 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-cqoe] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T16:48:16.321Z] 2024/01/18 16:48:14 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-cqoe] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T16:48:26.263Z] 2024/01/18 16:48:24 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-cqoe] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T16:48:36.205Z] INFO[2024-01-18 16:48:34] Verifying : Description : Deleting Backup Schedule [tp-backup-cqoe] for user [testuser1-1705595980] [2024-01-18T16:48:36.205Z] INFO[2024-01-18 16:48:34] Actual:true, Expected: true [2024-01-18T16:48:36.205Z] 2024-01-18 16:48:34 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Verify px-admin group user can list RBAC resources created by Infra-Admin User [2024-01-18T16:48:36.206Z] 2024-01-18 16:48:35 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.getAllCloudCredentials:#1624] - The cloud credentials and their UID are map[003d42b9-10c5-415b-860a-ef2401c4efe0:ibm-source-config-cloud-cred-xjkmp 05c1ce23-8059-4e66-b024-3fbbbc031f4b:ibm-destination-config-cloud-cred-ughzr 0c9455a6-d605-405b-845e-993a972b38c3:ibm-source-config-cloud-cred-ywkqe 400c89ab-c1e4-4fec-8856-9968bf6622b3:ibm-destination-config-cloud-cred-wlqhq 4526f4e4-5f42-4eb4-836b-4e562b4515b8:ibm-destination-config-cloud-cred-ggszx 4e20d0f0-1904-466a-ad3a-1e19d7169cae:ibm-destination-config-cloud-cred-buvoh 4f3c25d0-c35c-4838-8e74-c0eb5f9c2cf6:ibm-source-config-cloud-cred-jiipm 4f8fa3dc-340f-4d11-be2f-31c9cacee9aa:ibm-source-config-cloud-cred-eymzd 59e03021-b557-43b0-9579-f2246ae847e7:ibm-source-config-cloud-cred-htmnt 5c6b1236-8d6c-4087-8034-27705179aa52:ibm-source-config-cloud-cred-gbiga 611175c8-09e6-4f7c-a7ba-953634c139b0:cred-aws-kudis 7e619e3a-1834-4563-b6b5-5c6356601127:ibm-destination-config-cloud-cred-xejzw 7e6ad931-e339-4f31-9674-a563626aad91:ibm-source-config-cloud-cred-rcmdm 87a15256-4a8a-4244-8df9-2f79276576b6:cred-aws-1705577909 8c9a2571-b3b0-48bf-b1cf-eeef4421a95e:ibm-source-config-cloud-cred-jpmbk 91cb8736-840f-48d5-9a64-e36e454643a2:ibm-destination-config-cloud-cred-btcfu 9312af98-b05c-4c55-8f00-164bdb7b60f9:ibm-destination-config-cloud-cred-iuupt 9841bce2-74d5-4fc2-831e-caf981277058:ibm-destination-config-cloud-cred-zzzrt a0238056-dd69-42f2-ae5a-2f9e6cbbda5a:ibm-source-config-cloud-cred-kvjhs a12b5b6d-216c-433c-a2a3-62d797b4b228:ibm-source-config-cloud-cred-jyrnu a1e6b817-24c9-4a38-9596-199a019ea67e:ibm-destination-config-cloud-cred-zqepo a2e242fb-97cc-40d5-92e8-ab560f98bb36:ibm-source-config-cloud-cred-tgism ab4a9ad9-5834-4329-a070-07e162922643:ibm-destination-config-cloud-cred-oxkoo b446c8d6-6de7-4f2f-93fa-42dd0df7fd49:ibm-destination-config-cloud-cred-teqcf b5dc5235-d58e-4094-96e1-db8248a9f4e2:autogenerated-cred-testuser1-1705570507-vjsmz c51bf4bb-493a-4784-af65-fdbc0167456c:ibm-source-config-cloud-cred-bfxwa cd18a11c-b21b-4afb-b6fe-5651dc053d42:ibm-destination-config-cloud-cred-vhryz d3b1b808-0bab-4c91-a304-f60c8c64f1e0:ibm-destination-config-cloud-cred-eqjzn d64d258b-5f8c-4690-ad2a-05909c63cfa1:ibm-source-config-cloud-cred-fkfzj e754dfb8-04d0-4031-8571-aa8c1812669d:ibm-source-config-cloud-cred-jdydf e8391d79-31b8-49e8-8505-5fb84f53e3c4:autogenerated-cred-lvxlx ec6d2d18-1421-41d4-bad3-ceb41fd17fe0:ibm-destination-config-cloud-cred-jinlc fe53b7a2-3045-4494-90af-c5a010a82974:ibm-destination-config-cloud-cred-flqaz] [2024-01-18T16:48:36.206Z] 2024-01-18 16:48:35 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.getAllBackupLocations:#1603] - The backup locations and their UID are map[34518b95-86fb-40f8-a45d-df227357b0e1:autogenerated-backup-location-mpzou 4418c3bb-e07a-4cdc-83e7-b25be38d1e4c:aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 cd190df6-e916-41da-b969-ae70c2b830cb:autogenerated-backup-location-omphu] [2024-01-18T16:48:36.206Z] 2024-01-18 16:48:35 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Create Users with Different types of roles [2024-01-18T16:48:36.206Z] 2024-01-18 16:48:35 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Creating 1 users [2024-01-18T16:48:36.206Z] 2024-01-18 16:48:35 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.AddUser:#728] - Response for user [testuser1-1705596515] creation - [2024-01-18T16:48:36.206Z] 2024-01-18 16:48:35 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.AddUser:#733] - User [testuser1-1705596515] created [2024-01-18T16:48:36.206Z] INFO[2024-01-18 16:48:35] Verifying : Description : Creating user - testuser1-1705596515 [2024-01-18T16:48:36.206Z] INFO[2024-01-18 16:48:35] Actual:true, Expected: true [2024-01-18T16:48:36.206Z] 2024-01-18 16:48:35 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705596515 - ed18de04-e42b-48a4-9469-b00c20c8ddec [2024-01-18T16:48:37.574Z] 2024-01-18 16:48:37 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Creating 1 users [2024-01-18T16:48:38.543Z] 2024-01-18 16:48:38 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.AddUser:#728] - Response for user [testuser1-1705596517] creation - [2024-01-18T16:48:38.543Z] 2024-01-18 16:48:38 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.AddUser:#733] - User [testuser1-1705596517] created [2024-01-18T16:48:38.543Z] INFO[2024-01-18 16:48:38] Verifying : Description : Creating user - testuser1-1705596517 [2024-01-18T16:48:38.543Z] INFO[2024-01-18 16:48:38] Actual:true, Expected: true [2024-01-18T16:48:39.104Z] 2024-01-18 16:48:38 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705596517 - 88de5c6a-97e3-49f5-8bb8-8933dfbb79af [2024-01-18T16:48:41.002Z] 2024-01-18 16:48:40 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Creating 1 users [2024-01-18T16:48:41.563Z] 2024-01-18 16:48:41 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.AddUser:#728] - Response for user [testuser1-1705596520] creation - [2024-01-18T16:48:41.563Z] 2024-01-18 16:48:41 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.AddUser:#733] - User [testuser1-1705596520] created [2024-01-18T16:48:41.563Z] INFO[2024-01-18 16:48:41] Verifying : Description : Creating user - testuser1-1705596520 [2024-01-18T16:48:41.563Z] INFO[2024-01-18 16:48:41] Actual:true, Expected: true [2024-01-18T16:48:42.493Z] 2024-01-18 16:48:42 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705596520 - 0439fb16-36f9-47d1-88fe-1c5d55b096f3 [2024-01-18T16:48:43.862Z] 2024-01-18 16:48:43 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Verify Infra-Admin User has permission to share RBAC resources with non-admin users [2024-01-18T16:48:44.424Z] 2024-01-18 16:48:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.glob..func43.2.17:#336] - Update CloudAccount - autogenerated-cred-lvxlx ownership for users - [[testuser1-1705596515 testuser1-1705596517 testuser1-1705596520]] [2024-01-18T16:48:44.997Z] 2024-01-18 16:48:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705596515 - ed18de04-e42b-48a4-9469-b00c20c8ddec [2024-01-18T16:48:45.925Z] 2024-01-18 16:48:45 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705596517 - 88de5c6a-97e3-49f5-8bb8-8933dfbb79af [2024-01-18T16:48:46.851Z] 2024-01-18 16:48:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705596520 - 0439fb16-36f9-47d1-88fe-1c5d55b096f3 [2024-01-18T16:48:46.851Z] INFO[2024-01-18 16:48:46] Verifying : Description : Verifying updation of owbership for CloudCredential- autogenerated-cred-lvxlx [2024-01-18T16:48:46.851Z] INFO[2024-01-18 16:48:46] Actual:true, Expected: true [2024-01-18T16:48:46.851Z] 2024-01-18 16:48:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Update BackupLocation - autogenerated-backup-location-mpzou ownership for users - [[testuser1-1705596515 testuser1-1705596517 testuser1-1705596520]] [2024-01-18T16:48:47.412Z] 2024-01-18 16:48:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705596515 - ed18de04-e42b-48a4-9469-b00c20c8ddec [2024-01-18T16:48:48.340Z] 2024-01-18 16:48:48 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705596517 - 88de5c6a-97e3-49f5-8bb8-8933dfbb79af [2024-01-18T16:48:49.313Z] 2024-01-18 16:48:48 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705596520 - 0439fb16-36f9-47d1-88fe-1c5d55b096f3 [2024-01-18T16:48:49.313Z] INFO[2024-01-18 16:48:49] Verifying : Description : Verifying updation of owbership for backuplocation - autogenerated-backup-location-mpzou [2024-01-18T16:48:49.313Z] INFO[2024-01-18 16:48:49] Actual:true, Expected: true [2024-01-18T16:48:49.313Z] 2024-01-18 16:48:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Update SchedulePolicy - periodic-oyizj ownership for users - [[testuser1-1705596515 testuser1-1705596517 testuser1-1705596520]] [2024-01-18T16:48:49.877Z] 2024-01-18 16:48:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705596515 - ed18de04-e42b-48a4-9469-b00c20c8ddec [2024-01-18T16:48:50.805Z] 2024-01-18 16:48:50 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705596517 - 88de5c6a-97e3-49f5-8bb8-8933dfbb79af [2024-01-18T16:48:51.369Z] 2024-01-18 16:48:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705596520 - 0439fb16-36f9-47d1-88fe-1c5d55b096f3 [2024-01-18T16:48:51.627Z] INFO[2024-01-18 16:48:51] Verifying : Description : Verifying updation of ownership for schedulepolicy - periodic-oyizj [2024-01-18T16:48:51.627Z] INFO[2024-01-18 16:48:51] Actual:true, Expected: true [2024-01-18T16:48:51.627Z] 2024-01-18 16:48:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Update Application Rules ownership for users - [[testuser1-1705596515 testuser1-1705596517 testuser1-1705596520]] [2024-01-18T16:48:52.190Z] 2024-01-18 16:48:52 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705596515 - ed18de04-e42b-48a4-9469-b00c20c8ddec [2024-01-18T16:48:53.121Z] 2024-01-18 16:48:52 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705596517 - 88de5c6a-97e3-49f5-8bb8-8933dfbb79af [2024-01-18T16:48:54.050Z] 2024-01-18 16:48:53 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705596520 - 0439fb16-36f9-47d1-88fe-1c5d55b096f3 [2024-01-18T16:48:54.050Z] INFO[2024-01-18 16:48:53] Verifying : Description : Verifying updation of ownership for pre-rule of application [2024-01-18T16:48:54.050Z] INFO[2024-01-18 16:48:53] Actual:true, Expected: true [2024-01-18T16:48:54.613Z] 2024-01-18 16:48:54 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705596515 - ed18de04-e42b-48a4-9469-b00c20c8ddec [2024-01-18T16:48:55.544Z] 2024-01-18 16:48:55 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705596517 - 88de5c6a-97e3-49f5-8bb8-8933dfbb79af [2024-01-18T16:48:56.472Z] 2024-01-18 16:48:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705596520 - 0439fb16-36f9-47d1-88fe-1c5d55b096f3 [2024-01-18T16:48:56.472Z] INFO[2024-01-18 16:48:56] Verifying : Description : Verifying updation of ownership for post-rule of application [2024-01-18T16:48:56.472Z] INFO[2024-01-18 16:48:56] Actual:true, Expected: true [2024-01-18T16:48:56.472Z] 2024-01-18 16:48:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Creating source and destination cluster from the user testuser1-1705596515 [2024-01-18T16:48:56.728Z] INFO[2024-01-18 16:48:56] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T16:48:56.728Z] INFO[2024-01-18 16:48:56] Actual:true, Expected: true [2024-01-18T16:48:56.728Z] 2024-01-18 16:48:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T16:48:56.983Z] 2024-01-18 16:48:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T16:48:56.983Z] 2024-01-18 16:48:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T16:48:56.983Z] 2024-01-18 16:48:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T16:48:56.983Z] 2024-01-18 16:48:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Create cluster [source-cluster] in org [default] [2024-01-18T16:48:56.983Z] 2024-01-18 16:48:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T16:48:56.983Z] 2024-01-18 16:48:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T16:48:56.983Z] 2024-01-18 16:48:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-18T16:48:56.984Z] 2024-01-18 16:48:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T16:48:56.984Z] 2024-01-18 16:48:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T16:48:56.984Z] 2024-01-18 16:48:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-vrmuf] for IBM [2024-01-18T16:48:56.984Z] 2024-01-18 16:48:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-vrmuf] for org [default] with [ibm] as provider [2024-01-18T16:48:56.984Z] 2024-01-18 16:48:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:48:56.984Z] 2024-01-18 16:48:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:48:56.984Z] 2024-01-18 16:48:56 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-vrmuf] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user px-backup-app.admin]] [2024-01-18T16:48:56.984Z] 2024-01-18 16:48:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user px-backup-app.admin] [2024-01-18T16:48:58.348Z] 2024-01-18 16:48:58 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-vrmuf from admin context and sharing with all the users [2024-01-18T16:48:58.348Z] 2024-01-18 16:48:58 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-vrmuf] for org [default] with [ibm] as provider [2024-01-18T16:48:58.348Z] 2024-01-18 16:48:58 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:48:58.349Z] 2024-01-18 16:48:58 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:48:58.349Z] 2024-01-18 16:48:58 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-vrmuf] is present [2024-01-18T16:48:58.605Z] 2024-01-18 16:48:58 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Create cluster [source-cluster] in org [default] [2024-01-18T16:48:59.167Z] 2024-01-18 16:48:58 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-eoxdv] for IBM [2024-01-18T16:48:59.167Z] 2024-01-18 16:48:58 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-eoxdv] for org [default] with [ibm] as provider [2024-01-18T16:48:59.167Z] 2024-01-18 16:48:58 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:48:59.167Z] 2024-01-18 16:48:58 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:48:59.167Z] 2024-01-18 16:48:58 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-eoxdv] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user px-backup-app.admin]] [2024-01-18T16:48:59.167Z] 2024-01-18 16:48:58 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user px-backup-app.admin] [2024-01-18T16:48:59.768Z] 2024-01-18 16:48:59 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-eoxdv from admin context and sharing with all the users [2024-01-18T16:48:59.768Z] 2024-01-18 16:48:59 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-eoxdv] for org [default] with [ibm] as provider [2024-01-18T16:48:59.768Z] 2024-01-18 16:48:59 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:48:59.768Z] 2024-01-18 16:48:59 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:48:59.768Z] 2024-01-18 16:48:59 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-eoxdv] is present [2024-01-18T16:49:00.024Z] 2024-01-18 16:48:59 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-18T16:49:00.588Z] INFO[2024-01-18 16:49:00] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-18T16:49:00.588Z] INFO[2024-01-18 16:49:00] Actual:Online, Expected: Online [2024-01-18T16:49:00.588Z] INFO[2024-01-18 16:49:00] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-18T16:49:00.588Z] INFO[2024-01-18 16:49:00] Actual:true, Expected: true [2024-01-18T16:49:00.588Z] INFO[2024-01-18 16:49:00] Verifying : Description : Fetching [destination-cluster] cluster uid [2024-01-18T16:49:00.588Z] INFO[2024-01-18 16:49:00] Actual:true, Expected: true [2024-01-18T16:49:00.588Z] 2024-01-18 16:49:00 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Creating source and destination cluster from the user testuser1-1705596517 [2024-01-18T16:49:00.588Z] INFO[2024-01-18 16:49:00] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T16:49:00.588Z] INFO[2024-01-18 16:49:00] Actual:true, Expected: true [2024-01-18T16:49:00.588Z] 2024-01-18 16:49:00 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T16:49:00.589Z] 2024-01-18 16:49:00 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T16:49:00.589Z] 2024-01-18 16:49:00 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T16:49:00.589Z] 2024-01-18 16:49:00 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T16:49:00.589Z] 2024-01-18 16:49:00 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Create cluster [source-cluster] in org [default] [2024-01-18T16:49:00.589Z] 2024-01-18 16:49:00 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T16:49:00.589Z] 2024-01-18 16:49:00 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T16:49:00.589Z] 2024-01-18 16:49:00 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-18T16:49:00.589Z] 2024-01-18 16:49:00 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T16:49:00.589Z] 2024-01-18 16:49:00 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T16:49:00.589Z] 2024-01-18 16:49:00 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-vrzih] for IBM [2024-01-18T16:49:00.589Z] 2024-01-18 16:49:00 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-vrzih] for org [default] with [ibm] as provider [2024-01-18T16:49:00.589Z] 2024-01-18 16:49:00 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:49:00.589Z] 2024-01-18 16:49:00 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:49:00.845Z] 2024-01-18 16:49:00 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-vrzih] is present [2024-01-18T16:49:00.845Z] 2024-01-18 16:49:00 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Create cluster [source-cluster] in org [default] [2024-01-18T16:49:01.407Z] 2024-01-18 16:49:01 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-jeepg] for IBM [2024-01-18T16:49:01.407Z] 2024-01-18 16:49:01 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-jeepg] for org [default] with [ibm] as provider [2024-01-18T16:49:01.407Z] 2024-01-18 16:49:01 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:49:01.407Z] 2024-01-18 16:49:01 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:49:01.407Z] 2024-01-18 16:49:01 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-jeepg] is present [2024-01-18T16:49:01.407Z] 2024-01-18 16:49:01 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-18T16:49:01.968Z] INFO[2024-01-18 16:49:01] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-18T16:49:01.968Z] INFO[2024-01-18 16:49:01] Actual:Online, Expected: Online [2024-01-18T16:49:01.968Z] INFO[2024-01-18 16:49:01] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-18T16:49:01.968Z] INFO[2024-01-18 16:49:01] Actual:true, Expected: true [2024-01-18T16:49:01.968Z] INFO[2024-01-18 16:49:01] Verifying : Description : Fetching [destination-cluster] cluster uid [2024-01-18T16:49:01.969Z] INFO[2024-01-18 16:49:01] Actual:true, Expected: true [2024-01-18T16:49:01.969Z] 2024-01-18 16:49:01 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Creating source and destination cluster from the user testuser1-1705596520 [2024-01-18T16:49:02.224Z] INFO[2024-01-18 16:49:01] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T16:49:02.224Z] INFO[2024-01-18 16:49:01] Actual:true, Expected: true [2024-01-18T16:49:02.224Z] 2024-01-18 16:49:01 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T16:49:02.224Z] 2024-01-18 16:49:01 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T16:49:02.224Z] 2024-01-18 16:49:01 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T16:49:02.224Z] 2024-01-18 16:49:01 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T16:49:02.224Z] 2024-01-18 16:49:01 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Create cluster [source-cluster] in org [default] [2024-01-18T16:49:02.224Z] 2024-01-18 16:49:01 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T16:49:02.224Z] 2024-01-18 16:49:01 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T16:49:02.224Z] 2024-01-18 16:49:01 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-18T16:49:02.224Z] 2024-01-18 16:49:01 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T16:49:02.225Z] 2024-01-18 16:49:01 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T16:49:02.225Z] 2024-01-18 16:49:01 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-xoaah] for IBM [2024-01-18T16:49:02.225Z] 2024-01-18 16:49:01 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-xoaah] for org [default] with [ibm] as provider [2024-01-18T16:49:02.225Z] 2024-01-18 16:49:01 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:49:02.225Z] 2024-01-18 16:49:01 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:49:02.225Z] 2024-01-18 16:49:01 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-xoaah] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user]] [2024-01-18T16:49:02.225Z] 2024-01-18 16:49:01 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user] [2024-01-18T16:49:02.481Z] 2024-01-18 16:49:02 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-xoaah from admin context and sharing with all the users [2024-01-18T16:49:02.481Z] 2024-01-18 16:49:02 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-xoaah] for org [default] with [ibm] as provider [2024-01-18T16:49:02.481Z] 2024-01-18 16:49:02 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:49:02.481Z] 2024-01-18 16:49:02 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:49:02.481Z] 2024-01-18 16:49:02 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-xoaah] is present [2024-01-18T16:49:02.736Z] 2024-01-18 16:49:02 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Create cluster [source-cluster] in org [default] [2024-01-18T16:49:03.298Z] 2024-01-18 16:49:03 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-dksbt] for IBM [2024-01-18T16:49:03.298Z] 2024-01-18 16:49:03 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-dksbt] for org [default] with [ibm] as provider [2024-01-18T16:49:03.298Z] 2024-01-18 16:49:03 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:49:03.298Z] 2024-01-18 16:49:03 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:49:03.298Z] 2024-01-18 16:49:03 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-dksbt] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user]] [2024-01-18T16:49:03.298Z] 2024-01-18 16:49:03 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user] [2024-01-18T16:49:03.554Z] 2024-01-18 16:49:03 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-dksbt from admin context and sharing with all the users [2024-01-18T16:49:03.554Z] 2024-01-18 16:49:03 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-dksbt] for org [default] with [ibm] as provider [2024-01-18T16:49:03.554Z] 2024-01-18 16:49:03 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:49:03.555Z] 2024-01-18 16:49:03 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:49:03.836Z] 2024-01-18 16:49:03 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-dksbt] is present [2024-01-18T16:49:03.836Z] 2024-01-18 16:49:03 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-18T16:49:04.400Z] INFO[2024-01-18 16:49:04] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-18T16:49:04.400Z] INFO[2024-01-18 16:49:04] Actual:Online, Expected: Online [2024-01-18T16:49:04.400Z] INFO[2024-01-18 16:49:04] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-18T16:49:04.400Z] INFO[2024-01-18 16:49:04] Actual:true, Expected: true [2024-01-18T16:49:04.400Z] INFO[2024-01-18 16:49:04] Verifying : Description : Fetching [destination-cluster] cluster uid [2024-01-18T16:49:04.400Z] INFO[2024-01-18 16:49:04] Actual:true, Expected: true [2024-01-18T16:49:04.400Z] 2024-01-18 16:49:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Taking backup of applications from the user testuser1-1705596520 [2024-01-18T16:49:04.400Z] 2024-01-18 16:49:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Taking backup of applications from the user testuser1-1705596517 [2024-01-18T16:49:04.400Z] 2024-01-18 16:49:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Taking backup of applications from the user testuser1-1705596515 [2024-01-18T16:49:04.657Z] 2024-01-18 16:49:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T16:49:04.657Z] 2024-01-18 16:49:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Volume snapshot classes found in the cluster - [2024-01-18T16:49:04.657Z] 2024-01-18 16:49:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] ibmc-vpcblock-snapshot [2024-01-18T16:49:04.657Z] 2024-01-18 16:49:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T16:49:04.657Z] 2024-01-18 16:49:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T16:49:04.657Z] 2024-01-18 16:49:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Volume snapshot classes found in the cluster - [2024-01-18T16:49:04.657Z] 2024-01-18 16:49:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] ibmc-vpcblock-snapshot [2024-01-18T16:49:04.657Z] 2024-01-18 16:49:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T16:49:04.914Z] 2024-01-18 16:49:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T16:49:04.914Z] 2024-01-18 16:49:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Volume snapshot classes found in the cluster - [2024-01-18T16:49:04.914Z] 2024-01-18 16:49:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] ibmc-vpcblock-snapshot [2024-01-18T16:49:04.914Z] 2024-01-18 16:49:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T16:49:05.477Z] 2024/01/18 16:49:05 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705596515-with-rules-gidt] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T16:49:06.404Z] 2024/01/18 16:49:06 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705596517-with-rules-trui] expected was [[Success]] but got [InProgress] because of [Pre-Exec rules are being executed]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:49:06.661Z] 2024/01/18 16:49:06 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705596520-with-rules-kghk] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T16:49:38.733Z] 2024/01/18 16:49:35 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705596515-with-rules-gidt] expected was [[Success]] but got [InProgress] because of [Pre-Exec rules are being executed]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:49:38.733Z] 2024/01/18 16:49:36 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705596517-with-rules-trui] expected was [[Success]] but got [InProgress] because of [Pre-Exec rules are being executed]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:49:38.733Z] 2024/01/18 16:49:36 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705596520-with-rules-kghk] expected was [[Success]] but got [InProgress] because of [Pre-Exec rules are being executed]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:50:10.783Z] 2024/01/18 16:50:05 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705596515-with-rules-gidt] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:50:10.783Z] 2024/01/18 16:50:06 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705596517-with-rules-trui] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:50:10.783Z] 2024/01/18 16:50:06 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705596520-with-rules-kghk] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:50:37.286Z] 2024/01/18 16:50:35 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705596515-with-rules-gidt] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:50:37.286Z] 2024/01/18 16:50:36 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705596517-with-rules-trui] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:50:37.286Z] 2024/01/18 16:50:36 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705596520-with-rules-kghk] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:51:09.356Z] 2024/01/18 16:51:05 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705596515-with-rules-gidt] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:51:09.356Z] 2024/01/18 16:51:06 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705596517-with-rules-trui] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:51:09.356Z] 2024/01/18 16:51:06 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705596520-with-rules-kghk] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:51:35.859Z] 2024/01/18 16:51:35 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705596515-with-rules-gidt] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:51:36.787Z] 2024/01/18 16:51:36 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705596517-with-rules-trui] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:51:36.787Z] 2024/01/18 16:51:36 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705596520-with-rules-kghk] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:52:08.845Z] 2024-01-18 16:52:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.CreateBackup:#313] - Backup [tp-backup-manual-single-ns-testuser1-1705596515-with-rules-gidt] created successfully [2024-01-18T16:52:08.845Z] 2024-01-18 16:52:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating backup [tp-backup-manual-single-ns-testuser1-1705596515-with-rules-gidt] in org [default] [2024-01-18T16:52:08.845Z] 2024-01-18 16:52:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-manual-single-ns-testuser1-1705596515-with-rules-gidt] [2024-01-18T16:52:08.845Z] 2024-01-18 16:52:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in backup [tp-backup-manual-single-ns-testuser1-1705596515-with-rules-gidt] [2024-01-18T16:52:08.845Z] 2024-01-18 16:52:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in backup [tp-backup-manual-single-ns-testuser1-1705596515-with-rules-gidt] [2024-01-18T16:52:08.845Z] 2024-01-18 16:52:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T16:52:08.845Z] 2024-01-18 16:52:05 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:52:08.845Z] 2024-01-18 16:52:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:52:08.845Z] 2024-01-18 16:52:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[mysql-data]] [2024-01-18T16:52:08.845Z] 2024-01-18 16:52:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T16:52:08.845Z] 2024-01-18 16:52:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating CR cleanup [2024-01-18T16:52:08.845Z] 2024-01-18 16:52:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:52:08.845Z] 2024-01-18 16:52:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[]] [2024-01-18T16:52:08.846Z] INFO[2024-01-18 16:52:05] Verifying : Description : Verifying creation and validation of backup [tp-backup-manual-single-ns-testuser1-1705596515-with-rules-gidt] of namespace (scheduled Context) [[mysql-ibm-pxb-0-87886-01-18-16h38m08s]] [2024-01-18T16:52:08.846Z] INFO[2024-01-18 16:52:05] Actual:true, Expected: true [2024-01-18T16:52:08.846Z] 2024-01-18 16:52:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Taking schedule backup of applications from the user testuser1-1705596515 [2024-01-18T16:52:08.846Z] 2024-01-18 16:52:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-18T16:52:08.846Z] 2024-01-18 16:52:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Volume snapshot classes found in the cluster - [2024-01-18T16:52:08.846Z] 2024-01-18 16:52:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] ibmc-vpcblock-snapshot [2024-01-18T16:52:08.846Z] 2024-01-18 16:52:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T16:52:08.846Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.CreateBackup:#313] - Backup [tp-backup-manual-single-ns-testuser1-1705596517-with-rules-trui] created successfully [2024-01-18T16:52:08.846Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating backup [tp-backup-manual-single-ns-testuser1-1705596517-with-rules-trui] in org [default] [2024-01-18T16:52:08.846Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-manual-single-ns-testuser1-1705596517-with-rules-trui] [2024-01-18T16:52:08.846Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in backup [tp-backup-manual-single-ns-testuser1-1705596517-with-rules-trui] [2024-01-18T16:52:08.846Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in backup [tp-backup-manual-single-ns-testuser1-1705596517-with-rules-trui] [2024-01-18T16:52:08.846Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T16:52:08.846Z] 2024-01-18 16:52:06 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:52:08.846Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:52:08.846Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[mysql-data]] [2024-01-18T16:52:08.846Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.CreateBackup:#313] - Backup [tp-backup-manual-single-ns-testuser1-1705596520-with-rules-kghk] created successfully [2024-01-18T16:52:08.846Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating backup [tp-backup-manual-single-ns-testuser1-1705596520-with-rules-kghk] in org [default] [2024-01-18T16:52:08.846Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-manual-single-ns-testuser1-1705596520-with-rules-kghk] [2024-01-18T16:52:08.846Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T16:52:08.846Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating CR cleanup [2024-01-18T16:52:08.846Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in backup [tp-backup-manual-single-ns-testuser1-1705596520-with-rules-kghk] [2024-01-18T16:52:08.847Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in backup [tp-backup-manual-single-ns-testuser1-1705596520-with-rules-kghk] [2024-01-18T16:52:08.847Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T16:52:08.847Z] 2024-01-18 16:52:06 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:52:08.847Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:52:08.847Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[mysql-data]] [2024-01-18T16:52:08.847Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T16:52:08.847Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating CR cleanup [2024-01-18T16:52:08.847Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:52:08.847Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[backup-schedule-irumj-a613710-interval-2024-01-18-165206]] [2024-01-18T16:52:08.847Z] INFO[2024-01-18 16:52:06] Verifying : Description : Verifying creation and validation of backup [tp-backup-manual-single-ns-testuser1-1705596517-with-rules-trui] of namespace (scheduled Context) [[mysql-ibm-pxb-0-87886-01-18-16h38m08s]] [2024-01-18T16:52:08.847Z] INFO[2024-01-18 16:52:06] Actual:true, Expected: true [2024-01-18T16:52:08.847Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Taking schedule backup of applications from the user testuser1-1705596517 [2024-01-18T16:52:08.847Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:52:08.847Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[backup-schedule-irumj-a613710-interval-2024-01-18-165206]] [2024-01-18T16:52:08.847Z] INFO[2024-01-18 16:52:06] Verifying : Description : Verifying creation and validation of backup [tp-backup-manual-single-ns-testuser1-1705596520-with-rules-kghk] of namespace (scheduled Context) [[mysql-ibm-pxb-0-87886-01-18-16h38m08s]] [2024-01-18T16:52:08.847Z] INFO[2024-01-18 16:52:06] Actual:true, Expected: true [2024-01-18T16:52:08.847Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Taking schedule backup of applications from the user testuser1-1705596520 [2024-01-18T16:52:08.847Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-18T16:52:08.847Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Volume snapshot classes found in the cluster - [2024-01-18T16:52:08.847Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] ibmc-vpcblock-snapshot [2024-01-18T16:52:08.847Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T16:52:08.847Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-18T16:52:08.847Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Volume snapshot classes found in the cluster - [2024-01-18T16:52:08.847Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] ibmc-vpcblock-snapshot [2024-01-18T16:52:08.847Z] 2024-01-18 16:52:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T16:53:16.481Z] 2024-01-18 16:53:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] first schedule backup for schedule name [backup-schedule-irumj] is [backup-schedule-irumj-a613710-interval-2024-01-18-165206] [2024-01-18T16:53:16.481Z] 2024/01/18 16:53:06 DoRetryWithTimeout - Error: {backup status for [backup-schedule-irumj-a613710-interval-2024-01-18-165206] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:53:16.481Z] 2024-01-18 16:53:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] first schedule backup for schedule name [backup-schedule-empla] is [backup-schedule-empla-d7df443-interval-2024-01-18-165208] [2024-01-18T16:53:16.482Z] 2024/01/18 16:53:08 DoRetryWithTimeout - Error: {backup status for [backup-schedule-empla-d7df443-interval-2024-01-18-165208] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:53:16.482Z] 2024-01-18 16:53:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] first schedule backup for schedule name [backup-schedule-cwxez] is [backup-schedule-cwxez-6263ec8-interval-2024-01-18-165208] [2024-01-18T16:53:16.482Z] 2024/01/18 16:53:08 DoRetryWithTimeout - Error: {backup status for [backup-schedule-cwxez-6263ec8-interval-2024-01-18-165208] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:53:38.378Z] 2024/01/18 16:53:36 DoRetryWithTimeout - Error: {backup status for [backup-schedule-irumj-a613710-interval-2024-01-18-165206] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:53:38.939Z] 2024/01/18 16:53:38 DoRetryWithTimeout - Error: {backup status for [backup-schedule-empla-d7df443-interval-2024-01-18-165208] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:53:38.939Z] 2024/01/18 16:53:38 DoRetryWithTimeout - Error: {backup status for [backup-schedule-cwxez-6263ec8-interval-2024-01-18-165208] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:54:10.964Z] INFO[2024-01-18 16:54:05] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T16:54:10.964Z] INFO[2024-01-18 16:54:05] sched stats: tick interval: 4805 < 1.2s (100%) [2024-01-18T16:54:10.964Z] INFO[2024-01-18 16:54:05] sched stats: tick process durations: 4804 < 500ms (100%) [2024-01-18T16:54:10.964Z] 2024/01/18 16:54:06 DoRetryWithTimeout - Error: {backup status for [backup-schedule-irumj-a613710-interval-2024-01-18-165206] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:54:10.964Z] 2024/01/18 16:54:08 DoRetryWithTimeout - Error: {backup status for [backup-schedule-empla-d7df443-interval-2024-01-18-165208] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:54:10.964Z] 2024/01/18 16:54:08 DoRetryWithTimeout - Error: {backup status for [backup-schedule-cwxez-6263ec8-interval-2024-01-18-165208] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:54:37.524Z] 2024/01/18 16:54:36 DoRetryWithTimeout - Error: {backup status for [backup-schedule-irumj-a613710-interval-2024-01-18-165206] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:54:38.892Z] 2024/01/18 16:54:38 DoRetryWithTimeout - Error: {backup status for [backup-schedule-empla-d7df443-interval-2024-01-18-165208] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:54:38.892Z] 2024/01/18 16:54:38 DoRetryWithTimeout - Error: {backup status for [backup-schedule-cwxez-6263ec8-interval-2024-01-18-165208] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:55:10.915Z] 2024-01-18 16:55:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating backup [backup-schedule-irumj-a613710-interval-2024-01-18-165206] in org [default] [2024-01-18T16:55:10.915Z] 2024-01-18 16:55:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [backup-schedule-irumj-a613710-interval-2024-01-18-165206] [2024-01-18T16:55:10.915Z] 2024-01-18 16:55:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in backup [backup-schedule-irumj-a613710-interval-2024-01-18-165206] [2024-01-18T16:55:10.915Z] 2024-01-18 16:55:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in backup [backup-schedule-irumj-a613710-interval-2024-01-18-165206] [2024-01-18T16:55:10.915Z] 2024-01-18 16:55:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T16:55:10.915Z] 2024-01-18 16:55:06 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:55:10.916Z] 2024-01-18 16:55:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:55:10.916Z] 2024-01-18 16:55:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[mysql-data]] [2024-01-18T16:55:10.916Z] 2024-01-18 16:55:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T16:55:10.916Z] 2024-01-18 16:55:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating CR cleanup [2024-01-18T16:55:10.916Z] 2024-01-18 16:55:06 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:55:10.916Z] 2024-01-18 16:55:07 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[backup-schedule-empla-d7df443-interval-2024-01-18-165208 backup-schedule-irumj-a613710-interval-2024-01-18-165206]] [2024-01-18T16:55:10.916Z] 2024-01-18 16:55:07 +0000:[INFO] [{VerifyRBACForInfraAdmin}] CR found for [[backup-schedule-empla-d7df443-interval-2024-01-18-165208 backup-schedule-irumj-a613710-interval-2024-01-18-165206]] under [mysql-ibm-pxb-0-87886-01-18-16h38m08s] namespace [2024-01-18T16:55:10.916Z] 2024/01/18 16:55:07 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-irumj-a613710-interval-2024-01-18-165206]}, Next try in [5s], timeout [5m0s] [2024-01-18T16:55:10.916Z] 2024-01-18 16:55:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating backup [backup-schedule-empla-d7df443-interval-2024-01-18-165208] in org [default] [2024-01-18T16:55:10.916Z] 2024-01-18 16:55:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [backup-schedule-empla-d7df443-interval-2024-01-18-165208] [2024-01-18T16:55:10.916Z] 2024-01-18 16:55:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in backup [backup-schedule-empla-d7df443-interval-2024-01-18-165208] [2024-01-18T16:55:10.916Z] 2024-01-18 16:55:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in backup [backup-schedule-empla-d7df443-interval-2024-01-18-165208] [2024-01-18T16:55:10.916Z] 2024-01-18 16:55:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T16:55:10.916Z] 2024-01-18 16:55:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating backup [backup-schedule-cwxez-6263ec8-interval-2024-01-18-165208] in org [default] [2024-01-18T16:55:10.916Z] 2024-01-18 16:55:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [backup-schedule-cwxez-6263ec8-interval-2024-01-18-165208] [2024-01-18T16:55:10.916Z] 2024-01-18 16:55:08 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:55:10.916Z] 2024-01-18 16:55:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:55:10.916Z] 2024-01-18 16:55:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[mysql-data]] [2024-01-18T16:55:10.916Z] 2024-01-18 16:55:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T16:55:10.916Z] 2024-01-18 16:55:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating CR cleanup [2024-01-18T16:55:10.917Z] 2024-01-18 16:55:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in backup [backup-schedule-cwxez-6263ec8-interval-2024-01-18-165208] [2024-01-18T16:55:10.917Z] 2024-01-18 16:55:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in backup [backup-schedule-cwxez-6263ec8-interval-2024-01-18-165208] [2024-01-18T16:55:10.917Z] 2024-01-18 16:55:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T16:55:10.917Z] 2024-01-18 16:55:08 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T16:55:10.917Z] 2024-01-18 16:55:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T16:55:10.917Z] 2024-01-18 16:55:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[mysql-data]] [2024-01-18T16:55:10.917Z] 2024-01-18 16:55:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T16:55:10.917Z] 2024-01-18 16:55:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating CR cleanup [2024-01-18T16:55:10.917Z] 2024-01-18 16:55:08 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:55:10.917Z] 2024-01-18 16:55:09 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[backup-schedule-empla-d7df443-interval-2024-01-18-165208 backup-schedule-irumj-a613710-interval-2024-01-18-165206]] [2024-01-18T16:55:10.917Z] 2024-01-18 16:55:09 +0000:[INFO] [{VerifyRBACForInfraAdmin}] CR found for [[backup-schedule-empla-d7df443-interval-2024-01-18-165208 backup-schedule-irumj-a613710-interval-2024-01-18-165206]] under [mysql-ibm-pxb-0-87886-01-18-16h38m08s] namespace [2024-01-18T16:55:10.917Z] 2024/01/18 16:55:09 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-empla-d7df443-interval-2024-01-18-165208]}, Next try in [5s], timeout [5m0s] [2024-01-18T16:55:10.917Z] 2024-01-18 16:55:09 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T16:55:10.917Z] 2024-01-18 16:55:09 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[backup-schedule-empla-d7df443-interval-2024-01-18-165208 backup-schedule-irumj-a613710-interval-2024-01-18-165206]] [2024-01-18T16:55:10.917Z] INFO[2024-01-18 16:55:09] Verifying : Description : Verifying creation and validation of schedule backup with schedule name [backup-schedule-cwxez] [2024-01-18T16:55:10.917Z] INFO[2024-01-18 16:55:09] Actual:true, Expected: true [2024-01-18T16:55:10.917Z] INFO[2024-01-18 16:55:09] Verifying : Description : Suspending Backup Schedule [backup-schedule-cwxez] for user [testuser1-1705596520] [2024-01-18T16:55:10.917Z] INFO[2024-01-18 16:55:09] Actual:true, Expected: true [2024-01-18T16:55:12.299Z] 2024-01-18 16:55:12 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[backup-schedule-empla-d7df443-interval-2024-01-18-165208 backup-schedule-irumj-a613710-interval-2024-01-18-165206]] [2024-01-18T16:55:12.299Z] 2024-01-18 16:55:12 +0000:[INFO] [{VerifyRBACForInfraAdmin}] CR found for [[backup-schedule-empla-d7df443-interval-2024-01-18-165208 backup-schedule-irumj-a613710-interval-2024-01-18-165206]] under [mysql-ibm-pxb-0-87886-01-18-16h38m08s] namespace [2024-01-18T16:55:12.299Z] 2024/01/18 16:55:12 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-irumj-a613710-interval-2024-01-18-165206]}, Next try in [5s], timeout [5m0s] [2024-01-18T16:55:14.190Z] 2024-01-18 16:55:14 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[backup-schedule-empla-d7df443-interval-2024-01-18-165208 backup-schedule-irumj-a613710-interval-2024-01-18-165206]] [2024-01-18T16:55:14.190Z] 2024-01-18 16:55:14 +0000:[INFO] [{VerifyRBACForInfraAdmin}] CR found for [[backup-schedule-empla-d7df443-interval-2024-01-18-165208 backup-schedule-irumj-a613710-interval-2024-01-18-165206]] under [mysql-ibm-pxb-0-87886-01-18-16h38m08s] namespace [2024-01-18T16:55:14.190Z] 2024/01/18 16:55:14 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-empla-d7df443-interval-2024-01-18-165208]}, Next try in [5s], timeout [5m0s] [2024-01-18T16:55:17.455Z] 2024-01-18 16:55:17 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[backup-schedule-empla-d7df443-interval-2024-01-18-165208 backup-schedule-irumj-a613710-interval-2024-01-18-165206]] [2024-01-18T16:55:17.455Z] 2024-01-18 16:55:17 +0000:[INFO] [{VerifyRBACForInfraAdmin}] CR found for [[backup-schedule-empla-d7df443-interval-2024-01-18-165208 backup-schedule-irumj-a613710-interval-2024-01-18-165206]] under [mysql-ibm-pxb-0-87886-01-18-16h38m08s] namespace [2024-01-18T16:55:17.456Z] 2024/01/18 16:55:17 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-irumj-a613710-interval-2024-01-18-165206]}, Next try in [5s], timeout [5m0s] [2024-01-18T16:55:19.354Z] 2024-01-18 16:55:19 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[backup-schedule-empla-d7df443-interval-2024-01-18-165208 backup-schedule-irumj-a613710-interval-2024-01-18-165206]] [2024-01-18T16:55:19.354Z] 2024-01-18 16:55:19 +0000:[INFO] [{VerifyRBACForInfraAdmin}] CR found for [[backup-schedule-empla-d7df443-interval-2024-01-18-165208 backup-schedule-irumj-a613710-interval-2024-01-18-165206]] under [mysql-ibm-pxb-0-87886-01-18-16h38m08s] namespace [2024-01-18T16:55:19.354Z] 2024/01/18 16:55:19 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-empla-d7df443-interval-2024-01-18-165208]}, Next try in [5s], timeout [5m0s] [2024-01-18T16:55:22.632Z] 2024-01-18 16:55:22 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[backup-schedule-empla-d7df443-interval-2024-01-18-165208 backup-schedule-irumj-a613710-interval-2024-01-18-165206]] [2024-01-18T16:55:22.632Z] 2024-01-18 16:55:22 +0000:[INFO] [{VerifyRBACForInfraAdmin}] CR found for [[backup-schedule-empla-d7df443-interval-2024-01-18-165208 backup-schedule-irumj-a613710-interval-2024-01-18-165206]] under [mysql-ibm-pxb-0-87886-01-18-16h38m08s] namespace [2024-01-18T16:55:22.632Z] 2024/01/18 16:55:22 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-irumj-a613710-interval-2024-01-18-165206]}, Next try in [5s], timeout [5m0s] [2024-01-18T16:55:24.520Z] 2024-01-18 16:55:24 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[backup-schedule-empla-d7df443-interval-2024-01-18-165208 backup-schedule-irumj-a613710-interval-2024-01-18-165206]] [2024-01-18T16:55:24.520Z] 2024-01-18 16:55:24 +0000:[INFO] [{VerifyRBACForInfraAdmin}] CR found for [[backup-schedule-empla-d7df443-interval-2024-01-18-165208 backup-schedule-irumj-a613710-interval-2024-01-18-165206]] under [mysql-ibm-pxb-0-87886-01-18-16h38m08s] namespace [2024-01-18T16:55:24.520Z] 2024/01/18 16:55:24 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-empla-d7df443-interval-2024-01-18-165208]}, Next try in [5s], timeout [5m0s] [2024-01-18T16:55:27.788Z] 2024-01-18 16:55:27 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[backup-schedule-empla-d7df443-interval-2024-01-18-165208 backup-schedule-irumj-a613710-interval-2024-01-18-165206]] [2024-01-18T16:55:27.788Z] 2024-01-18 16:55:27 +0000:[INFO] [{VerifyRBACForInfraAdmin}] CR found for [[backup-schedule-empla-d7df443-interval-2024-01-18-165208 backup-schedule-irumj-a613710-interval-2024-01-18-165206]] under [mysql-ibm-pxb-0-87886-01-18-16h38m08s] namespace [2024-01-18T16:55:27.788Z] 2024/01/18 16:55:27 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-irumj-a613710-interval-2024-01-18-165206]}, Next try in [5s], timeout [5m0s] [2024-01-18T16:55:29.682Z] 2024-01-18 16:55:29 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[backup-schedule-empla-d7df443-interval-2024-01-18-165208]] [2024-01-18T16:55:29.682Z] 2024-01-18 16:55:29 +0000:[INFO] [{VerifyRBACForInfraAdmin}] CR found for [[backup-schedule-empla-d7df443-interval-2024-01-18-165208]] under [mysql-ibm-pxb-0-87886-01-18-16h38m08s] namespace [2024-01-18T16:55:29.682Z] 2024/01/18 16:55:29 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-empla-d7df443-interval-2024-01-18-165208]}, Next try in [5s], timeout [5m0s] [2024-01-18T16:55:32.948Z] 2024-01-18 16:55:32 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[backup-schedule-empla-d7df443-interval-2024-01-18-165208]] [2024-01-18T16:55:32.948Z] INFO[2024-01-18 16:55:32] Verifying : Description : Verifying creation and validation of schedule backup with schedule name [backup-schedule-irumj] [2024-01-18T16:55:32.948Z] INFO[2024-01-18 16:55:32] Actual:true, Expected: true [2024-01-18T16:55:32.948Z] INFO[2024-01-18 16:55:32] Verifying : Description : Suspending Backup Schedule [backup-schedule-irumj] for user [testuser1-1705596515] [2024-01-18T16:55:32.948Z] INFO[2024-01-18 16:55:32] Actual:true, Expected: true [2024-01-18T16:55:34.337Z] 2024-01-18 16:55:34 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[backup-schedule-empla-d7df443-interval-2024-01-18-165208]] [2024-01-18T16:55:34.337Z] 2024-01-18 16:55:34 +0000:[INFO] [{VerifyRBACForInfraAdmin}] CR found for [[backup-schedule-empla-d7df443-interval-2024-01-18-165208]] under [mysql-ibm-pxb-0-87886-01-18-16h38m08s] namespace [2024-01-18T16:55:34.337Z] 2024/01/18 16:55:34 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-empla-d7df443-interval-2024-01-18-165208]}, Next try in [5s], timeout [5m0s] [2024-01-18T16:55:39.580Z] 2024-01-18 16:55:39 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[backup-schedule-empla-d7df443-interval-2024-01-18-165208]] [2024-01-18T16:55:39.580Z] 2024-01-18 16:55:39 +0000:[INFO] [{VerifyRBACForInfraAdmin}] CR found for [[backup-schedule-empla-d7df443-interval-2024-01-18-165208]] under [mysql-ibm-pxb-0-87886-01-18-16h38m08s] namespace [2024-01-18T16:55:39.580Z] 2024/01/18 16:55:39 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-empla-d7df443-interval-2024-01-18-165208]}, Next try in [5s], timeout [5m0s] [2024-01-18T16:55:44.828Z] 2024-01-18 16:55:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[backup-schedule-empla-d7df443-interval-2024-01-18-165208]] [2024-01-18T16:55:44.828Z] 2024-01-18 16:55:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] CR found for [[backup-schedule-empla-d7df443-interval-2024-01-18-165208]] under [mysql-ibm-pxb-0-87886-01-18-16h38m08s] namespace [2024-01-18T16:55:44.828Z] 2024/01/18 16:55:44 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-empla-d7df443-interval-2024-01-18-165208]}, Next try in [5s], timeout [5m0s] [2024-01-18T16:55:50.071Z] 2024-01-18 16:55:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[backup-schedule-empla-d7df443-interval-2024-01-18-165208]] [2024-01-18T16:55:50.071Z] 2024-01-18 16:55:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] CR found for [[backup-schedule-empla-d7df443-interval-2024-01-18-165208]] under [mysql-ibm-pxb-0-87886-01-18-16h38m08s] namespace [2024-01-18T16:55:50.071Z] 2024/01/18 16:55:49 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-empla-d7df443-interval-2024-01-18-165208]}, Next try in [5s], timeout [5m0s] [2024-01-18T16:55:55.363Z] 2024-01-18 16:55:54 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[backup-schedule-empla-d7df443-interval-2024-01-18-165208]] [2024-01-18T16:55:55.363Z] 2024-01-18 16:55:54 +0000:[INFO] [{VerifyRBACForInfraAdmin}] CR found for [[backup-schedule-empla-d7df443-interval-2024-01-18-165208]] under [mysql-ibm-pxb-0-87886-01-18-16h38m08s] namespace [2024-01-18T16:55:55.363Z] 2024/01/18 16:55:54 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-empla-d7df443-interval-2024-01-18-165208]}, Next try in [5s], timeout [5m0s] [2024-01-18T16:55:59.529Z] 2024-01-18 16:55:59 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[backup-schedule-empla-d7df443-interval-2024-01-18-165208]] [2024-01-18T16:55:59.529Z] 2024-01-18 16:55:59 +0000:[INFO] [{VerifyRBACForInfraAdmin}] CR found for [[backup-schedule-empla-d7df443-interval-2024-01-18-165208]] under [mysql-ibm-pxb-0-87886-01-18-16h38m08s] namespace [2024-01-18T16:55:59.529Z] 2024/01/18 16:55:59 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-empla-d7df443-interval-2024-01-18-165208]}, Next try in [5s], timeout [5m0s] [2024-01-18T16:56:04.774Z] 2024-01-18 16:56:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[backup-schedule-empla-d7df443-interval-2024-01-18-165208]] [2024-01-18T16:56:04.774Z] 2024-01-18 16:56:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] CR found for [[backup-schedule-empla-d7df443-interval-2024-01-18-165208]] under [mysql-ibm-pxb-0-87886-01-18-16h38m08s] namespace [2024-01-18T16:56:04.774Z] 2024/01/18 16:56:04 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-empla-d7df443-interval-2024-01-18-165208]}, Next try in [5s], timeout [5m0s] [2024-01-18T16:56:10.019Z] 2024-01-18 16:56:09 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[backup-schedule-empla-d7df443-interval-2024-01-18-165208]] [2024-01-18T16:56:10.019Z] 2024-01-18 16:56:09 +0000:[INFO] [{VerifyRBACForInfraAdmin}] CR found for [[backup-schedule-empla-d7df443-interval-2024-01-18-165208]] under [mysql-ibm-pxb-0-87886-01-18-16h38m08s] namespace [2024-01-18T16:56:10.019Z] 2024/01/18 16:56:09 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-empla-d7df443-interval-2024-01-18-165208]}, Next try in [5s], timeout [5m0s] [2024-01-18T16:56:15.262Z] 2024-01-18 16:56:14 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[]] [2024-01-18T16:56:15.262Z] INFO[2024-01-18 16:56:14] Verifying : Description : Verifying creation and validation of schedule backup with schedule name [backup-schedule-empla] [2024-01-18T16:56:15.262Z] INFO[2024-01-18 16:56:14] Actual:true, Expected: true [2024-01-18T16:56:15.262Z] INFO[2024-01-18 16:56:14] Verifying : Description : Suspending Backup Schedule [backup-schedule-empla] for user [testuser1-1705596517] [2024-01-18T16:56:15.262Z] INFO[2024-01-18 16:56:14] Actual:true, Expected: true [2024-01-18T16:56:15.262Z] 2024-01-18 16:56:14 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Deleting user testuser1-1705596520 backups from the user context [2024-01-18T16:56:15.262Z] 2024-01-18 16:56:14 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Deleting user testuser1-1705596517 backups from the user context [2024-01-18T16:56:15.262Z] 2024-01-18 16:56:14 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Deleting user testuser1-1705596515 backups from the user context [2024-01-18T16:56:15.262Z] 2024/01/18 16:56:14 DoRetryWithTimeout - Error: {backupObject [tp-backup-manual-single-ns-testuser1-1705596515-with-rules-gidt] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-18T16:56:15.262Z] 2024/01/18 16:56:14 DoRetryWithTimeout - Error: {backupObject [tp-backup-manual-single-ns-testuser1-1705596520-with-rules-kghk] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-18T16:56:15.262Z] 2024/01/18 16:56:14 DoRetryWithTimeout - Error: {backupObject [tp-backup-manual-single-ns-testuser1-1705596517-with-rules-trui] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-18T16:56:47.293Z] 2024/01/18 16:56:45 DoRetryWithTimeout - Error: {backupObject [backup-schedule-irumj-a613710-interval-2024-01-18-165206] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-18T16:56:47.293Z] 2024/01/18 16:56:45 DoRetryWithTimeout - Error: {backupObject [backup-schedule-cwxez-6263ec8-interval-2024-01-18-165208] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-18T16:56:47.293Z] 2024/01/18 16:56:45 DoRetryWithTimeout - Error: {backupObject [backup-schedule-empla-d7df443-interval-2024-01-18-165208] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-18T16:57:19.317Z] 2024/01/18 16:57:15 DoRetryWithTimeout - Error: {backupObject [backup-schedule-irumj-a613710-interval-2024-01-18-165206] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-18T16:57:19.317Z] 2024/01/18 16:57:15 DoRetryWithTimeout - Error: {backupObject [backup-schedule-cwxez-6263ec8-interval-2024-01-18-165208] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-18T16:57:19.317Z] 2024/01/18 16:57:15 DoRetryWithTimeout - Error: {backupObject [backup-schedule-empla-d7df443-interval-2024-01-18-165208] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-18T16:57:45.811Z] 2024-01-18 16:57:45 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Delete user testuser1-1705596515 backup schedule [2024-01-18T16:57:45.811Z] 2024-01-18 16:57:45 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Delete user testuser1-1705596520 backup schedule [2024-01-18T16:57:45.811Z] 2024-01-18 16:57:45 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Delete user testuser1-1705596517 backup schedule [2024-01-18T16:57:45.811Z] 2024/01/18 16:57:45 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-cwxez] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T16:57:45.811Z] 2024/01/18 16:57:45 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-irumj] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T16:57:45.811Z] 2024/01/18 16:57:45 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-empla] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T16:57:55.755Z] 2024/01/18 16:57:55 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-cwxez] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T16:57:55.755Z] 2024/01/18 16:57:55 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-irumj] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T16:57:55.756Z] 2024/01/18 16:57:55 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-empla] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T16:58:05.699Z] 2024/01/18 16:58:05 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-cwxez] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T16:58:05.699Z] 2024/01/18 16:58:05 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-irumj] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T16:58:05.699Z] 2024/01/18 16:58:05 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-empla] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T16:58:15.639Z] 2024/01/18 16:58:15 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-cwxez] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T16:58:15.639Z] 2024/01/18 16:58:15 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-irumj] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T16:58:15.639Z] 2024/01/18 16:58:15 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-empla] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T16:58:25.577Z] 2024/01/18 16:58:25 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-cwxez] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T16:58:25.577Z] 2024/01/18 16:58:25 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-irumj] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T16:58:25.577Z] 2024/01/18 16:58:25 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-empla] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T16:58:35.516Z] INFO[2024-01-18 16:58:35] Verifying : Description : Deleting Backup Schedule [backup-schedule-cwxez] for user [testuser1-1705596520] [2024-01-18T16:58:35.516Z] INFO[2024-01-18 16:58:35] Actual:true, Expected: true [2024-01-18T16:58:35.516Z] 2024-01-18 16:58:35 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Deleting user testuser1-1705596520 source and destination cluster from the user context [2024-01-18T16:58:35.516Z] INFO[2024-01-18 16:58:35] Verifying : Description : Deleting Backup Schedule [backup-schedule-irumj] for user [testuser1-1705596515] [2024-01-18T16:58:35.516Z] INFO[2024-01-18 16:58:35] Actual:true, Expected: true [2024-01-18T16:58:35.516Z] 2024-01-18 16:58:35 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Deleting user testuser1-1705596515 source and destination cluster from the user context [2024-01-18T16:58:35.775Z] INFO[2024-01-18 16:58:35] Verifying : Description : Deleting Backup Schedule [backup-schedule-empla] for user [testuser1-1705596517] [2024-01-18T16:58:35.775Z] INFO[2024-01-18 16:58:35] Actual:true, Expected: true [2024-01-18T16:58:35.775Z] 2024-01-18 16:58:35 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Deleting user testuser1-1705596517 source and destination cluster from the user context [2024-01-18T16:58:36.031Z] INFO[2024-01-18 16:58:35] Verifying : Description : Verifying deletion of cluster [source-cluster] of the user testuser1-1705596517 [2024-01-18T16:58:36.031Z] INFO[2024-01-18 16:58:35] Actual:true, Expected: true [2024-01-18T16:58:36.031Z] INFO[2024-01-18 16:58:35] Verifying : Description : Verifying deletion of cluster [source-cluster] of the user testuser1-1705596515 [2024-01-18T16:58:36.031Z] INFO[2024-01-18 16:58:35] Actual:true, Expected: true [2024-01-18T16:58:36.286Z] INFO[2024-01-18 16:58:36] Verifying : Description : Verifying deletion of cluster [source-cluster] of the user testuser1-1705596520 [2024-01-18T16:58:36.286Z] INFO[2024-01-18 16:58:36] Actual:true, Expected: true [2024-01-18T16:58:36.286Z] INFO[2024-01-18 16:58:36] Verifying : Description : Verifying deletion of cluster [destination-cluster] of the user testuser1-1705596517 [2024-01-18T16:58:36.287Z] INFO[2024-01-18 16:58:36] Actual:true, Expected: true [2024-01-18T16:58:36.542Z] INFO[2024-01-18 16:58:36] Verifying : Description : Verifying deletion of cluster [destination-cluster] of the user testuser1-1705596515 [2024-01-18T16:58:36.542Z] INFO[2024-01-18 16:58:36] Actual:true, Expected: true [2024-01-18T16:58:36.542Z] INFO[2024-01-18 16:58:36] Verifying : Description : Verifying deletion of cluster [destination-cluster] of the user testuser1-1705596520 [2024-01-18T16:58:36.542Z] INFO[2024-01-18 16:58:36] Actual:true, Expected: true [2024-01-18T16:58:36.542Z] 2024-01-18 16:58:36 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Verify Infra-Admin User has permission to create new custom role [2024-01-18T16:58:37.104Z] INFO[2024-01-18 16:58:37] Verifying : Description : Verifying creation of role [custom-infra-admin-role-rscg] by the infra-admin user [2024-01-18T16:58:37.104Z] INFO[2024-01-18 16:58:37] Actual:true, Expected: true [2024-01-18T16:58:37.104Z] 2024-01-18 16:58:37 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Create a new user with custom role [custom-infra-admin-role-rscg] [2024-01-18T16:58:37.104Z] 2024-01-18 16:58:37 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Creating 1 users [2024-01-18T16:58:37.672Z] 2024-01-18 16:58:37 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.AddUser:#728] - Response for user [testuser1-1705597117] creation - [2024-01-18T16:58:37.672Z] 2024-01-18 16:58:37 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.AddUser:#733] - User [testuser1-1705597117] created [2024-01-18T16:58:37.672Z] INFO[2024-01-18 16:58:37] Verifying : Description : Creating user - testuser1-1705597117 [2024-01-18T16:58:37.672Z] INFO[2024-01-18 16:58:37] Actual:true, Expected: true [2024-01-18T16:58:38.603Z] 2024-01-18 16:58:38 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705597117 - 70a91b8d-127c-4fba-bbf6-15c954a0a95c [2024-01-18T16:58:39.970Z] 2024-01-18 16:58:39 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.glob..func43.2.22:#457] - username testuser1-1705595980 common password K1osV#93fgcmdm [2024-01-18T16:58:39.970Z] 2024-01-18 16:58:39 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Verify custom user has permission to create cloud credential and backup location [2024-01-18T16:58:40.531Z] 2024-01-18 16:58:40 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-cifdg] for org [default] with [aws] as provider [2024-01-18T16:58:40.531Z] 2024-01-18 16:58:40 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-18T16:58:40.531Z] 2024-01-18 16:58:40 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-cifdg] is present [2024-01-18T16:58:40.531Z] INFO[2024-01-18 16:58:40] Verifying : Description : Verifying creation of cloud credential autogenerated-cred-cifdg using provider aws for the user [2024-01-18T16:58:40.531Z] INFO[2024-01-18 16:58:40] Actual:true, Expected: true [2024-01-18T16:58:42.474Z] INFO[2024-01-18 16:58:42] Verifying : Description : Verifying creation of backup location autogenerated-backup-location-csaxc using provider aws for the custom user [testuser1-1705597117] [2024-01-18T16:58:42.474Z] INFO[2024-01-18 16:58:42] Actual:true, Expected: true [2024-01-18T16:58:42.474Z] 2024-01-18 16:58:42 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Adding labels to namespaces for custom user [2024-01-18T16:58:42.474Z] INFO[2024-01-18 16:58:42] Verifying : Description : Adding labels [map[0-93526b28-85ca-412f-bce9-f9484c29a40d:93526b28-85ca-412f-bce9-f9484c29a40d 1-93526b28-85ca-412f-bce9-f9484c29a40d:93526b28-85ca-412f-bce9-f9484c29a40d]] to namespaces [[mysql-ibm-pxb-0-87886-01-18-16h38m08s]] for custom user validation [2024-01-18T16:58:42.474Z] INFO[2024-01-18 16:58:42] Actual:true, Expected: true [2024-01-18T16:58:42.474Z] 2024-01-18 16:58:42 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Generating namespace label string from label map for namespaces [2024-01-18T16:58:42.474Z] 2024-01-18 16:58:42 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.glob..func43.2.25:#486] - label string for namespaces 0-93526b28-85ca-412f-bce9-f9484c29a40d=93526b28-85ca-412f-bce9-f9484c29a40d,1-93526b28-85ca-412f-bce9-f9484c29a40d=93526b28-85ca-412f-bce9-f9484c29a40d [2024-01-18T16:58:42.474Z] 2024-01-18 16:58:42 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Verify custom user has permission to create a schedule policy [2024-01-18T16:58:42.733Z] INFO[2024-01-18 16:58:42] Verifying : Description : Verifying creation of periodic schedule policy of interval [15] minutes named [periodic-xhovs] [2024-01-18T16:58:42.733Z] INFO[2024-01-18 16:58:42] Actual:true, Expected: true [2024-01-18T16:58:42.733Z] 2024-01-18 16:58:42 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Verify custom user has permission to create pre and post exec rules for applications [2024-01-18T16:58:42.733Z] 2024-01-18 16:58:42 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.CreateRuleForBackupWithMultipleApplications:#4309] - Validate pre-rules for backup [2024-01-18T16:58:42.733Z] 2024-01-18 16:58:42 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.CreateRuleForBackupWithMultipleApplications:#4322] - Validate post-rules for backup [2024-01-18T16:58:42.991Z] INFO[2024-01-18 16:58:42] Verifying : Description : Verifying creation of pre and post exec rules for applications from custom user [testuser1-1705597117] [2024-01-18T16:58:42.991Z] INFO[2024-01-18 16:58:42] Actual:true, Expected: true [2024-01-18T16:58:42.991Z] 2024-01-18 16:58:42 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Verify custom user doesn't have permission to create roles [2024-01-18T16:58:43.925Z] INFO[2024-01-18 16:58:43] Verifying : Description : Verifying custom user doesnt have permission for creating role [custom-user-role-aqug] [2024-01-18T16:58:43.925Z] INFO[2024-01-18 16:58:43] Actual:true, Expected: true [2024-01-18T16:58:43.925Z] 2024-01-18 16:58:43 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Creating source and destination cluster from the custom user testuser1-1705597117 [2024-01-18T16:58:44.185Z] INFO[2024-01-18 16:58:44] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T16:58:44.185Z] INFO[2024-01-18 16:58:44] Actual:true, Expected: true [2024-01-18T16:58:44.185Z] 2024-01-18 16:58:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T16:58:44.452Z] 2024-01-18 16:58:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T16:58:44.452Z] 2024-01-18 16:58:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T16:58:44.452Z] 2024-01-18 16:58:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T16:58:44.452Z] 2024-01-18 16:58:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Create cluster [source-cluster] in org [default] [2024-01-18T16:58:44.452Z] 2024-01-18 16:58:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T16:58:44.452Z] 2024-01-18 16:58:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T16:58:44.452Z] 2024-01-18 16:58:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-18T16:58:44.452Z] 2024-01-18 16:58:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T16:58:44.452Z] 2024-01-18 16:58:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T16:58:44.452Z] 2024-01-18 16:58:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-eockv] for IBM [2024-01-18T16:58:44.452Z] 2024-01-18 16:58:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-eockv] for org [default] with [ibm] as provider [2024-01-18T16:58:44.452Z] 2024-01-18 16:58:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:58:44.452Z] 2024-01-18 16:58:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:58:44.453Z] 2024-01-18 16:58:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-eockv] is present [2024-01-18T16:58:44.453Z] 2024-01-18 16:58:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Create cluster [source-cluster] in org [default] [2024-01-18T16:58:45.014Z] 2024-01-18 16:58:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-ansrj] for IBM [2024-01-18T16:58:45.014Z] 2024-01-18 16:58:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-ansrj] for org [default] with [ibm] as provider [2024-01-18T16:58:45.014Z] 2024-01-18 16:58:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T16:58:45.014Z] 2024-01-18 16:58:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T16:58:45.014Z] 2024-01-18 16:58:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-ansrj] is present [2024-01-18T16:58:45.014Z] 2024-01-18 16:58:44 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-18T16:58:45.575Z] INFO[2024-01-18 16:58:45] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-18T16:58:45.575Z] INFO[2024-01-18 16:58:45] Actual:Online, Expected: Online [2024-01-18T16:58:45.575Z] INFO[2024-01-18 16:58:45] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-18T16:58:45.575Z] INFO[2024-01-18 16:58:45] Actual:true, Expected: true [2024-01-18T16:58:45.575Z] INFO[2024-01-18 16:58:45] Verifying : Description : Fetching [destination-cluster] cluster uid [2024-01-18T16:58:45.575Z] INFO[2024-01-18 16:58:45] Actual:true, Expected: true [2024-01-18T16:58:45.575Z] 2024-01-18 16:58:45 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Taking backup of applications from the custom user testuser1-1705597117 [2024-01-18T16:58:45.575Z] 2024-01-18 16:58:45 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T16:58:45.575Z] 2024-01-18 16:58:45 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Volume snapshot classes found in the cluster - [2024-01-18T16:58:45.575Z] 2024-01-18 16:58:45 +0000:[INFO] [{VerifyRBACForInfraAdmin}] ibmc-vpcblock-snapshot [2024-01-18T16:58:45.575Z] 2024-01-18 16:58:45 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T16:58:46.136Z] 2024/01/18 16:58:45 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T16:59:18.175Z] 2024/01/18 16:59:15 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T16:59:50.217Z] 2024/01/18 16:59:45 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T17:00:16.713Z] 2024-01-18 17:00:15 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.CreateBackup:#313] - Backup [tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty] created successfully [2024-01-18T17:00:16.713Z] 2024-01-18 17:00:15 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating backup [tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty] in org [default] [2024-01-18T17:00:16.713Z] 2024-01-18 17:00:15 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty] [2024-01-18T17:00:16.713Z] 2024-01-18 17:00:15 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in backup [tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty] [2024-01-18T17:00:16.713Z] 2024-01-18 17:00:15 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in backup [tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty] [2024-01-18T17:00:16.713Z] 2024-01-18 17:00:15 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:00:16.713Z] 2024-01-18 17:00:16 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:00:16.713Z] 2024-01-18 17:00:16 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:00:16.714Z] 2024-01-18 17:00:16 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[mysql-data]] [2024-01-18T17:00:16.714Z] 2024-01-18 17:00:16 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T17:00:16.714Z] 2024-01-18 17:00:16 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating CR cleanup [2024-01-18T17:00:16.714Z] 2024-01-18 17:00:16 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T17:00:16.714Z] 2024-01-18 17:00:16 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[]] [2024-01-18T17:00:16.714Z] INFO[2024-01-18 17:00:16] Verifying : Description : Verifying creation and validation of backup [tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty] of namespace (scheduled Context) [[mysql-ibm-pxb-0-87886-01-18-16h38m08s]] [2024-01-18T17:00:16.714Z] INFO[2024-01-18 17:00:16] Actual:true, Expected: true [2024-01-18T17:00:16.714Z] 2024-01-18 17:00:16 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Taking restore of applications from the custom user testuser1-1705597117 [2024-01-18T17:00:17.276Z] 2024/01/18 17:00:17 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T17:00:49.326Z] 2024-01-18 17:00:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.CreateRestore:#936] - Restore [tp-restore-tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty] created successfully [2024-01-18T17:00:49.326Z] 2024-01-18 17:00:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switching cluster context to cluster [destination-cluster] [2024-01-18T17:00:49.326Z] 2024-01-18 17:00:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switching context to [/tmp/destination-config] [2024-01-18T17:00:49.326Z] 2024-01-18 17:00:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:00:49.326Z] 2024-01-18 17:00:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:00:49.326Z] 2024-01-18 17:00:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:00:49.326Z] 2024-01-18 17:00:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:00:49.326Z] 2024-01-18 17:00:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:00:49.326Z] 2024-01-18 17:00:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:00:49.326Z] 2024-01-18 17:00:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:00:49.326Z] 2024-01-18 17:00:48 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:00:49.327Z] 2024-01-18 17:00:48 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:00:49.327Z] 2024-01-18 17:00:48 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T17:00:49.327Z] 2024-01-18 17:00:49 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T17:00:49.327Z] 2024-01-18 17:00:49 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T17:00:49.327Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T17:00:49.327Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switched context to [/tmp/destination-config] [2024-01-18T17:00:49.327Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-87886-01-18-16h38m08s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-18T17:00:49.327Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:00:49.327Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T17:00:49.327Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T17:00:49.327Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating restore [tp-restore-tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty] in org [default] [2024-01-18T17:00:49.327Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-restore-tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty] [2024-01-18T17:00:49.327Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-restore-tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty] [2024-01-18T17:00:49.327Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in restore [tp-restore-tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty] [2024-01-18T17:00:49.327Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-87886-01-18-16h38m08s]) was found in restore [tp-restore-tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:00:49.585Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-87886-01-18-16h38m08s]) found in the restore [tp-restore-tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty] was also present on the cluster/namespace [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:00:49.585Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-87886-01-18-16h38m08s]) was found in restore [tp-restore-tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:00:49.585Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-87886-01-18-16h38m08s]) found in the restore [tp-restore-tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty] was also present on the cluster/namespace [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:00:49.585Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-87886-01-18-16h38m08s]) was found in restore [tp-restore-tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:00:49.842Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-87886-01-18-16h38m08s]) found in the restore [tp-restore-tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty] was also present on the cluster/namespace [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:00:49.842Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-87886-01-18-16h38m08s]) was found in restore [tp-restore-tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:00:49.842Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-87886-01-18-16h38m08s]) found in the restore [tp-restore-tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty] was also present on the cluster/namespace [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:00:49.842Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in restore [tp-restore-tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty] [2024-01-18T17:00:49.842Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in restore [tp-restore-tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty] [2024-01-18T17:00:49.842Z] 2024-01-18 17:00:49 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:00:49.842Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:00:49.842Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[mysql-data]] [2024-01-18T17:00:49.842Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2243] - in restore [tp-restore-tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty], the status of the restored volume [pvc-08811e3e-a792-4522-bacc-c14fe6aeb7de] was not Success. It was [Retained] with reason [Skipped from volume restore as policy is set to Retain and pvc already exists] [2024-01-18T17:00:49.842Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validate applications in restored namespace [mysql-ibm-pxb-0-87886-01-18-16h38m08s] due to restore [tp-restore-tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty] [2024-01-18T17:00:49.842Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating mysql-ibm app [2024-01-18T17:00:49.842Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] wait for mysql-ibm app to start running [2024-01-18T17:00:50.098Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T17:00:50.098Z] 2024-01-18 17:00:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T17:00:50.098Z] 2024-01-18 17:00:50 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T17:00:50.355Z] 2024-01-18 17:00:50 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating CR cleanup [2024-01-18T17:00:50.620Z] 2024-01-18 17:00:50 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T17:00:50.879Z] 2024-01-18 17:00:50 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[]] [2024-01-18T17:00:50.879Z] 2024-01-18 17:00:50 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switching cluster context back to cluster path [/tmp/source-config] [2024-01-18T17:00:50.879Z] 2024-01-18 17:00:50 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switching context to [/tmp/source-config] [2024-01-18T17:00:50.879Z] 2024-01-18 17:00:50 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:00:50.879Z] 2024-01-18 17:00:50 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:00:50.879Z] 2024-01-18 17:00:50 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:00:50.879Z] 2024-01-18 17:00:50 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:00:50.879Z] 2024-01-18 17:00:50 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:00:51.244Z] 2024-01-18 17:00:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:00:51.501Z] 2024-01-18 17:00:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:00:52.065Z] 2024-01-18 17:00:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:00:52.322Z] 2024-01-18 17:00:52 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:00:52.584Z] 2024-01-18 17:00:52 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T17:00:52.840Z] 2024-01-18 17:00:52 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T17:00:52.840Z] 2024-01-18 17:00:52 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T17:00:52.840Z] 2024-01-18 17:00:52 +0000:[INFO] [{VerifyRBACForInfraAdmin}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T17:00:52.840Z] 2024-01-18 17:00:52 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switched context to [/tmp/source-config] [2024-01-18T17:00:52.840Z] INFO[2024-01-18 17:00:52] Verifying : Description : Verifying creation of restore tp-restore-tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty of backup tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty [2024-01-18T17:00:52.840Z] INFO[2024-01-18 17:00:52] Actual:true, Expected: true [2024-01-18T17:00:52.840Z] 2024-01-18 17:00:52 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validate taking manual backup of applications with namespace for custom user [2024-01-18T17:00:53.096Z] 2024-01-18 17:00:53 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T17:00:53.352Z] 2024-01-18 17:00:53 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Volume snapshot classes found in the cluster - [2024-01-18T17:00:53.352Z] 2024-01-18 17:00:53 +0000:[INFO] [{VerifyRBACForInfraAdmin}] ibmc-vpcblock-snapshot [2024-01-18T17:00:53.353Z] 2024-01-18 17:00:53 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T17:00:53.915Z] 2024/01/18 17:00:53 DoRetryWithTimeout - Error: {backup status for [customuser-otjq] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T17:01:26.022Z] 2024/01/18 17:01:23 DoRetryWithTimeout - Error: {backup status for [customuser-otjq] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T17:01:58.051Z] 2024/01/18 17:01:53 DoRetryWithTimeout - Error: {backup status for [customuser-otjq] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T17:02:25.753Z] 2024-01-18 17:02:23 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.CreateBackupWithNamespaceLabel:#3030] - Successfully created backup [customuser-otjq] with namespace label [0-93526b28-85ca-412f-bce9-f9484c29a40d=93526b28-85ca-412f-bce9-f9484c29a40d,1-93526b28-85ca-412f-bce9-f9484c29a40d=93526b28-85ca-412f-bce9-f9484c29a40d] [2024-01-18T17:02:25.753Z] 2024-01-18 17:02:23 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating backup [customuser-otjq] in org [default] [2024-01-18T17:02:25.753Z] 2024-01-18 17:02:23 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [customuser-otjq] [2024-01-18T17:02:25.753Z] 2024-01-18 17:02:23 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in backup [customuser-otjq] [2024-01-18T17:02:25.753Z] 2024-01-18 17:02:23 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in backup [customuser-otjq] [2024-01-18T17:02:25.754Z] 2024-01-18 17:02:23 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:02:25.754Z] 2024-01-18 17:02:23 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:02:25.754Z] 2024-01-18 17:02:23 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:02:25.754Z] 2024-01-18 17:02:23 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[mysql-data]] [2024-01-18T17:02:25.754Z] 2024-01-18 17:02:23 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T17:02:25.754Z] 2024-01-18 17:02:23 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating CR cleanup [2024-01-18T17:02:25.754Z] 2024-01-18 17:02:23 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T17:02:25.754Z] 2024-01-18 17:02:24 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[]] [2024-01-18T17:02:25.754Z] INFO[2024-01-18 17:02:24] Verifying : Description : Verifying backup [customuser-otjq] creation with labels [0-93526b28-85ca-412f-bce9-f9484c29a40d=93526b28-85ca-412f-bce9-f9484c29a40d,1-93526b28-85ca-412f-bce9-f9484c29a40d=93526b28-85ca-412f-bce9-f9484c29a40d] [2024-01-18T17:02:25.754Z] INFO[2024-01-18 17:02:24] Actual:true, Expected: true [2024-01-18T17:02:25.754Z] 2024-01-18 17:02:24 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup customuser-otjq [2024-01-18T17:02:25.754Z] 2024-01-18 17:02:24 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:02:25.754Z] 2024-01-18 17:02:24 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are 0-93526b28-85ca-412f-bce9-f9484c29a40d=93526b28-85ca-412f-bce9-f9484c29a40d,1-93526b28-85ca-412f-bce9-f9484c29a40d=93526b28-85ca-412f-bce9-f9484c29a40d [2024-01-18T17:02:25.754Z] INFO[2024-01-18 17:02:24] Verifying : Description : Verifying if the labeled namespaces [[mysql-ibm-pxb-0-87886-01-18-16h38m08s]] are backed up, and check if labels [0-93526b28-85ca-412f-bce9-f9484c29a40d=93526b28-85ca-412f-bce9-f9484c29a40d,1-93526b28-85ca-412f-bce9-f9484c29a40d=93526b28-85ca-412f-bce9-f9484c29a40d] are applied to backups [customuser-otjq] [2024-01-18T17:02:25.754Z] INFO[2024-01-18 17:02:24] Actual:true, Expected: true [2024-01-18T17:02:25.754Z] 2024-01-18 17:02:24 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validate restoring manual backup of applications with namespace label [2024-01-18T17:02:25.754Z] 2024/01/18 17:02:24 DoRetryWithTimeout - Error: {restore status for [tp-restore-customuser-otjq] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T17:02:57.778Z] 2024-01-18 17:02:54 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.CreateRestore:#936] - Restore [tp-restore-customuser-otjq] created successfully [2024-01-18T17:02:57.779Z] 2024-01-18 17:02:54 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switching cluster context to cluster [destination-cluster] [2024-01-18T17:02:57.779Z] 2024-01-18 17:02:54 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switching context to [/tmp/destination-config] [2024-01-18T17:02:57.779Z] 2024-01-18 17:02:54 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:02:57.779Z] 2024-01-18 17:02:54 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:02:57.779Z] 2024-01-18 17:02:55 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:02:57.779Z] 2024-01-18 17:02:55 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:02:57.779Z] 2024-01-18 17:02:55 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:02:57.779Z] 2024-01-18 17:02:55 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:02:57.779Z] 2024-01-18 17:02:55 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:02:57.779Z] 2024-01-18 17:02:55 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:02:57.779Z] 2024-01-18 17:02:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:02:57.779Z] 2024-01-18 17:02:56 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T17:02:57.779Z] 2024-01-18 17:02:56 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T17:02:57.779Z] 2024-01-18 17:02:56 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T17:02:57.779Z] 2024-01-18 17:02:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T17:02:57.779Z] 2024-01-18 17:02:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switched context to [/tmp/destination-config] [2024-01-18T17:02:57.780Z] 2024-01-18 17:02:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-87886-01-18-16h38m08s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-18T17:02:57.780Z] 2024-01-18 17:02:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:02:57.780Z] 2024-01-18 17:02:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T17:02:57.780Z] 2024-01-18 17:02:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T17:02:57.780Z] 2024-01-18 17:02:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating restore [tp-restore-customuser-otjq] in org [default] [2024-01-18T17:02:57.780Z] 2024-01-18 17:02:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-restore-customuser-otjq] [2024-01-18T17:02:57.780Z] 2024-01-18 17:02:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-restore-customuser-otjq] [2024-01-18T17:02:57.780Z] 2024-01-18 17:02:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in restore [tp-restore-customuser-otjq] [2024-01-18T17:02:57.780Z] 2024-01-18 17:02:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-87886-01-18-16h38m08s]) was found in restore [tp-restore-customuser-otjq], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:02:57.780Z] 2024-01-18 17:02:57 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-87886-01-18-16h38m08s]) found in the restore [tp-restore-customuser-otjq] was also present on the cluster/namespace [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:02:57.780Z] 2024-01-18 17:02:57 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-87886-01-18-16h38m08s]) was found in restore [tp-restore-customuser-otjq], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:02:57.780Z] 2024-01-18 17:02:57 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-87886-01-18-16h38m08s]) found in the restore [tp-restore-customuser-otjq] was also present on the cluster/namespace [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:02:57.781Z] 2024-01-18 17:02:57 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-87886-01-18-16h38m08s]) was found in restore [tp-restore-customuser-otjq], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:02:57.781Z] 2024-01-18 17:02:57 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-87886-01-18-16h38m08s]) found in the restore [tp-restore-customuser-otjq] was also present on the cluster/namespace [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:02:57.781Z] 2024-01-18 17:02:57 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-87886-01-18-16h38m08s]) was found in restore [tp-restore-customuser-otjq], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:02:57.781Z] 2024-01-18 17:02:57 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-87886-01-18-16h38m08s]) found in the restore [tp-restore-customuser-otjq] was also present on the cluster/namespace [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:02:57.781Z] 2024-01-18 17:02:57 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in restore [tp-restore-customuser-otjq] [2024-01-18T17:02:57.781Z] 2024-01-18 17:02:57 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in restore [tp-restore-customuser-otjq] [2024-01-18T17:02:57.781Z] 2024-01-18 17:02:57 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:02:57.781Z] 2024-01-18 17:02:57 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:02:57.781Z] 2024-01-18 17:02:57 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[mysql-data]] [2024-01-18T17:02:57.781Z] 2024-01-18 17:02:57 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2243] - in restore [tp-restore-customuser-otjq], the status of the restored volume [pvc-08811e3e-a792-4522-bacc-c14fe6aeb7de] was not Success. It was [Retained] with reason [Skipped from volume restore as policy is set to Retain and pvc already exists] [2024-01-18T17:02:57.781Z] 2024-01-18 17:02:57 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validate applications in restored namespace [mysql-ibm-pxb-0-87886-01-18-16h38m08s] due to restore [tp-restore-customuser-otjq] [2024-01-18T17:02:57.781Z] 2024-01-18 17:02:57 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating mysql-ibm app [2024-01-18T17:02:57.781Z] 2024-01-18 17:02:57 +0000:[INFO] [{VerifyRBACForInfraAdmin}] wait for mysql-ibm app to start running [2024-01-18T17:02:57.781Z] 2024-01-18 17:02:57 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T17:02:57.781Z] 2024-01-18 17:02:57 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T17:02:58.040Z] 2024-01-18 17:02:57 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T17:02:58.040Z] 2024-01-18 17:02:58 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating CR cleanup [2024-01-18T17:02:58.297Z] 2024-01-18 17:02:58 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T17:02:58.555Z] 2024-01-18 17:02:58 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[]] [2024-01-18T17:02:58.555Z] 2024-01-18 17:02:58 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switching cluster context back to cluster path [/tmp/source-config] [2024-01-18T17:02:58.555Z] 2024-01-18 17:02:58 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switching context to [/tmp/source-config] [2024-01-18T17:02:58.555Z] 2024-01-18 17:02:58 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:02:58.555Z] 2024-01-18 17:02:58 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:02:58.555Z] 2024-01-18 17:02:58 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:02:58.555Z] 2024-01-18 17:02:58 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:02:58.812Z] 2024-01-18 17:02:58 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:02:58.812Z] 2024-01-18 17:02:58 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:02:59.376Z] 2024-01-18 17:02:59 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:02:59.632Z] 2024-01-18 17:02:59 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:03:00.192Z] 2024-01-18 17:02:59 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:03:00.192Z] 2024-01-18 17:03:00 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T17:03:00.448Z] 2024-01-18 17:03:00 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T17:03:00.448Z] 2024-01-18 17:03:00 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T17:03:00.448Z] 2024-01-18 17:03:00 +0000:[INFO] [{VerifyRBACForInfraAdmin}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T17:03:00.448Z] 2024-01-18 17:03:00 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switched context to [/tmp/source-config] [2024-01-18T17:03:00.448Z] INFO[2024-01-18 17:03:00] Verifying : Description : Verifying restoration of backup customuser-otjq [2024-01-18T17:03:00.448Z] INFO[2024-01-18 17:03:00] Actual:true, Expected: true [2024-01-18T17:03:00.448Z] 2024-01-18 17:03:00 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validate creating scheduled backup with namespace label for custom user [2024-01-18T17:03:01.012Z] 2024-01-18 17:03:00 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-18T17:03:01.012Z] 2024-01-18 17:03:00 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Volume snapshot classes found in the cluster - [2024-01-18T17:03:01.012Z] 2024-01-18 17:03:00 +0000:[INFO] [{VerifyRBACForInfraAdmin}] ibmc-vpcblock-snapshot [2024-01-18T17:03:01.012Z] 2024-01-18 17:03:00 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T17:04:08.627Z] 2024-01-18 17:04:01 +0000:[INFO] [{VerifyRBACForInfraAdmin}] first schedule backup for schedule name [tp-backup-tses] is [tp-backup-tses-6604692-interval-2024-01-18-170301] [2024-01-18T17:04:08.627Z] 2024/01/18 17:04:01 DoRetryWithTimeout - Error: {backup status for [tp-backup-tses-6604692-interval-2024-01-18-170301] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T17:04:08.627Z] INFO[2024-01-18 17:04:05] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T17:04:08.627Z] INFO[2024-01-18 17:04:05] sched stats: tick interval: 5405 < 1.2s (100%) [2024-01-18T17:04:08.627Z] INFO[2024-01-18 17:04:05] sched stats: tick process durations: 5404 < 500ms (100%) [2024-01-18T17:04:35.170Z] 2024-01-18 17:04:31 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating backup [tp-backup-tses-6604692-interval-2024-01-18-170301] in org [default] [2024-01-18T17:04:35.170Z] 2024-01-18 17:04:31 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-tses-6604692-interval-2024-01-18-170301] [2024-01-18T17:04:35.170Z] 2024-01-18 17:04:31 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in backup [tp-backup-tses-6604692-interval-2024-01-18-170301] [2024-01-18T17:04:35.171Z] 2024-01-18 17:04:31 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] in backup [tp-backup-tses-6604692-interval-2024-01-18-170301] [2024-01-18T17:04:35.171Z] 2024-01-18 17:04:31 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:04:35.171Z] 2024-01-18 17:04:31 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:04:35.171Z] 2024-01-18 17:04:31 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:04:35.171Z] 2024-01-18 17:04:31 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[mysql-data]] [2024-01-18T17:04:35.171Z] 2024-01-18 17:04:31 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T17:04:35.171Z] 2024-01-18 17:04:31 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating CR cleanup [2024-01-18T17:04:35.171Z] 2024-01-18 17:04:31 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T17:04:35.171Z] 2024-01-18 17:04:31 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [mysql-ibm-pxb-0-87886-01-18-16h38m08s] are [[]] [2024-01-18T17:04:35.171Z] INFO[2024-01-18 17:04:31] Verifying : Description : Verification of creating first schedule backup tp-backup-tses with labels [0-93526b28-85ca-412f-bce9-f9484c29a40d=93526b28-85ca-412f-bce9-f9484c29a40d,1-93526b28-85ca-412f-bce9-f9484c29a40d=93526b28-85ca-412f-bce9-f9484c29a40d] [2024-01-18T17:04:35.171Z] INFO[2024-01-18 17:04:31] Actual:true, Expected: true [2024-01-18T17:04:35.171Z] INFO[2024-01-18 17:04:32] Verifying : Description : Suspending Backup Schedule [tp-backup-tses] for user [testuser1-1705597117] [2024-01-18T17:04:35.171Z] INFO[2024-01-18 17:04:32] Actual:true, Expected: true [2024-01-18T17:04:35.171Z] 2024-01-18 17:04:32 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup tp-backup-tses-6604692-interval-2024-01-18-170301 [2024-01-18T17:04:35.171Z] 2024-01-18 17:04:32 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:04:35.171Z] 2024-01-18 17:04:32 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are 0-93526b28-85ca-412f-bce9-f9484c29a40d=93526b28-85ca-412f-bce9-f9484c29a40d,1-93526b28-85ca-412f-bce9-f9484c29a40d=93526b28-85ca-412f-bce9-f9484c29a40d [2024-01-18T17:04:35.171Z] INFO[2024-01-18 17:04:32] Verifying : Description : Verifying if the labeled namespaces [[mysql-ibm-pxb-0-87886-01-18-16h38m08s]] are backed up, and check if labels [0-93526b28-85ca-412f-bce9-f9484c29a40d=93526b28-85ca-412f-bce9-f9484c29a40d,1-93526b28-85ca-412f-bce9-f9484c29a40d=93526b28-85ca-412f-bce9-f9484c29a40d] are applied to backups [tp-backup-tses-6604692-interval-2024-01-18-170301] [2024-01-18T17:04:35.171Z] INFO[2024-01-18 17:04:32] Actual:true, Expected: true [2024-01-18T17:04:35.171Z] 2024-01-18 17:04:32 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validate restoring the scheduled backup with namespace label [2024-01-18T17:04:35.171Z] INFO[2024-01-18 17:04:32] Verifying : Description : Fetching namespaces [mysql-ibm-pxb-0-87886-01-18-16h38m08s] from backup tp-backup-tses-6604692-interval-2024-01-18-170301 [2024-01-18T17:04:35.172Z] INFO[2024-01-18 17:04:32] Actual:true, Expected: true [2024-01-18T17:04:35.172Z] 2024/01/18 17:04:32 DoRetryWithTimeout - Error: {restore status for [customrestore-adjh] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T17:05:07.226Z] 2024-01-18 17:05:02 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.CreateRestore:#936] - Restore [customrestore-adjh] created successfully [2024-01-18T17:05:07.226Z] 2024-01-18 17:05:02 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switching cluster context to cluster [destination-cluster] [2024-01-18T17:05:07.226Z] 2024-01-18 17:05:02 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switching context to [/tmp/destination-config] [2024-01-18T17:05:07.226Z] 2024-01-18 17:05:02 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:05:07.226Z] 2024-01-18 17:05:02 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:05:07.226Z] 2024-01-18 17:05:02 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:05:07.226Z] 2024-01-18 17:05:02 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:05:07.226Z] 2024-01-18 17:05:02 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:05:07.226Z] 2024-01-18 17:05:03 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:05:07.226Z] 2024-01-18 17:05:03 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:05:07.226Z] 2024-01-18 17:05:03 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:05:07.226Z] 2024-01-18 17:05:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:05:07.226Z] 2024-01-18 17:05:04 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T17:05:07.226Z] 2024-01-18 17:05:04 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T17:05:07.226Z] 2024-01-18 17:05:04 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T17:05:07.226Z] 2024-01-18 17:05:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T17:05:07.226Z] 2024-01-18 17:05:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switched context to [/tmp/destination-config] [2024-01-18T17:05:07.226Z] 2024-01-18 17:05:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-87886-01-18-16h38m08s] with namespace mapping [map[mysql-ibm-pxb-0-87886-01-18-16h38m08s:tp-backup-tses-6604692-interval-2024-01-18-170301-tdz]] and storage Class Mapping [map[]] [2024-01-18T17:05:07.227Z] 2024-01-18 17:05:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:05:07.227Z] 2024-01-18 17:05:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T17:05:07.227Z] 2024-01-18 17:05:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T17:05:07.227Z] 2024-01-18 17:05:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating restore [customrestore-adjh] in org [default] [2024-01-18T17:05:07.227Z] 2024-01-18 17:05:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [customrestore-adjh] [2024-01-18T17:05:07.227Z] 2024-01-18 17:05:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2110] - Checking status of restore [customrestore-adjh] [2024-01-18T17:05:07.227Z] 2024-01-18 17:05:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating specs for the namespace (restoredAppContext) [tp-backup-tses-6604692-interval-2024-01-18-170301-tdz] in restore [customrestore-adjh] [2024-01-18T17:05:07.227Z] 2024-01-18 17:05:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [tp-backup-tses-6604692-interval-2024-01-18-170301-tdz]) was found in restore [customrestore-adjh], as expected by presence in expectedRestoredAppContext [tp-backup-tses-6604692-interval-2024-01-18-170301-tdz] [2024-01-18T17:05:07.227Z] 2024-01-18 17:05:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [tp-backup-tses-6604692-interval-2024-01-18-170301-tdz]) found in the restore [customrestore-adjh] was also present on the cluster/namespace [tp-backup-tses-6604692-interval-2024-01-18-170301-tdz] [2024-01-18T17:05:07.227Z] 2024-01-18 17:05:04 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [tp-backup-tses-6604692-interval-2024-01-18-170301-tdz]) was found in restore [customrestore-adjh], as expected by presence in expectedRestoredAppContext [tp-backup-tses-6604692-interval-2024-01-18-170301-tdz] [2024-01-18T17:05:07.227Z] 2024-01-18 17:05:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [tp-backup-tses-6604692-interval-2024-01-18-170301-tdz]) found in the restore [customrestore-adjh] was also present on the cluster/namespace [tp-backup-tses-6604692-interval-2024-01-18-170301-tdz] [2024-01-18T17:05:07.227Z] 2024-01-18 17:05:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [tp-backup-tses-6604692-interval-2024-01-18-170301-tdz]) was found in restore [customrestore-adjh], as expected by presence in expectedRestoredAppContext [tp-backup-tses-6604692-interval-2024-01-18-170301-tdz] [2024-01-18T17:05:07.227Z] 2024-01-18 17:05:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [tp-backup-tses-6604692-interval-2024-01-18-170301-tdz]) found in the restore [customrestore-adjh] was also present on the cluster/namespace [tp-backup-tses-6604692-interval-2024-01-18-170301-tdz] [2024-01-18T17:05:07.227Z] 2024-01-18 17:05:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [tp-backup-tses-6604692-interval-2024-01-18-170301-tdz]) was found in restore [customrestore-adjh], as expected by presence in expectedRestoredAppContext [tp-backup-tses-6604692-interval-2024-01-18-170301-tdz] [2024-01-18T17:05:07.227Z] 2024-01-18 17:05:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [tp-backup-tses-6604692-interval-2024-01-18-170301-tdz]) found in the restore [customrestore-adjh] was also present on the cluster/namespace [tp-backup-tses-6604692-interval-2024-01-18-170301-tdz] [2024-01-18T17:05:07.228Z] 2024-01-18 17:05:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating Restored Volumes for the namespace (restoredAppContext) [tp-backup-tses-6604692-interval-2024-01-18-170301-tdz] in restore [customrestore-adjh] [2024-01-18T17:05:07.228Z] 2024-01-18 17:05:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [tp-backup-tses-6604692-interval-2024-01-18-170301-tdz] in restore [customrestore-adjh] [2024-01-18T17:05:07.228Z] 2024-01-18 17:05:05 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:05:07.228Z] 2024-01-18 17:05:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:05:07.228Z] 2024-01-18 17:05:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [tp-backup-tses-6604692-interval-2024-01-18-170301-tdz] are [[mysql-data]] [2024-01-18T17:05:07.228Z] 2024-01-18 17:05:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.ValidateRestore:#2240] - in restore [customrestore-adjh], the status of the restored volume [pvc-bfbe9aa1-d48e-4284-b4ad-7c5d202e7250] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-18T17:05:07.228Z] 2024-01-18 17:05:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validate applications in restored namespace [tp-backup-tses-6604692-interval-2024-01-18-170301-tdz] due to restore [customrestore-adjh] [2024-01-18T17:05:07.228Z] 2024-01-18 17:05:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating mysql-ibm app [2024-01-18T17:05:07.228Z] 2024-01-18 17:05:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] wait for mysql-ibm app to start running [2024-01-18T17:05:07.228Z] 2024-01-18 17:05:05 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T17:05:07.228Z] 2024/01/18 17:05:05 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:05:07.228Z] pod name:mysql-7d766cd65c-mvxlv namespace:tp-backup-tses-6604692-interval-2024-01-18-170301-tdz running:false ready:false node:10.240.64.6 [2024-01-18T17:05:07.228Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:05:17.169Z] 2024/01/18 17:05:15 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:05:17.169Z] pod name:mysql-7d766cd65c-mvxlv namespace:tp-backup-tses-6604692-interval-2024-01-18-170301-tdz running:false ready:false node:10.240.64.6 [2024-01-18T17:05:17.169Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:05:27.108Z] 2024/01/18 17:05:25 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:05:27.108Z] pod name:mysql-7d766cd65c-mvxlv namespace:tp-backup-tses-6604692-interval-2024-01-18-170301-tdz running:false ready:false node:10.240.64.6 [2024-01-18T17:05:27.108Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:05:37.061Z] 2024/01/18 17:05:35 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:05:37.061Z] pod name:mysql-7d766cd65c-mvxlv namespace:tp-backup-tses-6604692-interval-2024-01-18-170301-tdz running:false ready:false node:10.240.64.6 [2024-01-18T17:05:37.061Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:05:47.005Z] 2024/01/18 17:05:45 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:05:47.005Z] pod name:mysql-7d766cd65c-mvxlv namespace:tp-backup-tses-6604692-interval-2024-01-18-170301-tdz running:true ready:false node:10.240.64.6 [2024-01-18T17:05:47.005Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:05:56.953Z] 2024/01/18 17:05:55 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:05:56.953Z] pod name:mysql-7d766cd65c-mvxlv namespace:tp-backup-tses-6604692-interval-2024-01-18-170301-tdz running:true ready:false node:10.240.64.6 [2024-01-18T17:05:56.953Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:06:06.896Z] 2024/01/18 17:06:06 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:06:06.896Z] pod name:mysql-7d766cd65c-mvxlv namespace:tp-backup-tses-6604692-interval-2024-01-18-170301-tdz running:true ready:false node:10.240.64.6 [2024-01-18T17:06:06.896Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:06:16.857Z] 2024/01/18 17:06:16 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:06:16.857Z] pod name:mysql-7d766cd65c-mvxlv namespace:tp-backup-tses-6604692-interval-2024-01-18-170301-tdz running:true ready:false node:10.240.64.6 [2024-01-18T17:06:16.857Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:06:26.806Z] 2024/01/18 17:06:26 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:06:26.806Z] pod name:mysql-7d766cd65c-mvxlv namespace:tp-backup-tses-6604692-interval-2024-01-18-170301-tdz running:true ready:false node:10.240.64.6 [2024-01-18T17:06:26.806Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:06:36.746Z] 2024/01/18 17:06:36 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:06:36.746Z] pod name:mysql-7d766cd65c-mvxlv namespace:tp-backup-tses-6604692-interval-2024-01-18-170301-tdz running:true ready:false node:10.240.64.6 [2024-01-18T17:06:36.746Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:06:46.716Z] 2024/01/18 17:06:46 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:06:46.716Z] pod name:mysql-7d766cd65c-mvxlv namespace:tp-backup-tses-6604692-interval-2024-01-18-170301-tdz running:true ready:false node:10.240.64.6 [2024-01-18T17:06:46.716Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:06:56.696Z] 2024-01-18 17:06:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T17:06:56.696Z] 2024-01-18 17:06:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T17:06:56.696Z] 2024-01-18 17:06:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validating CR cleanup [2024-01-18T17:06:56.696Z] 2024-01-18 17:06:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T17:06:56.952Z] 2024-01-18 17:06:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] All CRs in [tp-backup-tses-6604692-interval-2024-01-18-170301-tdz] are [[]] [2024-01-18T17:06:56.952Z] 2024-01-18 17:06:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switching cluster context back to cluster path [/tmp/source-config] [2024-01-18T17:06:56.952Z] 2024-01-18 17:06:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switching context to [/tmp/source-config] [2024-01-18T17:06:56.952Z] 2024-01-18 17:06:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:06:56.952Z] 2024-01-18 17:06:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:06:57.208Z] 2024-01-18 17:06:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:06:57.208Z] 2024-01-18 17:06:57 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:06:57.209Z] 2024-01-18 17:06:57 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:06:57.209Z] 2024-01-18 17:06:57 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:06:57.770Z] 2024-01-18 17:06:57 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:06:58.025Z] 2024-01-18 17:06:57 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:06:58.587Z] 2024-01-18 17:06:58 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:06:58.587Z] 2024-01-18 17:06:58 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T17:06:58.842Z] 2024-01-18 17:06:58 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T17:06:58.842Z] 2024-01-18 17:06:58 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T17:06:58.842Z] 2024-01-18 17:06:58 +0000:[INFO] [{VerifyRBACForInfraAdmin}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T17:06:58.842Z] 2024-01-18 17:06:58 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switched context to [/tmp/source-config] [2024-01-18T17:06:58.843Z] INFO[2024-01-18 17:06:58] Verifying : Description : Verifying multiple backup restore [customrestore-adjh] in custom namespace [map[mysql-ibm-pxb-0-87886-01-18-16h38m08s:tp-backup-tses-6604692-interval-2024-01-18-170301-tdz]] [2024-01-18T17:06:58.843Z] INFO[2024-01-18 17:06:58] Actual:true, Expected: true [2024-01-18T17:06:58.843Z] 2024-01-18 17:06:58 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Validate deleting namespace labelled backups and restores from the custom user testuser1-1705597117 user context [2024-01-18T17:06:59.404Z] 2024/01/18 17:06:59 DoRetryWithTimeout - Error: {backupObject [tp-backup-tses-6604692-interval-2024-01-18-170301] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-18T17:06:59.404Z] 2024/01/18 17:06:59 DoRetryWithTimeout - Error: {backupObject [customuser-otjq] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-18T17:07:31.429Z] INFO[2024-01-18 17:07:29] Verifying : Description : Getting the backup driver [2024-01-18T17:07:31.429Z] INFO[2024-01-18 17:07:29] Actual:true, Expected: true [2024-01-18T17:07:31.429Z] INFO[2024-01-18 17:07:29] Verifying : Description : Verifying the deletion of the restore named [tp-restore-customuser-otjq] [2024-01-18T17:07:31.429Z] INFO[2024-01-18 17:07:29] Actual:true, Expected: true [2024-01-18T17:07:31.429Z] INFO[2024-01-18 17:07:29] Verifying : Description : Getting the backup driver [2024-01-18T17:07:31.429Z] INFO[2024-01-18 17:07:29] Actual:true, Expected: true [2024-01-18T17:07:31.429Z] INFO[2024-01-18 17:07:29] Verifying : Description : Verifying the deletion of the restore named [customrestore-adjh] [2024-01-18T17:07:31.429Z] INFO[2024-01-18 17:07:29] Actual:true, Expected: true [2024-01-18T17:07:31.429Z] 2024-01-18 17:07:29 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Delete custom user testuser1-1705597117 backup schedule [2024-01-18T17:07:31.429Z] 2024/01/18 17:07:29 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-tses] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T17:07:41.375Z] INFO[2024-01-18 17:07:39] Verifying : Description : Deleting Backup Schedule [tp-backup-tses] for user [testuser1-1705597117] [2024-01-18T17:07:41.375Z] INFO[2024-01-18 17:07:39] Actual:true, Expected: true [2024-01-18T17:07:41.375Z] 2024-01-18 17:07:39 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Deleting user testuser1-1705597117 backups from the user context [2024-01-18T17:07:41.375Z] 2024/01/18 17:07:39 DoRetryWithTimeout - Error: {backupObject [tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-18T17:08:13.409Z] 2024/01/18 17:08:09 DoRetryWithTimeout - Error: {backupObject [tp-backup-manual-single-ns-testuser1-1705597117-with-rules-yhty] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-18T17:08:39.904Z] 2024-01-18 17:08:39 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Deleting user testuser1-1705597117 restores from the user context [2024-01-18T17:08:40.159Z] INFO[2024-01-18 17:08:39] Verifying : Description : Getting the backup driver [2024-01-18T17:08:40.159Z] INFO[2024-01-18 17:08:39] Actual:true, Expected: true [2024-01-18T17:08:40.159Z] 2024-01-18 17:08:39 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Deleting user testuser1-1705597117 source and destination cluster from the user context [2024-01-18T17:08:40.415Z] INFO[2024-01-18 17:08:40] Verifying : Description : Verifying deletion of cluster [source-cluster] of the user testuser1-1705597117 [2024-01-18T17:08:40.415Z] INFO[2024-01-18 17:08:40] Actual:true, Expected: true [2024-01-18T17:08:40.671Z] INFO[2024-01-18 17:08:40] Verifying : Description : Verifying deletion of cluster [destination-cluster] of the user testuser1-1705597117 [2024-01-18T17:08:40.671Z] INFO[2024-01-18 17:08:40] Actual:true, Expected: true [2024-01-18T17:08:40.671Z] 2024-01-18 17:08:40 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Verify px-admin group user can delete RBAC resources created by customUser [testuser1-1705597117] [2024-01-18T17:08:40.928Z] 2024-01-18 17:08:40 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.glob..func43.2.41:#683] - Verify deletion of backup location [autogenerated-backup-location-csaxc] of user [testuser1-1705597117] from px-admin [2024-01-18T17:08:42.296Z] INFO[2024-01-18 17:08:41] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-csaxc] of the user testuser1-1705597117 [2024-01-18T17:08:42.296Z] INFO[2024-01-18 17:08:41] Actual:true, Expected: true [2024-01-18T17:08:42.296Z] 2024/01/18 17:08:41 DoRetryWithTimeout - Error: {backup location [autogenerated-backup-location-csaxc] is in [DeletePending] state}, Next try in [30s], timeout [1h0m0s] [2024-01-18T17:09:14.321Z] 2024/01/18 17:09:12 DoRetryWithTimeout - Error: {backup location [autogenerated-backup-location-csaxc] is in [DeletePending] state}, Next try in [30s], timeout [1h0m0s] [2024-01-18T17:09:46.352Z] INFO[2024-01-18 17:09:42] Verifying : Description : Verifying waiting for backup location [autogenerated-backup-location-csaxc] deletion of the user testuser1-1705597117 [2024-01-18T17:09:46.352Z] INFO[2024-01-18 17:09:42] Actual:true, Expected: true [2024-01-18T17:09:46.352Z] 2024-01-18 17:09:42 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.glob..func43.2.41:#688] - Verify deletion of schedule policy [periodic-xhovs] of user [testuser1-1705597117] from px-admin [2024-01-18T17:09:46.352Z] INFO[2024-01-18 17:09:42] Verifying : Description : Verifying deletion of schedule policy [periodic-xhovs] of the user testuser1-1705597117 [2024-01-18T17:09:46.352Z] INFO[2024-01-18 17:09:42] Actual:true, Expected: true [2024-01-18T17:09:46.352Z] 2024-01-18 17:09:42 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.glob..func43.2.41:#691] - Verify deletion of rules of user [testuser1-1705597117] from px-admin [2024-01-18T17:09:46.352Z] INFO[2024-01-18 17:09:42] Verifying : Description : Verifying deletion of rule [pre-rule-jbjjj] of the user testuser1-1705597117 [2024-01-18T17:09:46.352Z] INFO[2024-01-18 17:09:42] Actual:true, Expected: true [2024-01-18T17:09:46.352Z] INFO[2024-01-18 17:09:42] Verifying : Description : Verifying deletion of rule [post-rule-defbb] of the user testuser1-1705597117 [2024-01-18T17:09:46.352Z] INFO[2024-01-18 17:09:42] Actual:true, Expected: true [2024-01-18T17:09:46.352Z] 2024-01-18 17:09:42 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.glob..func43.2.41:#699] - Verify deletion of cloud credential [autogenerated-cred-cifdg] of user [testuser1-1705597117] from px-admin [2024-01-18T17:09:46.352Z] INFO[2024-01-18 17:09:43] Verifying : Description : Verifying deletion of cloud credential [autogenerated-cred-cifdg] of the user testuser1-1705597117 [2024-01-18T17:09:46.352Z] INFO[2024-01-18 17:09:43] Actual:true, Expected: true [2024-01-18T17:09:46.352Z] 2024-01-18 17:09:43 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Verify infra-admin user [testuser1-1705595980] can delete RBAC resources created [2024-01-18T17:09:46.352Z] 2024-01-18 17:09:43 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.glob..func43.2.42:#710] - Verify deletion of schedule policy [periodic-oyizj] of user [testuser1-1705595980] [2024-01-18T17:09:46.352Z] INFO[2024-01-18 17:09:43] Verifying : Description : Verifying deletion of schedule policy [periodic-oyizj] of the user testuser1-1705595980 [2024-01-18T17:09:46.352Z] INFO[2024-01-18 17:09:43] Actual:true, Expected: true [2024-01-18T17:09:46.352Z] 2024-01-18 17:09:43 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.glob..func43.2.42:#713] - Verify deletion of rules of user [testuser1-1705595980] [2024-01-18T17:09:46.352Z] INFO[2024-01-18 17:09:43] Verifying : Description : Verifying deletion of rule [pre-rule-jdrjo] of the user testuser1-1705595980 [2024-01-18T17:09:46.352Z] INFO[2024-01-18 17:09:43] Actual:true, Expected: true [2024-01-18T17:09:46.352Z] INFO[2024-01-18 17:09:43] Verifying : Description : Verifying deletion of rule [post-rule-lenid] of the user testuser1-1705595980 [2024-01-18T17:09:46.352Z] INFO[2024-01-18 17:09:43] Actual:true, Expected: true [2024-01-18T17:09:46.352Z] 2024-01-18 17:09:43 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.glob..func43.2.42:#719] - Verify deletion of role [custom-infra-admin-role-rscg] created by user [testuser1-1705595980] [2024-01-18T17:09:46.352Z] INFO[2024-01-18 17:09:45] Verifying : Description : Verifying deletion of role [custom-infra-admin-role-rscg] created by user testuser1-1705595980 [2024-01-18T17:09:46.352Z] INFO[2024-01-18 17:09:45] Actual:true, Expected: true [2024-01-18T17:09:46.609Z] 2024-01-18 17:09:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Deleting the deployed apps after the testcase [2024-01-18T17:09:46.609Z] 2024-01-18 17:09:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Destroying apps [2024-01-18T17:09:46.609Z] 2024-01-18 17:09:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T17:09:47.182Z] 2024-01-18 17:09:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T17:09:47.182Z] 2024-01-18 17:09:46 +0000:[INFO] [{VerifyRBACForInfraAdmin}] start destroying mysql-ibm app [2024-01-18T17:09:47.182Z] 2024-01-18 17:09:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T17:09:47.438Z] 2024-01-18 17:09:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T17:09:47.438Z] 2024-01-18 17:09:47 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T17:09:47.438Z] INFO[2024-01-18 17:09:47] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T17:09:47.438Z] INFO[2024-01-18 17:09:47] Actual:true, Expected: true [2024-01-18T17:09:47.438Z] 2024-01-18 17:09:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Deleting labels from namespaces - [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:09:47.695Z] INFO[2024-01-18 17:09:47] Verifying : Description : Deleting labels [map[0-0235ca01-2486-49e1-ba1d-6858b4c6d492:0235ca01-2486-49e1-ba1d-6858b4c6d492 1-0235ca01-2486-49e1-ba1d-6858b4c6d492:0235ca01-2486-49e1-ba1d-6858b4c6d492]] from namespaces [[mysql-ibm-pxb-0-87886-01-18-16h38m08s]] [2024-01-18T17:09:47.695Z] INFO[2024-01-18 17:09:47] Actual:true, Expected: true [2024-01-18T17:09:47.695Z] 2024-01-18 17:09:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Deleting the px-backup objects [2024-01-18T17:09:47.695Z] 2024-01-18 17:09:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Cleaning backup locations in map [map[34518b95-86fb-40f8-a45d-df227357b0e1:autogenerated-backup-location-mpzou]], cloud credential [autogenerated-cred-lvxlx], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T17:09:47.695Z] INFO[2024-01-18 17:09:47] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-mpzou] [2024-01-18T17:09:47.695Z] INFO[2024-01-18 17:09:47] Actual:true, Expected: true [2024-01-18T17:09:47.695Z] 2024-01-18 17:09:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-mpzou] is present [2024-01-18T17:09:47.696Z] 2024/01/18 17:09:47 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-mpzou is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T17:10:19.721Z] 2024-01-18 17:10:17 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-mpzou] is present [2024-01-18T17:10:19.721Z] 2024/01/18 17:10:17 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-mpzou is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T17:10:51.755Z] 2024-01-18 17:10:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [] [2024-01-18T17:10:51.755Z] INFO[2024-01-18 17:10:47] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-mpzou [2024-01-18T17:10:51.755Z] INFO[2024-01-18 17:10:47] Actual:true, Expected: true [2024-01-18T17:10:51.755Z] 2024-01-18 17:10:47 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-lvxlx] is present [2024-01-18T17:10:51.755Z] INFO[2024-01-18 17:10:47] Verifying : Description : Verifying if cloud cred [autogenerated-cred-lvxlx] is present [2024-01-18T17:10:51.755Z] INFO[2024-01-18 17:10:47] Actual:true, Expected: true [2024-01-18T17:10:51.755Z] INFO[2024-01-18 17:10:47] Verifying : Description : Verifying deletion of cloud cred [autogenerated-cred-lvxlx] [2024-01-18T17:10:51.755Z] INFO[2024-01-18 17:10:47] Actual:true, Expected: true [2024-01-18T17:10:51.755Z] INFO[2024-01-18 17:10:47] Verifying : Description : Deleting cloud cred autogenerated-cred-lvxlx [2024-01-18T17:10:51.755Z] INFO[2024-01-18 17:10:47] Actual:true, Expected: true [2024-01-18T17:10:51.755Z] INFO[2024-01-18 17:10:47] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T17:10:51.755Z] INFO[2024-01-18 17:10:47] Actual:true, Expected: true [2024-01-18T17:10:51.755Z] INFO[2024-01-18 17:10:48] Verifying : Description : Deleting cluster source-cluster [2024-01-18T17:10:51.755Z] INFO[2024-01-18 17:10:48] Actual:true, Expected: true [2024-01-18T17:10:51.755Z] INFO[2024-01-18 17:10:48] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-htmnt] [2024-01-18T17:10:51.755Z] INFO[2024-01-18 17:10:48] Actual:true, Expected: true [2024-01-18T17:10:51.755Z] INFO[2024-01-18 17:10:48] Verifying : Description : Deleting cluster destination-cluster [2024-01-18T17:10:51.755Z] INFO[2024-01-18 17:10:48] Actual:true, Expected: true [2024-01-18T17:10:51.755Z] INFO[2024-01-18 17:10:48] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-btcfu] [2024-01-18T17:10:51.756Z] INFO[2024-01-18 17:10:48] Actual:true, Expected: true [2024-01-18T17:10:51.756Z] 2024-01-18 17:10:48 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switching context to destination cluster for clean up [2024-01-18T17:10:51.756Z] 2024-01-18 17:10:48 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T17:10:51.756Z] 2024-01-18 17:10:48 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switching context to [/tmp/destination-config] [2024-01-18T17:10:51.756Z] 2024-01-18 17:10:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:10:51.756Z] 2024-01-18 17:10:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:10:51.756Z] 2024-01-18 17:10:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:10:51.756Z] 2024-01-18 17:10:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:10:51.756Z] 2024-01-18 17:10:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:10:51.756Z] 2024-01-18 17:10:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:10:51.756Z] 2024-01-18 17:10:49 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:10:51.756Z] 2024-01-18 17:10:50 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:10:51.756Z] 2024-01-18 17:10:50 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:10:51.756Z] 2024-01-18 17:10:50 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T17:10:51.756Z] 2024-01-18 17:10:51 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T17:10:51.756Z] 2024-01-18 17:10:51 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T17:10:51.756Z] 2024-01-18 17:10:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T17:10:51.756Z] 2024-01-18 17:10:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switched context to [/tmp/destination-config] [2024-01-18T17:10:51.756Z] 2024-01-18 17:10:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Destroying apps [2024-01-18T17:10:51.756Z] 2024-01-18 17:10:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T17:10:51.756Z] 2024-01-18 17:10:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T17:10:51.756Z] 2024-01-18 17:10:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] start destroying mysql-ibm app [2024-01-18T17:10:51.756Z] 2024-01-18 17:10:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T17:10:51.756Z] 2024-01-18 17:10:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T17:10:51.756Z] 2024-01-18 17:10:51 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T17:10:51.756Z] INFO[2024-01-18 17:10:51] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T17:10:51.756Z] INFO[2024-01-18 17:10:51] Actual:true, Expected: true [2024-01-18T17:10:51.757Z] 2024-01-18 17:10:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switching back context to Source cluster [2024-01-18T17:10:51.757Z] 2024-01-18 17:10:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T17:10:51.757Z] 2024-01-18 17:10:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switching context to [/tmp/source-config] [2024-01-18T17:10:52.013Z] 2024-01-18 17:10:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:10:52.013Z] 2024-01-18 17:10:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:10:52.013Z] 2024-01-18 17:10:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:10:52.013Z] 2024-01-18 17:10:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:10:52.013Z] 2024-01-18 17:10:51 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:10:52.269Z] 2024-01-18 17:10:52 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:10:52.830Z] 2024-01-18 17:10:52 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:10:53.086Z] 2024-01-18 17:10:52 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:10:53.342Z] 2024-01-18 17:10:53 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:10:53.597Z] 2024-01-18 17:10:53 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T17:10:53.853Z] 2024-01-18 17:10:53 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T17:10:53.853Z] 2024-01-18 17:10:53 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T17:10:53.853Z] 2024-01-18 17:10:53 +0000:[INFO] [{VerifyRBACForInfraAdmin}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T17:10:53.853Z] 2024-01-18 17:10:53 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switched context to [/tmp/source-config] [2024-01-18T17:10:53.853Z] INFO[2024-01-18 17:10:53] --------Test End------ [2024-01-18T17:10:53.853Z] INFO[2024-01-18 17:10:53] #Test: [2024-01-18T17:10:53.853Z] INFO[2024-01-18 17:10:53] #Description: [2024-01-18T17:10:53.853Z] INFO[2024-01-18 17:10:53] ------------------------ [2024-01-18T17:10:54.109Z] 2024-01-18 17:10:53 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:10:54.364Z] 2024-01-18 17:10:54 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T17:10:54.364Z] 2024-01-18 17:10:54 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switching context to [/tmp/destination-config] [2024-01-18T17:10:54.364Z] 2024-01-18 17:10:54 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:10:54.364Z] 2024-01-18 17:10:54 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:10:54.364Z] 2024-01-18 17:10:54 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:10:54.364Z] 2024-01-18 17:10:54 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:10:54.364Z] 2024-01-18 17:10:54 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:10:54.620Z] 2024-01-18 17:10:54 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:10:55.180Z] 2024-01-18 17:10:54 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:10:55.436Z] 2024-01-18 17:10:55 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:10:55.996Z] 2024-01-18 17:10:55 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:10:55.996Z] 2024-01-18 17:10:55 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T17:10:56.253Z] 2024-01-18 17:10:56 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T17:10:56.253Z] 2024-01-18 17:10:56 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T17:10:56.253Z] 2024-01-18 17:10:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T17:10:56.253Z] 2024-01-18 17:10:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switched context to [/tmp/destination-config] [2024-01-18T17:10:56.509Z] 2024-01-18 17:10:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-87886-01-18-16h38m08s] [2024-01-18T17:10:56.766Z] 2024-01-18 17:10:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {VerifyRBACForInfraAdmin} Validates the RBAC operation for infra-admin user [2024-01-18T17:10:56.766Z] 2024-01-18 17:10:56 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-18T17:10:56.767Z] 2024/01/18 17:10:56 DoRetryWithTimeout - Error: {dial tcp 10.240.128.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T17:11:02.008Z] 2024/01/18 17:11:01 DoRetryWithTimeout - Error: {dial tcp 10.240.128.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T17:11:07.252Z] 2024/01/18 17:11:06 DoRetryWithTimeout - Error: {dial tcp 10.240.128.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T17:11:12.534Z] 2024/01/18 17:11:11 DoRetryWithTimeout - Error: {dial tcp 10.240.128.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T17:11:16.698Z] 2024/01/18 17:11:16 DoRetryWithTimeout - Error: {dial tcp 10.240.128.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T17:11:21.950Z] 2024/01/18 17:11:21 DoRetryWithTimeout - Error: {dial tcp 10.240.128.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T17:11:27.199Z] 2024/01/18 17:11:26 DoRetryWithTimeout - Error: {dial tcp 10.240.128.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T17:11:32.442Z] 2024/01/18 17:11:31 DoRetryWithTimeout - Error: {dial tcp 10.240.128.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T17:11:36.613Z] 2024/01/18 17:11:36 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.128.4. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.4 10.240.128.4]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T17:11:36.613Z] {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T17:11:36.613Z] {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T17:11:36.613Z] {dial tcp 10.240.128.4:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-18T17:11:41.874Z] 2024-01-18 17:11:41 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.4. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.4 10.240.128.4]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T17:11:41.874Z] {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T17:11:41.874Z] {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T17:11:41.874Z] {dial tcp 10.240.128.4:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T17:11:41.874Z] 2024-01-18 17:11:41 +0000:[ERROR] [{VerifyRBACForInfraAdmin}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.4. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.4 10.240.128.4]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T17:11:41.874Z] {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T17:11:41.874Z] {dial tcp 10.240.128.4:22: connect: connection refused} [2024-01-18T17:11:41.874Z] {dial tcp 10.240.128.4:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T17:11:41.875Z] 2024-01-18 17:11:41 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T17:11:41.875Z] 2024-01-18 17:11:41 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switching context to [/tmp/source-config] [2024-01-18T17:11:41.875Z] 2024-01-18 17:11:41 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:11:41.875Z] 2024-01-18 17:11:41 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:11:41.875Z] 2024-01-18 17:11:41 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:11:41.875Z] 2024-01-18 17:11:41 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:11:41.875Z] 2024-01-18 17:11:41 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:11:42.131Z] 2024-01-18 17:11:41 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:11:42.692Z] 2024-01-18 17:11:42 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:11:42.949Z] 2024-01-18 17:11:42 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:11:43.205Z] 2024-01-18 17:11:43 +0000:[INFO] [{VerifyRBACForInfraAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:11:43.461Z] 2024-01-18 17:11:43 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T17:11:43.717Z] 2024-01-18 17:11:43 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T17:11:43.717Z] 2024-01-18 17:11:43 +0000:[WARNING] [{VerifyRBACForInfraAdmin}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T17:11:43.717Z] 2024-01-18 17:11:43 +0000:[INFO] [{VerifyRBACForInfraAdmin}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T17:11:43.717Z] 2024-01-18 17:11:43 +0000:[INFO] [{VerifyRBACForInfraAdmin}] Switched context to [/tmp/source-config] [2024-01-18T17:11:43.717Z] [2024-01-18T17:11:43.717Z] ------------------------------ [2024-01-18T17:11:43.717Z] • [SLOW TEST:2015.037 seconds] [2024-01-18T17:11:43.717Z] {VerifyRBACForInfraAdmin} [2024-01-18T17:11:43.718Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_rbac_test.go:19 [2024-01-18T17:11:43.718Z] Validates the RBAC operation for infra-admin user [2024-01-18T17:11:43.718Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_rbac_test.go:85 [2024-01-18T17:11:43.718Z] ------------------------------ [2024-01-18T17:11:43.718Z] 2024-01-18 17:11:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Log Dir: /testresults//ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit.log [2024-01-18T17:11:43.718Z] INFO[2024-01-18 17:11:43] --------Test Start------ [2024-01-18T17:11:43.718Z] INFO[2024-01-18 17:11:43] #Test: ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit [2024-01-18T17:11:43.718Z] INFO[2024-01-18 17:11:43] #Description: Manual, schedule backup and restore of single, multiple and all namespaces with namespace labeled as max character limit [2024-01-18T17:11:43.718Z] INFO[2024-01-18 17:11:43] ------------------------ [2024-01-18T17:11:43.718Z] 2024-01-18 17:11:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Deploy applications [2024-01-18T17:11:43.718Z] 2024-01-18 17:11:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T17:11:43.718Z] 2024-01-18 17:11:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T17:11:43.976Z] 2024-01-18 17:11:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T17:11:43.976Z] 2024-01-18 17:11:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T17:11:44.233Z] 2024-01-18 17:11:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T17:11:44.233Z] 2024-01-18 17:11:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T17:11:44.234Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T17:11:44.490Z] 2024-01-18 17:11:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T17:11:44.491Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T17:11:44.491Z] 2024-01-18 17:11:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T17:11:44.491Z] 2024-01-18 17:11:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T17:11:44.491Z] 2024-01-18 17:11:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T17:11:44.491Z] 2024-01-18 17:11:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T17:11:44.491Z] 2024-01-18 17:11:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T17:11:45.054Z] 2024-01-18 17:11:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T17:11:45.054Z] 2024-01-18 17:11:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T17:11:45.055Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T17:11:45.055Z] 2024-01-18 17:11:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T17:11:45.055Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T17:11:45.055Z] 2024-01-18 17:11:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T17:11:45.055Z] 2024-01-18 17:11:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T17:11:45.055Z] 2024-01-18 17:11:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T17:11:45.055Z] 2024-01-18 17:11:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T17:11:45.055Z] 2024-01-18 17:11:45 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T17:11:45.311Z] 2024-01-18 17:11:45 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T17:11:45.568Z] 2024-01-18 17:11:45 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T17:11:45.569Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T17:11:45.569Z] 2024-01-18 17:11:45 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T17:11:45.569Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T17:11:45.569Z] 2024-01-18 17:11:45 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T17:11:45.569Z] 2024-01-18 17:11:45 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Created namespaces [mysql-ibm-pxb-0-84853-01-18-17h11m43s mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:11:45.569Z] 2024-01-18 17:11:45 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validate applications [2024-01-18T17:11:45.569Z] 2024-01-18 17:11:45 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validate applications [2024-01-18T17:11:45.569Z] 2024-01-18 17:11:45 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating mysql-ibm app [2024-01-18T17:11:45.570Z] 2024-01-18 17:11:45 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating mysql-ibm app's volumes [2024-01-18T17:11:45.826Z] 2024-01-18 17:11:45 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:11:45.826Z] 2024/01/18 17:11:45 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T17:11:58.092Z] 2024-01-18 17:11:55 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-84853-01-18-17h11m43s [2024-01-18T17:11:58.092Z] 2024-01-18 17:11:55 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T17:11:58.092Z] 2024-01-18 17:11:55 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T17:11:58.092Z] 2024-01-18 17:11:55 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] wait for mysql-ibm app to start running [2024-01-18T17:11:58.092Z] 2024-01-18 17:11:55 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T17:11:58.092Z] 2024/01/18 17:11:56 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:11:58.092Z] pod name:mysql-7d766cd65c-lb2dx namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s running:true ready:false node: [2024-01-18T17:11:58.092Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:12:06.193Z] 2024/01/18 17:12:06 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:12:06.193Z] pod name:mysql-7d766cd65c-lb2dx namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s running:false ready:false node:10.240.64.9 [2024-01-18T17:12:06.193Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:12:18.373Z] 2024/01/18 17:12:16 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:12:18.373Z] pod name:mysql-7d766cd65c-lb2dx namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s running:true ready:false node:10.240.64.9 [2024-01-18T17:12:18.373Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:12:26.472Z] 2024/01/18 17:12:26 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:12:26.472Z] pod name:mysql-7d766cd65c-lb2dx namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s running:true ready:false node:10.240.64.9 [2024-01-18T17:12:26.472Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:12:38.649Z] 2024/01/18 17:12:36 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:12:38.649Z] pod name:mysql-7d766cd65c-lb2dx namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s running:true ready:false node:10.240.64.9 [2024-01-18T17:12:38.649Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:12:46.738Z] 2024/01/18 17:12:46 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:12:46.738Z] pod name:mysql-7d766cd65c-lb2dx namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s running:true ready:false node:10.240.64.9 [2024-01-18T17:12:46.738Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:12:56.685Z] 2024/01/18 17:12:56 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:12:56.685Z] pod name:mysql-7d766cd65c-lb2dx namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s running:true ready:false node:10.240.64.9 [2024-01-18T17:12:56.685Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:13:08.856Z] 2024/01/18 17:13:06 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:13:08.856Z] pod name:mysql-7d766cd65c-lb2dx namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s running:true ready:false node:10.240.64.9 [2024-01-18T17:13:08.857Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:13:16.942Z] 2024/01/18 17:13:16 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:13:16.942Z] pod name:mysql-7d766cd65c-lb2dx namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s running:true ready:false node:10.240.64.9 [2024-01-18T17:13:16.942Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:13:29.121Z] 2024-01-18 17:13:26 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T17:13:29.121Z] 2024-01-18 17:13:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T17:13:29.121Z] 2024-01-18 17:13:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] validate if mysql-ibm app's volumes are setup [2024-01-18T17:13:29.122Z] 2024-01-18 17:13:27 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:13:29.122Z] 2024-01-18 17:13:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:13:29.122Z] 2024-01-18 17:13:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T17:13:29.122Z] 2024-01-18 17:13:27 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T17:13:29.122Z] 2024-01-18 17:13:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating mysql-ibm app [2024-01-18T17:13:29.122Z] 2024-01-18 17:13:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating mysql-ibm app's volumes [2024-01-18T17:13:29.122Z] 2024-01-18 17:13:27 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:13:29.122Z] 2024-01-18 17:13:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:13:29.122Z] 2024-01-18 17:13:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-1-84853-01-18-17h11m43s [2024-01-18T17:13:29.122Z] 2024-01-18 17:13:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T17:13:29.122Z] 2024-01-18 17:13:27 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T17:13:29.122Z] 2024-01-18 17:13:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] wait for mysql-ibm app to start running [2024-01-18T17:13:29.122Z] 2024-01-18 17:13:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T17:13:29.122Z] 2024-01-18 17:13:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T17:13:29.122Z] 2024-01-18 17:13:28 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T17:13:29.122Z] 2024-01-18 17:13:28 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] validate if mysql-ibm app's volumes are setup [2024-01-18T17:13:29.122Z] 2024-01-18 17:13:28 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:13:29.122Z] 2024-01-18 17:13:28 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:13:29.122Z] 2024-01-18 17:13:28 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T17:13:29.122Z] 2024-01-18 17:13:28 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T17:13:29.122Z] 2024-01-18 17:13:28 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating mysql-ibm app [2024-01-18T17:13:29.122Z] 2024-01-18 17:13:28 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating mysql-ibm app's volumes [2024-01-18T17:13:29.123Z] 2024-01-18 17:13:28 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:13:29.123Z] 2024-01-18 17:13:28 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:13:29.123Z] 2024-01-18 17:13:29 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-2-84853-01-18-17h11m43s [2024-01-18T17:13:29.123Z] 2024-01-18 17:13:29 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T17:13:29.381Z] 2024-01-18 17:13:29 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T17:13:29.381Z] 2024-01-18 17:13:29 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] wait for mysql-ibm app to start running [2024-01-18T17:13:29.381Z] 2024-01-18 17:13:29 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T17:13:29.637Z] 2024-01-18 17:13:29 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T17:13:30.200Z] 2024-01-18 17:13:30 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T17:13:31.128Z] 2024-01-18 17:13:30 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] validate if mysql-ibm app's volumes are setup [2024-01-18T17:13:31.128Z] 2024-01-18 17:13:30 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:13:31.128Z] 2024-01-18 17:13:30 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:13:31.128Z] 2024-01-18 17:13:30 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T17:13:31.128Z] 2024-01-18 17:13:30 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T17:13:31.128Z] 2024-01-18 17:13:30 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Adding labels to resources [2024-01-18T17:13:31.128Z] INFO[2024-01-18 17:13:30] Verifying : Description : Fetching pvc list from namespace [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:13:31.128Z] INFO[2024-01-18 17:13:30] Actual:true, Expected: true [2024-01-18T17:13:31.129Z] INFO[2024-01-18 17:13:30] Verifying : Description : Fetching pvc pointer &PersistentVolumeClaim{ObjectMeta:{mysql-data mysql-ibm-pxb-0-84853-01-18-17h11m43s ef9dfb2b-b7e3-469f-8112-7ba5f3bca496 1286915 0 2024-01-18 17:11:43 +0000 UTC map[] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-18 17:11:43 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-18 17:11:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-18 17:11:54 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{2147483648 0} {} 2Gi BinarySI},},},VolumeName:pvc-ef9dfb2b-b7e3-469f-8112-7ba5f3bca496,Selector:nil,StorageClassName:*mysql-sc,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{10737418240 0} {} 10Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},} for pvc {{ } {mysql-data mysql-ibm-pxb-0-84853-01-18-17h11m43s ef9dfb2b-b7e3-469f-8112-7ba5f3bca496 1286915 0 2024-01-18 17:11:43 +0000 UTC map[] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-18 17:11:43 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-18 17:11:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-18 17:11:54 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{2147483648 0} {} 2Gi BinarySI}]} pvc-ef9dfb2b-b7e3-469f-8112-7ba5f3bca496 0xc00148e7c0 0xc00148e7d0 nil nil} {Bound [ReadWriteOnce] map[storage:{{10737418240 0} {} 10Gi BinarySI}] [] map[] }} [2024-01-18T17:13:31.129Z] INFO[2024-01-18 17:13:30] Actual:true, Expected: true [2024-01-18T17:13:31.129Z] 2024-01-18 17:13:30 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.AddLabelToResource:#5030] - Adding label [abe5cf54-4e74-45b4-882f-82b02f3dd9cf=cd7d2c21-9adb-41bf-b01f-ae218113bcc9] to PVC mysql-data [2024-01-18T17:13:31.129Z] INFO[2024-01-18 17:13:31] Verifying : Description : Adding labels abe5cf54-4e74-45b4-882f-82b02f3dd9cf=cd7d2c21-9adb-41bf-b01f-ae218113bcc9 to resource &PersistentVolumeClaim{ObjectMeta:{mysql-data mysql-ibm-pxb-0-84853-01-18-17h11m43s ef9dfb2b-b7e3-469f-8112-7ba5f3bca496 1286915 0 2024-01-18 17:11:43 +0000 UTC map[abe5cf54-4e74-45b4-882f-82b02f3dd9cf:cd7d2c21-9adb-41bf-b01f-ae218113bcc9] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-18 17:11:43 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-18 17:11:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-18 17:11:54 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{2147483648 0} {} 2Gi BinarySI},},},VolumeName:pvc-ef9dfb2b-b7e3-469f-8112-7ba5f3bca496,Selector:nil,StorageClassName:*mysql-sc,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{10737418240 0} {} 10Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},} [2024-01-18T17:13:31.129Z] INFO[2024-01-18 17:13:31] Actual:true, Expected: true [2024-01-18T17:13:31.129Z] INFO[2024-01-18 17:13:31] Verifying : Description : Fetching configmap list from namespace [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:13:31.129Z] INFO[2024-01-18 17:13:31] Actual:true, Expected: true [2024-01-18T17:13:31.387Z] INFO[2024-01-18 17:13:31] Verifying : Description : Fetching configmap pointer &ConfigMap{ObjectMeta:{kube-****-ca.crt mysql-ibm-pxb-0-84853-01-18-17h11m43s f271815a-afb4-492d-bcc1-aec45aab135a 1286617 0 2024-01-18 17:11:43 +0000 UTC map[] map[kubernetes.io/description:Contains a CA bundle that can be used to verify the kube-apiserver when using internal endpoints such as the internal service IP or kubernetes.default.svc. No other usage is guaranteed across distributions of Kubernetes clusters.] [] [] [{kube-controller-manager Update v1 2024-01-18 17:11:43 +0000 UTC FieldsV1 {"f:data":{".":{},"f:ca.crt":{}},"f:metadata":{"f:annotations":{".":{},"f:kubernetes.io/description":{}}}} }]},Data:map[string]string{ca.crt: -----BEGIN CERTIFICATE----- [2024-01-18T17:13:31.387Z] MIIDCjCCAfKgAwIBAgIUWkn6rgBaC+VWV1XdJjGA3DwoRRYwDQYJKoZIhvcNAQEL [2024-01-18T17:13:31.387Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-18T17:13:31.387Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-18T17:13:31.387Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-18T17:13:31.387Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-18T17:13:31.387Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-18T17:13:31.387Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-18T17:13:31.387Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-18T17:13:31.387Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-18T17:13:31.387Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-18T17:13:31.387Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAHbD [2024-01-18T17:13:31.387Z] 4nWTgt/8sat7SbbHxZ+JcMGRAIaeLAjMJxkxyqD8xUq6YQcgbjDVHo1Qbh3PXB3T [2024-01-18T17:13:31.387Z] 5BWUsRbDoKi0eRJx/5cyI3846SYEAOpuUfJrg9X2CH7ZoYhxeJ42dqt3MdoMJEM1 [2024-01-18T17:13:31.387Z] k3e2rRqAKuAxwNXugHryxQM+1ehOJjrEuT5DVK6qu8+JYRcT1hBZb6hrebKPp5eG [2024-01-18T17:13:31.387Z] JvBzHP4YOATIAXhfH0Uj5Y+JhwoOdpXmOFpz+6gITPD4aNzzJZe/0ADSFfOyUJaR [2024-01-18T17:13:31.387Z] g5NnJnkeGqMM3SLsur25r+m3y0i3t+RdocZ/4mHTEnF4KWVWvZy4GjENiEBcXEol [2024-01-18T17:13:31.387Z] o4SZVaP4AF7XQJ5vdA0= [2024-01-18T17:13:31.387Z] -----END CERTIFICATE----- [2024-01-18T17:13:31.387Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:31.387Z] MIIDCjCCAfKgAwIBAgIUXhIIFCc/NxZJn6hNEFoZuPQ8N5UwDQYJKoZIhvcNAQEL [2024-01-18T17:13:31.387Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-18T17:13:31.387Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-18T17:13:31.387Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-18T17:13:31.387Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-18T17:13:31.387Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-18T17:13:31.387Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-18T17:13:31.387Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-18T17:13:31.387Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-18T17:13:31.387Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-18T17:13:31.387Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAKG3 [2024-01-18T17:13:31.387Z] nVfv4IkgHVOxgbrNIoYhjvBvVJcLZub0C9yidexLA3A61p1a7b/OpidG5zShPhH2 [2024-01-18T17:13:31.387Z] HRdQcqgp0HIxaS+/oLD3eElOnnC/EKD359K6n3kN66sMgvNbHxNFM84Au0TXqmbT [2024-01-18T17:13:31.387Z] w8afrKjrfe/VglpHC2GfmgHMF4YH+ZF48C1RZA6z2oXj6Aj+uww0PjiClE9qUYqK [2024-01-18T17:13:31.387Z] R9uRa32prExihsi+Bto7n2Axo92qUiK2KrkEdQitnQfH5z/PojS78vbyIKp1O86N [2024-01-18T17:13:31.387Z] qRImvCYY17CPxP+JUQ7dxWDMF4NTgSnabkp+cd5i/pyoAb0guQFlQQUnHAZa4QNC [2024-01-18T17:13:31.387Z] 7fgTo1IUwWyDmwCnN5U= [2024-01-18T17:13:31.387Z] -----END CERTIFICATE----- [2024-01-18T17:13:31.387Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:31.387Z] MIIDCjCCAfKgAwIBAgIUJPzO3O+wmUt4gHCkl3ojsvwZL6owDQYJKoZIhvcNAQEL [2024-01-18T17:13:31.387Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-18T17:13:31.387Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-18T17:13:31.387Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-18T17:13:31.387Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-18T17:13:31.387Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-18T17:13:31.387Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-18T17:13:31.387Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-18T17:13:31.387Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-18T17:13:31.387Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-18T17:13:31.387Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAGVt [2024-01-18T17:13:31.387Z] g9qX+r8kyGYsvFavmnDxgMK1Zkz80irQJ2BJlDHdK1hp5aqpRbT2z2YgLog5ZMIZ [2024-01-18T17:13:31.387Z] uI36uPCOXo8+Gol1jT7D6vsgMWm0Tk9jCNbO4jPkv+wm2q1B9UZo94UGiS0K+G0S [2024-01-18T17:13:31.388Z] nccl4jmSq1gWpBZIVetdDFNjpipKjhEOVQM7x33APhACxSAWRp9iEr6V/CzMOgkf [2024-01-18T17:13:31.388Z] RHyz5D7ljh4a2XVH7++eKsj2QPfKyiKtIG72tCccQ9zGnzveiXXb4Ai8r+DyKH4Z [2024-01-18T17:13:31.388Z] zDt5Hb4ZFl72E3Rl4bIh/VQBVsr9u/SFR1p/Emyq+kGXsn+9YMiM9tK4mLUtjdox [2024-01-18T17:13:31.388Z] LMWUWTelNRGWAiaTVz8= [2024-01-18T17:13:31.388Z] -----END CERTIFICATE----- [2024-01-18T17:13:31.388Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:31.388Z] MIIDCjCCAfKgAwIBAgIUBxbEoan/HnRZ+2deRj+2RvGMrWQwDQYJKoZIhvcNAQEL [2024-01-18T17:13:31.388Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-18T17:13:31.388Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-18T17:13:31.388Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-18T17:13:31.388Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-18T17:13:31.388Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-18T17:13:31.388Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-18T17:13:31.388Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-18T17:13:31.388Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-18T17:13:31.388Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-18T17:13:31.388Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAB65 [2024-01-18T17:13:31.388Z] nHU+98oTP0U2J0/TsL5GFeBTH7YyUXUNRJvPWxkA8riB3swoScFKbavO6XJ/hrI2 [2024-01-18T17:13:31.388Z] ywApOz77Iyq58OLa5aIC5aaPRf3YE8LbJQYIHmtza21LXhtlKVSrLu4W58PauI8Z [2024-01-18T17:13:31.388Z] 7iaL9QPUihHFK7wrZ2ZpabN2rRJvTGWWKlhmgFTjI+71gPrXWmYBHIKmTv55V5m1 [2024-01-18T17:13:31.388Z] LxORTR/gBXeH9gmM7wz1cbaYNazqs8R1LS5SW76p/y9RLjTzU6N39VYefMMGwbsQ [2024-01-18T17:13:31.388Z] DlD11wvMBxTBp3hQYc3pdVLkVEQt8trVIwEUts7zfwUTCu/z8gH7H5XePpEVNana [2024-01-18T17:13:31.388Z] JhlAcXHCO7nN/ddJmoc= [2024-01-18T17:13:31.388Z] -----END CERTIFICATE----- [2024-01-18T17:13:31.388Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:31.388Z] MIIDGDCCAgCgAwIBAgIUEvLaarpXlv1cVvG6nMuxh2DNCKQwDQYJKoZIhvcNAQEL [2024-01-18T17:13:31.388Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-18T17:13:31.388Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-18T17:13:31.388Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-18T17:13:31.388Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-18T17:13:31.388Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-18T17:13:31.388Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-18T17:13:31.388Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-18T17:13:31.388Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-18T17:13:31.388Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-18T17:13:31.388Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-18T17:13:31.388Z] 9w0BAQsFAAOCAQEAOPakzVUmAqHhmQViZZHSQeRh/xYQBrtn6dnEo6X55SXNWcPU [2024-01-18T17:13:31.388Z] 4EEW3DvwBf75uqxKeGoXHTBpqNHG2x1+IncvIKJmi5OWM3aR/vEyJoBOchPBl+A4 [2024-01-18T17:13:31.388Z] 0xZO1vKiS2djxz8XHMkR70cxD8hv/zDJeIQbgBroF4Wa7rim5+y7jgO0O7Rs1zhT [2024-01-18T17:13:31.388Z] LDDStJ8aUKwcVRmOv7NFQxbQHXNx07kaUE2SQJfhryzZmotoBiymmQT2v0H4/4q0 [2024-01-18T17:13:31.388Z] 5dylRhSCWfwN5+It40/C3b1pbC2d9Iv/eO+B0wqg13dLF3xuA1JblEgnqA9mdNnp [2024-01-18T17:13:31.388Z] P6EgKTQBMiO6nPFyYIPqg1eWYkb1eXu9aEkmbw== [2024-01-18T17:13:31.388Z] -----END CERTIFICATE----- [2024-01-18T17:13:31.388Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:31.388Z] MIIDGDCCAgCgAwIBAgIUWf/qoBANFWF/a/yZr7wfLBMfItAwDQYJKoZIhvcNAQEL [2024-01-18T17:13:31.388Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-18T17:13:31.388Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-18T17:13:31.388Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-18T17:13:31.388Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-18T17:13:31.388Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-18T17:13:31.388Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-18T17:13:31.388Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-18T17:13:31.388Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-18T17:13:31.388Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-18T17:13:31.388Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-18T17:13:31.388Z] 9w0BAQsFAAOCAQEAdCguadZU6WJAuuAtWzvYgftPKkDKNBeGVgbqRwWRCB+QqVFl [2024-01-18T17:13:31.388Z] +3ZOQTStpnXJsY3Ig31hmL88BkBPQ47ENEzs7Czhuz59m+dcBhh/buUoIERSnfUY [2024-01-18T17:13:31.388Z] lWIfqxQevwHYZ3Kjx0l3qliYjKBQJo5xDPnukGb4yUnmrCdZ3fxMzlnr0Ais9rUH [2024-01-18T17:13:31.388Z] MjiDsjP/oUa8naQUTZO4DTRZqeEldzNXnxF6YH+3moTkwRLt9xzirNwECmJ9qxrC [2024-01-18T17:13:31.389Z] eCA7zORxbFSftYLnQiGWUS1dnxBuLOvJHjYPC6EkPPrvqnx2fXcTE1V8GizVRrA6 [2024-01-18T17:13:31.389Z] NPxt+e4qZgU/Nk/Jkpsb7h1FVcKtS8CvmtbdqQ== [2024-01-18T17:13:31.389Z] -----END CERTIFICATE----- [2024-01-18T17:13:31.389Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:31.389Z] MIIDGDCCAgCgAwIBAgIUI6D0WS4tl52N+w2lwAS+6V0DPyUwDQYJKoZIhvcNAQEL [2024-01-18T17:13:31.389Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-18T17:13:31.389Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-18T17:13:31.389Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-18T17:13:31.389Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-18T17:13:31.389Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-18T17:13:31.389Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-18T17:13:31.389Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-18T17:13:31.389Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-18T17:13:31.389Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-18T17:13:31.389Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-18T17:13:31.389Z] 9w0BAQsFAAOCAQEAAS4yElh6TMyP18ipzRA0Wenukevjs/vtkAFOsv1u5GNeDIrv [2024-01-18T17:13:31.389Z] lNi6/VjvtS+DoFNzQnTf3QoO6GTuoWHLQWt6Z/V5AwIsnOXduSXLugMwEv3j9l50 [2024-01-18T17:13:31.389Z] aDkA7vJ3OH5cGqIvIzxQN7IUOEvZNhY67GFP7+zAzVNRpJ1FHYbhQGPuFD//b9G3 [2024-01-18T17:13:31.389Z] hr4E3+iUEc+F/iBA+caA8wRx1zR383Jz6wuoFt7n7HpWNcgMvoh91JlUAK7yUp0B [2024-01-18T17:13:31.389Z] ekuG6q7HtVsk6X+aB1RyWfrSJjih75W7b+p+/bV4vNmQS1wSGSOaSIuVRvMhmpwf [2024-01-18T17:13:31.389Z] FpT+4rUpiCxklD69/MET/1cWxqvao/ppf6phzg== [2024-01-18T17:13:31.389Z] -----END CERTIFICATE----- [2024-01-18T17:13:31.389Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:31.389Z] MIIDGDCCAgCgAwIBAgIUIVCCgYV9enDck18n89YsA5b2IdgwDQYJKoZIhvcNAQEL [2024-01-18T17:13:31.389Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-18T17:13:31.389Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-18T17:13:31.389Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-18T17:13:31.389Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-18T17:13:31.389Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-18T17:13:31.389Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-18T17:13:31.389Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-18T17:13:31.389Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-18T17:13:31.389Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-18T17:13:31.389Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-18T17:13:31.389Z] 9w0BAQsFAAOCAQEAa3eFm1ivFf5Vom6a/7E1gDdjl7cjUZ5cqg332Ky1NdzEOiwW [2024-01-18T17:13:31.389Z] mx3yEk35qCd8LAoFMvPvPIuMx1NDS8kwy3FmIOjKnZapHA31ZUpn/gT/seH4GFIU [2024-01-18T17:13:31.389Z] oZw+rODV5h6jmtCVugAn48u/LxOXbY417JWwaeDx5uw+NzOSyebe+td2PbtUjf5X [2024-01-18T17:13:31.389Z] 6xcwYUmaUGsIAgfIMM12hzXOI2/q1qz5dAqwSgnR7dKBReOXnUoFpNOMpxlJ70p/ [2024-01-18T17:13:31.389Z] WUU1aWIWZqKrNI7Pg1T5jljq7+KbT3+A6+ux0FLExr2c1tw3NndXS2Y9fO55m5/q [2024-01-18T17:13:31.389Z] 0EcrTdKx4+QjbWcja3WdX2pGDk0XGNgx+EXn6Q== [2024-01-18T17:13:31.389Z] -----END CERTIFICATE----- [2024-01-18T17:13:31.389Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:31.389Z] MIIFATCCAumgAwIBAgIRAKc9ZKBASymy5TLOEp57N98wDQYJKoZIhvcNAQELBQAw [2024-01-18T17:13:31.389Z] GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDMyMzIyNTM0NloXDTM2 [2024-01-18T17:13:31.389Z] MDMyMzIyNTM0NlowGjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMIICIjANBgkq [2024-01-18T17:13:31.389Z] hkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA+pYHvQw5iU3v2b3iNuYNKYgsWD6KU7aJ [2024-01-18T17:13:31.389Z] diddtZQxSWYzUI3U0I1UsRPTxnhTifs/M9NW4ZlV13ZfB7APwC8oqKOIiwo7IwlP [2024-01-18T17:13:31.389Z] xg0VKgyz+kT8RJfYr66PPIYP0fpTeu42LpMJ+CKo9sbpgVNDZN2z/qiXrRNX/VtG [2024-01-18T17:13:31.389Z] TkPV7a44fZ5bHHVruAxvDnylpQxJobtCBWlJSsbIRGFHMc2z88eUz9NmIOWUKGGj [2024-01-18T17:13:31.389Z] EmP76x8OfRHpIpuxRSCjn0+i9+hR2siIOpcMOGd+40uVJxbRRP5ZXnUFa2fF5FWd [2024-01-18T17:13:31.389Z] O0u0RPI8HON0ovhrwPJY+4eWKkQzyC611oLPYGQ4EbifRsTsCxUZqyUuStGyp8oa [2024-01-18T17:13:31.389Z] aoSKfF6X0+KzGgwwnrjRTUpIl19A92KR0Noo6h622OX+4sZiO/JQdkuX5w/HupK0 [2024-01-18T17:13:31.389Z] A0M0WSMCvU6GOhjGotmh2VTEJwHHY4+TUk0iQYRtv1crONklyZoAQPD76hCrC8Cr [2024-01-18T17:13:31.389Z] IbgsZLfTMC8TWUoMbyUDgvgYkHKMoPm0VGVVuwpRKJxv7+2wXO+pivrrUl2Q9fPe [2024-01-18T17:13:31.389Z] Kk055nJLMV9yPUdig8othUKrRfSxli946AEV1eEOhxddfEwBE3Lt2xn0hhiIedbb [2024-01-18T17:13:31.389Z] Ftf/5kEWFZkXyUmMJK8Ra76Kus2ABueUVEcZ48hrRr1Hf1N9n59VbTUaXgeiZA50 [2024-01-18T17:13:31.389Z] qXf2bymE6F8CAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMB [2024-01-18T17:13:31.389Z] Af8wHQYDVR0OBBYEFMEmdKSKRKDm+iAo2FwjmkWIGHngMA0GCSqGSIb3DQEBCwUA [2024-01-18T17:13:31.389Z] A4ICAQBCPw74M9X/Xx04K1VAES3ypgQYH5bf9FXVDrwhRFSVckria/7dMzoF5wln [2024-01-18T17:13:31.389Z] uq9NGsjkkkDg17AohcQdr8alH4LvPdxpKr3BjpvEcmbqF8xH+MbbeUEnmbSfLI8H [2024-01-18T17:13:31.390Z] sefuhXF9AF/9iYvpVNC8FmJ0OhiVv13VgMQw0CRKkbtjZBf8xaEhq/YqxWVsgOjm [2024-01-18T17:13:31.390Z] dm5CAQ2X0aX7502x8wYRgMnZhA5goC1zVWBVAi8yhhmlhhoDUfg17cXkmaJC5pDd [2024-01-18T17:13:31.390Z] oenZ9NVhW8eDb03MFCrWNvIh89DDeCGWuWfDltDq0n3owyL0IeSn7RfpSclpxVmV [2024-01-18T17:13:31.390Z] /53jkYjwIgxIG7Gsv0LKMbsf6QdBcTjhvfZyMIpBRkTe3zuHd2feKzY9lEkbRvRQ [2024-01-18T17:13:31.390Z] zbh4Ps5YBnG6CKJPTbe2hfi3nhnw/MyEmF3zb0hzvLWNrR9XW3ibb2oL3424XOwc [2024-01-18T17:13:31.390Z] VjrTSCLzO9Rv6s5wi03qoWvKAQQAElqTYRHhynJ3w6wuvKYF5zcZF3MDnrVGLbh1 [2024-01-18T17:13:31.390Z] Q9ePRFBCiXOQ6wPLoUhrrbZ8LpFUFYDXHMtYM7P9sc9IAWoONXREJaO08zgFtMp4 [2024-01-18T17:13:31.390Z] 8iyIYUyQAbsvx8oD2M8kRvrIRSrRJSl6L957b4AFiLIQ/GgV2curs0jje7Edx34c [2024-01-18T17:13:31.390Z] idWw1VrejtwclobqNMVtG3EiPUIpJGpbMcJgbiLSmKkrvQtGng== [2024-01-18T17:13:31.390Z] -----END CERTIFICATE----- [2024-01-18T17:13:31.390Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} from namespace mysql-ibm-pxb-0-84853-01-18-17h11m43s [2024-01-18T17:13:31.390Z] INFO[2024-01-18 17:13:31] Actual:true, Expected: true [2024-01-18T17:13:31.390Z] 2024-01-18 17:13:31 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.AddLabelToResource:#5038] - Adding label [abe5cf54-4e74-45b4-882f-82b02f3dd9cf=cd7d2c21-9adb-41bf-b01f-ae218113bcc9] to ConfigMap kube-****-ca.crt [2024-01-18T17:13:31.390Z] INFO[2024-01-18 17:13:31] Verifying : Description : Adding labels abe5cf54-4e74-45b4-882f-82b02f3dd9cf=cd7d2c21-9adb-41bf-b01f-ae218113bcc9 to resource &ConfigMap{ObjectMeta:{kube-****-ca.crt mysql-ibm-pxb-0-84853-01-18-17h11m43s f271815a-afb4-492d-bcc1-aec45aab135a 1286617 0 2024-01-18 17:11:43 +0000 UTC map[abe5cf54-4e74-45b4-882f-82b02f3dd9cf:cd7d2c21-9adb-41bf-b01f-ae218113bcc9] map[kubernetes.io/description:Contains a CA bundle that can be used to verify the kube-apiserver when using internal endpoints such as the internal service IP or kubernetes.default.svc. No other usage is guaranteed across distributions of Kubernetes clusters.] [] [] [{kube-controller-manager Update v1 2024-01-18 17:11:43 +0000 UTC FieldsV1 {"f:data":{".":{},"f:ca.crt":{}},"f:metadata":{"f:annotations":{".":{},"f:kubernetes.io/description":{}}}} }]},Data:map[string]string{ca.crt: -----BEGIN CERTIFICATE----- [2024-01-18T17:13:31.390Z] MIIDCjCCAfKgAwIBAgIUWkn6rgBaC+VWV1XdJjGA3DwoRRYwDQYJKoZIhvcNAQEL [2024-01-18T17:13:31.390Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-18T17:13:31.390Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-18T17:13:31.390Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-18T17:13:31.391Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-18T17:13:31.391Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-18T17:13:31.391Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-18T17:13:31.391Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-18T17:13:31.391Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-18T17:13:31.391Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-18T17:13:31.391Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAHbD [2024-01-18T17:13:31.391Z] 4nWTgt/8sat7SbbHxZ+JcMGRAIaeLAjMJxkxyqD8xUq6YQcgbjDVHo1Qbh3PXB3T [2024-01-18T17:13:31.391Z] 5BWUsRbDoKi0eRJx/5cyI3846SYEAOpuUfJrg9X2CH7ZoYhxeJ42dqt3MdoMJEM1 [2024-01-18T17:13:31.391Z] k3e2rRqAKuAxwNXugHryxQM+1ehOJjrEuT5DVK6qu8+JYRcT1hBZb6hrebKPp5eG [2024-01-18T17:13:31.391Z] JvBzHP4YOATIAXhfH0Uj5Y+JhwoOdpXmOFpz+6gITPD4aNzzJZe/0ADSFfOyUJaR [2024-01-18T17:13:31.391Z] g5NnJnkeGqMM3SLsur25r+m3y0i3t+RdocZ/4mHTEnF4KWVWvZy4GjENiEBcXEol [2024-01-18T17:13:31.391Z] o4SZVaP4AF7XQJ5vdA0= [2024-01-18T17:13:31.391Z] -----END CERTIFICATE----- [2024-01-18T17:13:31.391Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:31.391Z] MIIDCjCCAfKgAwIBAgIUXhIIFCc/NxZJn6hNEFoZuPQ8N5UwDQYJKoZIhvcNAQEL [2024-01-18T17:13:31.391Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-18T17:13:31.391Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-18T17:13:31.391Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-18T17:13:31.391Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-18T17:13:31.391Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-18T17:13:31.391Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-18T17:13:31.391Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-18T17:13:31.391Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-18T17:13:31.391Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-18T17:13:31.391Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAKG3 [2024-01-18T17:13:31.391Z] nVfv4IkgHVOxgbrNIoYhjvBvVJcLZub0C9yidexLA3A61p1a7b/OpidG5zShPhH2 [2024-01-18T17:13:31.391Z] HRdQcqgp0HIxaS+/oLD3eElOnnC/EKD359K6n3kN66sMgvNbHxNFM84Au0TXqmbT [2024-01-18T17:13:31.391Z] w8afrKjrfe/VglpHC2GfmgHMF4YH+ZF48C1RZA6z2oXj6Aj+uww0PjiClE9qUYqK [2024-01-18T17:13:31.391Z] R9uRa32prExihsi+Bto7n2Axo92qUiK2KrkEdQitnQfH5z/PojS78vbyIKp1O86N [2024-01-18T17:13:31.391Z] qRImvCYY17CPxP+JUQ7dxWDMF4NTgSnabkp+cd5i/pyoAb0guQFlQQUnHAZa4QNC [2024-01-18T17:13:31.391Z] 7fgTo1IUwWyDmwCnN5U= [2024-01-18T17:13:31.391Z] -----END CERTIFICATE----- [2024-01-18T17:13:31.391Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:31.391Z] MIIDCjCCAfKgAwIBAgIUJPzO3O+wmUt4gHCkl3ojsvwZL6owDQYJKoZIhvcNAQEL [2024-01-18T17:13:31.391Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-18T17:13:31.391Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-18T17:13:31.391Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-18T17:13:31.391Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-18T17:13:31.391Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-18T17:13:31.391Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-18T17:13:31.391Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-18T17:13:31.391Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-18T17:13:31.391Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-18T17:13:31.391Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAGVt [2024-01-18T17:13:31.391Z] g9qX+r8kyGYsvFavmnDxgMK1Zkz80irQJ2BJlDHdK1hp5aqpRbT2z2YgLog5ZMIZ [2024-01-18T17:13:31.391Z] uI36uPCOXo8+Gol1jT7D6vsgMWm0Tk9jCNbO4jPkv+wm2q1B9UZo94UGiS0K+G0S [2024-01-18T17:13:31.391Z] nccl4jmSq1gWpBZIVetdDFNjpipKjhEOVQM7x33APhACxSAWRp9iEr6V/CzMOgkf [2024-01-18T17:13:31.391Z] RHyz5D7ljh4a2XVH7++eKsj2QPfKyiKtIG72tCccQ9zGnzveiXXb4Ai8r+DyKH4Z [2024-01-18T17:13:31.391Z] zDt5Hb4ZFl72E3Rl4bIh/VQBVsr9u/SFR1p/Emyq+kGXsn+9YMiM9tK4mLUtjdox [2024-01-18T17:13:31.391Z] LMWUWTelNRGWAiaTVz8= [2024-01-18T17:13:31.391Z] -----END CERTIFICATE----- [2024-01-18T17:13:31.391Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:31.392Z] MIIDCjCCAfKgAwIBAgIUBxbEoan/HnRZ+2deRj+2RvGMrWQwDQYJKoZIhvcNAQEL [2024-01-18T17:13:31.392Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-18T17:13:31.392Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-18T17:13:31.392Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-18T17:13:31.392Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-18T17:13:31.392Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-18T17:13:31.392Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-18T17:13:31.392Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-18T17:13:31.392Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-18T17:13:31.392Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-18T17:13:31.392Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAB65 [2024-01-18T17:13:31.392Z] nHU+98oTP0U2J0/TsL5GFeBTH7YyUXUNRJvPWxkA8riB3swoScFKbavO6XJ/hrI2 [2024-01-18T17:13:31.392Z] ywApOz77Iyq58OLa5aIC5aaPRf3YE8LbJQYIHmtza21LXhtlKVSrLu4W58PauI8Z [2024-01-18T17:13:31.392Z] 7iaL9QPUihHFK7wrZ2ZpabN2rRJvTGWWKlhmgFTjI+71gPrXWmYBHIKmTv55V5m1 [2024-01-18T17:13:31.392Z] LxORTR/gBXeH9gmM7wz1cbaYNazqs8R1LS5SW76p/y9RLjTzU6N39VYefMMGwbsQ [2024-01-18T17:13:31.392Z] DlD11wvMBxTBp3hQYc3pdVLkVEQt8trVIwEUts7zfwUTCu/z8gH7H5XePpEVNana [2024-01-18T17:13:31.392Z] JhlAcXHCO7nN/ddJmoc= [2024-01-18T17:13:31.392Z] -----END CERTIFICATE----- [2024-01-18T17:13:31.392Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:31.392Z] MIIDGDCCAgCgAwIBAgIUEvLaarpXlv1cVvG6nMuxh2DNCKQwDQYJKoZIhvcNAQEL [2024-01-18T17:13:31.392Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-18T17:13:31.392Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-18T17:13:31.392Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-18T17:13:31.392Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-18T17:13:31.392Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-18T17:13:31.392Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-18T17:13:31.392Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-18T17:13:31.394Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-18T17:13:31.394Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-18T17:13:31.394Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-18T17:13:31.394Z] 9w0BAQsFAAOCAQEAOPakzVUmAqHhmQViZZHSQeRh/xYQBrtn6dnEo6X55SXNWcPU [2024-01-18T17:13:31.394Z] 4EEW3DvwBf75uqxKeGoXHTBpqNHG2x1+IncvIKJmi5OWM3aR/vEyJoBOchPBl+A4 [2024-01-18T17:13:31.394Z] 0xZO1vKiS2djxz8XHMkR70cxD8hv/zDJeIQbgBroF4Wa7rim5+y7jgO0O7Rs1zhT [2024-01-18T17:13:31.394Z] LDDStJ8aUKwcVRmOv7NFQxbQHXNx07kaUE2SQJfhryzZmotoBiymmQT2v0H4/4q0 [2024-01-18T17:13:31.394Z] 5dylRhSCWfwN5+It40/C3b1pbC2d9Iv/eO+B0wqg13dLF3xuA1JblEgnqA9mdNnp [2024-01-18T17:13:31.394Z] P6EgKTQBMiO6nPFyYIPqg1eWYkb1eXu9aEkmbw== [2024-01-18T17:13:31.394Z] -----END CERTIFICATE----- [2024-01-18T17:13:31.394Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:31.394Z] MIIDGDCCAgCgAwIBAgIUWf/qoBANFWF/a/yZr7wfLBMfItAwDQYJKoZIhvcNAQEL [2024-01-18T17:13:31.395Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-18T17:13:31.405Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-18T17:13:31.405Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-18T17:13:31.405Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-18T17:13:31.405Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-18T17:13:31.405Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-18T17:13:31.405Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-18T17:13:31.405Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-18T17:13:31.405Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-18T17:13:31.405Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-18T17:13:31.405Z] 9w0BAQsFAAOCAQEAdCguadZU6WJAuuAtWzvYgftPKkDKNBeGVgbqRwWRCB+QqVFl [2024-01-18T17:13:31.405Z] +3ZOQTStpnXJsY3Ig31hmL88BkBPQ47ENEzs7Czhuz59m+dcBhh/buUoIERSnfUY [2024-01-18T17:13:31.405Z] lWIfqxQevwHYZ3Kjx0l3qliYjKBQJo5xDPnukGb4yUnmrCdZ3fxMzlnr0Ais9rUH [2024-01-18T17:13:31.405Z] MjiDsjP/oUa8naQUTZO4DTRZqeEldzNXnxF6YH+3moTkwRLt9xzirNwECmJ9qxrC [2024-01-18T17:13:31.405Z] eCA7zORxbFSftYLnQiGWUS1dnxBuLOvJHjYPC6EkPPrvqnx2fXcTE1V8GizVRrA6 [2024-01-18T17:13:31.405Z] NPxt+e4qZgU/Nk/Jkpsb7h1FVcKtS8CvmtbdqQ== [2024-01-18T17:13:31.405Z] -----END CERTIFICATE----- [2024-01-18T17:13:31.405Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:31.405Z] MIIDGDCCAgCgAwIBAgIUI6D0WS4tl52N+w2lwAS+6V0DPyUwDQYJKoZIhvcNAQEL [2024-01-18T17:13:31.405Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-18T17:13:31.405Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-18T17:13:31.405Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-18T17:13:31.405Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-18T17:13:31.405Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-18T17:13:31.405Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-18T17:13:31.405Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-18T17:13:31.405Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-18T17:13:31.405Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-18T17:13:31.405Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-18T17:13:31.405Z] 9w0BAQsFAAOCAQEAAS4yElh6TMyP18ipzRA0Wenukevjs/vtkAFOsv1u5GNeDIrv [2024-01-18T17:13:31.405Z] lNi6/VjvtS+DoFNzQnTf3QoO6GTuoWHLQWt6Z/V5AwIsnOXduSXLugMwEv3j9l50 [2024-01-18T17:13:31.405Z] aDkA7vJ3OH5cGqIvIzxQN7IUOEvZNhY67GFP7+zAzVNRpJ1FHYbhQGPuFD//b9G3 [2024-01-18T17:13:31.405Z] hr4E3+iUEc+F/iBA+caA8wRx1zR383Jz6wuoFt7n7HpWNcgMvoh91JlUAK7yUp0B [2024-01-18T17:13:31.405Z] ekuG6q7HtVsk6X+aB1RyWfrSJjih75W7b+p+/bV4vNmQS1wSGSOaSIuVRvMhmpwf [2024-01-18T17:13:31.405Z] FpT+4rUpiCxklD69/MET/1cWxqvao/ppf6phzg== [2024-01-18T17:13:31.405Z] -----END CERTIFICATE----- [2024-01-18T17:13:31.405Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:31.405Z] MIIDGDCCAgCgAwIBAgIUIVCCgYV9enDck18n89YsA5b2IdgwDQYJKoZIhvcNAQEL [2024-01-18T17:13:31.405Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-18T17:13:31.405Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-18T17:13:31.405Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-18T17:13:31.405Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-18T17:13:31.405Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-18T17:13:31.405Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-18T17:13:31.405Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-18T17:13:31.405Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-18T17:13:31.405Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-18T17:13:31.405Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-18T17:13:31.405Z] 9w0BAQsFAAOCAQEAa3eFm1ivFf5Vom6a/7E1gDdjl7cjUZ5cqg332Ky1NdzEOiwW [2024-01-18T17:13:31.405Z] mx3yEk35qCd8LAoFMvPvPIuMx1NDS8kwy3FmIOjKnZapHA31ZUpn/gT/seH4GFIU [2024-01-18T17:13:31.406Z] oZw+rODV5h6jmtCVugAn48u/LxOXbY417JWwaeDx5uw+NzOSyebe+td2PbtUjf5X [2024-01-18T17:13:31.406Z] 6xcwYUmaUGsIAgfIMM12hzXOI2/q1qz5dAqwSgnR7dKBReOXnUoFpNOMpxlJ70p/ [2024-01-18T17:13:31.406Z] WUU1aWIWZqKrNI7Pg1T5jljq7+KbT3+A6+ux0FLExr2c1tw3NndXS2Y9fO55m5/q [2024-01-18T17:13:31.406Z] 0EcrTdKx4+QjbWcja3WdX2pGDk0XGNgx+EXn6Q== [2024-01-18T17:13:31.406Z] -----END CERTIFICATE----- [2024-01-18T17:13:31.406Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:31.406Z] MIIFATCCAumgAwIBAgIRAKc9ZKBASymy5TLOEp57N98wDQYJKoZIhvcNAQELBQAw [2024-01-18T17:13:31.406Z] GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDMyMzIyNTM0NloXDTM2 [2024-01-18T17:13:31.406Z] MDMyMzIyNTM0NlowGjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMIICIjANBgkq [2024-01-18T17:13:31.406Z] hkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA+pYHvQw5iU3v2b3iNuYNKYgsWD6KU7aJ [2024-01-18T17:13:31.406Z] diddtZQxSWYzUI3U0I1UsRPTxnhTifs/M9NW4ZlV13ZfB7APwC8oqKOIiwo7IwlP [2024-01-18T17:13:31.406Z] xg0VKgyz+kT8RJfYr66PPIYP0fpTeu42LpMJ+CKo9sbpgVNDZN2z/qiXrRNX/VtG [2024-01-18T17:13:31.406Z] TkPV7a44fZ5bHHVruAxvDnylpQxJobtCBWlJSsbIRGFHMc2z88eUz9NmIOWUKGGj [2024-01-18T17:13:31.406Z] EmP76x8OfRHpIpuxRSCjn0+i9+hR2siIOpcMOGd+40uVJxbRRP5ZXnUFa2fF5FWd [2024-01-18T17:13:31.406Z] O0u0RPI8HON0ovhrwPJY+4eWKkQzyC611oLPYGQ4EbifRsTsCxUZqyUuStGyp8oa [2024-01-18T17:13:31.406Z] aoSKfF6X0+KzGgwwnrjRTUpIl19A92KR0Noo6h622OX+4sZiO/JQdkuX5w/HupK0 [2024-01-18T17:13:31.406Z] A0M0WSMCvU6GOhjGotmh2VTEJwHHY4+TUk0iQYRtv1crONklyZoAQPD76hCrC8Cr [2024-01-18T17:13:31.406Z] IbgsZLfTMC8TWUoMbyUDgvgYkHKMoPm0VGVVuwpRKJxv7+2wXO+pivrrUl2Q9fPe [2024-01-18T17:13:31.406Z] Kk055nJLMV9yPUdig8othUKrRfSxli946AEV1eEOhxddfEwBE3Lt2xn0hhiIedbb [2024-01-18T17:13:31.406Z] Ftf/5kEWFZkXyUmMJK8Ra76Kus2ABueUVEcZ48hrRr1Hf1N9n59VbTUaXgeiZA50 [2024-01-18T17:13:31.406Z] qXf2bymE6F8CAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMB [2024-01-18T17:13:31.406Z] Af8wHQYDVR0OBBYEFMEmdKSKRKDm+iAo2FwjmkWIGHngMA0GCSqGSIb3DQEBCwUA [2024-01-18T17:13:31.406Z] A4ICAQBCPw74M9X/Xx04K1VAES3ypgQYH5bf9FXVDrwhRFSVckria/7dMzoF5wln [2024-01-18T17:13:31.406Z] uq9NGsjkkkDg17AohcQdr8alH4LvPdxpKr3BjpvEcmbqF8xH+MbbeUEnmbSfLI8H [2024-01-18T17:13:31.406Z] sefuhXF9AF/9iYvpVNC8FmJ0OhiVv13VgMQw0CRKkbtjZBf8xaEhq/YqxWVsgOjm [2024-01-18T17:13:31.406Z] dm5CAQ2X0aX7502x8wYRgMnZhA5goC1zVWBVAi8yhhmlhhoDUfg17cXkmaJC5pDd [2024-01-18T17:13:31.406Z] oenZ9NVhW8eDb03MFCrWNvIh89DDeCGWuWfDltDq0n3owyL0IeSn7RfpSclpxVmV [2024-01-18T17:13:31.406Z] /53jkYjwIgxIG7Gsv0LKMbsf6QdBcTjhvfZyMIpBRkTe3zuHd2feKzY9lEkbRvRQ [2024-01-18T17:13:31.406Z] zbh4Ps5YBnG6CKJPTbe2hfi3nhnw/MyEmF3zb0hzvLWNrR9XW3ibb2oL3424XOwc [2024-01-18T17:13:31.406Z] VjrTSCLzO9Rv6s5wi03qoWvKAQQAElqTYRHhynJ3w6wuvKYF5zcZF3MDnrVGLbh1 [2024-01-18T17:13:31.406Z] Q9ePRFBCiXOQ6wPLoUhrrbZ8LpFUFYDXHMtYM7P9sc9IAWoONXREJaO08zgFtMp4 [2024-01-18T17:13:31.406Z] 8iyIYUyQAbsvx8oD2M8kRvrIRSrRJSl6L957b4AFiLIQ/GgV2curs0jje7Edx34c [2024-01-18T17:13:31.406Z] idWw1VrejtwclobqNMVtG3EiPUIpJGpbMcJgbiLSmKkrvQtGng== [2024-01-18T17:13:31.406Z] -----END CERTIFICATE----- [2024-01-18T17:13:31.406Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} [2024-01-18T17:13:31.406Z] INFO[2024-01-18 17:13:31] Actual:true, Expected: true [2024-01-18T17:13:31.406Z] INFO[2024-01-18 17:13:31] Verifying : Description : Fetching configmap pointer &ConfigMap{ObjectMeta:{openshift-service-ca.crt mysql-ibm-pxb-0-84853-01-18-17h11m43s f13aabdd-bbde-4cf7-8d1c-6471f7d45dd0 1286622 0 2024-01-18 17:11:43 +0000 UTC map[] map[service.beta.openshift.io/inject-cabundle:true] [] [] [{kube-controller-manager Update v1 2024-01-18 17:11:43 +0000 UTC FieldsV1 {"f:data":{},"f:metadata":{"f:annotations":{".":{},"f:service.beta.openshift.io/inject-cabundle":{}}}} } {service-ca-operator Update v1 2024-01-18 17:11:43 +0000 UTC FieldsV1 {"f:data":{"f:service-ca.crt":{}}} }]},Data:map[string]string{service-ca.crt: -----BEGIN CERTIFICATE----- [2024-01-18T17:13:31.406Z] MIIDUTCCAjmgAwIBAgIILvRjno8BjvIwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE [2024-01-18T17:13:31.406Z] Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwNTM5NjA4ODAe [2024-01-18T17:13:31.406Z] Fw0yNDAxMTYwOTA4MDhaFw0yNjAzMTYwOTA4MDlaMDYxNDAyBgNVBAMMK29wZW5z [2024-01-18T17:13:31.406Z] aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDUzOTYwODgwggEiMA0GCSqG [2024-01-18T17:13:31.406Z] SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVr8xmLKsMoemVtfnlzzil6B8CGocEhNRo [2024-01-18T17:13:31.407Z] fkzkD8aSOYgIj5zcZo/M0O7nXDXeg9dqx9/h2qrGk+SS2ikU2Xs4Q5gaQDh9U1lB [2024-01-18T17:13:31.407Z] y8paoOEvw06CmJ/1Qqur5A539/OnimRSTdejTu4lwaDwsJiyoGfhvrBJAdjagtYA [2024-01-18T17:13:31.407Z] 0sNMrCgEAT4NWHS5EDx5dF7nMDHmQ0h0gHYei6vUEtiPyLwcTH1T7uVSj22fCEXX [2024-01-18T17:13:31.407Z] LjgN3OSiquJ+Hf4+csPpwgxAEkfjdBldhnImYkunJbg/x4c90uFL1pygZusrbFAf [2024-01-18T17:13:31.407Z] cERW8GngZuL5JY8MQoIstMu+1qS390ouQDDbEcvWjxC8YFNFHywrAgMBAAGjYzBh [2024-01-18T17:13:31.407Z] MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQS5vRk [2024-01-18T17:13:31.407Z] jkBPISX2qYmk1VEQZIXSLzAfBgNVHSMEGDAWgBQS5vRkjkBPISX2qYmk1VEQZIXS [2024-01-18T17:13:31.407Z] LzANBgkqhkiG9w0BAQsFAAOCAQEAZ/qc2vFGJZgDJCEUaPYuufuLRPSF68ixV9fk [2024-01-18T17:13:31.407Z] e5tBkRN8atCsEBygC/RUcwULlhsdgUMGMcREgNC/FYlwkOsAj3gP8KmcXAZ4gHCa [2024-01-18T17:13:31.407Z] T6PIbIcj1hCX0mdX0tvtaj7PWvRUz9CHh6iFzmPsXK5xjaBA9vZbkUAa7odguUit [2024-01-18T17:13:31.407Z] vz1QnKoV1OaZyccDLeQL50khM1XEBn8L+zeY3k85uo2X514HZW4LDPTQ94cpyNnb [2024-01-18T17:13:31.407Z] zqI6utp7JoPJexsHMWNFKGb//HgjluNg7TsywXIeebk+/GZUEDfNKid0oiTpQvzR [2024-01-18T17:13:31.407Z] TFFF+5KO/sa4+qIh1ERm140JxeaurUMaJzn7cFQbekEgt/N+iA== [2024-01-18T17:13:31.407Z] -----END CERTIFICATE----- [2024-01-18T17:13:31.407Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} from namespace mysql-ibm-pxb-0-84853-01-18-17h11m43s [2024-01-18T17:13:31.407Z] INFO[2024-01-18 17:13:31] Actual:true, Expected: true [2024-01-18T17:13:31.407Z] 2024-01-18 17:13:31 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.AddLabelToResource:#5038] - Adding label [abe5cf54-4e74-45b4-882f-82b02f3dd9cf=cd7d2c21-9adb-41bf-b01f-ae218113bcc9] to ConfigMap openshift-service-ca.crt [2024-01-18T17:13:31.664Z] INFO[2024-01-18 17:13:31] Verifying : Description : Adding labels abe5cf54-4e74-45b4-882f-82b02f3dd9cf=cd7d2c21-9adb-41bf-b01f-ae218113bcc9 to resource &ConfigMap{ObjectMeta:{openshift-service-ca.crt mysql-ibm-pxb-0-84853-01-18-17h11m43s f13aabdd-bbde-4cf7-8d1c-6471f7d45dd0 1286622 0 2024-01-18 17:11:43 +0000 UTC map[abe5cf54-4e74-45b4-882f-82b02f3dd9cf:cd7d2c21-9adb-41bf-b01f-ae218113bcc9] map[service.beta.openshift.io/inject-cabundle:true] [] [] [{kube-controller-manager Update v1 2024-01-18 17:11:43 +0000 UTC FieldsV1 {"f:data":{},"f:metadata":{"f:annotations":{".":{},"f:service.beta.openshift.io/inject-cabundle":{}}}} } {service-ca-operator Update v1 2024-01-18 17:11:43 +0000 UTC FieldsV1 {"f:data":{"f:service-ca.crt":{}}} }]},Data:map[string]string{service-ca.crt: -----BEGIN CERTIFICATE----- [2024-01-18T17:13:31.664Z] MIIDUTCCAjmgAwIBAgIILvRjno8BjvIwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE [2024-01-18T17:13:31.664Z] Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwNTM5NjA4ODAe [2024-01-18T17:13:31.664Z] Fw0yNDAxMTYwOTA4MDhaFw0yNjAzMTYwOTA4MDlaMDYxNDAyBgNVBAMMK29wZW5z [2024-01-18T17:13:31.664Z] aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDUzOTYwODgwggEiMA0GCSqG [2024-01-18T17:13:31.664Z] SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVr8xmLKsMoemVtfnlzzil6B8CGocEhNRo [2024-01-18T17:13:31.664Z] fkzkD8aSOYgIj5zcZo/M0O7nXDXeg9dqx9/h2qrGk+SS2ikU2Xs4Q5gaQDh9U1lB [2024-01-18T17:13:31.664Z] y8paoOEvw06CmJ/1Qqur5A539/OnimRSTdejTu4lwaDwsJiyoGfhvrBJAdjagtYA [2024-01-18T17:13:31.664Z] 0sNMrCgEAT4NWHS5EDx5dF7nMDHmQ0h0gHYei6vUEtiPyLwcTH1T7uVSj22fCEXX [2024-01-18T17:13:31.664Z] LjgN3OSiquJ+Hf4+csPpwgxAEkfjdBldhnImYkunJbg/x4c90uFL1pygZusrbFAf [2024-01-18T17:13:31.664Z] cERW8GngZuL5JY8MQoIstMu+1qS390ouQDDbEcvWjxC8YFNFHywrAgMBAAGjYzBh [2024-01-18T17:13:31.664Z] MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQS5vRk [2024-01-18T17:13:31.664Z] jkBPISX2qYmk1VEQZIXSLzAfBgNVHSMEGDAWgBQS5vRkjkBPISX2qYmk1VEQZIXS [2024-01-18T17:13:31.664Z] LzANBgkqhkiG9w0BAQsFAAOCAQEAZ/qc2vFGJZgDJCEUaPYuufuLRPSF68ixV9fk [2024-01-18T17:13:31.664Z] e5tBkRN8atCsEBygC/RUcwULlhsdgUMGMcREgNC/FYlwkOsAj3gP8KmcXAZ4gHCa [2024-01-18T17:13:31.664Z] T6PIbIcj1hCX0mdX0tvtaj7PWvRUz9CHh6iFzmPsXK5xjaBA9vZbkUAa7odguUit [2024-01-18T17:13:31.664Z] vz1QnKoV1OaZyccDLeQL50khM1XEBn8L+zeY3k85uo2X514HZW4LDPTQ94cpyNnb [2024-01-18T17:13:31.664Z] zqI6utp7JoPJexsHMWNFKGb//HgjluNg7TsywXIeebk+/GZUEDfNKid0oiTpQvzR [2024-01-18T17:13:31.664Z] TFFF+5KO/sa4+qIh1ERm140JxeaurUMaJzn7cFQbekEgt/N+iA== [2024-01-18T17:13:31.664Z] -----END CERTIFICATE----- [2024-01-18T17:13:31.664Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} [2024-01-18T17:13:31.664Z] INFO[2024-01-18 17:13:31] Actual:true, Expected: true [2024-01-18T17:13:31.664Z] INFO[2024-01-18 17:13:31] Verifying : Description : Fetching pvc list from namespace [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T17:13:31.664Z] INFO[2024-01-18 17:13:31] Actual:true, Expected: true [2024-01-18T17:13:31.922Z] INFO[2024-01-18 17:13:31] Verifying : Description : Fetching pvc pointer &PersistentVolumeClaim{ObjectMeta:{mysql-data mysql-ibm-pxb-1-84853-01-18-17h11m43s 89b302bc-3457-4a65-bb98-d0cda5e7541c 1286922 0 2024-01-18 17:11:44 +0000 UTC map[] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-18 17:11:44 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-18 17:11:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-18 17:11:55 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{2147483648 0} {} 2Gi BinarySI},},},VolumeName:pvc-89b302bc-3457-4a65-bb98-d0cda5e7541c,Selector:nil,StorageClassName:*mysql-sc,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{10737418240 0} {} 10Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},} for pvc {{ } {mysql-data mysql-ibm-pxb-1-84853-01-18-17h11m43s 89b302bc-3457-4a65-bb98-d0cda5e7541c 1286922 0 2024-01-18 17:11:44 +0000 UTC map[] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-18 17:11:44 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-18 17:11:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-18 17:11:55 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{2147483648 0} {} 2Gi BinarySI}]} pvc-89b302bc-3457-4a65-bb98-d0cda5e7541c 0xc00148f7f0 0xc00148f800 nil nil} {Bound [ReadWriteOnce] map[storage:{{10737418240 0} {} 10Gi BinarySI}] [] map[] }} [2024-01-18T17:13:31.922Z] INFO[2024-01-18 17:13:31] Actual:true, Expected: true [2024-01-18T17:13:31.922Z] 2024-01-18 17:13:31 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.AddLabelToResource:#5030] - Adding label [abe5cf54-4e74-45b4-882f-82b02f3dd9cf=cd7d2c21-9adb-41bf-b01f-ae218113bcc9] to PVC mysql-data [2024-01-18T17:13:32.179Z] INFO[2024-01-18 17:13:32] Verifying : Description : Adding labels abe5cf54-4e74-45b4-882f-82b02f3dd9cf=cd7d2c21-9adb-41bf-b01f-ae218113bcc9 to resource &PersistentVolumeClaim{ObjectMeta:{mysql-data mysql-ibm-pxb-1-84853-01-18-17h11m43s 89b302bc-3457-4a65-bb98-d0cda5e7541c 1286922 0 2024-01-18 17:11:44 +0000 UTC map[abe5cf54-4e74-45b4-882f-82b02f3dd9cf:cd7d2c21-9adb-41bf-b01f-ae218113bcc9] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-18 17:11:44 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-18 17:11:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-18 17:11:55 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{2147483648 0} {} 2Gi BinarySI},},},VolumeName:pvc-89b302bc-3457-4a65-bb98-d0cda5e7541c,Selector:nil,StorageClassName:*mysql-sc,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{10737418240 0} {} 10Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},} [2024-01-18T17:13:32.179Z] INFO[2024-01-18 17:13:32] Actual:true, Expected: true [2024-01-18T17:13:32.436Z] INFO[2024-01-18 17:13:32] Verifying : Description : Fetching configmap list from namespace [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T17:13:32.436Z] INFO[2024-01-18 17:13:32] Actual:true, Expected: true [2024-01-18T17:13:32.694Z] INFO[2024-01-18 17:13:32] Verifying : Description : Fetching configmap pointer &ConfigMap{ObjectMeta:{kube-****-ca.crt mysql-ibm-pxb-1-84853-01-18-17h11m43s 7a5b1227-b15c-41ac-8a02-d15eca58a803 1286679 0 2024-01-18 17:11:44 +0000 UTC map[] map[kubernetes.io/description:Contains a CA bundle that can be used to verify the kube-apiserver when using internal endpoints such as the internal service IP or kubernetes.default.svc. No other usage is guaranteed across distributions of Kubernetes clusters.] [] [] [{kube-controller-manager Update v1 2024-01-18 17:11:44 +0000 UTC FieldsV1 {"f:data":{".":{},"f:ca.crt":{}},"f:metadata":{"f:annotations":{".":{},"f:kubernetes.io/description":{}}}} }]},Data:map[string]string{ca.crt: -----BEGIN CERTIFICATE----- [2024-01-18T17:13:32.694Z] MIIDCjCCAfKgAwIBAgIUWkn6rgBaC+VWV1XdJjGA3DwoRRYwDQYJKoZIhvcNAQEL [2024-01-18T17:13:32.694Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-18T17:13:32.694Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-18T17:13:32.694Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-18T17:13:32.694Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-18T17:13:32.694Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-18T17:13:32.694Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-18T17:13:32.694Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-18T17:13:32.694Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-18T17:13:32.694Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-18T17:13:32.694Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAHbD [2024-01-18T17:13:32.694Z] 4nWTgt/8sat7SbbHxZ+JcMGRAIaeLAjMJxkxyqD8xUq6YQcgbjDVHo1Qbh3PXB3T [2024-01-18T17:13:32.694Z] 5BWUsRbDoKi0eRJx/5cyI3846SYEAOpuUfJrg9X2CH7ZoYhxeJ42dqt3MdoMJEM1 [2024-01-18T17:13:32.694Z] k3e2rRqAKuAxwNXugHryxQM+1ehOJjrEuT5DVK6qu8+JYRcT1hBZb6hrebKPp5eG [2024-01-18T17:13:32.694Z] JvBzHP4YOATIAXhfH0Uj5Y+JhwoOdpXmOFpz+6gITPD4aNzzJZe/0ADSFfOyUJaR [2024-01-18T17:13:32.694Z] g5NnJnkeGqMM3SLsur25r+m3y0i3t+RdocZ/4mHTEnF4KWVWvZy4GjENiEBcXEol [2024-01-18T17:13:32.694Z] o4SZVaP4AF7XQJ5vdA0= [2024-01-18T17:13:32.694Z] -----END CERTIFICATE----- [2024-01-18T17:13:32.694Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:32.694Z] MIIDCjCCAfKgAwIBAgIUXhIIFCc/NxZJn6hNEFoZuPQ8N5UwDQYJKoZIhvcNAQEL [2024-01-18T17:13:32.694Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-18T17:13:32.694Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-18T17:13:32.694Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-18T17:13:32.694Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-18T17:13:32.694Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-18T17:13:32.695Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-18T17:13:32.695Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-18T17:13:32.695Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-18T17:13:32.695Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-18T17:13:32.695Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAKG3 [2024-01-18T17:13:32.695Z] nVfv4IkgHVOxgbrNIoYhjvBvVJcLZub0C9yidexLA3A61p1a7b/OpidG5zShPhH2 [2024-01-18T17:13:32.695Z] HRdQcqgp0HIxaS+/oLD3eElOnnC/EKD359K6n3kN66sMgvNbHxNFM84Au0TXqmbT [2024-01-18T17:13:32.695Z] w8afrKjrfe/VglpHC2GfmgHMF4YH+ZF48C1RZA6z2oXj6Aj+uww0PjiClE9qUYqK [2024-01-18T17:13:32.695Z] R9uRa32prExihsi+Bto7n2Axo92qUiK2KrkEdQitnQfH5z/PojS78vbyIKp1O86N [2024-01-18T17:13:32.695Z] qRImvCYY17CPxP+JUQ7dxWDMF4NTgSnabkp+cd5i/pyoAb0guQFlQQUnHAZa4QNC [2024-01-18T17:13:32.695Z] 7fgTo1IUwWyDmwCnN5U= [2024-01-18T17:13:32.695Z] -----END CERTIFICATE----- [2024-01-18T17:13:32.695Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:32.695Z] MIIDCjCCAfKgAwIBAgIUJPzO3O+wmUt4gHCkl3ojsvwZL6owDQYJKoZIhvcNAQEL [2024-01-18T17:13:32.695Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-18T17:13:32.695Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-18T17:13:32.695Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-18T17:13:32.695Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-18T17:13:32.695Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-18T17:13:32.695Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-18T17:13:32.695Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-18T17:13:32.695Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-18T17:13:32.695Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-18T17:13:32.695Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAGVt [2024-01-18T17:13:32.695Z] g9qX+r8kyGYsvFavmnDxgMK1Zkz80irQJ2BJlDHdK1hp5aqpRbT2z2YgLog5ZMIZ [2024-01-18T17:13:32.695Z] uI36uPCOXo8+Gol1jT7D6vsgMWm0Tk9jCNbO4jPkv+wm2q1B9UZo94UGiS0K+G0S [2024-01-18T17:13:32.695Z] nccl4jmSq1gWpBZIVetdDFNjpipKjhEOVQM7x33APhACxSAWRp9iEr6V/CzMOgkf [2024-01-18T17:13:32.695Z] RHyz5D7ljh4a2XVH7++eKsj2QPfKyiKtIG72tCccQ9zGnzveiXXb4Ai8r+DyKH4Z [2024-01-18T17:13:32.695Z] zDt5Hb4ZFl72E3Rl4bIh/VQBVsr9u/SFR1p/Emyq+kGXsn+9YMiM9tK4mLUtjdox [2024-01-18T17:13:32.695Z] LMWUWTelNRGWAiaTVz8= [2024-01-18T17:13:32.695Z] -----END CERTIFICATE----- [2024-01-18T17:13:32.695Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:32.695Z] MIIDCjCCAfKgAwIBAgIUBxbEoan/HnRZ+2deRj+2RvGMrWQwDQYJKoZIhvcNAQEL [2024-01-18T17:13:32.695Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-18T17:13:32.695Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-18T17:13:32.695Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-18T17:13:32.695Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-18T17:13:32.695Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-18T17:13:32.695Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-18T17:13:32.695Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-18T17:13:32.695Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-18T17:13:32.695Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-18T17:13:32.695Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAB65 [2024-01-18T17:13:32.695Z] nHU+98oTP0U2J0/TsL5GFeBTH7YyUXUNRJvPWxkA8riB3swoScFKbavO6XJ/hrI2 [2024-01-18T17:13:32.695Z] ywApOz77Iyq58OLa5aIC5aaPRf3YE8LbJQYIHmtza21LXhtlKVSrLu4W58PauI8Z [2024-01-18T17:13:32.695Z] 7iaL9QPUihHFK7wrZ2ZpabN2rRJvTGWWKlhmgFTjI+71gPrXWmYBHIKmTv55V5m1 [2024-01-18T17:13:32.695Z] LxORTR/gBXeH9gmM7wz1cbaYNazqs8R1LS5SW76p/y9RLjTzU6N39VYefMMGwbsQ [2024-01-18T17:13:32.695Z] DlD11wvMBxTBp3hQYc3pdVLkVEQt8trVIwEUts7zfwUTCu/z8gH7H5XePpEVNana [2024-01-18T17:13:32.695Z] JhlAcXHCO7nN/ddJmoc= [2024-01-18T17:13:32.695Z] -----END CERTIFICATE----- [2024-01-18T17:13:32.695Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:32.695Z] MIIDGDCCAgCgAwIBAgIUEvLaarpXlv1cVvG6nMuxh2DNCKQwDQYJKoZIhvcNAQEL [2024-01-18T17:13:32.695Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-18T17:13:32.695Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-18T17:13:32.695Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-18T17:13:32.695Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-18T17:13:32.695Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-18T17:13:32.695Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-18T17:13:32.695Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-18T17:13:32.695Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-18T17:13:32.695Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-18T17:13:32.696Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-18T17:13:32.696Z] 9w0BAQsFAAOCAQEAOPakzVUmAqHhmQViZZHSQeRh/xYQBrtn6dnEo6X55SXNWcPU [2024-01-18T17:13:32.696Z] 4EEW3DvwBf75uqxKeGoXHTBpqNHG2x1+IncvIKJmi5OWM3aR/vEyJoBOchPBl+A4 [2024-01-18T17:13:32.696Z] 0xZO1vKiS2djxz8XHMkR70cxD8hv/zDJeIQbgBroF4Wa7rim5+y7jgO0O7Rs1zhT [2024-01-18T17:13:32.696Z] LDDStJ8aUKwcVRmOv7NFQxbQHXNx07kaUE2SQJfhryzZmotoBiymmQT2v0H4/4q0 [2024-01-18T17:13:32.696Z] 5dylRhSCWfwN5+It40/C3b1pbC2d9Iv/eO+B0wqg13dLF3xuA1JblEgnqA9mdNnp [2024-01-18T17:13:32.696Z] P6EgKTQBMiO6nPFyYIPqg1eWYkb1eXu9aEkmbw== [2024-01-18T17:13:32.696Z] -----END CERTIFICATE----- [2024-01-18T17:13:32.696Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:32.696Z] MIIDGDCCAgCgAwIBAgIUWf/qoBANFWF/a/yZr7wfLBMfItAwDQYJKoZIhvcNAQEL [2024-01-18T17:13:32.696Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-18T17:13:32.696Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-18T17:13:32.696Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-18T17:13:32.696Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-18T17:13:32.696Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-18T17:13:32.696Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-18T17:13:32.696Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-18T17:13:32.696Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-18T17:13:32.696Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-18T17:13:32.696Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-18T17:13:32.696Z] 9w0BAQsFAAOCAQEAdCguadZU6WJAuuAtWzvYgftPKkDKNBeGVgbqRwWRCB+QqVFl [2024-01-18T17:13:32.696Z] +3ZOQTStpnXJsY3Ig31hmL88BkBPQ47ENEzs7Czhuz59m+dcBhh/buUoIERSnfUY [2024-01-18T17:13:32.696Z] lWIfqxQevwHYZ3Kjx0l3qliYjKBQJo5xDPnukGb4yUnmrCdZ3fxMzlnr0Ais9rUH [2024-01-18T17:13:32.696Z] MjiDsjP/oUa8naQUTZO4DTRZqeEldzNXnxF6YH+3moTkwRLt9xzirNwECmJ9qxrC [2024-01-18T17:13:32.696Z] eCA7zORxbFSftYLnQiGWUS1dnxBuLOvJHjYPC6EkPPrvqnx2fXcTE1V8GizVRrA6 [2024-01-18T17:13:32.696Z] NPxt+e4qZgU/Nk/Jkpsb7h1FVcKtS8CvmtbdqQ== [2024-01-18T17:13:32.696Z] -----END CERTIFICATE----- [2024-01-18T17:13:32.696Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:32.696Z] MIIDGDCCAgCgAwIBAgIUI6D0WS4tl52N+w2lwAS+6V0DPyUwDQYJKoZIhvcNAQEL [2024-01-18T17:13:32.696Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-18T17:13:32.696Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-18T17:13:32.696Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-18T17:13:32.696Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-18T17:13:32.696Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-18T17:13:32.696Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-18T17:13:32.696Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-18T17:13:32.696Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-18T17:13:32.696Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-18T17:13:32.696Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-18T17:13:32.696Z] 9w0BAQsFAAOCAQEAAS4yElh6TMyP18ipzRA0Wenukevjs/vtkAFOsv1u5GNeDIrv [2024-01-18T17:13:32.696Z] lNi6/VjvtS+DoFNzQnTf3QoO6GTuoWHLQWt6Z/V5AwIsnOXduSXLugMwEv3j9l50 [2024-01-18T17:13:32.696Z] aDkA7vJ3OH5cGqIvIzxQN7IUOEvZNhY67GFP7+zAzVNRpJ1FHYbhQGPuFD//b9G3 [2024-01-18T17:13:32.696Z] hr4E3+iUEc+F/iBA+caA8wRx1zR383Jz6wuoFt7n7HpWNcgMvoh91JlUAK7yUp0B [2024-01-18T17:13:32.696Z] ekuG6q7HtVsk6X+aB1RyWfrSJjih75W7b+p+/bV4vNmQS1wSGSOaSIuVRvMhmpwf [2024-01-18T17:13:32.696Z] FpT+4rUpiCxklD69/MET/1cWxqvao/ppf6phzg== [2024-01-18T17:13:32.696Z] -----END CERTIFICATE----- [2024-01-18T17:13:32.696Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:32.696Z] MIIDGDCCAgCgAwIBAgIUIVCCgYV9enDck18n89YsA5b2IdgwDQYJKoZIhvcNAQEL [2024-01-18T17:13:32.696Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-18T17:13:32.696Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-18T17:13:32.696Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-18T17:13:32.696Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-18T17:13:32.696Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-18T17:13:32.696Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-18T17:13:32.696Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-18T17:13:32.696Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-18T17:13:32.696Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-18T17:13:32.697Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-18T17:13:32.697Z] 9w0BAQsFAAOCAQEAa3eFm1ivFf5Vom6a/7E1gDdjl7cjUZ5cqg332Ky1NdzEOiwW [2024-01-18T17:13:32.697Z] mx3yEk35qCd8LAoFMvPvPIuMx1NDS8kwy3FmIOjKnZapHA31ZUpn/gT/seH4GFIU [2024-01-18T17:13:32.697Z] oZw+rODV5h6jmtCVugAn48u/LxOXbY417JWwaeDx5uw+NzOSyebe+td2PbtUjf5X [2024-01-18T17:13:32.697Z] 6xcwYUmaUGsIAgfIMM12hzXOI2/q1qz5dAqwSgnR7dKBReOXnUoFpNOMpxlJ70p/ [2024-01-18T17:13:32.697Z] WUU1aWIWZqKrNI7Pg1T5jljq7+KbT3+A6+ux0FLExr2c1tw3NndXS2Y9fO55m5/q [2024-01-18T17:13:32.697Z] 0EcrTdKx4+QjbWcja3WdX2pGDk0XGNgx+EXn6Q== [2024-01-18T17:13:32.697Z] -----END CERTIFICATE----- [2024-01-18T17:13:32.697Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:32.697Z] MIIFATCCAumgAwIBAgIRAKc9ZKBASymy5TLOEp57N98wDQYJKoZIhvcNAQELBQAw [2024-01-18T17:13:32.697Z] GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDMyMzIyNTM0NloXDTM2 [2024-01-18T17:13:32.697Z] MDMyMzIyNTM0NlowGjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMIICIjANBgkq [2024-01-18T17:13:32.697Z] hkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA+pYHvQw5iU3v2b3iNuYNKYgsWD6KU7aJ [2024-01-18T17:13:32.697Z] diddtZQxSWYzUI3U0I1UsRPTxnhTifs/M9NW4ZlV13ZfB7APwC8oqKOIiwo7IwlP [2024-01-18T17:13:32.697Z] xg0VKgyz+kT8RJfYr66PPIYP0fpTeu42LpMJ+CKo9sbpgVNDZN2z/qiXrRNX/VtG [2024-01-18T17:13:32.697Z] TkPV7a44fZ5bHHVruAxvDnylpQxJobtCBWlJSsbIRGFHMc2z88eUz9NmIOWUKGGj [2024-01-18T17:13:32.697Z] EmP76x8OfRHpIpuxRSCjn0+i9+hR2siIOpcMOGd+40uVJxbRRP5ZXnUFa2fF5FWd [2024-01-18T17:13:32.697Z] O0u0RPI8HON0ovhrwPJY+4eWKkQzyC611oLPYGQ4EbifRsTsCxUZqyUuStGyp8oa [2024-01-18T17:13:32.697Z] aoSKfF6X0+KzGgwwnrjRTUpIl19A92KR0Noo6h622OX+4sZiO/JQdkuX5w/HupK0 [2024-01-18T17:13:32.697Z] A0M0WSMCvU6GOhjGotmh2VTEJwHHY4+TUk0iQYRtv1crONklyZoAQPD76hCrC8Cr [2024-01-18T17:13:32.697Z] IbgsZLfTMC8TWUoMbyUDgvgYkHKMoPm0VGVVuwpRKJxv7+2wXO+pivrrUl2Q9fPe [2024-01-18T17:13:32.697Z] Kk055nJLMV9yPUdig8othUKrRfSxli946AEV1eEOhxddfEwBE3Lt2xn0hhiIedbb [2024-01-18T17:13:32.697Z] Ftf/5kEWFZkXyUmMJK8Ra76Kus2ABueUVEcZ48hrRr1Hf1N9n59VbTUaXgeiZA50 [2024-01-18T17:13:32.697Z] qXf2bymE6F8CAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMB [2024-01-18T17:13:32.697Z] Af8wHQYDVR0OBBYEFMEmdKSKRKDm+iAo2FwjmkWIGHngMA0GCSqGSIb3DQEBCwUA [2024-01-18T17:13:32.697Z] A4ICAQBCPw74M9X/Xx04K1VAES3ypgQYH5bf9FXVDrwhRFSVckria/7dMzoF5wln [2024-01-18T17:13:32.697Z] uq9NGsjkkkDg17AohcQdr8alH4LvPdxpKr3BjpvEcmbqF8xH+MbbeUEnmbSfLI8H [2024-01-18T17:13:32.697Z] sefuhXF9AF/9iYvpVNC8FmJ0OhiVv13VgMQw0CRKkbtjZBf8xaEhq/YqxWVsgOjm [2024-01-18T17:13:32.697Z] dm5CAQ2X0aX7502x8wYRgMnZhA5goC1zVWBVAi8yhhmlhhoDUfg17cXkmaJC5pDd [2024-01-18T17:13:32.697Z] oenZ9NVhW8eDb03MFCrWNvIh89DDeCGWuWfDltDq0n3owyL0IeSn7RfpSclpxVmV [2024-01-18T17:13:32.697Z] /53jkYjwIgxIG7Gsv0LKMbsf6QdBcTjhvfZyMIpBRkTe3zuHd2feKzY9lEkbRvRQ [2024-01-18T17:13:32.697Z] zbh4Ps5YBnG6CKJPTbe2hfi3nhnw/MyEmF3zb0hzvLWNrR9XW3ibb2oL3424XOwc [2024-01-18T17:13:32.697Z] VjrTSCLzO9Rv6s5wi03qoWvKAQQAElqTYRHhynJ3w6wuvKYF5zcZF3MDnrVGLbh1 [2024-01-18T17:13:32.697Z] Q9ePRFBCiXOQ6wPLoUhrrbZ8LpFUFYDXHMtYM7P9sc9IAWoONXREJaO08zgFtMp4 [2024-01-18T17:13:32.697Z] 8iyIYUyQAbsvx8oD2M8kRvrIRSrRJSl6L957b4AFiLIQ/GgV2curs0jje7Edx34c [2024-01-18T17:13:32.697Z] idWw1VrejtwclobqNMVtG3EiPUIpJGpbMcJgbiLSmKkrvQtGng== [2024-01-18T17:13:32.697Z] -----END CERTIFICATE----- [2024-01-18T17:13:32.697Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} from namespace mysql-ibm-pxb-1-84853-01-18-17h11m43s [2024-01-18T17:13:32.697Z] INFO[2024-01-18 17:13:32] Actual:true, Expected: true [2024-01-18T17:13:32.697Z] 2024-01-18 17:13:32 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.AddLabelToResource:#5038] - Adding label [abe5cf54-4e74-45b4-882f-82b02f3dd9cf=cd7d2c21-9adb-41bf-b01f-ae218113bcc9] to ConfigMap kube-****-ca.crt [2024-01-18T17:13:32.698Z] INFO[2024-01-18 17:13:32] Verifying : Description : Adding labels abe5cf54-4e74-45b4-882f-82b02f3dd9cf=cd7d2c21-9adb-41bf-b01f-ae218113bcc9 to resource &ConfigMap{ObjectMeta:{kube-****-ca.crt mysql-ibm-pxb-1-84853-01-18-17h11m43s 7a5b1227-b15c-41ac-8a02-d15eca58a803 1286679 0 2024-01-18 17:11:44 +0000 UTC map[abe5cf54-4e74-45b4-882f-82b02f3dd9cf:cd7d2c21-9adb-41bf-b01f-ae218113bcc9] map[kubernetes.io/description:Contains a CA bundle that can be used to verify the kube-apiserver when using internal endpoints such as the internal service IP or kubernetes.default.svc. No other usage is guaranteed across distributions of Kubernetes clusters.] [] [] [{kube-controller-manager Update v1 2024-01-18 17:11:44 +0000 UTC FieldsV1 {"f:data":{".":{},"f:ca.crt":{}},"f:metadata":{"f:annotations":{".":{},"f:kubernetes.io/description":{}}}} }]},Data:map[string]string{ca.crt: -----BEGIN CERTIFICATE----- [2024-01-18T17:13:32.698Z] MIIDCjCCAfKgAwIBAgIUWkn6rgBaC+VWV1XdJjGA3DwoRRYwDQYJKoZIhvcNAQEL [2024-01-18T17:13:32.698Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-18T17:13:32.698Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-18T17:13:32.698Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-18T17:13:32.698Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-18T17:13:32.698Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-18T17:13:32.698Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-18T17:13:32.698Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-18T17:13:32.698Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-18T17:13:32.698Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-18T17:13:32.698Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAHbD [2024-01-18T17:13:32.698Z] 4nWTgt/8sat7SbbHxZ+JcMGRAIaeLAjMJxkxyqD8xUq6YQcgbjDVHo1Qbh3PXB3T [2024-01-18T17:13:32.698Z] 5BWUsRbDoKi0eRJx/5cyI3846SYEAOpuUfJrg9X2CH7ZoYhxeJ42dqt3MdoMJEM1 [2024-01-18T17:13:32.698Z] k3e2rRqAKuAxwNXugHryxQM+1ehOJjrEuT5DVK6qu8+JYRcT1hBZb6hrebKPp5eG [2024-01-18T17:13:32.698Z] JvBzHP4YOATIAXhfH0Uj5Y+JhwoOdpXmOFpz+6gITPD4aNzzJZe/0ADSFfOyUJaR [2024-01-18T17:13:32.698Z] g5NnJnkeGqMM3SLsur25r+m3y0i3t+RdocZ/4mHTEnF4KWVWvZy4GjENiEBcXEol [2024-01-18T17:13:32.698Z] o4SZVaP4AF7XQJ5vdA0= [2024-01-18T17:13:32.698Z] -----END CERTIFICATE----- [2024-01-18T17:13:32.698Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:32.698Z] MIIDCjCCAfKgAwIBAgIUXhIIFCc/NxZJn6hNEFoZuPQ8N5UwDQYJKoZIhvcNAQEL [2024-01-18T17:13:32.698Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-18T17:13:32.698Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-18T17:13:32.698Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-18T17:13:32.698Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-18T17:13:32.698Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-18T17:13:32.698Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-18T17:13:32.698Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-18T17:13:32.698Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-18T17:13:32.698Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-18T17:13:32.698Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAKG3 [2024-01-18T17:13:32.698Z] nVfv4IkgHVOxgbrNIoYhjvBvVJcLZub0C9yidexLA3A61p1a7b/OpidG5zShPhH2 [2024-01-18T17:13:32.698Z] HRdQcqgp0HIxaS+/oLD3eElOnnC/EKD359K6n3kN66sMgvNbHxNFM84Au0TXqmbT [2024-01-18T17:13:32.698Z] w8afrKjrfe/VglpHC2GfmgHMF4YH+ZF48C1RZA6z2oXj6Aj+uww0PjiClE9qUYqK [2024-01-18T17:13:32.698Z] R9uRa32prExihsi+Bto7n2Axo92qUiK2KrkEdQitnQfH5z/PojS78vbyIKp1O86N [2024-01-18T17:13:32.698Z] qRImvCYY17CPxP+JUQ7dxWDMF4NTgSnabkp+cd5i/pyoAb0guQFlQQUnHAZa4QNC [2024-01-18T17:13:32.698Z] 7fgTo1IUwWyDmwCnN5U= [2024-01-18T17:13:32.698Z] -----END CERTIFICATE----- [2024-01-18T17:13:32.698Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:32.698Z] MIIDCjCCAfKgAwIBAgIUJPzO3O+wmUt4gHCkl3ojsvwZL6owDQYJKoZIhvcNAQEL [2024-01-18T17:13:32.698Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-18T17:13:32.698Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-18T17:13:32.698Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-18T17:13:32.698Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-18T17:13:32.698Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-18T17:13:32.698Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-18T17:13:32.698Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-18T17:13:32.698Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-18T17:13:32.698Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-18T17:13:32.698Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAGVt [2024-01-18T17:13:32.698Z] g9qX+r8kyGYsvFavmnDxgMK1Zkz80irQJ2BJlDHdK1hp5aqpRbT2z2YgLog5ZMIZ [2024-01-18T17:13:32.698Z] uI36uPCOXo8+Gol1jT7D6vsgMWm0Tk9jCNbO4jPkv+wm2q1B9UZo94UGiS0K+G0S [2024-01-18T17:13:32.698Z] nccl4jmSq1gWpBZIVetdDFNjpipKjhEOVQM7x33APhACxSAWRp9iEr6V/CzMOgkf [2024-01-18T17:13:32.698Z] RHyz5D7ljh4a2XVH7++eKsj2QPfKyiKtIG72tCccQ9zGnzveiXXb4Ai8r+DyKH4Z [2024-01-18T17:13:32.698Z] zDt5Hb4ZFl72E3Rl4bIh/VQBVsr9u/SFR1p/Emyq+kGXsn+9YMiM9tK4mLUtjdox [2024-01-18T17:13:32.698Z] LMWUWTelNRGWAiaTVz8= [2024-01-18T17:13:32.698Z] -----END CERTIFICATE----- [2024-01-18T17:13:32.698Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:32.699Z] MIIDCjCCAfKgAwIBAgIUBxbEoan/HnRZ+2deRj+2RvGMrWQwDQYJKoZIhvcNAQEL [2024-01-18T17:13:32.699Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-18T17:13:32.699Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-18T17:13:32.699Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-18T17:13:32.699Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-18T17:13:32.699Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-18T17:13:32.699Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-18T17:13:32.699Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-18T17:13:32.699Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-18T17:13:32.699Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-18T17:13:32.699Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAB65 [2024-01-18T17:13:32.699Z] nHU+98oTP0U2J0/TsL5GFeBTH7YyUXUNRJvPWxkA8riB3swoScFKbavO6XJ/hrI2 [2024-01-18T17:13:32.699Z] ywApOz77Iyq58OLa5aIC5aaPRf3YE8LbJQYIHmtza21LXhtlKVSrLu4W58PauI8Z [2024-01-18T17:13:32.699Z] 7iaL9QPUihHFK7wrZ2ZpabN2rRJvTGWWKlhmgFTjI+71gPrXWmYBHIKmTv55V5m1 [2024-01-18T17:13:32.699Z] LxORTR/gBXeH9gmM7wz1cbaYNazqs8R1LS5SW76p/y9RLjTzU6N39VYefMMGwbsQ [2024-01-18T17:13:32.699Z] DlD11wvMBxTBp3hQYc3pdVLkVEQt8trVIwEUts7zfwUTCu/z8gH7H5XePpEVNana [2024-01-18T17:13:32.699Z] JhlAcXHCO7nN/ddJmoc= [2024-01-18T17:13:32.699Z] -----END CERTIFICATE----- [2024-01-18T17:13:32.699Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:32.699Z] MIIDGDCCAgCgAwIBAgIUEvLaarpXlv1cVvG6nMuxh2DNCKQwDQYJKoZIhvcNAQEL [2024-01-18T17:13:32.699Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-18T17:13:32.699Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-18T17:13:32.699Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-18T17:13:32.699Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-18T17:13:32.699Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-18T17:13:32.699Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-18T17:13:32.699Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-18T17:13:32.699Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-18T17:13:32.699Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-18T17:13:32.699Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-18T17:13:32.699Z] 9w0BAQsFAAOCAQEAOPakzVUmAqHhmQViZZHSQeRh/xYQBrtn6dnEo6X55SXNWcPU [2024-01-18T17:13:32.699Z] 4EEW3DvwBf75uqxKeGoXHTBpqNHG2x1+IncvIKJmi5OWM3aR/vEyJoBOchPBl+A4 [2024-01-18T17:13:32.699Z] 0xZO1vKiS2djxz8XHMkR70cxD8hv/zDJeIQbgBroF4Wa7rim5+y7jgO0O7Rs1zhT [2024-01-18T17:13:32.699Z] LDDStJ8aUKwcVRmOv7NFQxbQHXNx07kaUE2SQJfhryzZmotoBiymmQT2v0H4/4q0 [2024-01-18T17:13:32.699Z] 5dylRhSCWfwN5+It40/C3b1pbC2d9Iv/eO+B0wqg13dLF3xuA1JblEgnqA9mdNnp [2024-01-18T17:13:32.699Z] P6EgKTQBMiO6nPFyYIPqg1eWYkb1eXu9aEkmbw== [2024-01-18T17:13:32.699Z] -----END CERTIFICATE----- [2024-01-18T17:13:32.699Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:32.699Z] MIIDGDCCAgCgAwIBAgIUWf/qoBANFWF/a/yZr7wfLBMfItAwDQYJKoZIhvcNAQEL [2024-01-18T17:13:32.699Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-18T17:13:32.699Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-18T17:13:32.699Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-18T17:13:32.699Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-18T17:13:32.699Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-18T17:13:32.699Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-18T17:13:32.699Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-18T17:13:32.699Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-18T17:13:32.699Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-18T17:13:32.699Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-18T17:13:32.699Z] 9w0BAQsFAAOCAQEAdCguadZU6WJAuuAtWzvYgftPKkDKNBeGVgbqRwWRCB+QqVFl [2024-01-18T17:13:32.699Z] +3ZOQTStpnXJsY3Ig31hmL88BkBPQ47ENEzs7Czhuz59m+dcBhh/buUoIERSnfUY [2024-01-18T17:13:32.699Z] lWIfqxQevwHYZ3Kjx0l3qliYjKBQJo5xDPnukGb4yUnmrCdZ3fxMzlnr0Ais9rUH [2024-01-18T17:13:32.699Z] MjiDsjP/oUa8naQUTZO4DTRZqeEldzNXnxF6YH+3moTkwRLt9xzirNwECmJ9qxrC [2024-01-18T17:13:32.699Z] eCA7zORxbFSftYLnQiGWUS1dnxBuLOvJHjYPC6EkPPrvqnx2fXcTE1V8GizVRrA6 [2024-01-18T17:13:32.699Z] NPxt+e4qZgU/Nk/Jkpsb7h1FVcKtS8CvmtbdqQ== [2024-01-18T17:13:32.699Z] -----END CERTIFICATE----- [2024-01-18T17:13:32.699Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:32.699Z] MIIDGDCCAgCgAwIBAgIUI6D0WS4tl52N+w2lwAS+6V0DPyUwDQYJKoZIhvcNAQEL [2024-01-18T17:13:32.699Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-18T17:13:32.699Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-18T17:13:32.700Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-18T17:13:32.700Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-18T17:13:32.700Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-18T17:13:32.700Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-18T17:13:32.700Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-18T17:13:32.700Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-18T17:13:32.700Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-18T17:13:32.700Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-18T17:13:32.700Z] 9w0BAQsFAAOCAQEAAS4yElh6TMyP18ipzRA0Wenukevjs/vtkAFOsv1u5GNeDIrv [2024-01-18T17:13:32.700Z] lNi6/VjvtS+DoFNzQnTf3QoO6GTuoWHLQWt6Z/V5AwIsnOXduSXLugMwEv3j9l50 [2024-01-18T17:13:32.700Z] aDkA7vJ3OH5cGqIvIzxQN7IUOEvZNhY67GFP7+zAzVNRpJ1FHYbhQGPuFD//b9G3 [2024-01-18T17:13:32.700Z] hr4E3+iUEc+F/iBA+caA8wRx1zR383Jz6wuoFt7n7HpWNcgMvoh91JlUAK7yUp0B [2024-01-18T17:13:32.700Z] ekuG6q7HtVsk6X+aB1RyWfrSJjih75W7b+p+/bV4vNmQS1wSGSOaSIuVRvMhmpwf [2024-01-18T17:13:32.700Z] FpT+4rUpiCxklD69/MET/1cWxqvao/ppf6phzg== [2024-01-18T17:13:32.700Z] -----END CERTIFICATE----- [2024-01-18T17:13:32.700Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:32.700Z] MIIDGDCCAgCgAwIBAgIUIVCCgYV9enDck18n89YsA5b2IdgwDQYJKoZIhvcNAQEL [2024-01-18T17:13:32.700Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-18T17:13:32.700Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-18T17:13:32.700Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-18T17:13:32.700Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-18T17:13:32.700Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-18T17:13:32.700Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-18T17:13:32.700Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-18T17:13:32.700Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-18T17:13:32.700Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-18T17:13:32.700Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-18T17:13:32.700Z] 9w0BAQsFAAOCAQEAa3eFm1ivFf5Vom6a/7E1gDdjl7cjUZ5cqg332Ky1NdzEOiwW [2024-01-18T17:13:32.700Z] mx3yEk35qCd8LAoFMvPvPIuMx1NDS8kwy3FmIOjKnZapHA31ZUpn/gT/seH4GFIU [2024-01-18T17:13:32.700Z] oZw+rODV5h6jmtCVugAn48u/LxOXbY417JWwaeDx5uw+NzOSyebe+td2PbtUjf5X [2024-01-18T17:13:32.700Z] 6xcwYUmaUGsIAgfIMM12hzXOI2/q1qz5dAqwSgnR7dKBReOXnUoFpNOMpxlJ70p/ [2024-01-18T17:13:32.700Z] WUU1aWIWZqKrNI7Pg1T5jljq7+KbT3+A6+ux0FLExr2c1tw3NndXS2Y9fO55m5/q [2024-01-18T17:13:32.700Z] 0EcrTdKx4+QjbWcja3WdX2pGDk0XGNgx+EXn6Q== [2024-01-18T17:13:32.700Z] -----END CERTIFICATE----- [2024-01-18T17:13:32.700Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:32.700Z] MIIFATCCAumgAwIBAgIRAKc9ZKBASymy5TLOEp57N98wDQYJKoZIhvcNAQELBQAw [2024-01-18T17:13:32.700Z] GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDMyMzIyNTM0NloXDTM2 [2024-01-18T17:13:32.700Z] MDMyMzIyNTM0NlowGjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMIICIjANBgkq [2024-01-18T17:13:32.700Z] hkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA+pYHvQw5iU3v2b3iNuYNKYgsWD6KU7aJ [2024-01-18T17:13:32.700Z] diddtZQxSWYzUI3U0I1UsRPTxnhTifs/M9NW4ZlV13ZfB7APwC8oqKOIiwo7IwlP [2024-01-18T17:13:32.700Z] xg0VKgyz+kT8RJfYr66PPIYP0fpTeu42LpMJ+CKo9sbpgVNDZN2z/qiXrRNX/VtG [2024-01-18T17:13:32.700Z] TkPV7a44fZ5bHHVruAxvDnylpQxJobtCBWlJSsbIRGFHMc2z88eUz9NmIOWUKGGj [2024-01-18T17:13:32.700Z] EmP76x8OfRHpIpuxRSCjn0+i9+hR2siIOpcMOGd+40uVJxbRRP5ZXnUFa2fF5FWd [2024-01-18T17:13:32.700Z] O0u0RPI8HON0ovhrwPJY+4eWKkQzyC611oLPYGQ4EbifRsTsCxUZqyUuStGyp8oa [2024-01-18T17:13:32.700Z] aoSKfF6X0+KzGgwwnrjRTUpIl19A92KR0Noo6h622OX+4sZiO/JQdkuX5w/HupK0 [2024-01-18T17:13:32.700Z] A0M0WSMCvU6GOhjGotmh2VTEJwHHY4+TUk0iQYRtv1crONklyZoAQPD76hCrC8Cr [2024-01-18T17:13:32.700Z] IbgsZLfTMC8TWUoMbyUDgvgYkHKMoPm0VGVVuwpRKJxv7+2wXO+pivrrUl2Q9fPe [2024-01-18T17:13:32.700Z] Kk055nJLMV9yPUdig8othUKrRfSxli946AEV1eEOhxddfEwBE3Lt2xn0hhiIedbb [2024-01-18T17:13:32.700Z] Ftf/5kEWFZkXyUmMJK8Ra76Kus2ABueUVEcZ48hrRr1Hf1N9n59VbTUaXgeiZA50 [2024-01-18T17:13:32.700Z] qXf2bymE6F8CAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMB [2024-01-18T17:13:32.700Z] Af8wHQYDVR0OBBYEFMEmdKSKRKDm+iAo2FwjmkWIGHngMA0GCSqGSIb3DQEBCwUA [2024-01-18T17:13:32.700Z] A4ICAQBCPw74M9X/Xx04K1VAES3ypgQYH5bf9FXVDrwhRFSVckria/7dMzoF5wln [2024-01-18T17:13:32.700Z] uq9NGsjkkkDg17AohcQdr8alH4LvPdxpKr3BjpvEcmbqF8xH+MbbeUEnmbSfLI8H [2024-01-18T17:13:32.700Z] sefuhXF9AF/9iYvpVNC8FmJ0OhiVv13VgMQw0CRKkbtjZBf8xaEhq/YqxWVsgOjm [2024-01-18T17:13:32.700Z] dm5CAQ2X0aX7502x8wYRgMnZhA5goC1zVWBVAi8yhhmlhhoDUfg17cXkmaJC5pDd [2024-01-18T17:13:32.700Z] oenZ9NVhW8eDb03MFCrWNvIh89DDeCGWuWfDltDq0n3owyL0IeSn7RfpSclpxVmV [2024-01-18T17:13:32.700Z] /53jkYjwIgxIG7Gsv0LKMbsf6QdBcTjhvfZyMIpBRkTe3zuHd2feKzY9lEkbRvRQ [2024-01-18T17:13:32.700Z] zbh4Ps5YBnG6CKJPTbe2hfi3nhnw/MyEmF3zb0hzvLWNrR9XW3ibb2oL3424XOwc [2024-01-18T17:13:32.700Z] VjrTSCLzO9Rv6s5wi03qoWvKAQQAElqTYRHhynJ3w6wuvKYF5zcZF3MDnrVGLbh1 [2024-01-18T17:13:32.700Z] Q9ePRFBCiXOQ6wPLoUhrrbZ8LpFUFYDXHMtYM7P9sc9IAWoONXREJaO08zgFtMp4 [2024-01-18T17:13:32.701Z] 8iyIYUyQAbsvx8oD2M8kRvrIRSrRJSl6L957b4AFiLIQ/GgV2curs0jje7Edx34c [2024-01-18T17:13:32.701Z] idWw1VrejtwclobqNMVtG3EiPUIpJGpbMcJgbiLSmKkrvQtGng== [2024-01-18T17:13:32.701Z] -----END CERTIFICATE----- [2024-01-18T17:13:32.701Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} [2024-01-18T17:13:32.701Z] INFO[2024-01-18 17:13:32] Actual:true, Expected: true [2024-01-18T17:13:32.958Z] INFO[2024-01-18 17:13:32] Verifying : Description : Fetching configmap pointer &ConfigMap{ObjectMeta:{openshift-service-ca.crt mysql-ibm-pxb-1-84853-01-18-17h11m43s 93687f4d-8152-4f9a-97d1-2f8e8d8a2db0 1286692 0 2024-01-18 17:11:44 +0000 UTC map[] map[service.beta.openshift.io/inject-cabundle:true] [] [] [{kube-controller-manager Update v1 2024-01-18 17:11:44 +0000 UTC FieldsV1 {"f:data":{},"f:metadata":{"f:annotations":{".":{},"f:service.beta.openshift.io/inject-cabundle":{}}}} } {service-ca-operator Update v1 2024-01-18 17:11:44 +0000 UTC FieldsV1 {"f:data":{"f:service-ca.crt":{}}} }]},Data:map[string]string{service-ca.crt: -----BEGIN CERTIFICATE----- [2024-01-18T17:13:32.958Z] MIIDUTCCAjmgAwIBAgIILvRjno8BjvIwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE [2024-01-18T17:13:32.958Z] Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwNTM5NjA4ODAe [2024-01-18T17:13:32.958Z] Fw0yNDAxMTYwOTA4MDhaFw0yNjAzMTYwOTA4MDlaMDYxNDAyBgNVBAMMK29wZW5z [2024-01-18T17:13:32.958Z] aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDUzOTYwODgwggEiMA0GCSqG [2024-01-18T17:13:32.958Z] SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVr8xmLKsMoemVtfnlzzil6B8CGocEhNRo [2024-01-18T17:13:32.958Z] fkzkD8aSOYgIj5zcZo/M0O7nXDXeg9dqx9/h2qrGk+SS2ikU2Xs4Q5gaQDh9U1lB [2024-01-18T17:13:32.958Z] y8paoOEvw06CmJ/1Qqur5A539/OnimRSTdejTu4lwaDwsJiyoGfhvrBJAdjagtYA [2024-01-18T17:13:32.958Z] 0sNMrCgEAT4NWHS5EDx5dF7nMDHmQ0h0gHYei6vUEtiPyLwcTH1T7uVSj22fCEXX [2024-01-18T17:13:32.958Z] LjgN3OSiquJ+Hf4+csPpwgxAEkfjdBldhnImYkunJbg/x4c90uFL1pygZusrbFAf [2024-01-18T17:13:32.958Z] cERW8GngZuL5JY8MQoIstMu+1qS390ouQDDbEcvWjxC8YFNFHywrAgMBAAGjYzBh [2024-01-18T17:13:32.958Z] MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQS5vRk [2024-01-18T17:13:32.958Z] jkBPISX2qYmk1VEQZIXSLzAfBgNVHSMEGDAWgBQS5vRkjkBPISX2qYmk1VEQZIXS [2024-01-18T17:13:32.959Z] LzANBgkqhkiG9w0BAQsFAAOCAQEAZ/qc2vFGJZgDJCEUaPYuufuLRPSF68ixV9fk [2024-01-18T17:13:32.959Z] e5tBkRN8atCsEBygC/RUcwULlhsdgUMGMcREgNC/FYlwkOsAj3gP8KmcXAZ4gHCa [2024-01-18T17:13:32.959Z] T6PIbIcj1hCX0mdX0tvtaj7PWvRUz9CHh6iFzmPsXK5xjaBA9vZbkUAa7odguUit [2024-01-18T17:13:32.959Z] vz1QnKoV1OaZyccDLeQL50khM1XEBn8L+zeY3k85uo2X514HZW4LDPTQ94cpyNnb [2024-01-18T17:13:32.959Z] zqI6utp7JoPJexsHMWNFKGb//HgjluNg7TsywXIeebk+/GZUEDfNKid0oiTpQvzR [2024-01-18T17:13:32.959Z] TFFF+5KO/sa4+qIh1ERm140JxeaurUMaJzn7cFQbekEgt/N+iA== [2024-01-18T17:13:32.959Z] -----END CERTIFICATE----- [2024-01-18T17:13:32.959Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} from namespace mysql-ibm-pxb-1-84853-01-18-17h11m43s [2024-01-18T17:13:32.959Z] INFO[2024-01-18 17:13:32] Actual:true, Expected: true [2024-01-18T17:13:32.959Z] 2024-01-18 17:13:32 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.AddLabelToResource:#5038] - Adding label [abe5cf54-4e74-45b4-882f-82b02f3dd9cf=cd7d2c21-9adb-41bf-b01f-ae218113bcc9] to ConfigMap openshift-service-ca.crt [2024-01-18T17:13:33.215Z] INFO[2024-01-18 17:13:33] Verifying : Description : Adding labels abe5cf54-4e74-45b4-882f-82b02f3dd9cf=cd7d2c21-9adb-41bf-b01f-ae218113bcc9 to resource &ConfigMap{ObjectMeta:{openshift-service-ca.crt mysql-ibm-pxb-1-84853-01-18-17h11m43s 93687f4d-8152-4f9a-97d1-2f8e8d8a2db0 1286692 0 2024-01-18 17:11:44 +0000 UTC map[abe5cf54-4e74-45b4-882f-82b02f3dd9cf:cd7d2c21-9adb-41bf-b01f-ae218113bcc9] map[service.beta.openshift.io/inject-cabundle:true] [] [] [{kube-controller-manager Update v1 2024-01-18 17:11:44 +0000 UTC FieldsV1 {"f:data":{},"f:metadata":{"f:annotations":{".":{},"f:service.beta.openshift.io/inject-cabundle":{}}}} } {service-ca-operator Update v1 2024-01-18 17:11:44 +0000 UTC FieldsV1 {"f:data":{"f:service-ca.crt":{}}} }]},Data:map[string]string{service-ca.crt: -----BEGIN CERTIFICATE----- [2024-01-18T17:13:33.215Z] MIIDUTCCAjmgAwIBAgIILvRjno8BjvIwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE [2024-01-18T17:13:33.215Z] Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwNTM5NjA4ODAe [2024-01-18T17:13:33.215Z] Fw0yNDAxMTYwOTA4MDhaFw0yNjAzMTYwOTA4MDlaMDYxNDAyBgNVBAMMK29wZW5z [2024-01-18T17:13:33.215Z] aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDUzOTYwODgwggEiMA0GCSqG [2024-01-18T17:13:33.215Z] SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVr8xmLKsMoemVtfnlzzil6B8CGocEhNRo [2024-01-18T17:13:33.215Z] fkzkD8aSOYgIj5zcZo/M0O7nXDXeg9dqx9/h2qrGk+SS2ikU2Xs4Q5gaQDh9U1lB [2024-01-18T17:13:33.215Z] y8paoOEvw06CmJ/1Qqur5A539/OnimRSTdejTu4lwaDwsJiyoGfhvrBJAdjagtYA [2024-01-18T17:13:33.215Z] 0sNMrCgEAT4NWHS5EDx5dF7nMDHmQ0h0gHYei6vUEtiPyLwcTH1T7uVSj22fCEXX [2024-01-18T17:13:33.215Z] LjgN3OSiquJ+Hf4+csPpwgxAEkfjdBldhnImYkunJbg/x4c90uFL1pygZusrbFAf [2024-01-18T17:13:33.215Z] cERW8GngZuL5JY8MQoIstMu+1qS390ouQDDbEcvWjxC8YFNFHywrAgMBAAGjYzBh [2024-01-18T17:13:33.215Z] MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQS5vRk [2024-01-18T17:13:33.215Z] jkBPISX2qYmk1VEQZIXSLzAfBgNVHSMEGDAWgBQS5vRkjkBPISX2qYmk1VEQZIXS [2024-01-18T17:13:33.215Z] LzANBgkqhkiG9w0BAQsFAAOCAQEAZ/qc2vFGJZgDJCEUaPYuufuLRPSF68ixV9fk [2024-01-18T17:13:33.215Z] e5tBkRN8atCsEBygC/RUcwULlhsdgUMGMcREgNC/FYlwkOsAj3gP8KmcXAZ4gHCa [2024-01-18T17:13:33.215Z] T6PIbIcj1hCX0mdX0tvtaj7PWvRUz9CHh6iFzmPsXK5xjaBA9vZbkUAa7odguUit [2024-01-18T17:13:33.215Z] vz1QnKoV1OaZyccDLeQL50khM1XEBn8L+zeY3k85uo2X514HZW4LDPTQ94cpyNnb [2024-01-18T17:13:33.216Z] zqI6utp7JoPJexsHMWNFKGb//HgjluNg7TsywXIeebk+/GZUEDfNKid0oiTpQvzR [2024-01-18T17:13:33.216Z] TFFF+5KO/sa4+qIh1ERm140JxeaurUMaJzn7cFQbekEgt/N+iA== [2024-01-18T17:13:33.216Z] -----END CERTIFICATE----- [2024-01-18T17:13:33.216Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} [2024-01-18T17:13:33.216Z] INFO[2024-01-18 17:13:33] Actual:true, Expected: true [2024-01-18T17:13:33.471Z] INFO[2024-01-18 17:13:33] Verifying : Description : Fetching pvc list from namespace [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:13:33.471Z] INFO[2024-01-18 17:13:33] Actual:true, Expected: true [2024-01-18T17:13:33.472Z] INFO[2024-01-18 17:13:33] Verifying : Description : Fetching pvc pointer &PersistentVolumeClaim{ObjectMeta:{mysql-data mysql-ibm-pxb-2-84853-01-18-17h11m43s 84174695-6ec8-4f5f-a002-f00fd67e8e6e 1286932 0 2024-01-18 17:11:45 +0000 UTC map[] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-18 17:11:45 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-18 17:11:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-18 17:11:55 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{2147483648 0} {} 2Gi BinarySI},},},VolumeName:pvc-84174695-6ec8-4f5f-a002-f00fd67e8e6e,Selector:nil,StorageClassName:*mysql-sc,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{10737418240 0} {} 10Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},} for pvc {{ } {mysql-data mysql-ibm-pxb-2-84853-01-18-17h11m43s 84174695-6ec8-4f5f-a002-f00fd67e8e6e 1286932 0 2024-01-18 17:11:45 +0000 UTC map[] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-18 17:11:45 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-18 17:11:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-18 17:11:55 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{2147483648 0} {} 2Gi BinarySI}]} pvc-84174695-6ec8-4f5f-a002-f00fd67e8e6e 0xc0010fb440 0xc0010fb450 nil nil} {Bound [ReadWriteOnce] map[storage:{{10737418240 0} {} 10Gi BinarySI}] [] map[] }} [2024-01-18T17:13:33.472Z] INFO[2024-01-18 17:13:33] Actual:true, Expected: true [2024-01-18T17:13:33.472Z] 2024-01-18 17:13:33 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.AddLabelToResource:#5030] - Adding label [abe5cf54-4e74-45b4-882f-82b02f3dd9cf=cd7d2c21-9adb-41bf-b01f-ae218113bcc9] to PVC mysql-data [2024-01-18T17:13:33.728Z] INFO[2024-01-18 17:13:33] Verifying : Description : Adding labels abe5cf54-4e74-45b4-882f-82b02f3dd9cf=cd7d2c21-9adb-41bf-b01f-ae218113bcc9 to resource &PersistentVolumeClaim{ObjectMeta:{mysql-data mysql-ibm-pxb-2-84853-01-18-17h11m43s 84174695-6ec8-4f5f-a002-f00fd67e8e6e 1286932 0 2024-01-18 17:11:45 +0000 UTC map[abe5cf54-4e74-45b4-882f-82b02f3dd9cf:cd7d2c21-9adb-41bf-b01f-ae218113bcc9] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-18 17:11:45 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-18 17:11:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-18 17:11:55 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{2147483648 0} {} 2Gi BinarySI},},},VolumeName:pvc-84174695-6ec8-4f5f-a002-f00fd67e8e6e,Selector:nil,StorageClassName:*mysql-sc,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{10737418240 0} {} 10Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},} [2024-01-18T17:13:33.728Z] INFO[2024-01-18 17:13:33] Actual:true, Expected: true [2024-01-18T17:13:33.984Z] INFO[2024-01-18 17:13:33] Verifying : Description : Fetching configmap list from namespace [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:13:33.984Z] INFO[2024-01-18 17:13:33] Actual:true, Expected: true [2024-01-18T17:13:34.240Z] INFO[2024-01-18 17:13:34] Verifying : Description : Fetching configmap pointer &ConfigMap{ObjectMeta:{kube-****-ca.crt mysql-ibm-pxb-2-84853-01-18-17h11m43s a74c378c-86f9-45da-bcdc-e5d7c6e58ca4 1286735 0 2024-01-18 17:11:44 +0000 UTC map[] map[kubernetes.io/description:Contains a CA bundle that can be used to verify the kube-apiserver when using internal endpoints such as the internal service IP or kubernetes.default.svc. No other usage is guaranteed across distributions of Kubernetes clusters.] [] [] [{kube-controller-manager Update v1 2024-01-18 17:11:44 +0000 UTC FieldsV1 {"f:data":{".":{},"f:ca.crt":{}},"f:metadata":{"f:annotations":{".":{},"f:kubernetes.io/description":{}}}} }]},Data:map[string]string{ca.crt: -----BEGIN CERTIFICATE----- [2024-01-18T17:13:34.240Z] MIIDCjCCAfKgAwIBAgIUWkn6rgBaC+VWV1XdJjGA3DwoRRYwDQYJKoZIhvcNAQEL [2024-01-18T17:13:34.240Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-18T17:13:34.240Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-18T17:13:34.240Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-18T17:13:34.240Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-18T17:13:34.240Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-18T17:13:34.240Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-18T17:13:34.240Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-18T17:13:34.240Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-18T17:13:34.240Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-18T17:13:34.240Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAHbD [2024-01-18T17:13:34.240Z] 4nWTgt/8sat7SbbHxZ+JcMGRAIaeLAjMJxkxyqD8xUq6YQcgbjDVHo1Qbh3PXB3T [2024-01-18T17:13:34.240Z] 5BWUsRbDoKi0eRJx/5cyI3846SYEAOpuUfJrg9X2CH7ZoYhxeJ42dqt3MdoMJEM1 [2024-01-18T17:13:34.240Z] k3e2rRqAKuAxwNXugHryxQM+1ehOJjrEuT5DVK6qu8+JYRcT1hBZb6hrebKPp5eG [2024-01-18T17:13:34.240Z] JvBzHP4YOATIAXhfH0Uj5Y+JhwoOdpXmOFpz+6gITPD4aNzzJZe/0ADSFfOyUJaR [2024-01-18T17:13:34.240Z] g5NnJnkeGqMM3SLsur25r+m3y0i3t+RdocZ/4mHTEnF4KWVWvZy4GjENiEBcXEol [2024-01-18T17:13:34.240Z] o4SZVaP4AF7XQJ5vdA0= [2024-01-18T17:13:34.240Z] -----END CERTIFICATE----- [2024-01-18T17:13:34.240Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:34.240Z] MIIDCjCCAfKgAwIBAgIUXhIIFCc/NxZJn6hNEFoZuPQ8N5UwDQYJKoZIhvcNAQEL [2024-01-18T17:13:34.240Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-18T17:13:34.240Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-18T17:13:34.240Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-18T17:13:34.240Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-18T17:13:34.240Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-18T17:13:34.240Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-18T17:13:34.240Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-18T17:13:34.241Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-18T17:13:34.241Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-18T17:13:34.241Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAKG3 [2024-01-18T17:13:34.241Z] nVfv4IkgHVOxgbrNIoYhjvBvVJcLZub0C9yidexLA3A61p1a7b/OpidG5zShPhH2 [2024-01-18T17:13:34.241Z] HRdQcqgp0HIxaS+/oLD3eElOnnC/EKD359K6n3kN66sMgvNbHxNFM84Au0TXqmbT [2024-01-18T17:13:34.241Z] w8afrKjrfe/VglpHC2GfmgHMF4YH+ZF48C1RZA6z2oXj6Aj+uww0PjiClE9qUYqK [2024-01-18T17:13:34.241Z] R9uRa32prExihsi+Bto7n2Axo92qUiK2KrkEdQitnQfH5z/PojS78vbyIKp1O86N [2024-01-18T17:13:34.241Z] qRImvCYY17CPxP+JUQ7dxWDMF4NTgSnabkp+cd5i/pyoAb0guQFlQQUnHAZa4QNC [2024-01-18T17:13:34.241Z] 7fgTo1IUwWyDmwCnN5U= [2024-01-18T17:13:34.241Z] -----END CERTIFICATE----- [2024-01-18T17:13:34.241Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:34.241Z] MIIDCjCCAfKgAwIBAgIUJPzO3O+wmUt4gHCkl3ojsvwZL6owDQYJKoZIhvcNAQEL [2024-01-18T17:13:34.241Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-18T17:13:34.241Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-18T17:13:34.241Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-18T17:13:34.241Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-18T17:13:34.241Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-18T17:13:34.241Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-18T17:13:34.241Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-18T17:13:34.241Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-18T17:13:34.241Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-18T17:13:34.241Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAGVt [2024-01-18T17:13:34.241Z] g9qX+r8kyGYsvFavmnDxgMK1Zkz80irQJ2BJlDHdK1hp5aqpRbT2z2YgLog5ZMIZ [2024-01-18T17:13:34.241Z] uI36uPCOXo8+Gol1jT7D6vsgMWm0Tk9jCNbO4jPkv+wm2q1B9UZo94UGiS0K+G0S [2024-01-18T17:13:34.241Z] nccl4jmSq1gWpBZIVetdDFNjpipKjhEOVQM7x33APhACxSAWRp9iEr6V/CzMOgkf [2024-01-18T17:13:34.241Z] RHyz5D7ljh4a2XVH7++eKsj2QPfKyiKtIG72tCccQ9zGnzveiXXb4Ai8r+DyKH4Z [2024-01-18T17:13:34.241Z] zDt5Hb4ZFl72E3Rl4bIh/VQBVsr9u/SFR1p/Emyq+kGXsn+9YMiM9tK4mLUtjdox [2024-01-18T17:13:34.241Z] LMWUWTelNRGWAiaTVz8= [2024-01-18T17:13:34.241Z] -----END CERTIFICATE----- [2024-01-18T17:13:34.241Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:34.241Z] MIIDCjCCAfKgAwIBAgIUBxbEoan/HnRZ+2deRj+2RvGMrWQwDQYJKoZIhvcNAQEL [2024-01-18T17:13:34.241Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-18T17:13:34.241Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-18T17:13:34.241Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-18T17:13:34.241Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-18T17:13:34.241Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-18T17:13:34.241Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-18T17:13:34.241Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-18T17:13:34.241Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-18T17:13:34.241Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-18T17:13:34.241Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAB65 [2024-01-18T17:13:34.241Z] nHU+98oTP0U2J0/TsL5GFeBTH7YyUXUNRJvPWxkA8riB3swoScFKbavO6XJ/hrI2 [2024-01-18T17:13:34.241Z] ywApOz77Iyq58OLa5aIC5aaPRf3YE8LbJQYIHmtza21LXhtlKVSrLu4W58PauI8Z [2024-01-18T17:13:34.241Z] 7iaL9QPUihHFK7wrZ2ZpabN2rRJvTGWWKlhmgFTjI+71gPrXWmYBHIKmTv55V5m1 [2024-01-18T17:13:34.241Z] LxORTR/gBXeH9gmM7wz1cbaYNazqs8R1LS5SW76p/y9RLjTzU6N39VYefMMGwbsQ [2024-01-18T17:13:34.241Z] DlD11wvMBxTBp3hQYc3pdVLkVEQt8trVIwEUts7zfwUTCu/z8gH7H5XePpEVNana [2024-01-18T17:13:34.241Z] JhlAcXHCO7nN/ddJmoc= [2024-01-18T17:13:34.241Z] -----END CERTIFICATE----- [2024-01-18T17:13:34.241Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:34.241Z] MIIDGDCCAgCgAwIBAgIUEvLaarpXlv1cVvG6nMuxh2DNCKQwDQYJKoZIhvcNAQEL [2024-01-18T17:13:34.241Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-18T17:13:34.241Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-18T17:13:34.241Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-18T17:13:34.241Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-18T17:13:34.241Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-18T17:13:34.241Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-18T17:13:34.241Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-18T17:13:34.241Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-18T17:13:34.241Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-18T17:13:34.241Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-18T17:13:34.241Z] 9w0BAQsFAAOCAQEAOPakzVUmAqHhmQViZZHSQeRh/xYQBrtn6dnEo6X55SXNWcPU [2024-01-18T17:13:34.241Z] 4EEW3DvwBf75uqxKeGoXHTBpqNHG2x1+IncvIKJmi5OWM3aR/vEyJoBOchPBl+A4 [2024-01-18T17:13:34.241Z] 0xZO1vKiS2djxz8XHMkR70cxD8hv/zDJeIQbgBroF4Wa7rim5+y7jgO0O7Rs1zhT [2024-01-18T17:13:34.241Z] LDDStJ8aUKwcVRmOv7NFQxbQHXNx07kaUE2SQJfhryzZmotoBiymmQT2v0H4/4q0 [2024-01-18T17:13:34.242Z] 5dylRhSCWfwN5+It40/C3b1pbC2d9Iv/eO+B0wqg13dLF3xuA1JblEgnqA9mdNnp [2024-01-18T17:13:34.242Z] P6EgKTQBMiO6nPFyYIPqg1eWYkb1eXu9aEkmbw== [2024-01-18T17:13:34.242Z] -----END CERTIFICATE----- [2024-01-18T17:13:34.242Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:34.242Z] MIIDGDCCAgCgAwIBAgIUWf/qoBANFWF/a/yZr7wfLBMfItAwDQYJKoZIhvcNAQEL [2024-01-18T17:13:34.242Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-18T17:13:34.242Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-18T17:13:34.242Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-18T17:13:34.242Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-18T17:13:34.242Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-18T17:13:34.242Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-18T17:13:34.242Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-18T17:13:34.242Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-18T17:13:34.242Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-18T17:13:34.242Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-18T17:13:34.242Z] 9w0BAQsFAAOCAQEAdCguadZU6WJAuuAtWzvYgftPKkDKNBeGVgbqRwWRCB+QqVFl [2024-01-18T17:13:34.242Z] +3ZOQTStpnXJsY3Ig31hmL88BkBPQ47ENEzs7Czhuz59m+dcBhh/buUoIERSnfUY [2024-01-18T17:13:34.242Z] lWIfqxQevwHYZ3Kjx0l3qliYjKBQJo5xDPnukGb4yUnmrCdZ3fxMzlnr0Ais9rUH [2024-01-18T17:13:34.242Z] MjiDsjP/oUa8naQUTZO4DTRZqeEldzNXnxF6YH+3moTkwRLt9xzirNwECmJ9qxrC [2024-01-18T17:13:34.242Z] eCA7zORxbFSftYLnQiGWUS1dnxBuLOvJHjYPC6EkPPrvqnx2fXcTE1V8GizVRrA6 [2024-01-18T17:13:34.242Z] NPxt+e4qZgU/Nk/Jkpsb7h1FVcKtS8CvmtbdqQ== [2024-01-18T17:13:34.242Z] -----END CERTIFICATE----- [2024-01-18T17:13:34.242Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:34.242Z] MIIDGDCCAgCgAwIBAgIUI6D0WS4tl52N+w2lwAS+6V0DPyUwDQYJKoZIhvcNAQEL [2024-01-18T17:13:34.242Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-18T17:13:34.242Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-18T17:13:34.242Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-18T17:13:34.242Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-18T17:13:34.242Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-18T17:13:34.242Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-18T17:13:34.242Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-18T17:13:34.242Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-18T17:13:34.242Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-18T17:13:34.242Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-18T17:13:34.242Z] 9w0BAQsFAAOCAQEAAS4yElh6TMyP18ipzRA0Wenukevjs/vtkAFOsv1u5GNeDIrv [2024-01-18T17:13:34.242Z] lNi6/VjvtS+DoFNzQnTf3QoO6GTuoWHLQWt6Z/V5AwIsnOXduSXLugMwEv3j9l50 [2024-01-18T17:13:34.242Z] aDkA7vJ3OH5cGqIvIzxQN7IUOEvZNhY67GFP7+zAzVNRpJ1FHYbhQGPuFD//b9G3 [2024-01-18T17:13:34.242Z] hr4E3+iUEc+F/iBA+caA8wRx1zR383Jz6wuoFt7n7HpWNcgMvoh91JlUAK7yUp0B [2024-01-18T17:13:34.242Z] ekuG6q7HtVsk6X+aB1RyWfrSJjih75W7b+p+/bV4vNmQS1wSGSOaSIuVRvMhmpwf [2024-01-18T17:13:34.242Z] FpT+4rUpiCxklD69/MET/1cWxqvao/ppf6phzg== [2024-01-18T17:13:34.242Z] -----END CERTIFICATE----- [2024-01-18T17:13:34.242Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:34.242Z] MIIDGDCCAgCgAwIBAgIUIVCCgYV9enDck18n89YsA5b2IdgwDQYJKoZIhvcNAQEL [2024-01-18T17:13:34.242Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-18T17:13:34.242Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-18T17:13:34.242Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-18T17:13:34.242Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-18T17:13:34.242Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-18T17:13:34.242Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-18T17:13:34.242Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-18T17:13:34.242Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-18T17:13:34.242Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-18T17:13:34.242Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-18T17:13:34.242Z] 9w0BAQsFAAOCAQEAa3eFm1ivFf5Vom6a/7E1gDdjl7cjUZ5cqg332Ky1NdzEOiwW [2024-01-18T17:13:34.242Z] mx3yEk35qCd8LAoFMvPvPIuMx1NDS8kwy3FmIOjKnZapHA31ZUpn/gT/seH4GFIU [2024-01-18T17:13:34.242Z] oZw+rODV5h6jmtCVugAn48u/LxOXbY417JWwaeDx5uw+NzOSyebe+td2PbtUjf5X [2024-01-18T17:13:34.242Z] 6xcwYUmaUGsIAgfIMM12hzXOI2/q1qz5dAqwSgnR7dKBReOXnUoFpNOMpxlJ70p/ [2024-01-18T17:13:34.242Z] WUU1aWIWZqKrNI7Pg1T5jljq7+KbT3+A6+ux0FLExr2c1tw3NndXS2Y9fO55m5/q [2024-01-18T17:13:34.242Z] 0EcrTdKx4+QjbWcja3WdX2pGDk0XGNgx+EXn6Q== [2024-01-18T17:13:34.242Z] -----END CERTIFICATE----- [2024-01-18T17:13:34.242Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:34.242Z] MIIFATCCAumgAwIBAgIRAKc9ZKBASymy5TLOEp57N98wDQYJKoZIhvcNAQELBQAw [2024-01-18T17:13:34.242Z] GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDMyMzIyNTM0NloXDTM2 [2024-01-18T17:13:34.243Z] MDMyMzIyNTM0NlowGjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMIICIjANBgkq [2024-01-18T17:13:34.243Z] hkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA+pYHvQw5iU3v2b3iNuYNKYgsWD6KU7aJ [2024-01-18T17:13:34.243Z] diddtZQxSWYzUI3U0I1UsRPTxnhTifs/M9NW4ZlV13ZfB7APwC8oqKOIiwo7IwlP [2024-01-18T17:13:34.243Z] xg0VKgyz+kT8RJfYr66PPIYP0fpTeu42LpMJ+CKo9sbpgVNDZN2z/qiXrRNX/VtG [2024-01-18T17:13:34.243Z] TkPV7a44fZ5bHHVruAxvDnylpQxJobtCBWlJSsbIRGFHMc2z88eUz9NmIOWUKGGj [2024-01-18T17:13:34.243Z] EmP76x8OfRHpIpuxRSCjn0+i9+hR2siIOpcMOGd+40uVJxbRRP5ZXnUFa2fF5FWd [2024-01-18T17:13:34.243Z] O0u0RPI8HON0ovhrwPJY+4eWKkQzyC611oLPYGQ4EbifRsTsCxUZqyUuStGyp8oa [2024-01-18T17:13:34.243Z] aoSKfF6X0+KzGgwwnrjRTUpIl19A92KR0Noo6h622OX+4sZiO/JQdkuX5w/HupK0 [2024-01-18T17:13:34.243Z] A0M0WSMCvU6GOhjGotmh2VTEJwHHY4+TUk0iQYRtv1crONklyZoAQPD76hCrC8Cr [2024-01-18T17:13:34.243Z] IbgsZLfTMC8TWUoMbyUDgvgYkHKMoPm0VGVVuwpRKJxv7+2wXO+pivrrUl2Q9fPe [2024-01-18T17:13:34.243Z] Kk055nJLMV9yPUdig8othUKrRfSxli946AEV1eEOhxddfEwBE3Lt2xn0hhiIedbb [2024-01-18T17:13:34.243Z] Ftf/5kEWFZkXyUmMJK8Ra76Kus2ABueUVEcZ48hrRr1Hf1N9n59VbTUaXgeiZA50 [2024-01-18T17:13:34.243Z] qXf2bymE6F8CAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMB [2024-01-18T17:13:34.243Z] Af8wHQYDVR0OBBYEFMEmdKSKRKDm+iAo2FwjmkWIGHngMA0GCSqGSIb3DQEBCwUA [2024-01-18T17:13:34.243Z] A4ICAQBCPw74M9X/Xx04K1VAES3ypgQYH5bf9FXVDrwhRFSVckria/7dMzoF5wln [2024-01-18T17:13:34.243Z] uq9NGsjkkkDg17AohcQdr8alH4LvPdxpKr3BjpvEcmbqF8xH+MbbeUEnmbSfLI8H [2024-01-18T17:13:34.243Z] sefuhXF9AF/9iYvpVNC8FmJ0OhiVv13VgMQw0CRKkbtjZBf8xaEhq/YqxWVsgOjm [2024-01-18T17:13:34.243Z] dm5CAQ2X0aX7502x8wYRgMnZhA5goC1zVWBVAi8yhhmlhhoDUfg17cXkmaJC5pDd [2024-01-18T17:13:34.243Z] oenZ9NVhW8eDb03MFCrWNvIh89DDeCGWuWfDltDq0n3owyL0IeSn7RfpSclpxVmV [2024-01-18T17:13:34.243Z] /53jkYjwIgxIG7Gsv0LKMbsf6QdBcTjhvfZyMIpBRkTe3zuHd2feKzY9lEkbRvRQ [2024-01-18T17:13:34.243Z] zbh4Ps5YBnG6CKJPTbe2hfi3nhnw/MyEmF3zb0hzvLWNrR9XW3ibb2oL3424XOwc [2024-01-18T17:13:34.243Z] VjrTSCLzO9Rv6s5wi03qoWvKAQQAElqTYRHhynJ3w6wuvKYF5zcZF3MDnrVGLbh1 [2024-01-18T17:13:34.243Z] Q9ePRFBCiXOQ6wPLoUhrrbZ8LpFUFYDXHMtYM7P9sc9IAWoONXREJaO08zgFtMp4 [2024-01-18T17:13:34.243Z] 8iyIYUyQAbsvx8oD2M8kRvrIRSrRJSl6L957b4AFiLIQ/GgV2curs0jje7Edx34c [2024-01-18T17:13:34.243Z] idWw1VrejtwclobqNMVtG3EiPUIpJGpbMcJgbiLSmKkrvQtGng== [2024-01-18T17:13:34.243Z] -----END CERTIFICATE----- [2024-01-18T17:13:34.243Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} from namespace mysql-ibm-pxb-2-84853-01-18-17h11m43s [2024-01-18T17:13:34.243Z] INFO[2024-01-18 17:13:34] Actual:true, Expected: true [2024-01-18T17:13:34.243Z] 2024-01-18 17:13:34 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.AddLabelToResource:#5038] - Adding label [abe5cf54-4e74-45b4-882f-82b02f3dd9cf=cd7d2c21-9adb-41bf-b01f-ae218113bcc9] to ConfigMap kube-****-ca.crt [2024-01-18T17:13:34.500Z] INFO[2024-01-18 17:13:34] Verifying : Description : Adding labels abe5cf54-4e74-45b4-882f-82b02f3dd9cf=cd7d2c21-9adb-41bf-b01f-ae218113bcc9 to resource &ConfigMap{ObjectMeta:{kube-****-ca.crt mysql-ibm-pxb-2-84853-01-18-17h11m43s a74c378c-86f9-45da-bcdc-e5d7c6e58ca4 1286735 0 2024-01-18 17:11:44 +0000 UTC map[abe5cf54-4e74-45b4-882f-82b02f3dd9cf:cd7d2c21-9adb-41bf-b01f-ae218113bcc9] map[kubernetes.io/description:Contains a CA bundle that can be used to verify the kube-apiserver when using internal endpoints such as the internal service IP or kubernetes.default.svc. No other usage is guaranteed across distributions of Kubernetes clusters.] [] [] [{kube-controller-manager Update v1 2024-01-18 17:11:44 +0000 UTC FieldsV1 {"f:data":{".":{},"f:ca.crt":{}},"f:metadata":{"f:annotations":{".":{},"f:kubernetes.io/description":{}}}} }]},Data:map[string]string{ca.crt: -----BEGIN CERTIFICATE----- [2024-01-18T17:13:34.500Z] MIIDCjCCAfKgAwIBAgIUWkn6rgBaC+VWV1XdJjGA3DwoRRYwDQYJKoZIhvcNAQEL [2024-01-18T17:13:34.500Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-18T17:13:34.500Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-18T17:13:34.500Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-18T17:13:34.500Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-18T17:13:34.500Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-18T17:13:34.500Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-18T17:13:34.500Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-18T17:13:34.500Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-18T17:13:34.500Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-18T17:13:34.500Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAHbD [2024-01-18T17:13:34.500Z] 4nWTgt/8sat7SbbHxZ+JcMGRAIaeLAjMJxkxyqD8xUq6YQcgbjDVHo1Qbh3PXB3T [2024-01-18T17:13:34.500Z] 5BWUsRbDoKi0eRJx/5cyI3846SYEAOpuUfJrg9X2CH7ZoYhxeJ42dqt3MdoMJEM1 [2024-01-18T17:13:34.500Z] k3e2rRqAKuAxwNXugHryxQM+1ehOJjrEuT5DVK6qu8+JYRcT1hBZb6hrebKPp5eG [2024-01-18T17:13:34.500Z] JvBzHP4YOATIAXhfH0Uj5Y+JhwoOdpXmOFpz+6gITPD4aNzzJZe/0ADSFfOyUJaR [2024-01-18T17:13:34.500Z] g5NnJnkeGqMM3SLsur25r+m3y0i3t+RdocZ/4mHTEnF4KWVWvZy4GjENiEBcXEol [2024-01-18T17:13:34.500Z] o4SZVaP4AF7XQJ5vdA0= [2024-01-18T17:13:34.500Z] -----END CERTIFICATE----- [2024-01-18T17:13:34.500Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:34.500Z] MIIDCjCCAfKgAwIBAgIUXhIIFCc/NxZJn6hNEFoZuPQ8N5UwDQYJKoZIhvcNAQEL [2024-01-18T17:13:34.500Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-18T17:13:34.500Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-18T17:13:34.500Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-18T17:13:34.500Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-18T17:13:34.500Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-18T17:13:34.500Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-18T17:13:34.500Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-18T17:13:34.500Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-18T17:13:34.500Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-18T17:13:34.500Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAKG3 [2024-01-18T17:13:34.501Z] nVfv4IkgHVOxgbrNIoYhjvBvVJcLZub0C9yidexLA3A61p1a7b/OpidG5zShPhH2 [2024-01-18T17:13:34.501Z] HRdQcqgp0HIxaS+/oLD3eElOnnC/EKD359K6n3kN66sMgvNbHxNFM84Au0TXqmbT [2024-01-18T17:13:34.501Z] w8afrKjrfe/VglpHC2GfmgHMF4YH+ZF48C1RZA6z2oXj6Aj+uww0PjiClE9qUYqK [2024-01-18T17:13:34.501Z] R9uRa32prExihsi+Bto7n2Axo92qUiK2KrkEdQitnQfH5z/PojS78vbyIKp1O86N [2024-01-18T17:13:34.501Z] qRImvCYY17CPxP+JUQ7dxWDMF4NTgSnabkp+cd5i/pyoAb0guQFlQQUnHAZa4QNC [2024-01-18T17:13:34.501Z] 7fgTo1IUwWyDmwCnN5U= [2024-01-18T17:13:34.501Z] -----END CERTIFICATE----- [2024-01-18T17:13:34.501Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:34.501Z] MIIDCjCCAfKgAwIBAgIUJPzO3O+wmUt4gHCkl3ojsvwZL6owDQYJKoZIhvcNAQEL [2024-01-18T17:13:34.501Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-18T17:13:34.501Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-18T17:13:34.501Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-18T17:13:34.501Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-18T17:13:34.501Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-18T17:13:34.501Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-18T17:13:34.501Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-18T17:13:34.501Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-18T17:13:34.501Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-18T17:13:34.501Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAGVt [2024-01-18T17:13:34.501Z] g9qX+r8kyGYsvFavmnDxgMK1Zkz80irQJ2BJlDHdK1hp5aqpRbT2z2YgLog5ZMIZ [2024-01-18T17:13:34.501Z] uI36uPCOXo8+Gol1jT7D6vsgMWm0Tk9jCNbO4jPkv+wm2q1B9UZo94UGiS0K+G0S [2024-01-18T17:13:34.501Z] nccl4jmSq1gWpBZIVetdDFNjpipKjhEOVQM7x33APhACxSAWRp9iEr6V/CzMOgkf [2024-01-18T17:13:34.501Z] RHyz5D7ljh4a2XVH7++eKsj2QPfKyiKtIG72tCccQ9zGnzveiXXb4Ai8r+DyKH4Z [2024-01-18T17:13:34.501Z] zDt5Hb4ZFl72E3Rl4bIh/VQBVsr9u/SFR1p/Emyq+kGXsn+9YMiM9tK4mLUtjdox [2024-01-18T17:13:34.501Z] LMWUWTelNRGWAiaTVz8= [2024-01-18T17:13:34.501Z] -----END CERTIFICATE----- [2024-01-18T17:13:34.501Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:34.501Z] MIIDCjCCAfKgAwIBAgIUBxbEoan/HnRZ+2deRj+2RvGMrWQwDQYJKoZIhvcNAQEL [2024-01-18T17:13:34.501Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-18T17:13:34.501Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-18T17:13:34.501Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-18T17:13:34.501Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-18T17:13:34.501Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-18T17:13:34.501Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-18T17:13:34.501Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-18T17:13:34.501Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-18T17:13:34.501Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-18T17:13:34.501Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAB65 [2024-01-18T17:13:34.501Z] nHU+98oTP0U2J0/TsL5GFeBTH7YyUXUNRJvPWxkA8riB3swoScFKbavO6XJ/hrI2 [2024-01-18T17:13:34.501Z] ywApOz77Iyq58OLa5aIC5aaPRf3YE8LbJQYIHmtza21LXhtlKVSrLu4W58PauI8Z [2024-01-18T17:13:34.501Z] 7iaL9QPUihHFK7wrZ2ZpabN2rRJvTGWWKlhmgFTjI+71gPrXWmYBHIKmTv55V5m1 [2024-01-18T17:13:34.501Z] LxORTR/gBXeH9gmM7wz1cbaYNazqs8R1LS5SW76p/y9RLjTzU6N39VYefMMGwbsQ [2024-01-18T17:13:34.501Z] DlD11wvMBxTBp3hQYc3pdVLkVEQt8trVIwEUts7zfwUTCu/z8gH7H5XePpEVNana [2024-01-18T17:13:34.501Z] JhlAcXHCO7nN/ddJmoc= [2024-01-18T17:13:34.501Z] -----END CERTIFICATE----- [2024-01-18T17:13:34.501Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:34.501Z] MIIDGDCCAgCgAwIBAgIUEvLaarpXlv1cVvG6nMuxh2DNCKQwDQYJKoZIhvcNAQEL [2024-01-18T17:13:34.501Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-18T17:13:34.501Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-18T17:13:34.501Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-18T17:13:34.501Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-18T17:13:34.501Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-18T17:13:34.501Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-18T17:13:34.501Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-18T17:13:34.501Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-18T17:13:34.501Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-18T17:13:34.501Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-18T17:13:34.501Z] 9w0BAQsFAAOCAQEAOPakzVUmAqHhmQViZZHSQeRh/xYQBrtn6dnEo6X55SXNWcPU [2024-01-18T17:13:34.501Z] 4EEW3DvwBf75uqxKeGoXHTBpqNHG2x1+IncvIKJmi5OWM3aR/vEyJoBOchPBl+A4 [2024-01-18T17:13:34.501Z] 0xZO1vKiS2djxz8XHMkR70cxD8hv/zDJeIQbgBroF4Wa7rim5+y7jgO0O7Rs1zhT [2024-01-18T17:13:34.501Z] LDDStJ8aUKwcVRmOv7NFQxbQHXNx07kaUE2SQJfhryzZmotoBiymmQT2v0H4/4q0 [2024-01-18T17:13:34.501Z] 5dylRhSCWfwN5+It40/C3b1pbC2d9Iv/eO+B0wqg13dLF3xuA1JblEgnqA9mdNnp [2024-01-18T17:13:34.502Z] P6EgKTQBMiO6nPFyYIPqg1eWYkb1eXu9aEkmbw== [2024-01-18T17:13:34.502Z] -----END CERTIFICATE----- [2024-01-18T17:13:34.502Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:34.502Z] MIIDGDCCAgCgAwIBAgIUWf/qoBANFWF/a/yZr7wfLBMfItAwDQYJKoZIhvcNAQEL [2024-01-18T17:13:34.502Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-18T17:13:34.502Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-18T17:13:34.502Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-18T17:13:34.502Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-18T17:13:34.502Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-18T17:13:34.502Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-18T17:13:34.502Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-18T17:13:34.502Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-18T17:13:34.502Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-18T17:13:34.502Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-18T17:13:34.502Z] 9w0BAQsFAAOCAQEAdCguadZU6WJAuuAtWzvYgftPKkDKNBeGVgbqRwWRCB+QqVFl [2024-01-18T17:13:34.502Z] +3ZOQTStpnXJsY3Ig31hmL88BkBPQ47ENEzs7Czhuz59m+dcBhh/buUoIERSnfUY [2024-01-18T17:13:34.502Z] lWIfqxQevwHYZ3Kjx0l3qliYjKBQJo5xDPnukGb4yUnmrCdZ3fxMzlnr0Ais9rUH [2024-01-18T17:13:34.502Z] MjiDsjP/oUa8naQUTZO4DTRZqeEldzNXnxF6YH+3moTkwRLt9xzirNwECmJ9qxrC [2024-01-18T17:13:34.502Z] eCA7zORxbFSftYLnQiGWUS1dnxBuLOvJHjYPC6EkPPrvqnx2fXcTE1V8GizVRrA6 [2024-01-18T17:13:34.502Z] NPxt+e4qZgU/Nk/Jkpsb7h1FVcKtS8CvmtbdqQ== [2024-01-18T17:13:34.502Z] -----END CERTIFICATE----- [2024-01-18T17:13:34.502Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:34.502Z] MIIDGDCCAgCgAwIBAgIUI6D0WS4tl52N+w2lwAS+6V0DPyUwDQYJKoZIhvcNAQEL [2024-01-18T17:13:34.502Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-18T17:13:34.502Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-18T17:13:34.502Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-18T17:13:34.502Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-18T17:13:34.502Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-18T17:13:34.502Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-18T17:13:34.502Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-18T17:13:34.502Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-18T17:13:34.502Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-18T17:13:34.502Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-18T17:13:34.502Z] 9w0BAQsFAAOCAQEAAS4yElh6TMyP18ipzRA0Wenukevjs/vtkAFOsv1u5GNeDIrv [2024-01-18T17:13:34.502Z] lNi6/VjvtS+DoFNzQnTf3QoO6GTuoWHLQWt6Z/V5AwIsnOXduSXLugMwEv3j9l50 [2024-01-18T17:13:34.502Z] aDkA7vJ3OH5cGqIvIzxQN7IUOEvZNhY67GFP7+zAzVNRpJ1FHYbhQGPuFD//b9G3 [2024-01-18T17:13:34.502Z] hr4E3+iUEc+F/iBA+caA8wRx1zR383Jz6wuoFt7n7HpWNcgMvoh91JlUAK7yUp0B [2024-01-18T17:13:34.502Z] ekuG6q7HtVsk6X+aB1RyWfrSJjih75W7b+p+/bV4vNmQS1wSGSOaSIuVRvMhmpwf [2024-01-18T17:13:34.502Z] FpT+4rUpiCxklD69/MET/1cWxqvao/ppf6phzg== [2024-01-18T17:13:34.502Z] -----END CERTIFICATE----- [2024-01-18T17:13:34.502Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:34.502Z] MIIDGDCCAgCgAwIBAgIUIVCCgYV9enDck18n89YsA5b2IdgwDQYJKoZIhvcNAQEL [2024-01-18T17:13:34.502Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-18T17:13:34.502Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-18T17:13:34.502Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-18T17:13:34.502Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-18T17:13:34.502Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-18T17:13:34.502Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-18T17:13:34.502Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-18T17:13:34.502Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-18T17:13:34.502Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-18T17:13:34.502Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-18T17:13:34.502Z] 9w0BAQsFAAOCAQEAa3eFm1ivFf5Vom6a/7E1gDdjl7cjUZ5cqg332Ky1NdzEOiwW [2024-01-18T17:13:34.502Z] mx3yEk35qCd8LAoFMvPvPIuMx1NDS8kwy3FmIOjKnZapHA31ZUpn/gT/seH4GFIU [2024-01-18T17:13:34.502Z] oZw+rODV5h6jmtCVugAn48u/LxOXbY417JWwaeDx5uw+NzOSyebe+td2PbtUjf5X [2024-01-18T17:13:34.502Z] 6xcwYUmaUGsIAgfIMM12hzXOI2/q1qz5dAqwSgnR7dKBReOXnUoFpNOMpxlJ70p/ [2024-01-18T17:13:34.502Z] WUU1aWIWZqKrNI7Pg1T5jljq7+KbT3+A6+ux0FLExr2c1tw3NndXS2Y9fO55m5/q [2024-01-18T17:13:34.502Z] 0EcrTdKx4+QjbWcja3WdX2pGDk0XGNgx+EXn6Q== [2024-01-18T17:13:34.502Z] -----END CERTIFICATE----- [2024-01-18T17:13:34.502Z] -----BEGIN CERTIFICATE----- [2024-01-18T17:13:34.502Z] MIIFATCCAumgAwIBAgIRAKc9ZKBASymy5TLOEp57N98wDQYJKoZIhvcNAQELBQAw [2024-01-18T17:13:34.502Z] GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDMyMzIyNTM0NloXDTM2 [2024-01-18T17:13:34.503Z] MDMyMzIyNTM0NlowGjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMIICIjANBgkq [2024-01-18T17:13:34.503Z] hkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA+pYHvQw5iU3v2b3iNuYNKYgsWD6KU7aJ [2024-01-18T17:13:34.503Z] diddtZQxSWYzUI3U0I1UsRPTxnhTifs/M9NW4ZlV13ZfB7APwC8oqKOIiwo7IwlP [2024-01-18T17:13:34.503Z] xg0VKgyz+kT8RJfYr66PPIYP0fpTeu42LpMJ+CKo9sbpgVNDZN2z/qiXrRNX/VtG [2024-01-18T17:13:34.503Z] TkPV7a44fZ5bHHVruAxvDnylpQxJobtCBWlJSsbIRGFHMc2z88eUz9NmIOWUKGGj [2024-01-18T17:13:34.503Z] EmP76x8OfRHpIpuxRSCjn0+i9+hR2siIOpcMOGd+40uVJxbRRP5ZXnUFa2fF5FWd [2024-01-18T17:13:34.503Z] O0u0RPI8HON0ovhrwPJY+4eWKkQzyC611oLPYGQ4EbifRsTsCxUZqyUuStGyp8oa [2024-01-18T17:13:34.503Z] aoSKfF6X0+KzGgwwnrjRTUpIl19A92KR0Noo6h622OX+4sZiO/JQdkuX5w/HupK0 [2024-01-18T17:13:34.503Z] A0M0WSMCvU6GOhjGotmh2VTEJwHHY4+TUk0iQYRtv1crONklyZoAQPD76hCrC8Cr [2024-01-18T17:13:34.503Z] IbgsZLfTMC8TWUoMbyUDgvgYkHKMoPm0VGVVuwpRKJxv7+2wXO+pivrrUl2Q9fPe [2024-01-18T17:13:34.503Z] Kk055nJLMV9yPUdig8othUKrRfSxli946AEV1eEOhxddfEwBE3Lt2xn0hhiIedbb [2024-01-18T17:13:34.503Z] Ftf/5kEWFZkXyUmMJK8Ra76Kus2ABueUVEcZ48hrRr1Hf1N9n59VbTUaXgeiZA50 [2024-01-18T17:13:34.503Z] qXf2bymE6F8CAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMB [2024-01-18T17:13:34.503Z] Af8wHQYDVR0OBBYEFMEmdKSKRKDm+iAo2FwjmkWIGHngMA0GCSqGSIb3DQEBCwUA [2024-01-18T17:13:34.503Z] A4ICAQBCPw74M9X/Xx04K1VAES3ypgQYH5bf9FXVDrwhRFSVckria/7dMzoF5wln [2024-01-18T17:13:34.503Z] uq9NGsjkkkDg17AohcQdr8alH4LvPdxpKr3BjpvEcmbqF8xH+MbbeUEnmbSfLI8H [2024-01-18T17:13:34.503Z] sefuhXF9AF/9iYvpVNC8FmJ0OhiVv13VgMQw0CRKkbtjZBf8xaEhq/YqxWVsgOjm [2024-01-18T17:13:34.503Z] dm5CAQ2X0aX7502x8wYRgMnZhA5goC1zVWBVAi8yhhmlhhoDUfg17cXkmaJC5pDd [2024-01-18T17:13:34.503Z] oenZ9NVhW8eDb03MFCrWNvIh89DDeCGWuWfDltDq0n3owyL0IeSn7RfpSclpxVmV [2024-01-18T17:13:34.503Z] /53jkYjwIgxIG7Gsv0LKMbsf6QdBcTjhvfZyMIpBRkTe3zuHd2feKzY9lEkbRvRQ [2024-01-18T17:13:34.503Z] zbh4Ps5YBnG6CKJPTbe2hfi3nhnw/MyEmF3zb0hzvLWNrR9XW3ibb2oL3424XOwc [2024-01-18T17:13:34.503Z] VjrTSCLzO9Rv6s5wi03qoWvKAQQAElqTYRHhynJ3w6wuvKYF5zcZF3MDnrVGLbh1 [2024-01-18T17:13:34.503Z] Q9ePRFBCiXOQ6wPLoUhrrbZ8LpFUFYDXHMtYM7P9sc9IAWoONXREJaO08zgFtMp4 [2024-01-18T17:13:34.503Z] 8iyIYUyQAbsvx8oD2M8kRvrIRSrRJSl6L957b4AFiLIQ/GgV2curs0jje7Edx34c [2024-01-18T17:13:34.503Z] idWw1VrejtwclobqNMVtG3EiPUIpJGpbMcJgbiLSmKkrvQtGng== [2024-01-18T17:13:34.503Z] -----END CERTIFICATE----- [2024-01-18T17:13:34.503Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} [2024-01-18T17:13:34.503Z] INFO[2024-01-18 17:13:34] Actual:true, Expected: true [2024-01-18T17:13:34.503Z] INFO[2024-01-18 17:13:34] Verifying : Description : Fetching configmap pointer &ConfigMap{ObjectMeta:{openshift-service-ca.crt mysql-ibm-pxb-2-84853-01-18-17h11m43s a4ab431f-6248-42d2-ba7f-75a461cc07ad 1286745 0 2024-01-18 17:11:44 +0000 UTC map[] map[service.beta.openshift.io/inject-cabundle:true] [] [] [{kube-controller-manager Update v1 2024-01-18 17:11:44 +0000 UTC FieldsV1 {"f:data":{},"f:metadata":{"f:annotations":{".":{},"f:service.beta.openshift.io/inject-cabundle":{}}}} } {service-ca-operator Update v1 2024-01-18 17:11:45 +0000 UTC FieldsV1 {"f:data":{"f:service-ca.crt":{}}} }]},Data:map[string]string{service-ca.crt: -----BEGIN CERTIFICATE----- [2024-01-18T17:13:34.503Z] MIIDUTCCAjmgAwIBAgIILvRjno8BjvIwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE [2024-01-18T17:13:34.503Z] Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwNTM5NjA4ODAe [2024-01-18T17:13:34.503Z] Fw0yNDAxMTYwOTA4MDhaFw0yNjAzMTYwOTA4MDlaMDYxNDAyBgNVBAMMK29wZW5z [2024-01-18T17:13:34.503Z] aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDUzOTYwODgwggEiMA0GCSqG [2024-01-18T17:13:34.503Z] SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVr8xmLKsMoemVtfnlzzil6B8CGocEhNRo [2024-01-18T17:13:34.503Z] fkzkD8aSOYgIj5zcZo/M0O7nXDXeg9dqx9/h2qrGk+SS2ikU2Xs4Q5gaQDh9U1lB [2024-01-18T17:13:34.503Z] y8paoOEvw06CmJ/1Qqur5A539/OnimRSTdejTu4lwaDwsJiyoGfhvrBJAdjagtYA [2024-01-18T17:13:34.503Z] 0sNMrCgEAT4NWHS5EDx5dF7nMDHmQ0h0gHYei6vUEtiPyLwcTH1T7uVSj22fCEXX [2024-01-18T17:13:34.503Z] LjgN3OSiquJ+Hf4+csPpwgxAEkfjdBldhnImYkunJbg/x4c90uFL1pygZusrbFAf [2024-01-18T17:13:34.503Z] cERW8GngZuL5JY8MQoIstMu+1qS390ouQDDbEcvWjxC8YFNFHywrAgMBAAGjYzBh [2024-01-18T17:13:34.503Z] MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQS5vRk [2024-01-18T17:13:34.503Z] jkBPISX2qYmk1VEQZIXSLzAfBgNVHSMEGDAWgBQS5vRkjkBPISX2qYmk1VEQZIXS [2024-01-18T17:13:34.503Z] LzANBgkqhkiG9w0BAQsFAAOCAQEAZ/qc2vFGJZgDJCEUaPYuufuLRPSF68ixV9fk [2024-01-18T17:13:34.503Z] e5tBkRN8atCsEBygC/RUcwULlhsdgUMGMcREgNC/FYlwkOsAj3gP8KmcXAZ4gHCa [2024-01-18T17:13:34.503Z] T6PIbIcj1hCX0mdX0tvtaj7PWvRUz9CHh6iFzmPsXK5xjaBA9vZbkUAa7odguUit [2024-01-18T17:13:34.503Z] vz1QnKoV1OaZyccDLeQL50khM1XEBn8L+zeY3k85uo2X514HZW4LDPTQ94cpyNnb [2024-01-18T17:13:34.503Z] zqI6utp7JoPJexsHMWNFKGb//HgjluNg7TsywXIeebk+/GZUEDfNKid0oiTpQvzR [2024-01-18T17:13:34.503Z] TFFF+5KO/sa4+qIh1ERm140JxeaurUMaJzn7cFQbekEgt/N+iA== [2024-01-18T17:13:34.503Z] -----END CERTIFICATE----- [2024-01-18T17:13:34.503Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} from namespace mysql-ibm-pxb-2-84853-01-18-17h11m43s [2024-01-18T17:13:34.504Z] INFO[2024-01-18 17:13:34] Actual:true, Expected: true [2024-01-18T17:13:34.504Z] 2024-01-18 17:13:34 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.AddLabelToResource:#5038] - Adding label [abe5cf54-4e74-45b4-882f-82b02f3dd9cf=cd7d2c21-9adb-41bf-b01f-ae218113bcc9] to ConfigMap openshift-service-ca.crt [2024-01-18T17:13:34.767Z] INFO[2024-01-18 17:13:34] Verifying : Description : Adding labels abe5cf54-4e74-45b4-882f-82b02f3dd9cf=cd7d2c21-9adb-41bf-b01f-ae218113bcc9 to resource &ConfigMap{ObjectMeta:{openshift-service-ca.crt mysql-ibm-pxb-2-84853-01-18-17h11m43s a4ab431f-6248-42d2-ba7f-75a461cc07ad 1286745 0 2024-01-18 17:11:44 +0000 UTC map[abe5cf54-4e74-45b4-882f-82b02f3dd9cf:cd7d2c21-9adb-41bf-b01f-ae218113bcc9] map[service.beta.openshift.io/inject-cabundle:true] [] [] [{kube-controller-manager Update v1 2024-01-18 17:11:44 +0000 UTC FieldsV1 {"f:data":{},"f:metadata":{"f:annotations":{".":{},"f:service.beta.openshift.io/inject-cabundle":{}}}} } {service-ca-operator Update v1 2024-01-18 17:11:45 +0000 UTC FieldsV1 {"f:data":{"f:service-ca.crt":{}}} }]},Data:map[string]string{service-ca.crt: -----BEGIN CERTIFICATE----- [2024-01-18T17:13:34.767Z] MIIDUTCCAjmgAwIBAgIILvRjno8BjvIwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE [2024-01-18T17:13:34.767Z] Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwNTM5NjA4ODAe [2024-01-18T17:13:34.767Z] Fw0yNDAxMTYwOTA4MDhaFw0yNjAzMTYwOTA4MDlaMDYxNDAyBgNVBAMMK29wZW5z [2024-01-18T17:13:34.767Z] aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDUzOTYwODgwggEiMA0GCSqG [2024-01-18T17:13:34.767Z] SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVr8xmLKsMoemVtfnlzzil6B8CGocEhNRo [2024-01-18T17:13:34.767Z] fkzkD8aSOYgIj5zcZo/M0O7nXDXeg9dqx9/h2qrGk+SS2ikU2Xs4Q5gaQDh9U1lB [2024-01-18T17:13:34.767Z] y8paoOEvw06CmJ/1Qqur5A539/OnimRSTdejTu4lwaDwsJiyoGfhvrBJAdjagtYA [2024-01-18T17:13:34.767Z] 0sNMrCgEAT4NWHS5EDx5dF7nMDHmQ0h0gHYei6vUEtiPyLwcTH1T7uVSj22fCEXX [2024-01-18T17:13:34.767Z] LjgN3OSiquJ+Hf4+csPpwgxAEkfjdBldhnImYkunJbg/x4c90uFL1pygZusrbFAf [2024-01-18T17:13:34.767Z] cERW8GngZuL5JY8MQoIstMu+1qS390ouQDDbEcvWjxC8YFNFHywrAgMBAAGjYzBh [2024-01-18T17:13:34.767Z] MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQS5vRk [2024-01-18T17:13:34.767Z] jkBPISX2qYmk1VEQZIXSLzAfBgNVHSMEGDAWgBQS5vRkjkBPISX2qYmk1VEQZIXS [2024-01-18T17:13:34.767Z] LzANBgkqhkiG9w0BAQsFAAOCAQEAZ/qc2vFGJZgDJCEUaPYuufuLRPSF68ixV9fk [2024-01-18T17:13:34.767Z] e5tBkRN8atCsEBygC/RUcwULlhsdgUMGMcREgNC/FYlwkOsAj3gP8KmcXAZ4gHCa [2024-01-18T17:13:34.767Z] T6PIbIcj1hCX0mdX0tvtaj7PWvRUz9CHh6iFzmPsXK5xjaBA9vZbkUAa7odguUit [2024-01-18T17:13:34.767Z] vz1QnKoV1OaZyccDLeQL50khM1XEBn8L+zeY3k85uo2X514HZW4LDPTQ94cpyNnb [2024-01-18T17:13:34.767Z] zqI6utp7JoPJexsHMWNFKGb//HgjluNg7TsywXIeebk+/GZUEDfNKid0oiTpQvzR [2024-01-18T17:13:34.767Z] TFFF+5KO/sa4+qIh1ERm140JxeaurUMaJzn7cFQbekEgt/N+iA== [2024-01-18T17:13:34.767Z] -----END CERTIFICATE----- [2024-01-18T17:13:34.767Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} [2024-01-18T17:13:34.767Z] INFO[2024-01-18 17:13:34] Actual:true, Expected: true [2024-01-18T17:13:34.767Z] 2024-01-18 17:13:34 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Adding labels to namespaces [2024-01-18T17:13:35.698Z] INFO[2024-01-18 17:13:35] Verifying : Description : Adding labels [map[axfchgaztneftcrwfiuzngrdjlhfjpfqhoadujitcdsfqebybvkdpxdojoxrrqh:b3f564ec-25e4-43f7-b0fd-003090761a0c fmbpuazkdhcielxgcaoirvnatdpjhpmiyfuabhvcrjeduooptlnkbesnlbytjqe:71b9d259-202c-44a0-b8b0-fb880171bf46 tiekdrxgrofqdhbevpnfwsbzriqbtjgorhpewobxvvycasbdnbezkuzoftyjmgj:cf04b0a1-41c3-4f3a-a0fe-7b20ea61e8f5]] to multiple namespaces [[mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s]] [2024-01-18T17:13:35.698Z] INFO[2024-01-18 17:13:35] Actual:true, Expected: true [2024-01-18T17:13:35.954Z] INFO[2024-01-18 17:13:35] Verifying : Description : Adding labels [map[etpumwvxiorhlbedvndojkerbnitppaqpiaidghvaapdhztwkeaykpoopimvmlq:38ff710d-1022-456d-920f-0fb90d5d7c37 yqypwkjtptjodgpzmboldyekuvskosannzdwscnxvotgpilczrakescfqcqoour:f7dde987-af16-415a-9f4c-9f117f216254 zxhacjspvspdddqsnndtkbjshqnlnqutzovrzdmqwhauuaqyrygurqzelvdfelx:72be52d0-458b-40c7-8c0f-9d90281b5bfd]] to single namespaces [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:13:35.954Z] INFO[2024-01-18 17:13:35] Actual:true, Expected: true [2024-01-18T17:13:37.324Z] INFO[2024-01-18 17:13:37] Verifying : Description : Adding labels [map[jcscxcmokfqjigdhdauujdccwnvaobakkvwqlltpoeesjygjfuoqvhovwgswoor:68f8b453-b3cf-4ffa-a7ad-1537518cf8ff kcvuvkfgekhphvfawturizbjgpfprhugkdjnfcqmvmiltrwcppbbuevnzswppph:ad9b899a-faee-4ea4-97f3-d0c839228f51 tyffzmgvkruasitedxdahjkjaihjewomwkwwxqkbhdbmghhnduauyfjbmlgvsbu:05b20d94-ef28-480f-be7d-3465fb05d059]] to all namespaces [[mysql-ibm-pxb-0-84853-01-18-17h11m43s mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s]] [2024-01-18T17:13:37.324Z] INFO[2024-01-18 17:13:37] Actual:true, Expected: true [2024-01-18T17:13:37.324Z] 2024-01-18 17:13:37 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Generating namespace label string from label map for single and multiple namespace [2024-01-18T17:13:37.324Z] 2024-01-18 17:13:37 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.glob..func22.2.4:#965] - Labels for single namespace [zxhacjspvspdddqsnndtkbjshqnlnqutzovrzdmqwhauuaqyrygurqzelvdfelx=72be52d0-458b-40c7-8c0f-9d90281b5bfd,yqypwkjtptjodgpzmboldyekuvskosannzdwscnxvotgpilczrakescfqcqoour=f7dde987-af16-415a-9f4c-9f117f216254,etpumwvxiorhlbedvndojkerbnitppaqpiaidghvaapdhztwkeaykpoopimvmlq=38ff710d-1022-456d-920f-0fb90d5d7c37] [2024-01-18T17:13:37.324Z] 2024-01-18 17:13:37 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.glob..func22.2.4:#967] - Labels for multiple namespace [axfchgaztneftcrwfiuzngrdjlhfjpfqhoadujitcdsfqebybvkdpxdojoxrrqh=b3f564ec-25e4-43f7-b0fd-003090761a0c,fmbpuazkdhcielxgcaoirvnatdpjhpmiyfuabhvcrjeduooptlnkbesnlbytjqe=71b9d259-202c-44a0-b8b0-fb880171bf46,tiekdrxgrofqdhbevpnfwsbzriqbtjgorhpewobxvvycasbdnbezkuzoftyjmgj=cf04b0a1-41c3-4f3a-a0fe-7b20ea61e8f5] [2024-01-18T17:13:37.324Z] 2024-01-18 17:13:37 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.glob..func22.2.4:#969] - Labels for all namespaces [kcvuvkfgekhphvfawturizbjgpfprhugkdjnfcqmvmiltrwcppbbuevnzswppph=ad9b899a-faee-4ea4-97f3-d0c839228f51,jcscxcmokfqjigdhdauujdccwnvaobakkvwqlltpoeesjygjfuoqvhovwgswoor=68f8b453-b3cf-4ffa-a7ad-1537518cf8ff,tyffzmgvkruasitedxdahjkjaihjewomwkwwxqkbhdbmghhnduauyfjbmlgvsbu=05b20d94-ef28-480f-be7d-3465fb05d059] [2024-01-18T17:13:37.324Z] 2024-01-18 17:13:37 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Creating cloud credentials and backup location [2024-01-18T17:13:38.698Z] 2024-01-18 17:13:38 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-1705598018] for org [default] with [aws] as provider [2024-01-18T17:13:38.698Z] 2024-01-18 17:13:38 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-18T17:13:38.698Z] 2024-01-18 17:13:38 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-1705598018] is present [2024-01-18T17:13:38.698Z] INFO[2024-01-18 17:13:38] Verifying : Description : Creating cloud credential named autogenerated-cred-1705598018 [2024-01-18T17:13:38.698Z] INFO[2024-01-18 17:13:38] Actual:true, Expected: true [2024-01-18T17:13:38.698Z] 2024-01-18 17:13:38 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Created Cloud Credentials with name - autogenerated-cred-1705598018 [2024-01-18T17:13:43.947Z] INFO[2024-01-18 17:13:43] Verifying : Description : Creating backup location autogenerated-backup-location-1705598018 [2024-01-18T17:13:43.947Z] INFO[2024-01-18 17:13:43] Actual:true, Expected: true [2024-01-18T17:13:43.947Z] 2024-01-18 17:13:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Adding source and destination clusters with px-central-admin ctx [2024-01-18T17:13:43.947Z] INFO[2024-01-18 17:13:43] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T17:13:43.947Z] INFO[2024-01-18 17:13:43] Actual:true, Expected: true [2024-01-18T17:13:43.947Z] 2024-01-18 17:13:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T17:13:43.947Z] 2024-01-18 17:13:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T17:13:43.947Z] 2024-01-18 17:13:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T17:13:43.947Z] 2024-01-18 17:13:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T17:13:43.948Z] 2024-01-18 17:13:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Create cluster [source-cluster] in org [default] [2024-01-18T17:13:43.948Z] 2024-01-18 17:13:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T17:13:43.948Z] 2024-01-18 17:13:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T17:13:43.948Z] 2024-01-18 17:13:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Create cluster [destination-cluster] in org [default] [2024-01-18T17:13:43.948Z] 2024-01-18 17:13:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T17:13:43.948Z] 2024-01-18 17:13:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T17:13:43.948Z] 2024-01-18 17:13:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-eopef] for IBM [2024-01-18T17:13:43.948Z] 2024-01-18 17:13:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-eopef] for org [default] with [ibm] as provider [2024-01-18T17:13:43.948Z] 2024-01-18 17:13:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T17:13:43.948Z] 2024-01-18 17:13:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T17:13:43.948Z] 2024-01-18 17:13:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-eopef] is present [2024-01-18T17:13:43.948Z] 2024-01-18 17:13:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Create cluster [source-cluster] in org [default] [2024-01-18T17:13:44.514Z] 2024-01-18 17:13:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-vmbmv] for IBM [2024-01-18T17:13:44.514Z] 2024-01-18 17:13:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-vmbmv] for org [default] with [ibm] as provider [2024-01-18T17:13:44.514Z] 2024-01-18 17:13:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T17:13:44.515Z] 2024-01-18 17:13:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T17:13:44.515Z] 2024-01-18 17:13:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-vmbmv] is present [2024-01-18T17:13:44.515Z] 2024-01-18 17:13:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Create cluster [destination-cluster] in org [default] [2024-01-18T17:13:45.079Z] INFO[2024-01-18 17:13:44] Verifying : Description : Verifying addition of source [source-cluster] and destination [destination-cluster] clusters [2024-01-18T17:13:45.079Z] INFO[2024-01-18 17:13:44] Actual:true, Expected: true [2024-01-18T17:13:45.079Z] INFO[2024-01-18 17:13:44] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-18T17:13:45.079Z] INFO[2024-01-18 17:13:44] Actual:Online, Expected: Online [2024-01-18T17:13:45.079Z] INFO[2024-01-18 17:13:44] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-18T17:13:45.079Z] INFO[2024-01-18 17:13:44] Actual:true, Expected: true [2024-01-18T17:13:45.079Z] 2024-01-18 17:13:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] UID of [source-cluster] cluster is 4e2d1620-3c92-4b20-8df4-0c099bb380b8 [2024-01-18T17:13:45.079Z] INFO[2024-01-18 17:13:44] Verifying : Description : Verifying if [destination-cluster] cluster is online [2024-01-18T17:13:45.079Z] INFO[2024-01-18 17:13:44] Actual:Online, Expected: Online [2024-01-18T17:13:45.079Z] 2024-01-18 17:13:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Taking manual backup of single application with namespace label filter [2024-01-18T17:13:45.336Z] 2024-01-18 17:13:45 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T17:13:45.336Z] 2024-01-18 17:13:45 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Volume snapshot classes found in the cluster - [2024-01-18T17:13:45.336Z] 2024-01-18 17:13:45 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] ibmc-vpcblock-snapshot [2024-01-18T17:13:45.336Z] 2024-01-18 17:13:45 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T17:13:45.898Z] 2024/01/18 17:13:45 DoRetryWithTimeout - Error: {backup status for [single-namespace-backup-1705598025] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T17:14:07.799Z] INFO[2024-01-18 17:14:06] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T17:14:07.799Z] INFO[2024-01-18 17:14:06] sched stats: tick interval: 6006 < 1.2s (100%) [2024-01-18T17:14:07.799Z] INFO[2024-01-18 17:14:06] sched stats: tick process durations: 6005 < 500ms (100%) [2024-01-18T17:14:15.885Z] 2024/01/18 17:14:15 DoRetryWithTimeout - Error: {backup status for [single-namespace-backup-1705598025] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T17:14:47.911Z] 2024/01/18 17:14:45 DoRetryWithTimeout - Error: {backup status for [single-namespace-backup-1705598025] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T17:15:19.944Z] 2024-01-18 17:15:15 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CreateBackupWithNamespaceLabel:#3030] - Successfully created backup [single-namespace-backup-1705598025] with namespace label [zxhacjspvspdddqsnndtkbjshqnlnqutzovrzdmqwhauuaqyrygurqzelvdfelx=72be52d0-458b-40c7-8c0f-9d90281b5bfd,yqypwkjtptjodgpzmboldyekuvskosannzdwscnxvotgpilczrakescfqcqoour=f7dde987-af16-415a-9f4c-9f117f216254,etpumwvxiorhlbedvndojkerbnitppaqpiaidghvaapdhztwkeaykpoopimvmlq=38ff710d-1022-456d-920f-0fb90d5d7c37] [2024-01-18T17:15:19.944Z] 2024-01-18 17:15:15 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating backup [single-namespace-backup-1705598025] in org [default] [2024-01-18T17:15:19.944Z] 2024-01-18 17:15:15 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [single-namespace-backup-1705598025] [2024-01-18T17:15:19.944Z] 2024-01-18 17:15:15 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84853-01-18-17h11m43s] in backup [single-namespace-backup-1705598025] [2024-01-18T17:15:19.945Z] 2024-01-18 17:15:15 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84853-01-18-17h11m43s] in backup [single-namespace-backup-1705598025] [2024-01-18T17:15:19.945Z] 2024-01-18 17:15:15 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:15:19.945Z] 2024-01-18 17:15:15 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:15:19.945Z] 2024-01-18 17:15:15 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:15:19.945Z] 2024-01-18 17:15:15 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-84853-01-18-17h11m43s] are [[mysql-data]] [2024-01-18T17:15:19.945Z] 2024-01-18 17:15:15 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T17:15:19.945Z] 2024-01-18 17:15:15 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating CR cleanup [2024-01-18T17:15:19.945Z] 2024-01-18 17:15:16 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T17:15:19.946Z] 2024-01-18 17:15:16 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] All CRs in [mysql-ibm-pxb-0-84853-01-18-17h11m43s] are [[]] [2024-01-18T17:15:19.946Z] INFO[2024-01-18 17:15:16] Verifying : Description : Verifying status of backup [single-namespace-backup-1705598025] creation with label [zxhacjspvspdddqsnndtkbjshqnlnqutzovrzdmqwhauuaqyrygurqzelvdfelx=72be52d0-458b-40c7-8c0f-9d90281b5bfd,yqypwkjtptjodgpzmboldyekuvskosannzdwscnxvotgpilczrakescfqcqoour=f7dde987-af16-415a-9f4c-9f117f216254,etpumwvxiorhlbedvndojkerbnitppaqpiaidghvaapdhztwkeaykpoopimvmlq=38ff710d-1022-456d-920f-0fb90d5d7c37] [2024-01-18T17:15:19.946Z] INFO[2024-01-18 17:15:16] Actual:true, Expected: true [2024-01-18T17:15:19.946Z] 2024-01-18 17:15:16 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup single-namespace-backup-1705598025 [2024-01-18T17:15:19.946Z] 2024-01-18 17:15:16 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:15:19.946Z] 2024-01-18 17:15:16 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are zxhacjspvspdddqsnndtkbjshqnlnqutzovrzdmqwhauuaqyrygurqzelvdfelx=72be52d0-458b-40c7-8c0f-9d90281b5bfd,yqypwkjtptjodgpzmboldyekuvskosannzdwscnxvotgpilczrakescfqcqoour=f7dde987-af16-415a-9f4c-9f117f216254,etpumwvxiorhlbedvndojkerbnitppaqpiaidghvaapdhztwkeaykpoopimvmlq=38ff710d-1022-456d-920f-0fb90d5d7c37 [2024-01-18T17:15:19.946Z] INFO[2024-01-18 17:15:16] Verifying : Description : Verifying if the labeled namespace [mysql-ibm-pxb-0-84853-01-18-17h11m43s] is backed up and checks for labels [zxhacjspvspdddqsnndtkbjshqnlnqutzovrzdmqwhauuaqyrygurqzelvdfelx=72be52d0-458b-40c7-8c0f-9d90281b5bfd,yqypwkjtptjodgpzmboldyekuvskosannzdwscnxvotgpilczrakescfqcqoour=f7dde987-af16-415a-9f4c-9f117f216254,etpumwvxiorhlbedvndojkerbnitppaqpiaidghvaapdhztwkeaykpoopimvmlq=38ff710d-1022-456d-920f-0fb90d5d7c37] applied to backup [single-namespace-backup-1705598025] [2024-01-18T17:15:19.947Z] INFO[2024-01-18 17:15:16] Actual:true, Expected: true [2024-01-18T17:15:19.947Z] 2024-01-18 17:15:16 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Taking manual backup of multiple applications with namespace label filter [2024-01-18T17:15:19.947Z] 2024-01-18 17:15:16 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T17:15:19.947Z] 2024-01-18 17:15:16 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Volume snapshot classes found in the cluster - [2024-01-18T17:15:19.947Z] 2024-01-18 17:15:16 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] ibmc-vpcblock-snapshot [2024-01-18T17:15:19.947Z] 2024-01-18 17:15:16 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T17:15:19.947Z] 2024/01/18 17:15:17 DoRetryWithTimeout - Error: {backup status for [multiple-namespace-backup-1705598116] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T17:15:52.033Z] 2024/01/18 17:15:47 DoRetryWithTimeout - Error: {backup status for [multiple-namespace-backup-1705598116] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T17:16:18.575Z] 2024/01/18 17:16:17 DoRetryWithTimeout - Error: {backup status for [multiple-namespace-backup-1705598116] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T17:16:50.604Z] 2024-01-18 17:16:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CreateBackupWithNamespaceLabel:#3030] - Successfully created backup [multiple-namespace-backup-1705598116] with namespace label [axfchgaztneftcrwfiuzngrdjlhfjpfqhoadujitcdsfqebybvkdpxdojoxrrqh=b3f564ec-25e4-43f7-b0fd-003090761a0c,fmbpuazkdhcielxgcaoirvnatdpjhpmiyfuabhvcrjeduooptlnkbesnlbytjqe=71b9d259-202c-44a0-b8b0-fb880171bf46,tiekdrxgrofqdhbevpnfwsbzriqbtjgorhpewobxvvycasbdnbezkuzoftyjmgj=cf04b0a1-41c3-4f3a-a0fe-7b20ea61e8f5] [2024-01-18T17:16:50.604Z] 2024-01-18 17:16:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating backup [multiple-namespace-backup-1705598116] in org [default] [2024-01-18T17:16:50.604Z] 2024-01-18 17:16:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [multiple-namespace-backup-1705598116] [2024-01-18T17:16:50.604Z] 2024-01-18 17:16:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84853-01-18-17h11m43s] in backup [multiple-namespace-backup-1705598116] [2024-01-18T17:16:50.604Z] 2024-01-18 17:16:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84853-01-18-17h11m43s] in backup [multiple-namespace-backup-1705598116] [2024-01-18T17:16:50.604Z] 2024-01-18 17:16:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T17:16:50.604Z] 2024-01-18 17:16:47 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:16:50.604Z] 2024-01-18 17:16:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:16:50.604Z] 2024-01-18 17:16:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-84853-01-18-17h11m43s] are [[mysql-data]] [2024-01-18T17:16:50.604Z] 2024-01-18 17:16:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T17:16:50.604Z] 2024-01-18 17:16:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84853-01-18-17h11m43s] in backup [multiple-namespace-backup-1705598116] [2024-01-18T17:16:50.605Z] 2024-01-18 17:16:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84853-01-18-17h11m43s] in backup [multiple-namespace-backup-1705598116] [2024-01-18T17:16:50.605Z] 2024-01-18 17:16:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:16:50.605Z] 2024-01-18 17:16:47 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:16:50.605Z] 2024-01-18 17:16:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:16:50.605Z] 2024-01-18 17:16:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-2-84853-01-18-17h11m43s] are [[mysql-data]] [2024-01-18T17:16:50.605Z] 2024-01-18 17:16:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T17:16:50.605Z] 2024-01-18 17:16:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating CR cleanup [2024-01-18T17:16:50.605Z] 2024-01-18 17:16:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T17:16:50.605Z] 2024-01-18 17:16:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] All CRs in [kube-system] are [[]] [2024-01-18T17:16:50.605Z] INFO[2024-01-18 17:16:47] Verifying : Description : Verifying backup [multiple-namespace-backup-1705598116] creation with labels [axfchgaztneftcrwfiuzngrdjlhfjpfqhoadujitcdsfqebybvkdpxdojoxrrqh=b3f564ec-25e4-43f7-b0fd-003090761a0c,fmbpuazkdhcielxgcaoirvnatdpjhpmiyfuabhvcrjeduooptlnkbesnlbytjqe=71b9d259-202c-44a0-b8b0-fb880171bf46,tiekdrxgrofqdhbevpnfwsbzriqbtjgorhpewobxvvycasbdnbezkuzoftyjmgj=cf04b0a1-41c3-4f3a-a0fe-7b20ea61e8f5] [2024-01-18T17:16:50.605Z] INFO[2024-01-18 17:16:47] Actual:true, Expected: true [2024-01-18T17:16:50.605Z] 2024-01-18 17:16:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup multiple-namespace-backup-1705598116 [2024-01-18T17:16:50.605Z] 2024-01-18 17:16:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:16:50.605Z] 2024-01-18 17:16:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are axfchgaztneftcrwfiuzngrdjlhfjpfqhoadujitcdsfqebybvkdpxdojoxrrqh=b3f564ec-25e4-43f7-b0fd-003090761a0c,fmbpuazkdhcielxgcaoirvnatdpjhpmiyfuabhvcrjeduooptlnkbesnlbytjqe=71b9d259-202c-44a0-b8b0-fb880171bf46,tiekdrxgrofqdhbevpnfwsbzriqbtjgorhpewobxvvycasbdnbezkuzoftyjmgj=cf04b0a1-41c3-4f3a-a0fe-7b20ea61e8f5 [2024-01-18T17:16:50.605Z] INFO[2024-01-18 17:16:47] Verifying : Description : Verifying if the labeled namespaces [[mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s]] are backed up and check for labels [axfchgaztneftcrwfiuzngrdjlhfjpfqhoadujitcdsfqebybvkdpxdojoxrrqh=b3f564ec-25e4-43f7-b0fd-003090761a0c,fmbpuazkdhcielxgcaoirvnatdpjhpmiyfuabhvcrjeduooptlnkbesnlbytjqe=71b9d259-202c-44a0-b8b0-fb880171bf46,tiekdrxgrofqdhbevpnfwsbzriqbtjgorhpewobxvvycasbdnbezkuzoftyjmgj=cf04b0a1-41c3-4f3a-a0fe-7b20ea61e8f5] applied to backups [multiple-namespace-backup-1705598116] [2024-01-18T17:16:50.605Z] INFO[2024-01-18 17:16:47] Actual:true, Expected: true [2024-01-18T17:16:50.606Z] 2024-01-18 17:16:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Taking manual backup of all applications with namespace label filter [2024-01-18T17:16:50.606Z] 2024-01-18 17:16:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T17:16:50.606Z] 2024-01-18 17:16:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Volume snapshot classes found in the cluster - [2024-01-18T17:16:50.606Z] 2024-01-18 17:16:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] ibmc-vpcblock-snapshot [2024-01-18T17:16:50.606Z] 2024-01-18 17:16:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T17:16:50.606Z] 2024/01/18 17:16:48 DoRetryWithTimeout - Error: {backup status for [all-namespace-backup-1705598207] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T17:17:22.637Z] 2024/01/18 17:17:18 DoRetryWithTimeout - Error: {backup status for [all-namespace-backup-1705598207] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T17:17:49.136Z] 2024/01/18 17:17:48 DoRetryWithTimeout - Error: {backup status for [all-namespace-backup-1705598207] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T17:18:21.182Z] 2024-01-18 17:18:18 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CreateBackupWithNamespaceLabel:#3030] - Successfully created backup [all-namespace-backup-1705598207] with namespace label [kcvuvkfgekhphvfawturizbjgpfprhugkdjnfcqmvmiltrwcppbbuevnzswppph=ad9b899a-faee-4ea4-97f3-d0c839228f51,jcscxcmokfqjigdhdauujdccwnvaobakkvwqlltpoeesjygjfuoqvhovwgswoor=68f8b453-b3cf-4ffa-a7ad-1537518cf8ff,tyffzmgvkruasitedxdahjkjaihjewomwkwwxqkbhdbmghhnduauyfjbmlgvsbu=05b20d94-ef28-480f-be7d-3465fb05d059] [2024-01-18T17:18:21.182Z] 2024-01-18 17:18:18 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating backup [all-namespace-backup-1705598207] in org [default] [2024-01-18T17:18:21.182Z] 2024-01-18 17:18:18 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [all-namespace-backup-1705598207] [2024-01-18T17:18:21.182Z] 2024-01-18 17:18:18 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84853-01-18-17h11m43s] in backup [all-namespace-backup-1705598207] [2024-01-18T17:18:21.182Z] 2024-01-18 17:18:18 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84853-01-18-17h11m43s] in backup [all-namespace-backup-1705598207] [2024-01-18T17:18:21.182Z] 2024-01-18 17:18:18 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:18:21.182Z] 2024-01-18 17:18:18 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:18:21.182Z] 2024-01-18 17:18:18 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:18:21.182Z] 2024-01-18 17:18:18 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-84853-01-18-17h11m43s] are [[mysql-data]] [2024-01-18T17:18:21.183Z] 2024-01-18 17:18:18 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T17:18:21.183Z] 2024-01-18 17:18:18 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84853-01-18-17h11m43s] in backup [all-namespace-backup-1705598207] [2024-01-18T17:18:21.183Z] 2024-01-18 17:18:18 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84853-01-18-17h11m43s] in backup [all-namespace-backup-1705598207] [2024-01-18T17:18:21.183Z] 2024-01-18 17:18:18 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T17:18:21.183Z] 2024-01-18 17:18:18 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:18:21.183Z] 2024-01-18 17:18:18 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:18:21.183Z] 2024-01-18 17:18:18 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-84853-01-18-17h11m43s] are [[mysql-data]] [2024-01-18T17:18:21.183Z] 2024-01-18 17:18:18 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T17:18:21.183Z] 2024-01-18 17:18:18 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84853-01-18-17h11m43s] in backup [all-namespace-backup-1705598207] [2024-01-18T17:18:21.183Z] 2024-01-18 17:18:18 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84853-01-18-17h11m43s] in backup [all-namespace-backup-1705598207] [2024-01-18T17:18:21.183Z] 2024-01-18 17:18:18 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:18:21.183Z] 2024-01-18 17:18:19 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:18:21.183Z] 2024-01-18 17:18:19 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:18:21.183Z] 2024-01-18 17:18:19 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-2-84853-01-18-17h11m43s] are [[mysql-data]] [2024-01-18T17:18:21.183Z] 2024-01-18 17:18:19 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T17:18:21.183Z] 2024-01-18 17:18:19 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating CR cleanup [2024-01-18T17:18:21.183Z] 2024-01-18 17:18:19 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T17:18:21.184Z] 2024-01-18 17:18:19 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] All CRs in [kube-system] are [[all-namespace-backup-1705598207-8739a0a]] [2024-01-18T17:18:21.184Z] 2024-01-18 17:18:19 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] CR found for [[all-namespace-backup-1705598207-8739a0a]] under [kube-system] namespace [2024-01-18T17:18:21.184Z] 2024/01/18 17:18:19 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [all-namespace-backup-1705598207]}, Next try in [5s], timeout [5m0s] [2024-01-18T17:18:24.452Z] 2024-01-18 17:18:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] All CRs in [kube-system] are [[]] [2024-01-18T17:18:24.453Z] INFO[2024-01-18 17:18:24] Verifying : Description : Verifying backup [all-namespace-backup-1705598207] creation with labels [kcvuvkfgekhphvfawturizbjgpfprhugkdjnfcqmvmiltrwcppbbuevnzswppph=ad9b899a-faee-4ea4-97f3-d0c839228f51,jcscxcmokfqjigdhdauujdccwnvaobakkvwqlltpoeesjygjfuoqvhovwgswoor=68f8b453-b3cf-4ffa-a7ad-1537518cf8ff,tyffzmgvkruasitedxdahjkjaihjewomwkwwxqkbhdbmghhnduauyfjbmlgvsbu=05b20d94-ef28-480f-be7d-3465fb05d059] [2024-01-18T17:18:24.453Z] INFO[2024-01-18 17:18:24] Actual:true, Expected: true [2024-01-18T17:18:24.453Z] 2024-01-18 17:18:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup all-namespace-backup-1705598207 [2024-01-18T17:18:24.453Z] 2024-01-18 17:18:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [mysql-ibm-pxb-0-84853-01-18-17h11m43s mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:18:24.453Z] 2024-01-18 17:18:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are kcvuvkfgekhphvfawturizbjgpfprhugkdjnfcqmvmiltrwcppbbuevnzswppph=ad9b899a-faee-4ea4-97f3-d0c839228f51,jcscxcmokfqjigdhdauujdccwnvaobakkvwqlltpoeesjygjfuoqvhovwgswoor=68f8b453-b3cf-4ffa-a7ad-1537518cf8ff,tyffzmgvkruasitedxdahjkjaihjewomwkwwxqkbhdbmghhnduauyfjbmlgvsbu=05b20d94-ef28-480f-be7d-3465fb05d059 [2024-01-18T17:18:24.453Z] INFO[2024-01-18 17:18:24] Verifying : Description : Verifying if the labeled namespaces [[mysql-ibm-pxb-0-84853-01-18-17h11m43s mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s]] are backed up and check for labels [kcvuvkfgekhphvfawturizbjgpfprhugkdjnfcqmvmiltrwcppbbuevnzswppph=ad9b899a-faee-4ea4-97f3-d0c839228f51,jcscxcmokfqjigdhdauujdccwnvaobakkvwqlltpoeesjygjfuoqvhovwgswoor=68f8b453-b3cf-4ffa-a7ad-1537518cf8ff,tyffzmgvkruasitedxdahjkjaihjewomwkwwxqkbhdbmghhnduauyfjbmlgvsbu=05b20d94-ef28-480f-be7d-3465fb05d059] applied to backups [all-namespace-backup-1705598207] [2024-01-18T17:18:24.453Z] INFO[2024-01-18 17:18:24] Actual:true, Expected: true [2024-01-18T17:18:24.453Z] 2024-01-18 17:18:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Creating a schedule policy [2024-01-18T17:18:24.709Z] 2024-01-18 17:18:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Create Backup Schedule Policy [2024-01-18T17:18:25.271Z] INFO[2024-01-18 17:18:25] Verifying : Description : Verifying creation of periodic schedule policy of interval 15 minutes named [periodic-1705598304] [2024-01-18T17:18:25.271Z] INFO[2024-01-18 17:18:25] Actual:true, Expected: true [2024-01-18T17:18:25.527Z] INFO[2024-01-18 17:18:25] Verifying : Description : Fetching uid of periodic schedule policy named [periodic-1705598304] [2024-01-18T17:18:25.527Z] INFO[2024-01-18 17:18:25] Actual:true, Expected: true [2024-01-18T17:18:25.527Z] 2024-01-18 17:18:25 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Creating a schedule backup for single namespace with namespace label filter [2024-01-18T17:18:26.892Z] 2024-01-18 17:18:26 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-18T17:18:26.892Z] 2024-01-18 17:18:26 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Volume snapshot classes found in the cluster - [2024-01-18T17:18:26.892Z] 2024-01-18 17:18:26 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] ibmc-vpcblock-snapshot [2024-01-18T17:18:26.892Z] 2024-01-18 17:18:26 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T17:19:34.525Z] 2024-01-18 17:19:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] first schedule backup for schedule name [tp-backup-1705598306] is [tp-backup-1705598306-7f77056-interval-2024-01-18-171827] [2024-01-18T17:19:34.525Z] 2024-01-18 17:19:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating backup [tp-backup-1705598306-7f77056-interval-2024-01-18-171827] in org [default] [2024-01-18T17:19:34.525Z] 2024-01-18 17:19:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705598306-7f77056-interval-2024-01-18-171827] [2024-01-18T17:19:34.525Z] 2024-01-18 17:19:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84853-01-18-17h11m43s] in backup [tp-backup-1705598306-7f77056-interval-2024-01-18-171827] [2024-01-18T17:19:34.525Z] 2024-01-18 17:19:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84853-01-18-17h11m43s] in backup [tp-backup-1705598306-7f77056-interval-2024-01-18-171827] [2024-01-18T17:19:34.525Z] 2024-01-18 17:19:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:19:34.525Z] 2024-01-18 17:19:27 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:19:34.525Z] 2024-01-18 17:19:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:19:34.525Z] 2024-01-18 17:19:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-84853-01-18-17h11m43s] are [[mysql-data]] [2024-01-18T17:19:34.525Z] 2024-01-18 17:19:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T17:19:34.525Z] 2024-01-18 17:19:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating CR cleanup [2024-01-18T17:19:34.525Z] 2024-01-18 17:19:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T17:19:34.525Z] 2024-01-18 17:19:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] All CRs in [mysql-ibm-pxb-0-84853-01-18-17h11m43s] are [[]] [2024-01-18T17:19:34.526Z] INFO[2024-01-18 17:19:27] Verifying : Description : Verifying creation of first scheduled backup [tp-backup-1705598306-7f77056-interval-2024-01-18-171827] for single namespace with labels [zxhacjspvspdddqsnndtkbjshqnlnqutzovrzdmqwhauuaqyrygurqzelvdfelx=72be52d0-458b-40c7-8c0f-9d90281b5bfd,yqypwkjtptjodgpzmboldyekuvskosannzdwscnxvotgpilczrakescfqcqoour=f7dde987-af16-415a-9f4c-9f117f216254,etpumwvxiorhlbedvndojkerbnitppaqpiaidghvaapdhztwkeaykpoopimvmlq=38ff710d-1022-456d-920f-0fb90d5d7c37] [2024-01-18T17:19:34.526Z] INFO[2024-01-18 17:19:27] Actual:true, Expected: true [2024-01-18T17:19:34.526Z] 2024-01-18 17:19:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup tp-backup-1705598306-7f77056-interval-2024-01-18-171827 [2024-01-18T17:19:34.526Z] 2024-01-18 17:19:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:19:34.526Z] 2024-01-18 17:19:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are zxhacjspvspdddqsnndtkbjshqnlnqutzovrzdmqwhauuaqyrygurqzelvdfelx=72be52d0-458b-40c7-8c0f-9d90281b5bfd,yqypwkjtptjodgpzmboldyekuvskosannzdwscnxvotgpilczrakescfqcqoour=f7dde987-af16-415a-9f4c-9f117f216254,etpumwvxiorhlbedvndojkerbnitppaqpiaidghvaapdhztwkeaykpoopimvmlq=38ff710d-1022-456d-920f-0fb90d5d7c37 [2024-01-18T17:19:34.526Z] INFO[2024-01-18 17:19:27] Verifying : Description : Verifying if the labeled namespace [mysql-ibm-pxb-0-84853-01-18-17h11m43s] is backed up and checks for labels [zxhacjspvspdddqsnndtkbjshqnlnqutzovrzdmqwhauuaqyrygurqzelvdfelx=72be52d0-458b-40c7-8c0f-9d90281b5bfd,yqypwkjtptjodgpzmboldyekuvskosannzdwscnxvotgpilczrakescfqcqoour=f7dde987-af16-415a-9f4c-9f117f216254,etpumwvxiorhlbedvndojkerbnitppaqpiaidghvaapdhztwkeaykpoopimvmlq=38ff710d-1022-456d-920f-0fb90d5d7c37] applied to backup [tp-backup-1705598306-7f77056-interval-2024-01-18-171827] [2024-01-18T17:19:34.526Z] INFO[2024-01-18 17:19:27] Actual:true, Expected: true [2024-01-18T17:19:34.526Z] 2024-01-18 17:19:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Creating a schedule backup for multiple applications with namespace label filter [2024-01-18T17:19:34.526Z] 2024-01-18 17:19:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-18T17:19:34.526Z] 2024-01-18 17:19:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Volume snapshot classes found in the cluster - [2024-01-18T17:19:34.527Z] 2024-01-18 17:19:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] ibmc-vpcblock-snapshot [2024-01-18T17:19:34.527Z] 2024-01-18 17:19:27 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T17:20:30.698Z] 2024-01-18 17:20:28 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] first schedule backup for schedule name [tp-backup-1705598367] is [tp-backup-1705598367-d71ecc5-interval-2024-01-18-171928] [2024-01-18T17:20:30.698Z] 2024/01/18 17:20:28 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705598367-d71ecc5-interval-2024-01-18-171928] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T17:21:02.812Z] 2024-01-18 17:20:58 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating backup [tp-backup-1705598367-d71ecc5-interval-2024-01-18-171928] in org [default] [2024-01-18T17:21:02.812Z] 2024-01-18 17:20:58 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705598367-d71ecc5-interval-2024-01-18-171928] [2024-01-18T17:21:02.812Z] 2024-01-18 17:20:58 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84853-01-18-17h11m43s] in backup [tp-backup-1705598367-d71ecc5-interval-2024-01-18-171928] [2024-01-18T17:21:02.812Z] 2024-01-18 17:20:58 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84853-01-18-17h11m43s] in backup [tp-backup-1705598367-d71ecc5-interval-2024-01-18-171928] [2024-01-18T17:21:02.812Z] 2024-01-18 17:20:58 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T17:21:02.812Z] 2024-01-18 17:20:58 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:21:02.812Z] 2024-01-18 17:20:58 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:21:02.812Z] 2024-01-18 17:20:58 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-84853-01-18-17h11m43s] are [[mysql-data]] [2024-01-18T17:21:02.812Z] 2024-01-18 17:20:58 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T17:21:02.812Z] 2024-01-18 17:20:58 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84853-01-18-17h11m43s] in backup [tp-backup-1705598367-d71ecc5-interval-2024-01-18-171928] [2024-01-18T17:21:02.812Z] 2024-01-18 17:20:58 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84853-01-18-17h11m43s] in backup [tp-backup-1705598367-d71ecc5-interval-2024-01-18-171928] [2024-01-18T17:21:02.812Z] 2024-01-18 17:20:58 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:21:02.812Z] 2024-01-18 17:20:58 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:21:02.812Z] 2024-01-18 17:20:58 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:21:02.813Z] 2024-01-18 17:20:58 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-2-84853-01-18-17h11m43s] are [[mysql-data]] [2024-01-18T17:21:02.813Z] 2024-01-18 17:20:58 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T17:21:02.813Z] 2024-01-18 17:20:58 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating CR cleanup [2024-01-18T17:21:02.813Z] 2024-01-18 17:20:58 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T17:21:02.813Z] 2024-01-18 17:20:58 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] All CRs in [kube-system] are [[tp-backup-1705598367-d71ecc5-interval-2024-01-18-171928]] [2024-01-18T17:21:02.813Z] 2024-01-18 17:20:58 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] CR found for [[tp-backup-1705598367-d71ecc5-interval-2024-01-18-171928]] under [kube-system] namespace [2024-01-18T17:21:02.813Z] 2024/01/18 17:20:58 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-1705598367-d71ecc5-interval-2024-01-18-171928]}, Next try in [5s], timeout [5m0s] [2024-01-18T17:21:04.190Z] 2024-01-18 17:21:03 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] All CRs in [kube-system] are [[tp-backup-1705598367-d71ecc5-interval-2024-01-18-171928]] [2024-01-18T17:21:04.190Z] 2024-01-18 17:21:03 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] CR found for [[tp-backup-1705598367-d71ecc5-interval-2024-01-18-171928]] under [kube-system] namespace [2024-01-18T17:21:04.190Z] 2024/01/18 17:21:03 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-1705598367-d71ecc5-interval-2024-01-18-171928]}, Next try in [5s], timeout [5m0s] [2024-01-18T17:21:09.441Z] 2024-01-18 17:21:09 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] All CRs in [kube-system] are [[tp-backup-1705598367-d71ecc5-interval-2024-01-18-171928]] [2024-01-18T17:21:09.441Z] 2024-01-18 17:21:09 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] CR found for [[tp-backup-1705598367-d71ecc5-interval-2024-01-18-171928]] under [kube-system] namespace [2024-01-18T17:21:09.441Z] 2024/01/18 17:21:09 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-1705598367-d71ecc5-interval-2024-01-18-171928]}, Next try in [5s], timeout [5m0s] [2024-01-18T17:21:14.685Z] 2024-01-18 17:21:14 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] All CRs in [kube-system] are [[tp-backup-1705598367-d71ecc5-interval-2024-01-18-171928]] [2024-01-18T17:21:14.685Z] 2024-01-18 17:21:14 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] CR found for [[tp-backup-1705598367-d71ecc5-interval-2024-01-18-171928]] under [kube-system] namespace [2024-01-18T17:21:14.685Z] 2024/01/18 17:21:14 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-1705598367-d71ecc5-interval-2024-01-18-171928]}, Next try in [5s], timeout [5m0s] [2024-01-18T17:21:19.936Z] 2024-01-18 17:21:19 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] All CRs in [kube-system] are [[]] [2024-01-18T17:21:19.936Z] INFO[2024-01-18 17:21:19] Verifying : Description : Verifying creation of first scheduled backup [tp-backup-1705598367-d71ecc5-interval-2024-01-18-171928] for multiple namespaces with labels [axfchgaztneftcrwfiuzngrdjlhfjpfqhoadujitcdsfqebybvkdpxdojoxrrqh=b3f564ec-25e4-43f7-b0fd-003090761a0c,fmbpuazkdhcielxgcaoirvnatdpjhpmiyfuabhvcrjeduooptlnkbesnlbytjqe=71b9d259-202c-44a0-b8b0-fb880171bf46,tiekdrxgrofqdhbevpnfwsbzriqbtjgorhpewobxvvycasbdnbezkuzoftyjmgj=cf04b0a1-41c3-4f3a-a0fe-7b20ea61e8f5] [2024-01-18T17:21:19.936Z] INFO[2024-01-18 17:21:19] Actual:true, Expected: true [2024-01-18T17:21:19.936Z] 2024-01-18 17:21:19 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup tp-backup-1705598367-d71ecc5-interval-2024-01-18-171928 [2024-01-18T17:21:19.936Z] 2024-01-18 17:21:19 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:21:19.936Z] 2024-01-18 17:21:19 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are axfchgaztneftcrwfiuzngrdjlhfjpfqhoadujitcdsfqebybvkdpxdojoxrrqh=b3f564ec-25e4-43f7-b0fd-003090761a0c,fmbpuazkdhcielxgcaoirvnatdpjhpmiyfuabhvcrjeduooptlnkbesnlbytjqe=71b9d259-202c-44a0-b8b0-fb880171bf46,tiekdrxgrofqdhbevpnfwsbzriqbtjgorhpewobxvvycasbdnbezkuzoftyjmgj=cf04b0a1-41c3-4f3a-a0fe-7b20ea61e8f5 [2024-01-18T17:21:19.936Z] INFO[2024-01-18 17:21:19] Verifying : Description : Verifying if the labeled namespaces [[mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s]] are backed up and check for labels [axfchgaztneftcrwfiuzngrdjlhfjpfqhoadujitcdsfqebybvkdpxdojoxrrqh=b3f564ec-25e4-43f7-b0fd-003090761a0c,fmbpuazkdhcielxgcaoirvnatdpjhpmiyfuabhvcrjeduooptlnkbesnlbytjqe=71b9d259-202c-44a0-b8b0-fb880171bf46,tiekdrxgrofqdhbevpnfwsbzriqbtjgorhpewobxvvycasbdnbezkuzoftyjmgj=cf04b0a1-41c3-4f3a-a0fe-7b20ea61e8f5] applied to backups [tp-backup-1705598367-d71ecc5-interval-2024-01-18-171928] [2024-01-18T17:21:19.936Z] INFO[2024-01-18 17:21:19] Actual:true, Expected: true [2024-01-18T17:21:19.936Z] 2024-01-18 17:21:19 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Creating a schedule backup for all applications with namespace label filter [2024-01-18T17:21:19.936Z] 2024-01-18 17:21:19 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-18T17:21:19.936Z] 2024-01-18 17:21:19 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Volume snapshot classes found in the cluster - [2024-01-18T17:21:19.936Z] 2024-01-18 17:21:19 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] ibmc-vpcblock-snapshot [2024-01-18T17:21:19.937Z] 2024-01-18 17:21:19 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T17:22:27.562Z] 2024-01-18 17:22:20 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] first schedule backup for schedule name [tp-backup-1705598479] is [tp-backup-1705598479-b6d689c-interval-2024-01-18-172120] [2024-01-18T17:22:27.562Z] 2024/01/18 17:22:20 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705598479-b6d689c-interval-2024-01-18-172120] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T17:22:54.069Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating backup [tp-backup-1705598479-b6d689c-interval-2024-01-18-172120] in org [default] [2024-01-18T17:22:54.069Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705598479-b6d689c-interval-2024-01-18-172120] [2024-01-18T17:22:54.069Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84853-01-18-17h11m43s] in backup [tp-backup-1705598479-b6d689c-interval-2024-01-18-172120] [2024-01-18T17:22:54.069Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84853-01-18-17h11m43s] in backup [tp-backup-1705598479-b6d689c-interval-2024-01-18-172120] [2024-01-18T17:22:54.069Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:22:54.069Z] 2024-01-18 17:22:50 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:22:54.069Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:22:54.069Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-84853-01-18-17h11m43s] are [[mysql-data]] [2024-01-18T17:22:54.069Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T17:22:54.069Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84853-01-18-17h11m43s] in backup [tp-backup-1705598479-b6d689c-interval-2024-01-18-172120] [2024-01-18T17:22:54.069Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84853-01-18-17h11m43s] in backup [tp-backup-1705598479-b6d689c-interval-2024-01-18-172120] [2024-01-18T17:22:54.069Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T17:22:54.069Z] 2024-01-18 17:22:50 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:22:54.069Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:22:54.069Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-84853-01-18-17h11m43s] are [[mysql-data]] [2024-01-18T17:22:54.069Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T17:22:54.069Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84853-01-18-17h11m43s] in backup [tp-backup-1705598479-b6d689c-interval-2024-01-18-172120] [2024-01-18T17:22:54.070Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84853-01-18-17h11m43s] in backup [tp-backup-1705598479-b6d689c-interval-2024-01-18-172120] [2024-01-18T17:22:54.070Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:22:54.070Z] 2024-01-18 17:22:50 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:22:54.070Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:22:54.070Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-2-84853-01-18-17h11m43s] are [[mysql-data]] [2024-01-18T17:22:54.070Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T17:22:54.070Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating CR cleanup [2024-01-18T17:22:54.070Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T17:22:54.070Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] All CRs in [kube-system] are [[]] [2024-01-18T17:22:54.070Z] INFO[2024-01-18 17:22:50] Verifying : Description : Verifying creation of first schedule backup [tp-backup-1705598479-b6d689c-interval-2024-01-18-172120] for all namespaces with labels [kcvuvkfgekhphvfawturizbjgpfprhugkdjnfcqmvmiltrwcppbbuevnzswppph=ad9b899a-faee-4ea4-97f3-d0c839228f51,jcscxcmokfqjigdhdauujdccwnvaobakkvwqlltpoeesjygjfuoqvhovwgswoor=68f8b453-b3cf-4ffa-a7ad-1537518cf8ff,tyffzmgvkruasitedxdahjkjaihjewomwkwwxqkbhdbmghhnduauyfjbmlgvsbu=05b20d94-ef28-480f-be7d-3465fb05d059] [2024-01-18T17:22:54.070Z] INFO[2024-01-18 17:22:50] Actual:true, Expected: true [2024-01-18T17:22:54.070Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup tp-backup-1705598479-b6d689c-interval-2024-01-18-172120 [2024-01-18T17:22:54.070Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [mysql-ibm-pxb-0-84853-01-18-17h11m43s mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:22:54.070Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are kcvuvkfgekhphvfawturizbjgpfprhugkdjnfcqmvmiltrwcppbbuevnzswppph=ad9b899a-faee-4ea4-97f3-d0c839228f51,jcscxcmokfqjigdhdauujdccwnvaobakkvwqlltpoeesjygjfuoqvhovwgswoor=68f8b453-b3cf-4ffa-a7ad-1537518cf8ff,tyffzmgvkruasitedxdahjkjaihjewomwkwwxqkbhdbmghhnduauyfjbmlgvsbu=05b20d94-ef28-480f-be7d-3465fb05d059 [2024-01-18T17:22:54.071Z] INFO[2024-01-18 17:22:50] Verifying : Description : Verifying if the labeled namespaces [[mysql-ibm-pxb-0-84853-01-18-17h11m43s mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s]] are backed up and check for labels [kcvuvkfgekhphvfawturizbjgpfprhugkdjnfcqmvmiltrwcppbbuevnzswppph=ad9b899a-faee-4ea4-97f3-d0c839228f51,jcscxcmokfqjigdhdauujdccwnvaobakkvwqlltpoeesjygjfuoqvhovwgswoor=68f8b453-b3cf-4ffa-a7ad-1537518cf8ff,tyffzmgvkruasitedxdahjkjaihjewomwkwwxqkbhdbmghhnduauyfjbmlgvsbu=05b20d94-ef28-480f-be7d-3465fb05d059] applied to backups [tp-backup-1705598479-b6d689c-interval-2024-01-18-172120] [2024-01-18T17:22:54.071Z] INFO[2024-01-18 17:22:50] Actual:true, Expected: true [2024-01-18T17:22:54.071Z] 2024-01-18 17:22:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Restoring backup of single application [2024-01-18T17:22:54.071Z] 2024/01/18 17:22:51 DoRetryWithTimeout - Error: {restore status for [single-namespace-backup-1705598025-1705598571] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T17:23:26.119Z] 2024-01-18 17:23:21 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CreateRestore:#936] - Restore [single-namespace-backup-1705598025-1705598571] created successfully [2024-01-18T17:23:26.119Z] 2024-01-18 17:23:21 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switching cluster context to cluster [destination-cluster] [2024-01-18T17:23:26.119Z] 2024-01-18 17:23:21 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switching context to [/tmp/destination-config] [2024-01-18T17:23:26.119Z] 2024-01-18 17:23:21 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:23:26.119Z] 2024-01-18 17:23:21 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:23:26.119Z] 2024-01-18 17:23:21 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:23:26.119Z] 2024-01-18 17:23:22 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:23:26.120Z] 2024-01-18 17:23:22 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:23:26.120Z] 2024-01-18 17:23:22 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:23:26.120Z] 2024-01-18 17:23:22 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:23:26.120Z] 2024-01-18 17:23:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:23:26.120Z] 2024-01-18 17:23:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:23:26.120Z] 2024-01-18 17:23:23 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T17:23:26.120Z] 2024-01-18 17:23:23 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T17:23:26.120Z] 2024-01-18 17:23:23 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T17:23:26.120Z] 2024-01-18 17:23:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T17:23:26.120Z] 2024-01-18 17:23:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switched context to [/tmp/destination-config] [2024-01-18T17:23:26.120Z] 2024-01-18 17:23:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-84853-01-18-17h11m43s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-18T17:23:26.120Z] 2024-01-18 17:23:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:23:26.120Z] 2024-01-18 17:23:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T17:23:26.120Z] 2024-01-18 17:23:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T17:23:26.120Z] 2024-01-18 17:23:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating restore [single-namespace-backup-1705598025-1705598571] in org [default] [2024-01-18T17:23:26.120Z] 2024-01-18 17:23:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [single-namespace-backup-1705598025-1705598571] [2024-01-18T17:23:26.120Z] 2024-01-18 17:23:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2110] - Checking status of restore [single-namespace-backup-1705598025-1705598571] [2024-01-18T17:23:26.120Z] 2024-01-18 17:23:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-84853-01-18-17h11m43s] in restore [single-namespace-backup-1705598025-1705598571] [2024-01-18T17:23:26.121Z] 2024-01-18 17:23:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-84853-01-18-17h11m43s]) was found in restore [single-namespace-backup-1705598025-1705598571], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:23:26.121Z] 2024-01-18 17:23:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-84853-01-18-17h11m43s]) found in the restore [single-namespace-backup-1705598025-1705598571] was also present on the cluster/namespace [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:23:26.121Z] 2024-01-18 17:23:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-84853-01-18-17h11m43s]) was found in restore [single-namespace-backup-1705598025-1705598571], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:23:26.121Z] 2024-01-18 17:23:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-84853-01-18-17h11m43s]) found in the restore [single-namespace-backup-1705598025-1705598571] was also present on the cluster/namespace [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:23:26.121Z] 2024-01-18 17:23:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-84853-01-18-17h11m43s]) was found in restore [single-namespace-backup-1705598025-1705598571], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:23:26.121Z] 2024-01-18 17:23:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-84853-01-18-17h11m43s]) found in the restore [single-namespace-backup-1705598025-1705598571] was also present on the cluster/namespace [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:23:26.121Z] 2024-01-18 17:23:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-84853-01-18-17h11m43s]) was found in restore [single-namespace-backup-1705598025-1705598571], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:23:26.121Z] 2024-01-18 17:23:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-84853-01-18-17h11m43s]) found in the restore [single-namespace-backup-1705598025-1705598571] was also present on the cluster/namespace [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:23:26.121Z] 2024-01-18 17:23:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-84853-01-18-17h11m43s] in restore [single-namespace-backup-1705598025-1705598571] [2024-01-18T17:23:26.121Z] 2024-01-18 17:23:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-84853-01-18-17h11m43s] in restore [single-namespace-backup-1705598025-1705598571] [2024-01-18T17:23:26.121Z] 2024-01-18 17:23:24 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:23:26.122Z] 2024-01-18 17:23:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:23:26.122Z] 2024-01-18 17:23:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-84853-01-18-17h11m43s] are [[mysql-data]] [2024-01-18T17:23:26.122Z] 2024-01-18 17:23:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2240] - in restore [single-namespace-backup-1705598025-1705598571], the status of the restored volume [pvc-a5f54e44-a149-4a7d-9ea4-e533a0ed861c] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-18T17:23:26.122Z] 2024-01-18 17:23:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validate applications in restored namespace [mysql-ibm-pxb-0-84853-01-18-17h11m43s] due to restore [single-namespace-backup-1705598025-1705598571] [2024-01-18T17:23:26.122Z] 2024-01-18 17:23:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating mysql-ibm app [2024-01-18T17:23:26.122Z] 2024-01-18 17:23:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] wait for mysql-ibm app to start running [2024-01-18T17:23:26.122Z] 2024-01-18 17:23:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T17:23:26.122Z] 2024/01/18 17:23:24 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:23:26.122Z] pod name:mysql-7d766cd65c-znq4d namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s running:false ready:false node:10.240.64.7 [2024-01-18T17:23:26.122Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:23:36.065Z] 2024/01/18 17:23:34 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:23:36.065Z] pod name:mysql-7d766cd65c-znq4d namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s running:false ready:false node:10.240.64.7 [2024-01-18T17:23:36.065Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:23:46.008Z] 2024/01/18 17:23:44 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:23:46.008Z] pod name:mysql-7d766cd65c-znq4d namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s running:true ready:false node:10.240.64.7 [2024-01-18T17:23:46.008Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:23:55.965Z] 2024/01/18 17:23:55 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:23:55.965Z] pod name:mysql-7d766cd65c-znq4d namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s running:true ready:false node:10.240.64.7 [2024-01-18T17:23:55.965Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:24:05.916Z] 2024/01/18 17:24:05 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:24:05.916Z] pod name:mysql-7d766cd65c-znq4d namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s running:true ready:false node:10.240.64.7 [2024-01-18T17:24:05.916Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:24:07.821Z] INFO[2024-01-18 17:24:07] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T17:24:07.821Z] INFO[2024-01-18 17:24:07] sched stats: tick interval: 6607 < 1.2s (100%) [2024-01-18T17:24:07.821Z] INFO[2024-01-18 17:24:07] sched stats: tick process durations: 6606 < 500ms (100%) [2024-01-18T17:24:15.910Z] 2024/01/18 17:24:15 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:24:15.910Z] pod name:mysql-7d766cd65c-znq4d namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s running:true ready:false node:10.240.64.7 [2024-01-18T17:24:15.910Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:24:25.857Z] 2024/01/18 17:24:25 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:24:25.857Z] pod name:mysql-7d766cd65c-znq4d namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s running:true ready:false node:10.240.64.7 [2024-01-18T17:24:25.857Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:24:35.818Z] 2024/01/18 17:24:35 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:24:35.818Z] pod name:mysql-7d766cd65c-znq4d namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s running:true ready:false node:10.240.64.7 [2024-01-18T17:24:35.818Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:24:45.761Z] 2024/01/18 17:24:45 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:24:45.761Z] pod name:mysql-7d766cd65c-znq4d namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s running:true ready:false node:10.240.64.7 [2024-01-18T17:24:45.761Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:24:55.725Z] 2024-01-18 17:24:55 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T17:24:55.982Z] 2024-01-18 17:24:55 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T17:24:55.982Z] 2024-01-18 17:24:55 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating CR cleanup [2024-01-18T17:24:56.240Z] 2024-01-18 17:24:56 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T17:24:56.240Z] 2024-01-18 17:24:56 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] All CRs in [mysql-ibm-pxb-0-84853-01-18-17h11m43s] are [[]] [2024-01-18T17:24:56.240Z] 2024-01-18 17:24:56 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switching cluster context back to cluster path [/tmp/source-config] [2024-01-18T17:24:56.240Z] 2024-01-18 17:24:56 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switching context to [/tmp/source-config] [2024-01-18T17:24:56.240Z] 2024-01-18 17:24:56 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:24:56.504Z] 2024-01-18 17:24:56 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:24:56.504Z] 2024-01-18 17:24:56 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:24:56.504Z] 2024-01-18 17:24:56 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:24:56.504Z] 2024-01-18 17:24:56 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:24:56.760Z] 2024-01-18 17:24:56 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:24:57.016Z] 2024-01-18 17:24:56 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:24:57.580Z] 2024-01-18 17:24:57 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:24:57.836Z] 2024-01-18 17:24:57 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:24:58.093Z] 2024-01-18 17:24:57 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T17:24:58.349Z] 2024-01-18 17:24:58 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T17:24:58.349Z] 2024-01-18 17:24:58 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T17:24:58.350Z] 2024-01-18 17:24:58 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T17:24:58.350Z] 2024-01-18 17:24:58 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switched context to [/tmp/source-config] [2024-01-18T17:24:58.350Z] INFO[2024-01-18 17:24:58] Verifying : Description : Verifying backup restore with name [single-namespace-backup-1705598025-1705598571] in default namespace [2024-01-18T17:24:58.350Z] INFO[2024-01-18 17:24:58] Actual:true, Expected: true [2024-01-18T17:24:58.350Z] 2024-01-18 17:24:58 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Restoring multiple applications backup [2024-01-18T17:25:00.866Z] 2024/01/18 17:25:00 DoRetryWithTimeout - Error: {restore status for [multiple-namespace-backup-1705598116-1705598699] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T17:25:32.926Z] 2024/01/18 17:25:30 DoRetryWithTimeout - Error: {restore status for [multiple-namespace-backup-1705598116-1705598699] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T17:26:05.008Z] 2024-01-18 17:26:00 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CreateRestore:#936] - Restore [multiple-namespace-backup-1705598116-1705598699] created successfully [2024-01-18T17:26:05.008Z] 2024-01-18 17:26:00 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switching cluster context to cluster [destination-cluster] [2024-01-18T17:26:05.008Z] 2024-01-18 17:26:00 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switching context to [/tmp/destination-config] [2024-01-18T17:26:05.008Z] 2024-01-18 17:26:00 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:26:05.008Z] 2024-01-18 17:26:00 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:26:05.008Z] 2024-01-18 17:26:00 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:26:05.008Z] 2024-01-18 17:26:00 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:26:05.008Z] 2024-01-18 17:26:00 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:26:05.008Z] 2024-01-18 17:26:00 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:26:05.008Z] 2024-01-18 17:26:01 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:26:05.008Z] 2024-01-18 17:26:01 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:26:05.008Z] 2024-01-18 17:26:01 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:26:05.008Z] 2024-01-18 17:26:02 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T17:26:05.008Z] 2024-01-18 17:26:02 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T17:26:05.009Z] 2024-01-18 17:26:02 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T17:26:05.009Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T17:26:05.009Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switched context to [/tmp/destination-config] [2024-01-18T17:26:05.009Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-1-84853-01-18-17h11m43s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-18T17:26:05.009Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T17:26:05.009Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T17:26:05.009Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T17:26:05.009Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-2-84853-01-18-17h11m43s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-18T17:26:05.009Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:26:05.009Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T17:26:05.009Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T17:26:05.009Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating restore [multiple-namespace-backup-1705598116-1705598699] in org [default] [2024-01-18T17:26:05.009Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [multiple-namespace-backup-1705598116-1705598699] [2024-01-18T17:26:05.009Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2110] - Checking status of restore [multiple-namespace-backup-1705598116-1705598699] [2024-01-18T17:26:05.009Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-1-84853-01-18-17h11m43s] in restore [multiple-namespace-backup-1705598116-1705598699] [2024-01-18T17:26:05.009Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-1-84853-01-18-17h11m43s]) was found in restore [multiple-namespace-backup-1705598116-1705598699], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T17:26:05.009Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-1-84853-01-18-17h11m43s]) found in the restore [multiple-namespace-backup-1705598116-1705598699] was also present on the cluster/namespace [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T17:26:05.010Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-1-84853-01-18-17h11m43s]) was found in restore [multiple-namespace-backup-1705598116-1705598699], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T17:26:05.010Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-1-84853-01-18-17h11m43s]) found in the restore [multiple-namespace-backup-1705598116-1705598699] was also present on the cluster/namespace [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T17:26:05.010Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-1-84853-01-18-17h11m43s]) was found in restore [multiple-namespace-backup-1705598116-1705598699], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T17:26:05.010Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-1-84853-01-18-17h11m43s]) found in the restore [multiple-namespace-backup-1705598116-1705598699] was also present on the cluster/namespace [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T17:26:05.010Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-1-84853-01-18-17h11m43s]) was found in restore [multiple-namespace-backup-1705598116-1705598699], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T17:26:05.010Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-1-84853-01-18-17h11m43s]) found in the restore [multiple-namespace-backup-1705598116-1705598699] was also present on the cluster/namespace [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T17:26:05.010Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-1-84853-01-18-17h11m43s] in restore [multiple-namespace-backup-1705598116-1705598699] [2024-01-18T17:26:05.010Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-1-84853-01-18-17h11m43s] in restore [multiple-namespace-backup-1705598116-1705598699] [2024-01-18T17:26:05.010Z] 2024-01-18 17:26:02 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:26:05.010Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:26:05.010Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-84853-01-18-17h11m43s] are [[mysql-data]] [2024-01-18T17:26:05.010Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2240] - in restore [multiple-namespace-backup-1705598116-1705598699], the status of the restored volume [pvc-c36528d8-4b34-4855-ae22-8c705b1f1fd2] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-18T17:26:05.010Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validate applications in restored namespace [mysql-ibm-pxb-1-84853-01-18-17h11m43s] due to restore [multiple-namespace-backup-1705598116-1705598699] [2024-01-18T17:26:05.011Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating mysql-ibm app [2024-01-18T17:26:05.011Z] 2024-01-18 17:26:02 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] wait for mysql-ibm app to start running [2024-01-18T17:26:05.011Z] 2024-01-18 17:26:03 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T17:26:05.011Z] 2024/01/18 17:26:03 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:26:05.011Z] pod name:mysql-7d766cd65c-s6k48 namespace:mysql-ibm-pxb-1-84853-01-18-17h11m43s running:true ready:false node:10.240.64.7 [2024-01-18T17:26:05.011Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:26:14.962Z] 2024/01/18 17:26:13 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:26:14.962Z] pod name:mysql-7d766cd65c-s6k48 namespace:mysql-ibm-pxb-1-84853-01-18-17h11m43s running:true ready:false node:10.240.64.7 [2024-01-18T17:26:14.962Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:26:24.901Z] 2024/01/18 17:26:23 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:26:24.901Z] pod name:mysql-7d766cd65c-s6k48 namespace:mysql-ibm-pxb-1-84853-01-18-17h11m43s running:true ready:false node:10.240.64.7 [2024-01-18T17:26:24.901Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:26:34.842Z] 2024/01/18 17:26:33 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:26:34.842Z] pod name:mysql-7d766cd65c-s6k48 namespace:mysql-ibm-pxb-1-84853-01-18-17h11m43s running:true ready:false node:10.240.64.7 [2024-01-18T17:26:34.842Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:26:44.785Z] 2024/01/18 17:26:43 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:26:44.785Z] pod name:mysql-7d766cd65c-s6k48 namespace:mysql-ibm-pxb-1-84853-01-18-17h11m43s running:true ready:false node:10.240.64.7 [2024-01-18T17:26:44.785Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:26:53.993Z] 2024/01/18 17:26:53 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:26:53.993Z] pod name:mysql-7d766cd65c-s6k48 namespace:mysql-ibm-pxb-1-84853-01-18-17h11m43s running:true ready:false node:10.240.64.7 [2024-01-18T17:26:53.993Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:27:03.942Z] 2024-01-18 17:27:03 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T17:27:04.198Z] 2024-01-18 17:27:03 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T17:27:04.198Z] 2024-01-18 17:27:04 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-2-84853-01-18-17h11m43s] in restore [multiple-namespace-backup-1705598116-1705598699] [2024-01-18T17:27:04.198Z] 2024-01-18 17:27:04 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-2-84853-01-18-17h11m43s]) was found in restore [multiple-namespace-backup-1705598116-1705598699], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:27:04.198Z] 2024-01-18 17:27:04 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-2-84853-01-18-17h11m43s]) found in the restore [multiple-namespace-backup-1705598116-1705598699] was also present on the cluster/namespace [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:27:04.198Z] 2024-01-18 17:27:04 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-2-84853-01-18-17h11m43s]) was found in restore [multiple-namespace-backup-1705598116-1705598699], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:27:04.454Z] 2024-01-18 17:27:04 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-2-84853-01-18-17h11m43s]) found in the restore [multiple-namespace-backup-1705598116-1705598699] was also present on the cluster/namespace [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:27:04.454Z] 2024-01-18 17:27:04 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-2-84853-01-18-17h11m43s]) was found in restore [multiple-namespace-backup-1705598116-1705598699], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:27:04.454Z] 2024-01-18 17:27:04 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-2-84853-01-18-17h11m43s]) found in the restore [multiple-namespace-backup-1705598116-1705598699] was also present on the cluster/namespace [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:27:04.454Z] 2024-01-18 17:27:04 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-2-84853-01-18-17h11m43s]) was found in restore [multiple-namespace-backup-1705598116-1705598699], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:27:04.454Z] 2024-01-18 17:27:04 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-2-84853-01-18-17h11m43s]) found in the restore [multiple-namespace-backup-1705598116-1705598699] was also present on the cluster/namespace [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:27:04.454Z] 2024-01-18 17:27:04 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-2-84853-01-18-17h11m43s] in restore [multiple-namespace-backup-1705598116-1705598699] [2024-01-18T17:27:04.454Z] 2024-01-18 17:27:04 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-2-84853-01-18-17h11m43s] in restore [multiple-namespace-backup-1705598116-1705598699] [2024-01-18T17:27:04.454Z] 2024-01-18 17:27:04 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:27:04.455Z] 2024-01-18 17:27:04 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:27:04.455Z] 2024-01-18 17:27:04 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-2-84853-01-18-17h11m43s] are [[mysql-data]] [2024-01-18T17:27:04.455Z] 2024-01-18 17:27:04 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2240] - in restore [multiple-namespace-backup-1705598116-1705598699], the status of the restored volume [pvc-b299a5cc-080c-41de-be2d-f3b96005c78e] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-18T17:27:04.455Z] 2024-01-18 17:27:04 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validate applications in restored namespace [mysql-ibm-pxb-2-84853-01-18-17h11m43s] due to restore [multiple-namespace-backup-1705598116-1705598699] [2024-01-18T17:27:04.455Z] 2024-01-18 17:27:04 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating mysql-ibm app [2024-01-18T17:27:04.455Z] 2024-01-18 17:27:04 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] wait for mysql-ibm app to start running [2024-01-18T17:27:04.455Z] 2024-01-18 17:27:04 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T17:27:05.018Z] 2024-01-18 17:27:04 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T17:27:05.583Z] 2024-01-18 17:27:05 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T17:27:06.510Z] 2024-01-18 17:27:06 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating CR cleanup [2024-01-18T17:27:06.510Z] 2024-01-18 17:27:06 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T17:27:06.510Z] 2024-01-18 17:27:06 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] All CRs in [kube-system] are [[]] [2024-01-18T17:27:06.510Z] 2024-01-18 17:27:06 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switching cluster context back to cluster path [/tmp/source-config] [2024-01-18T17:27:06.510Z] 2024-01-18 17:27:06 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switching context to [/tmp/source-config] [2024-01-18T17:27:06.772Z] 2024-01-18 17:27:06 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:27:06.772Z] 2024-01-18 17:27:06 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:27:06.772Z] 2024-01-18 17:27:06 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:27:06.772Z] 2024-01-18 17:27:06 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:27:06.772Z] 2024-01-18 17:27:06 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:27:07.028Z] 2024-01-18 17:27:06 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:27:07.284Z] 2024-01-18 17:27:07 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:27:07.846Z] 2024-01-18 17:27:07 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:27:08.123Z] 2024-01-18 17:27:08 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:27:08.380Z] 2024-01-18 17:27:08 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T17:27:08.636Z] 2024-01-18 17:27:08 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T17:27:08.636Z] 2024-01-18 17:27:08 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T17:27:08.636Z] 2024-01-18 17:27:08 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T17:27:08.636Z] 2024-01-18 17:27:08 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switched context to [/tmp/source-config] [2024-01-18T17:27:08.636Z] INFO[2024-01-18 17:27:08] Verifying : Description : Verifying multiple backups [multiple-namespace-backup-1705598116-1705598699] restore in default namespace [2024-01-18T17:27:08.636Z] INFO[2024-01-18 17:27:08] Actual:true, Expected: true [2024-01-18T17:27:08.636Z] 2024-01-18 17:27:08 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Restoring all applications backup [2024-01-18T17:27:11.154Z] 2024/01/18 17:27:10 DoRetryWithTimeout - Error: {restore status for [all-namespace-backup-1705598207-1705598829] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T17:27:43.188Z] 2024-01-18 17:27:40 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CreateRestore:#936] - Restore [all-namespace-backup-1705598207-1705598829] created successfully [2024-01-18T17:27:43.189Z] 2024-01-18 17:27:40 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switching cluster context to cluster [destination-cluster] [2024-01-18T17:27:43.189Z] 2024-01-18 17:27:40 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switching context to [/tmp/destination-config] [2024-01-18T17:27:43.189Z] 2024-01-18 17:27:40 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:27:43.189Z] 2024-01-18 17:27:40 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:27:43.189Z] 2024-01-18 17:27:40 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:27:43.189Z] 2024-01-18 17:27:40 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:27:43.189Z] 2024-01-18 17:27:41 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:27:43.189Z] 2024-01-18 17:27:41 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:27:43.189Z] 2024-01-18 17:27:41 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:27:43.189Z] 2024-01-18 17:27:41 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:27:43.189Z] 2024-01-18 17:27:42 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:27:43.189Z] 2024-01-18 17:27:42 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T17:27:43.189Z] 2024-01-18 17:27:42 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T17:27:43.189Z] 2024-01-18 17:27:42 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T17:27:43.189Z] 2024-01-18 17:27:42 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T17:27:43.189Z] 2024-01-18 17:27:42 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switched context to [/tmp/destination-config] [2024-01-18T17:27:43.189Z] 2024-01-18 17:27:42 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-84853-01-18-17h11m43s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-18T17:27:43.189Z] 2024-01-18 17:27:42 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:27:43.189Z] 2024-01-18 17:27:42 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T17:27:43.190Z] 2024-01-18 17:27:42 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T17:27:43.190Z] 2024-01-18 17:27:42 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-1-84853-01-18-17h11m43s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-18T17:27:43.190Z] 2024-01-18 17:27:42 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T17:27:43.190Z] 2024-01-18 17:27:42 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T17:27:43.190Z] 2024-01-18 17:27:42 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T17:27:43.190Z] 2024-01-18 17:27:42 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-2-84853-01-18-17h11m43s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-18T17:27:43.190Z] 2024-01-18 17:27:42 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:27:43.190Z] 2024-01-18 17:27:42 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T17:27:43.190Z] 2024-01-18 17:27:42 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T17:27:43.190Z] 2024-01-18 17:27:42 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating restore [all-namespace-backup-1705598207-1705598829] in org [default] [2024-01-18T17:27:43.190Z] 2024-01-18 17:27:42 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [all-namespace-backup-1705598207-1705598829] [2024-01-18T17:27:43.190Z] 2024-01-18 17:27:42 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2110] - Checking status of restore [all-namespace-backup-1705598207-1705598829] [2024-01-18T17:27:43.190Z] 2024-01-18 17:27:42 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-84853-01-18-17h11m43s] in restore [all-namespace-backup-1705598207-1705598829] [2024-01-18T17:27:43.190Z] 2024-01-18 17:27:42 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-84853-01-18-17h11m43s]) was found in restore [all-namespace-backup-1705598207-1705598829], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:27:43.190Z] 2024-01-18 17:27:42 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-84853-01-18-17h11m43s]) found in the restore [all-namespace-backup-1705598207-1705598829] was also present on the cluster/namespace [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:27:43.191Z] 2024-01-18 17:27:42 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-84853-01-18-17h11m43s]) was found in restore [all-namespace-backup-1705598207-1705598829], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:27:43.191Z] 2024-01-18 17:27:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-84853-01-18-17h11m43s]) found in the restore [all-namespace-backup-1705598207-1705598829] was also present on the cluster/namespace [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:27:43.191Z] 2024-01-18 17:27:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-84853-01-18-17h11m43s]) was found in restore [all-namespace-backup-1705598207-1705598829], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:27:43.449Z] 2024-01-18 17:27:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-84853-01-18-17h11m43s]) found in the restore [all-namespace-backup-1705598207-1705598829] was also present on the cluster/namespace [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:27:43.449Z] 2024-01-18 17:27:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-84853-01-18-17h11m43s]) was found in restore [all-namespace-backup-1705598207-1705598829], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:27:43.449Z] 2024-01-18 17:27:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-84853-01-18-17h11m43s]) found in the restore [all-namespace-backup-1705598207-1705598829] was also present on the cluster/namespace [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:27:43.449Z] 2024-01-18 17:27:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-84853-01-18-17h11m43s] in restore [all-namespace-backup-1705598207-1705598829] [2024-01-18T17:27:43.449Z] 2024-01-18 17:27:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-84853-01-18-17h11m43s] in restore [all-namespace-backup-1705598207-1705598829] [2024-01-18T17:27:43.449Z] 2024-01-18 17:27:43 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:27:43.449Z] 2024-01-18 17:27:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:27:43.449Z] 2024-01-18 17:27:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-84853-01-18-17h11m43s] are [[mysql-data]] [2024-01-18T17:27:43.449Z] 2024-01-18 17:27:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2243] - in restore [all-namespace-backup-1705598207-1705598829], the status of the restored volume [pvc-a5f54e44-a149-4a7d-9ea4-e533a0ed861c] was not Success. It was [Retained] with reason [Skipped from volume restore as policy is set to Retain and pvc already exists] [2024-01-18T17:27:43.449Z] 2024-01-18 17:27:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validate applications in restored namespace [mysql-ibm-pxb-0-84853-01-18-17h11m43s] due to restore [all-namespace-backup-1705598207-1705598829] [2024-01-18T17:27:43.449Z] 2024-01-18 17:27:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating mysql-ibm app [2024-01-18T17:27:43.449Z] 2024-01-18 17:27:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] wait for mysql-ibm app to start running [2024-01-18T17:27:43.705Z] 2024-01-18 17:27:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T17:27:43.705Z] 2024-01-18 17:27:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T17:27:43.962Z] 2024-01-18 17:27:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T17:27:44.218Z] 2024-01-18 17:27:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-1-84853-01-18-17h11m43s] in restore [all-namespace-backup-1705598207-1705598829] [2024-01-18T17:27:44.219Z] 2024-01-18 17:27:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-1-84853-01-18-17h11m43s]) was found in restore [all-namespace-backup-1705598207-1705598829], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T17:27:44.219Z] 2024-01-18 17:27:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-1-84853-01-18-17h11m43s]) found in the restore [all-namespace-backup-1705598207-1705598829] was also present on the cluster/namespace [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T17:27:44.219Z] 2024-01-18 17:27:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-1-84853-01-18-17h11m43s]) was found in restore [all-namespace-backup-1705598207-1705598829], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T17:27:44.474Z] 2024-01-18 17:27:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-1-84853-01-18-17h11m43s]) found in the restore [all-namespace-backup-1705598207-1705598829] was also present on the cluster/namespace [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T17:27:44.474Z] 2024-01-18 17:27:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-1-84853-01-18-17h11m43s]) was found in restore [all-namespace-backup-1705598207-1705598829], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T17:27:44.475Z] 2024-01-18 17:27:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-1-84853-01-18-17h11m43s]) found in the restore [all-namespace-backup-1705598207-1705598829] was also present on the cluster/namespace [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T17:27:44.475Z] 2024-01-18 17:27:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-1-84853-01-18-17h11m43s]) was found in restore [all-namespace-backup-1705598207-1705598829], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T17:27:44.475Z] 2024-01-18 17:27:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-1-84853-01-18-17h11m43s]) found in the restore [all-namespace-backup-1705598207-1705598829] was also present on the cluster/namespace [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T17:27:44.475Z] 2024-01-18 17:27:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-1-84853-01-18-17h11m43s] in restore [all-namespace-backup-1705598207-1705598829] [2024-01-18T17:27:44.475Z] 2024-01-18 17:27:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-1-84853-01-18-17h11m43s] in restore [all-namespace-backup-1705598207-1705598829] [2024-01-18T17:27:44.730Z] 2024-01-18 17:27:44 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:27:44.731Z] 2024-01-18 17:27:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:27:44.731Z] 2024-01-18 17:27:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-84853-01-18-17h11m43s] are [[mysql-data]] [2024-01-18T17:27:44.731Z] 2024-01-18 17:27:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2243] - in restore [all-namespace-backup-1705598207-1705598829], the status of the restored volume [pvc-c36528d8-4b34-4855-ae22-8c705b1f1fd2] was not Success. It was [Retained] with reason [Skipped from volume restore as policy is set to Retain and pvc already exists] [2024-01-18T17:27:44.731Z] 2024-01-18 17:27:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validate applications in restored namespace [mysql-ibm-pxb-1-84853-01-18-17h11m43s] due to restore [all-namespace-backup-1705598207-1705598829] [2024-01-18T17:27:44.731Z] 2024-01-18 17:27:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating mysql-ibm app [2024-01-18T17:27:44.731Z] 2024-01-18 17:27:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] wait for mysql-ibm app to start running [2024-01-18T17:27:44.988Z] 2024-01-18 17:27:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T17:27:44.988Z] 2024-01-18 17:27:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T17:27:45.250Z] 2024-01-18 17:27:45 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T17:27:46.180Z] 2024-01-18 17:27:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-2-84853-01-18-17h11m43s] in restore [all-namespace-backup-1705598207-1705598829] [2024-01-18T17:27:46.180Z] 2024-01-18 17:27:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-2-84853-01-18-17h11m43s]) was found in restore [all-namespace-backup-1705598207-1705598829], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:27:46.180Z] 2024-01-18 17:27:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-2-84853-01-18-17h11m43s]) found in the restore [all-namespace-backup-1705598207-1705598829] was also present on the cluster/namespace [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:27:46.180Z] 2024-01-18 17:27:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-2-84853-01-18-17h11m43s]) was found in restore [all-namespace-backup-1705598207-1705598829], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:27:46.180Z] 2024-01-18 17:27:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-2-84853-01-18-17h11m43s]) found in the restore [all-namespace-backup-1705598207-1705598829] was also present on the cluster/namespace [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:27:46.181Z] 2024-01-18 17:27:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-2-84853-01-18-17h11m43s]) was found in restore [all-namespace-backup-1705598207-1705598829], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:27:46.436Z] 2024-01-18 17:27:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-2-84853-01-18-17h11m43s]) found in the restore [all-namespace-backup-1705598207-1705598829] was also present on the cluster/namespace [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:27:46.436Z] 2024-01-18 17:27:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-2-84853-01-18-17h11m43s]) was found in restore [all-namespace-backup-1705598207-1705598829], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:27:46.436Z] 2024-01-18 17:27:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-2-84853-01-18-17h11m43s]) found in the restore [all-namespace-backup-1705598207-1705598829] was also present on the cluster/namespace [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T17:27:46.436Z] 2024-01-18 17:27:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-2-84853-01-18-17h11m43s] in restore [all-namespace-backup-1705598207-1705598829] [2024-01-18T17:27:46.437Z] 2024-01-18 17:27:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-2-84853-01-18-17h11m43s] in restore [all-namespace-backup-1705598207-1705598829] [2024-01-18T17:27:46.692Z] 2024-01-18 17:27:46 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:27:46.692Z] 2024-01-18 17:27:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:27:46.693Z] 2024-01-18 17:27:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-2-84853-01-18-17h11m43s] are [[mysql-data]] [2024-01-18T17:27:46.693Z] 2024-01-18 17:27:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2243] - in restore [all-namespace-backup-1705598207-1705598829], the status of the restored volume [pvc-b299a5cc-080c-41de-be2d-f3b96005c78e] was not Success. It was [Retained] with reason [Skipped from volume restore as policy is set to Retain and pvc already exists] [2024-01-18T17:27:46.693Z] 2024-01-18 17:27:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validate applications in restored namespace [mysql-ibm-pxb-2-84853-01-18-17h11m43s] due to restore [all-namespace-backup-1705598207-1705598829] [2024-01-18T17:27:46.693Z] 2024-01-18 17:27:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating mysql-ibm app [2024-01-18T17:27:46.693Z] 2024-01-18 17:27:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] wait for mysql-ibm app to start running [2024-01-18T17:27:46.693Z] 2024-01-18 17:27:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T17:27:47.255Z] 2024-01-18 17:27:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T17:27:47.816Z] 2024-01-18 17:27:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T17:27:48.745Z] 2024-01-18 17:27:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating CR cleanup [2024-01-18T17:27:48.745Z] 2024-01-18 17:27:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T17:27:48.745Z] 2024-01-18 17:27:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] All CRs in [kube-system] are [[]] [2024-01-18T17:27:48.745Z] 2024-01-18 17:27:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switching cluster context back to cluster path [/tmp/source-config] [2024-01-18T17:27:48.745Z] 2024-01-18 17:27:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switching context to [/tmp/source-config] [2024-01-18T17:27:48.745Z] 2024-01-18 17:27:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:27:48.745Z] 2024-01-18 17:27:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:27:49.001Z] 2024-01-18 17:27:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:27:49.001Z] 2024-01-18 17:27:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:27:49.001Z] 2024-01-18 17:27:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:27:49.257Z] 2024-01-18 17:27:49 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:27:49.513Z] 2024-01-18 17:27:49 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:27:50.083Z] 2024-01-18 17:27:49 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:27:50.339Z] 2024-01-18 17:27:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:27:50.597Z] 2024-01-18 17:27:50 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T17:27:50.853Z] 2024-01-18 17:27:50 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T17:27:50.853Z] 2024-01-18 17:27:50 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T17:27:50.853Z] 2024-01-18 17:27:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T17:27:50.853Z] 2024-01-18 17:27:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switched context to [/tmp/source-config] [2024-01-18T17:27:50.853Z] INFO[2024-01-18 17:27:50] Verifying : Description : Verifying all backups [all-namespace-backup-1705598207-1705598829] restore in default namespace [2024-01-18T17:27:50.853Z] INFO[2024-01-18 17:27:50] Actual:true, Expected: true [2024-01-18T17:27:50.853Z] 2024-01-18 17:27:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Restoring the incremental scheduled backup of single application [2024-01-18T17:27:52.218Z] 2024-01-18 17:27:52 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Waiting for the incremental scheduled backup for single namespace to be triggered [2024-01-18T17:27:52.218Z] 2024-01-18 17:27:52 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Waiting for 15ns minutes for the next schedule backup to be triggered [2024-01-18T17:34:13.751Z] INFO[2024-01-18 17:34:08] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T17:34:13.751Z] INFO[2024-01-18 17:34:08] sched stats: tick interval: 7208 < 1.2s (100%) [2024-01-18T17:34:13.751Z] INFO[2024-01-18 17:34:08] sched stats: tick process durations: 7207 < 500ms (100%) [2024-01-18T17:43:05.340Z] 2024-01-18 17:42:52 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Next schedule backup name [tp-backup-1705598306-7f77056-interval-2024-01-18-173330] [2024-01-18T17:43:05.340Z] INFO[2024-01-18 17:42:52] Verifying : Description : Fetching the name of the second schedule backup [tp-backup-1705598306-7f77056-interval-2024-01-18-173330] [2024-01-18T17:43:05.340Z] INFO[2024-01-18 17:42:52] Actual:true, Expected: true [2024-01-18T17:43:05.340Z] 2024-01-18 17:42:52 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup tp-backup-1705598306-7f77056-interval-2024-01-18-173330 [2024-01-18T17:43:05.340Z] 2024-01-18 17:42:52 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:43:05.340Z] 2024-01-18 17:42:52 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are zxhacjspvspdddqsnndtkbjshqnlnqutzovrzdmqwhauuaqyrygurqzelvdfelx=72be52d0-458b-40c7-8c0f-9d90281b5bfd,yqypwkjtptjodgpzmboldyekuvskosannzdwscnxvotgpilczrakescfqcqoour=f7dde987-af16-415a-9f4c-9f117f216254,etpumwvxiorhlbedvndojkerbnitppaqpiaidghvaapdhztwkeaykpoopimvmlq=38ff710d-1022-456d-920f-0fb90d5d7c37 [2024-01-18T17:43:05.341Z] INFO[2024-01-18 17:42:52] Verifying : Description : Verifying if the labeled namespace [mysql-ibm-pxb-0-84853-01-18-17h11m43s] is backed up and checks for labels [zxhacjspvspdddqsnndtkbjshqnlnqutzovrzdmqwhauuaqyrygurqzelvdfelx=72be52d0-458b-40c7-8c0f-9d90281b5bfd,yqypwkjtptjodgpzmboldyekuvskosannzdwscnxvotgpilczrakescfqcqoour=f7dde987-af16-415a-9f4c-9f117f216254,etpumwvxiorhlbedvndojkerbnitppaqpiaidghvaapdhztwkeaykpoopimvmlq=38ff710d-1022-456d-920f-0fb90d5d7c37] applied to backup [tp-backup-1705598306-7f77056-interval-2024-01-18-173330] [2024-01-18T17:43:05.341Z] INFO[2024-01-18 17:42:52] Actual:true, Expected: true [2024-01-18T17:43:05.341Z] INFO[2024-01-18 17:42:52] Verifying : Description : Fetching namespaces [mysql-ibm-pxb-0-84853-01-18-17h11m43s] from schedule backup tp-backup-1705598306-7f77056-interval-2024-01-18-173330 [2024-01-18T17:43:05.341Z] INFO[2024-01-18 17:42:52] Actual:true, Expected: true [2024-01-18T17:43:05.341Z] 2024/01/18 17:42:52 DoRetryWithTimeout - Error: {restore status for [tp-backup-1705598306-1705599772] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T17:43:23.416Z] 2024-01-18 17:43:22 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CreateRestore:#936] - Restore [tp-backup-1705598306-1705599772] created successfully [2024-01-18T17:43:23.416Z] 2024-01-18 17:43:22 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switching cluster context to cluster [destination-cluster] [2024-01-18T17:43:23.416Z] 2024-01-18 17:43:22 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switching context to [/tmp/destination-config] [2024-01-18T17:43:23.416Z] 2024-01-18 17:43:22 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:43:23.416Z] 2024-01-18 17:43:22 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:43:23.416Z] 2024-01-18 17:43:22 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:43:23.416Z] 2024-01-18 17:43:22 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:43:23.416Z] 2024-01-18 17:43:22 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:43:23.416Z] 2024-01-18 17:43:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:43:23.672Z] 2024-01-18 17:43:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:43:23.928Z] 2024-01-18 17:43:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:43:24.492Z] 2024-01-18 17:43:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:43:24.750Z] 2024-01-18 17:43:24 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T17:43:24.750Z] 2024-01-18 17:43:24 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T17:43:24.750Z] 2024-01-18 17:43:24 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T17:43:24.750Z] 2024-01-18 17:43:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T17:43:24.750Z] 2024-01-18 17:43:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switched context to [/tmp/destination-config] [2024-01-18T17:43:24.750Z] 2024-01-18 17:43:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-84853-01-18-17h11m43s] with namespace mapping [map[mysql-ibm-pxb-0-84853-01-18-17h11m43s:mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772]] and storage Class Mapping [map[]] [2024-01-18T17:43:24.750Z] 2024-01-18 17:43:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T17:43:24.750Z] 2024-01-18 17:43:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T17:43:24.750Z] 2024-01-18 17:43:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T17:43:24.750Z] 2024-01-18 17:43:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating restore [tp-backup-1705598306-1705599772] in org [default] [2024-01-18T17:43:24.750Z] 2024-01-18 17:43:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-backup-1705598306-1705599772] [2024-01-18T17:43:25.007Z] 2024-01-18 17:43:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-backup-1705598306-1705599772] [2024-01-18T17:43:25.007Z] 2024-01-18 17:43:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772] in restore [tp-backup-1705598306-1705599772] [2024-01-18T17:43:25.007Z] 2024-01-18 17:43:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772]) was found in restore [tp-backup-1705598306-1705599772], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772] [2024-01-18T17:43:25.007Z] 2024-01-18 17:43:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772]) found in the restore [tp-backup-1705598306-1705599772] was also present on the cluster/namespace [mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772] [2024-01-18T17:43:25.007Z] 2024-01-18 17:43:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772]) was found in restore [tp-backup-1705598306-1705599772], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772] [2024-01-18T17:43:25.263Z] 2024-01-18 17:43:25 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772]) found in the restore [tp-backup-1705598306-1705599772] was also present on the cluster/namespace [mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772] [2024-01-18T17:43:25.263Z] 2024-01-18 17:43:25 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772]) was found in restore [tp-backup-1705598306-1705599772], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772] [2024-01-18T17:43:25.263Z] 2024-01-18 17:43:25 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772]) found in the restore [tp-backup-1705598306-1705599772] was also present on the cluster/namespace [mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772] [2024-01-18T17:43:25.263Z] 2024-01-18 17:43:25 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772]) was found in restore [tp-backup-1705598306-1705599772], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772] [2024-01-18T17:43:25.263Z] 2024-01-18 17:43:25 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772]) found in the restore [tp-backup-1705598306-1705599772] was also present on the cluster/namespace [mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772] [2024-01-18T17:43:25.263Z] 2024-01-18 17:43:25 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772] in restore [tp-backup-1705598306-1705599772] [2024-01-18T17:43:25.263Z] 2024-01-18 17:43:25 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772] in restore [tp-backup-1705598306-1705599772] [2024-01-18T17:43:25.530Z] 2024-01-18 17:43:25 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T17:43:25.530Z] 2024-01-18 17:43:25 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T17:43:25.530Z] 2024-01-18 17:43:25 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772] are [[mysql-data]] [2024-01-18T17:43:25.530Z] 2024-01-18 17:43:25 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2240] - in restore [tp-backup-1705598306-1705599772], the status of the restored volume [pvc-055ede56-da88-4a5e-822f-7d6b7a776b6f] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-18T17:43:25.530Z] 2024-01-18 17:43:25 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validate applications in restored namespace [mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772] due to restore [tp-backup-1705598306-1705599772] [2024-01-18T17:43:25.530Z] 2024-01-18 17:43:25 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating mysql-ibm app [2024-01-18T17:43:25.530Z] 2024-01-18 17:43:25 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] wait for mysql-ibm app to start running [2024-01-18T17:43:25.786Z] 2024-01-18 17:43:25 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T17:43:25.786Z] 2024/01/18 17:43:25 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:43:25.786Z] pod name:mysql-7d766cd65c-mbc5g namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772 running:false ready:false node:10.240.64.6 [2024-01-18T17:43:25.786Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:43:35.744Z] 2024/01/18 17:43:35 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:43:35.744Z] pod name:mysql-7d766cd65c-mbc5g namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772 running:false ready:false node:10.240.64.6 [2024-01-18T17:43:35.744Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:43:47.914Z] 2024/01/18 17:43:45 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:43:47.914Z] pod name:mysql-7d766cd65c-mbc5g namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772 running:false ready:false node:10.240.64.6 [2024-01-18T17:43:47.914Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:43:56.000Z] 2024/01/18 17:43:55 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:43:56.001Z] pod name:mysql-7d766cd65c-mbc5g namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772 running:false ready:false node:10.240.64.6 [2024-01-18T17:43:56.001Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:44:05.950Z] 2024/01/18 17:44:05 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:44:05.950Z] pod name:mysql-7d766cd65c-mbc5g namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772 running:true ready:false node:10.240.64.6 [2024-01-18T17:44:05.950Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:44:10.128Z] INFO[2024-01-18 17:44:09] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T17:44:10.128Z] INFO[2024-01-18 17:44:09] sched stats: tick interval: 7809 < 1.2s (100%) [2024-01-18T17:44:10.128Z] INFO[2024-01-18 17:44:09] sched stats: tick process durations: 7808 < 500ms (100%) [2024-01-18T17:44:16.669Z] 2024/01/18 17:44:15 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:44:16.669Z] pod name:mysql-7d766cd65c-mbc5g namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772 running:true ready:false node:10.240.64.6 [2024-01-18T17:44:16.669Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:44:26.611Z] 2024/01/18 17:44:25 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:44:26.611Z] pod name:mysql-7d766cd65c-mbc5g namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772 running:true ready:false node:10.240.64.6 [2024-01-18T17:44:26.611Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:44:36.561Z] 2024/01/18 17:44:36 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:44:36.561Z] pod name:mysql-7d766cd65c-mbc5g namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772 running:true ready:false node:10.240.64.6 [2024-01-18T17:44:36.561Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:44:46.509Z] 2024/01/18 17:44:46 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:44:46.509Z] pod name:mysql-7d766cd65c-mbc5g namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772 running:true ready:false node:10.240.64.6 [2024-01-18T17:44:46.509Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:44:56.464Z] 2024/01/18 17:44:56 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T17:44:56.464Z] pod name:mysql-7d766cd65c-mbc5g namespace:mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772 running:true ready:false node:10.240.64.6 [2024-01-18T17:44:56.464Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T17:45:06.407Z] 2024-01-18 17:45:06 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T17:45:06.408Z] 2024-01-18 17:45:06 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T17:45:06.664Z] 2024-01-18 17:45:06 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating CR cleanup [2024-01-18T17:45:06.664Z] 2024-01-18 17:45:06 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T17:45:06.664Z] 2024-01-18 17:45:06 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] All CRs in [mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772] are [[]] [2024-01-18T17:45:06.664Z] 2024-01-18 17:45:06 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switching cluster context back to cluster path [/tmp/source-config] [2024-01-18T17:45:06.664Z] 2024-01-18 17:45:06 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switching context to [/tmp/source-config] [2024-01-18T17:45:06.921Z] 2024-01-18 17:45:06 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:45:06.921Z] 2024-01-18 17:45:06 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:45:06.921Z] 2024-01-18 17:45:06 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T17:45:07.179Z] 2024-01-18 17:45:06 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:45:07.179Z] 2024-01-18 17:45:06 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:45:07.179Z] 2024-01-18 17:45:07 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T17:45:07.741Z] 2024-01-18 17:45:07 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:45:07.999Z] 2024-01-18 17:45:07 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:45:08.563Z] 2024-01-18 17:45:08 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T17:45:08.563Z] 2024-01-18 17:45:08 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T17:45:08.819Z] 2024-01-18 17:45:08 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T17:45:08.819Z] 2024-01-18 17:45:08 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T17:45:08.819Z] 2024-01-18 17:45:08 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T17:45:08.819Z] 2024-01-18 17:45:08 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switched context to [/tmp/source-config] [2024-01-18T17:45:08.819Z] INFO[2024-01-18 17:45:08] Verifying : Description : Verification of restoring scheduled backups tp-backup-1705598306-1705599772 in custom namespace map[mysql-ibm-pxb-0-84853-01-18-17h11m43s:mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772] [2024-01-18T17:45:08.819Z] INFO[2024-01-18 17:45:08] Actual:true, Expected: true [2024-01-18T17:45:08.819Z] 2024-01-18 17:45:08 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Restoring the incremental scheduled backup of multiple applications [2024-01-18T17:45:10.185Z] 2024-01-18 17:45:10 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Waiting for the incremental scheduled backup for multiple namespace to be triggered [2024-01-18T17:45:10.185Z] 2024-01-18 17:45:10 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Waiting for 15ns minutes for the next schedule backup to be triggered [2024-01-18T17:54:18.227Z] INFO[2024-01-18 17:54:10] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T17:54:18.227Z] INFO[2024-01-18 17:54:10] sched stats: tick interval: 8410 < 1.2s (100%) [2024-01-18T17:54:18.227Z] INFO[2024-01-18 17:54:10] sched stats: tick process durations: 8409 < 500ms (100%) [2024-01-18T18:00:24.869Z] 2024-01-18 18:00:10 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Next schedule backup name [tp-backup-1705598367-d71ecc5-interval-2024-01-18-174934] [2024-01-18T18:00:24.869Z] INFO[2024-01-18 18:00:10] Verifying : Description : Fetching the name of the second schedule backup [tp-backup-1705598367-d71ecc5-interval-2024-01-18-173431] [2024-01-18T18:00:24.869Z] INFO[2024-01-18 18:00:10] Actual:true, Expected: true [2024-01-18T18:00:24.869Z] 2024-01-18 18:00:10 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup tp-backup-1705598367-d71ecc5-interval-2024-01-18-173431 [2024-01-18T18:00:24.869Z] 2024-01-18 18:00:10 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T18:00:24.869Z] 2024-01-18 18:00:10 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are axfchgaztneftcrwfiuzngrdjlhfjpfqhoadujitcdsfqebybvkdpxdojoxrrqh=b3f564ec-25e4-43f7-b0fd-003090761a0c,fmbpuazkdhcielxgcaoirvnatdpjhpmiyfuabhvcrjeduooptlnkbesnlbytjqe=71b9d259-202c-44a0-b8b0-fb880171bf46,tiekdrxgrofqdhbevpnfwsbzriqbtjgorhpewobxvvycasbdnbezkuzoftyjmgj=cf04b0a1-41c3-4f3a-a0fe-7b20ea61e8f5 [2024-01-18T18:00:24.870Z] INFO[2024-01-18 18:00:10] Verifying : Description : Verifying if the labeled namespace [[mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s]] is backed up and checks for labels [axfchgaztneftcrwfiuzngrdjlhfjpfqhoadujitcdsfqebybvkdpxdojoxrrqh=b3f564ec-25e4-43f7-b0fd-003090761a0c,fmbpuazkdhcielxgcaoirvnatdpjhpmiyfuabhvcrjeduooptlnkbesnlbytjqe=71b9d259-202c-44a0-b8b0-fb880171bf46,tiekdrxgrofqdhbevpnfwsbzriqbtjgorhpewobxvvycasbdnbezkuzoftyjmgj=cf04b0a1-41c3-4f3a-a0fe-7b20ea61e8f5] applied to backup [tp-backup-1705598367-d71ecc5-interval-2024-01-18-173431] [2024-01-18T18:00:24.870Z] INFO[2024-01-18 18:00:10] Actual:true, Expected: true [2024-01-18T18:00:24.870Z] INFO[2024-01-18 18:00:10] Verifying : Description : Fetching namespaces [mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s] from schedule backup tp-backup-1705598367-d71ecc5-interval-2024-01-18-173431 [2024-01-18T18:00:24.870Z] INFO[2024-01-18 18:00:10] Actual:true, Expected: true [2024-01-18T18:00:24.870Z] 2024/01/18 18:00:10 DoRetryWithTimeout - Error: {restore status for [tp-backup-1705598367-1705600810] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T18:00:42.927Z] 2024/01/18 18:00:40 DoRetryWithTimeout - Error: {restore status for [tp-backup-1705598367-1705600810] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T18:01:14.957Z] 2024-01-18 18:01:10 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CreateRestore:#936] - Restore [tp-backup-1705598367-1705600810] created successfully [2024-01-18T18:01:14.957Z] 2024-01-18 18:01:10 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switching cluster context to cluster [source-cluster] [2024-01-18T18:01:14.957Z] 2024-01-18 18:01:10 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switching context: The context is already [/tmp/source-config] [2024-01-18T18:01:14.957Z] 2024-01-18 18:01:10 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-1-84853-01-18-17h11m43s] with namespace mapping [map[mysql-ibm-pxb-1-84853-01-18-17h11m43s:tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s:tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s]] and storage Class Mapping [map[]] [2024-01-18T18:01:14.957Z] 2024-01-18 18:01:10 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T18:01:14.957Z] 2024-01-18 18:01:10 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T18:01:14.957Z] 2024-01-18 18:01:10 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T18:01:14.957Z] 2024-01-18 18:01:10 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-2-84853-01-18-17h11m43s] with namespace mapping [map[mysql-ibm-pxb-1-84853-01-18-17h11m43s:tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s:tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s]] and storage Class Mapping [map[]] [2024-01-18T18:01:14.957Z] 2024-01-18 18:01:10 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T18:01:14.957Z] 2024-01-18 18:01:10 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T18:01:14.957Z] 2024-01-18 18:01:10 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T18:01:14.957Z] 2024-01-18 18:01:10 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating restore [tp-backup-1705598367-1705600810] in org [default] [2024-01-18T18:01:14.957Z] 2024-01-18 18:01:10 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-backup-1705598367-1705600810] [2024-01-18T18:01:14.958Z] 2024-01-18 18:01:10 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-backup-1705598367-1705600810] [2024-01-18T18:01:14.958Z] 2024-01-18 18:01:10 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating specs for the namespace (restoredAppContext) [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] in restore [tp-backup-1705598367-1705600810] [2024-01-18T18:01:14.958Z] 2024-01-18 18:01:10 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s]) was found in restore [tp-backup-1705598367-1705600810], as expected by presence in expectedRestoredAppContext [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T18:01:14.958Z] 2024-01-18 18:01:11 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s]) found in the restore [tp-backup-1705598367-1705600810] was also present on the cluster/namespace [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T18:01:14.958Z] 2024-01-18 18:01:11 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s]) was found in restore [tp-backup-1705598367-1705600810], as expected by presence in expectedRestoredAppContext [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T18:01:14.958Z] 2024-01-18 18:01:11 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s]) found in the restore [tp-backup-1705598367-1705600810] was also present on the cluster/namespace [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T18:01:14.958Z] 2024-01-18 18:01:11 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s]) was found in restore [tp-backup-1705598367-1705600810], as expected by presence in expectedRestoredAppContext [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T18:01:14.958Z] 2024-01-18 18:01:11 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s]) found in the restore [tp-backup-1705598367-1705600810] was also present on the cluster/namespace [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T18:01:14.958Z] 2024-01-18 18:01:11 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s]) was found in restore [tp-backup-1705598367-1705600810], as expected by presence in expectedRestoredAppContext [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T18:01:14.958Z] 2024-01-18 18:01:11 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s]) found in the restore [tp-backup-1705598367-1705600810] was also present on the cluster/namespace [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T18:01:14.958Z] 2024-01-18 18:01:11 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating Restored Volumes for the namespace (restoredAppContext) [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] in restore [tp-backup-1705598367-1705600810] [2024-01-18T18:01:14.959Z] 2024-01-18 18:01:11 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] in restore [tp-backup-1705598367-1705600810] [2024-01-18T18:01:14.959Z] 2024-01-18 18:01:11 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T18:01:14.959Z] 2024-01-18 18:01:11 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T18:01:14.959Z] 2024-01-18 18:01:11 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] are [[mysql-data]] [2024-01-18T18:01:14.959Z] 2024-01-18 18:01:11 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2240] - in restore [tp-backup-1705598367-1705600810], the status of the restored volume [pvc-36a201fc-9ab6-49ca-9d5c-0b316e6c925e] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-18T18:01:14.959Z] 2024-01-18 18:01:11 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validate applications in restored namespace [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] due to restore [tp-backup-1705598367-1705600810] [2024-01-18T18:01:14.960Z] 2024-01-18 18:01:11 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating mysql-ibm app [2024-01-18T18:01:14.960Z] 2024-01-18 18:01:11 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] wait for mysql-ibm app to start running [2024-01-18T18:01:14.960Z] 2024-01-18 18:01:11 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T18:01:14.960Z] 2024/01/18 18:01:11 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:01:14.960Z] pod name:mysql-7d766cd65c-8cq5q namespace:tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s running:true ready:false node:10.240.64.9 [2024-01-18T18:01:14.960Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:01:21.498Z] 2024/01/18 18:01:21 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:01:21.499Z] pod name:mysql-7d766cd65c-8cq5q namespace:tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s running:true ready:false node:10.240.64.9 [2024-01-18T18:01:21.499Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:01:33.669Z] 2024/01/18 18:01:31 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:01:33.669Z] pod name:mysql-7d766cd65c-8cq5q namespace:tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s running:true ready:false node:10.240.64.9 [2024-01-18T18:01:33.669Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:01:41.759Z] 2024/01/18 18:01:41 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:01:41.759Z] pod name:mysql-7d766cd65c-8cq5q namespace:tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s running:true ready:false node:10.240.64.9 [2024-01-18T18:01:41.759Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:01:53.939Z] 2024/01/18 18:01:51 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:01:53.939Z] pod name:mysql-7d766cd65c-8cq5q namespace:tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s running:true ready:false node:10.240.64.9 [2024-01-18T18:01:53.939Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:02:02.024Z] 2024/01/18 18:02:01 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:02:02.024Z] pod name:mysql-7d766cd65c-8cq5q namespace:tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s running:true ready:false node:10.240.64.9 [2024-01-18T18:02:02.024Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:02:14.192Z] 2024-01-18 18:02:12 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T18:02:14.192Z] 2024-01-18 18:02:12 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T18:02:14.192Z] 2024-01-18 18:02:12 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating specs for the namespace (restoredAppContext) [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] in restore [tp-backup-1705598367-1705600810] [2024-01-18T18:02:14.193Z] 2024-01-18 18:02:12 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s]) was found in restore [tp-backup-1705598367-1705600810], as expected by presence in expectedRestoredAppContext [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T18:02:14.193Z] 2024-01-18 18:02:12 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s]) found in the restore [tp-backup-1705598367-1705600810] was also present on the cluster/namespace [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T18:02:14.193Z] 2024-01-18 18:02:12 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s]) was found in restore [tp-backup-1705598367-1705600810], as expected by presence in expectedRestoredAppContext [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T18:02:14.193Z] 2024-01-18 18:02:12 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s]) found in the restore [tp-backup-1705598367-1705600810] was also present on the cluster/namespace [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T18:02:14.193Z] 2024-01-18 18:02:12 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s]) was found in restore [tp-backup-1705598367-1705600810], as expected by presence in expectedRestoredAppContext [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T18:02:14.193Z] 2024-01-18 18:02:12 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s]) found in the restore [tp-backup-1705598367-1705600810] was also present on the cluster/namespace [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T18:02:14.193Z] 2024-01-18 18:02:12 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s]) was found in restore [tp-backup-1705598367-1705600810], as expected by presence in expectedRestoredAppContext [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T18:02:14.193Z] 2024-01-18 18:02:12 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s]) found in the restore [tp-backup-1705598367-1705600810] was also present on the cluster/namespace [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T18:02:14.193Z] 2024-01-18 18:02:12 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating Restored Volumes for the namespace (restoredAppContext) [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] in restore [tp-backup-1705598367-1705600810] [2024-01-18T18:02:14.194Z] 2024-01-18 18:02:12 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] in restore [tp-backup-1705598367-1705600810] [2024-01-18T18:02:14.194Z] 2024-01-18 18:02:12 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T18:02:14.194Z] 2024-01-18 18:02:12 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T18:02:14.194Z] 2024-01-18 18:02:12 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] are [[mysql-data]] [2024-01-18T18:02:14.194Z] 2024-01-18 18:02:12 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2240] - in restore [tp-backup-1705598367-1705600810], the status of the restored volume [pvc-de275780-a0db-4c53-a8ee-2c26f42e1539] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-18T18:02:14.194Z] 2024-01-18 18:02:12 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validate applications in restored namespace [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] due to restore [tp-backup-1705598367-1705600810] [2024-01-18T18:02:14.194Z] 2024-01-18 18:02:12 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating mysql-ibm app [2024-01-18T18:02:14.194Z] 2024-01-18 18:02:12 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] wait for mysql-ibm app to start running [2024-01-18T18:02:14.194Z] 2024-01-18 18:02:12 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T18:02:14.194Z] 2024/01/18 18:02:12 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:02:14.194Z] pod name:mysql-7d766cd65c-48gtr namespace:tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s running:true ready:false node:10.240.64.8 [2024-01-18T18:02:14.194Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:02:24.135Z] 2024-01-18 18:02:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T18:02:24.135Z] 2024-01-18 18:02:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T18:02:24.135Z] 2024-01-18 18:02:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating CR cleanup [2024-01-18T18:02:24.135Z] 2024-01-18 18:02:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T18:02:24.135Z] 2024-01-18 18:02:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] All CRs in [kube-system] are [[]] [2024-01-18T18:02:24.135Z] 2024-01-18 18:02:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switching cluster context back to cluster path [/tmp/source-config] [2024-01-18T18:02:24.135Z] 2024-01-18 18:02:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switching context: The context is already [/tmp/source-config] [2024-01-18T18:02:24.135Z] INFO[2024-01-18 18:02:23] Verifying : Description : Verification of restoring scheduled backups for multiple application [tp-backup-1705598367-1705600810] in custom namespace [map[mysql-ibm-pxb-1-84853-01-18-17h11m43s:tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s:tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s]] [2024-01-18T18:02:24.135Z] INFO[2024-01-18 18:02:23] Actual:true, Expected: true [2024-01-18T18:02:24.135Z] 2024-01-18 18:02:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Restoring the incremental scheduled backup of all applications [2024-01-18T18:02:24.135Z] 2024-01-18 18:02:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Waiting for the incremental scheduled backup for all namespace to be triggered [2024-01-18T18:02:24.135Z] 2024-01-18 18:02:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Waiting for 15ns minutes for the next schedule backup to be triggered [2024-01-18T18:04:16.966Z] INFO[2024-01-18 18:04:10] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T18:04:16.966Z] INFO[2024-01-18 18:04:10] sched stats: tick interval: 9010 < 1.2s (100%) [2024-01-18T18:04:16.966Z] INFO[2024-01-18 18:04:10] sched stats: tick process durations: 9009 < 500ms (100%) [2024-01-18T18:14:23.570Z] INFO[2024-01-18 18:14:11] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T18:14:23.570Z] INFO[2024-01-18 18:14:11] sched stats: tick interval: 9611 < 1.2s (100%) [2024-01-18T18:14:23.570Z] INFO[2024-01-18 18:14:11] sched stats: tick process durations: 9610 < 500ms (100%) [2024-01-18T18:17:29.977Z] 2024-01-18 18:17:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Next schedule backup name [tp-backup-1705598479-b6d689c-interval-2024-01-18-180629] [2024-01-18T18:17:29.977Z] INFO[2024-01-18 18:17:23] Verifying : Description : Fetching the name of the second schedule backup [tp-backup-1705598479-b6d689c-interval-2024-01-18-173623] [2024-01-18T18:17:29.977Z] INFO[2024-01-18 18:17:23] Actual:true, Expected: true [2024-01-18T18:17:29.977Z] 2024-01-18 18:17:23 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup tp-backup-1705598479-b6d689c-interval-2024-01-18-173623 [2024-01-18T18:17:29.977Z] 2024-01-18 18:17:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [mysql-ibm-pxb-0-84853-01-18-17h11m43s mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T18:17:29.977Z] 2024-01-18 18:17:24 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are kcvuvkfgekhphvfawturizbjgpfprhugkdjnfcqmvmiltrwcppbbuevnzswppph=ad9b899a-faee-4ea4-97f3-d0c839228f51,jcscxcmokfqjigdhdauujdccwnvaobakkvwqlltpoeesjygjfuoqvhovwgswoor=68f8b453-b3cf-4ffa-a7ad-1537518cf8ff,tyffzmgvkruasitedxdahjkjaihjewomwkwwxqkbhdbmghhnduauyfjbmlgvsbu=05b20d94-ef28-480f-be7d-3465fb05d059 [2024-01-18T18:17:29.978Z] INFO[2024-01-18 18:17:24] Verifying : Description : Verifying if the labeled namespace [[mysql-ibm-pxb-0-84853-01-18-17h11m43s mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s]] is backed up and checks for labels [kcvuvkfgekhphvfawturizbjgpfprhugkdjnfcqmvmiltrwcppbbuevnzswppph=ad9b899a-faee-4ea4-97f3-d0c839228f51,jcscxcmokfqjigdhdauujdccwnvaobakkvwqlltpoeesjygjfuoqvhovwgswoor=68f8b453-b3cf-4ffa-a7ad-1537518cf8ff,tyffzmgvkruasitedxdahjkjaihjewomwkwwxqkbhdbmghhnduauyfjbmlgvsbu=05b20d94-ef28-480f-be7d-3465fb05d059] applied to backup [tp-backup-1705598479-b6d689c-interval-2024-01-18-173623] [2024-01-18T18:17:29.978Z] INFO[2024-01-18 18:17:24] Actual:true, Expected: true [2024-01-18T18:17:29.978Z] INFO[2024-01-18 18:17:24] Verifying : Description : Fetching namespaces [mysql-ibm-pxb-0-84853-01-18-17h11m43s mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s] from schedule backup tp-backup-1705598479-b6d689c-interval-2024-01-18-173623 [2024-01-18T18:17:29.978Z] INFO[2024-01-18 18:17:24] Actual:true, Expected: true [2024-01-18T18:17:29.978Z] 2024/01/18 18:17:24 DoRetryWithTimeout - Error: {restore status for [tp-backup-1705598479-1705601844] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T18:17:56.481Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CreateRestore:#936] - Restore [tp-backup-1705598479-1705601844] created successfully [2024-01-18T18:17:56.481Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switching cluster context to cluster [source-cluster] [2024-01-18T18:17:56.481Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switching context: The context is already [/tmp/source-config] [2024-01-18T18:17:56.481Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-84853-01-18-17h11m43s] with namespace mapping [map[mysql-ibm-pxb-0-84853-01-18-17h11m43s:tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s mysql-ibm-pxb-1-84853-01-18-17h11m43s:tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s:tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s]] and storage Class Mapping [map[]] [2024-01-18T18:17:56.481Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T18:17:56.481Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T18:17:56.481Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T18:17:56.481Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-1-84853-01-18-17h11m43s] with namespace mapping [map[mysql-ibm-pxb-0-84853-01-18-17h11m43s:tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s mysql-ibm-pxb-1-84853-01-18-17h11m43s:tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s:tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s]] and storage Class Mapping [map[]] [2024-01-18T18:17:56.481Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T18:17:56.481Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T18:17:56.481Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T18:17:56.482Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-2-84853-01-18-17h11m43s] with namespace mapping [map[mysql-ibm-pxb-0-84853-01-18-17h11m43s:tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s mysql-ibm-pxb-1-84853-01-18-17h11m43s:tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s:tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s]] and storage Class Mapping [map[]] [2024-01-18T18:17:56.482Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T18:17:56.482Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T18:17:56.482Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T18:17:56.482Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating restore [tp-backup-1705598479-1705601844] in org [default] [2024-01-18T18:17:56.482Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-backup-1705598479-1705601844] [2024-01-18T18:17:56.482Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-backup-1705598479-1705601844] [2024-01-18T18:17:56.482Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating specs for the namespace (restoredAppContext) [tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s] in restore [tp-backup-1705598479-1705601844] [2024-01-18T18:17:56.483Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s]) was found in restore [tp-backup-1705598479-1705601844], as expected by presence in expectedRestoredAppContext [tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T18:17:56.483Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s]) found in the restore [tp-backup-1705598479-1705601844] was also present on the cluster/namespace [tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T18:17:56.483Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s]) was found in restore [tp-backup-1705598479-1705601844], as expected by presence in expectedRestoredAppContext [tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T18:17:56.483Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s]) found in the restore [tp-backup-1705598479-1705601844] was also present on the cluster/namespace [tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T18:17:56.484Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s]) was found in restore [tp-backup-1705598479-1705601844], as expected by presence in expectedRestoredAppContext [tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T18:17:56.484Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s]) found in the restore [tp-backup-1705598479-1705601844] was also present on the cluster/namespace [tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T18:17:56.484Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s]) was found in restore [tp-backup-1705598479-1705601844], as expected by presence in expectedRestoredAppContext [tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T18:17:56.484Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s]) found in the restore [tp-backup-1705598479-1705601844] was also present on the cluster/namespace [tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T18:17:56.484Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating Restored Volumes for the namespace (restoredAppContext) [tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s] in restore [tp-backup-1705598479-1705601844] [2024-01-18T18:17:56.484Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s] in restore [tp-backup-1705598479-1705601844] [2024-01-18T18:17:56.484Z] 2024-01-18 18:17:54 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T18:17:56.484Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T18:17:56.485Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s] are [[mysql-data]] [2024-01-18T18:17:56.485Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2240] - in restore [tp-backup-1705598479-1705601844], the status of the restored volume [pvc-d5746687-7194-471d-9b58-5aac00cb04c9] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-18T18:17:56.485Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validate applications in restored namespace [tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s] due to restore [tp-backup-1705598479-1705601844] [2024-01-18T18:17:56.485Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating mysql-ibm app [2024-01-18T18:17:56.485Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] wait for mysql-ibm app to start running [2024-01-18T18:17:56.485Z] 2024-01-18 18:17:54 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T18:17:56.485Z] 2024/01/18 18:17:55 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:17:56.485Z] pod name:mysql-7d766cd65c-d9llq namespace:tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s running:false ready:false node:10.240.64.27 [2024-01-18T18:17:56.485Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:18:06.428Z] 2024/01/18 18:18:05 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:18:06.428Z] pod name:mysql-7d766cd65c-d9llq namespace:tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s running:false ready:false node:10.240.64.27 [2024-01-18T18:18:06.428Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:18:16.390Z] 2024/01/18 18:18:15 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:18:16.390Z] pod name:mysql-7d766cd65c-d9llq namespace:tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s running:false ready:false node:10.240.64.27 [2024-01-18T18:18:16.390Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:18:26.336Z] 2024/01/18 18:18:25 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:18:26.336Z] pod name:mysql-7d766cd65c-d9llq namespace:tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s running:false ready:false node:10.240.64.27 [2024-01-18T18:18:26.336Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:18:36.277Z] 2024/01/18 18:18:35 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:18:36.277Z] pod name:mysql-7d766cd65c-d9llq namespace:tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s running:true ready:false node:10.240.64.27 [2024-01-18T18:18:36.277Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:18:46.230Z] 2024/01/18 18:18:45 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:18:46.230Z] pod name:mysql-7d766cd65c-d9llq namespace:tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s running:true ready:false node:10.240.64.27 [2024-01-18T18:18:46.230Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:18:56.182Z] 2024/01/18 18:18:55 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:18:56.182Z] pod name:mysql-7d766cd65c-d9llq namespace:tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s running:true ready:false node:10.240.64.27 [2024-01-18T18:18:56.182Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:19:06.133Z] 2024/01/18 18:19:05 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:19:06.133Z] pod name:mysql-7d766cd65c-d9llq namespace:tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s running:true ready:false node:10.240.64.27 [2024-01-18T18:19:06.133Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:19:16.081Z] 2024/01/18 18:19:15 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:19:16.081Z] pod name:mysql-7d766cd65c-d9llq namespace:tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s running:true ready:false node:10.240.64.27 [2024-01-18T18:19:16.081Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:19:26.040Z] 2024/01/18 18:19:25 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:19:26.040Z] pod name:mysql-7d766cd65c-d9llq namespace:tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s running:true ready:false node:10.240.64.27 [2024-01-18T18:19:26.040Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:19:38.229Z] 2024/01/18 18:19:36 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:19:38.229Z] pod name:mysql-7d766cd65c-d9llq namespace:tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s running:true ready:false node:10.240.64.27 [2024-01-18T18:19:38.229Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:19:47.448Z] 2024-01-18 18:19:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T18:19:47.448Z] 2024-01-18 18:19:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T18:19:47.448Z] 2024-01-18 18:19:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating specs for the namespace (restoredAppContext) [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] in restore [tp-backup-1705598479-1705601844] [2024-01-18T18:19:47.449Z] 2024-01-18 18:19:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s]) was found in restore [tp-backup-1705598479-1705601844], as expected by presence in expectedRestoredAppContext [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T18:19:47.449Z] 2024-01-18 18:19:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s]) found in the restore [tp-backup-1705598479-1705601844] was also present on the cluster/namespace [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T18:19:47.449Z] 2024-01-18 18:19:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s]) was found in restore [tp-backup-1705598479-1705601844], as expected by presence in expectedRestoredAppContext [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T18:19:47.449Z] 2024-01-18 18:19:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s]) found in the restore [tp-backup-1705598479-1705601844] was also present on the cluster/namespace [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T18:19:47.449Z] 2024-01-18 18:19:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s]) was found in restore [tp-backup-1705598479-1705601844], as expected by presence in expectedRestoredAppContext [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T18:19:47.449Z] 2024-01-18 18:19:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s]) found in the restore [tp-backup-1705598479-1705601844] was also present on the cluster/namespace [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T18:19:47.449Z] 2024-01-18 18:19:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s]) was found in restore [tp-backup-1705598479-1705601844], as expected by presence in expectedRestoredAppContext [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T18:19:47.449Z] 2024-01-18 18:19:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s]) found in the restore [tp-backup-1705598479-1705601844] was also present on the cluster/namespace [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T18:19:47.449Z] 2024-01-18 18:19:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating Restored Volumes for the namespace (restoredAppContext) [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] in restore [tp-backup-1705598479-1705601844] [2024-01-18T18:19:47.449Z] 2024-01-18 18:19:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] in restore [tp-backup-1705598479-1705601844] [2024-01-18T18:19:47.449Z] 2024-01-18 18:19:46 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T18:19:47.449Z] 2024-01-18 18:19:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T18:19:47.450Z] 2024-01-18 18:19:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] are [[mysql-data]] [2024-01-18T18:19:47.450Z] 2024-01-18 18:19:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2243] - in restore [tp-backup-1705598479-1705601844], the status of the restored volume [pvc-36a201fc-9ab6-49ca-9d5c-0b316e6c925e] was not Success. It was [Retained] with reason [Skipped from volume restore as policy is set to Retain and pvc already exists] [2024-01-18T18:19:47.450Z] 2024-01-18 18:19:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validate applications in restored namespace [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] due to restore [tp-backup-1705598479-1705601844] [2024-01-18T18:19:47.450Z] 2024-01-18 18:19:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating mysql-ibm app [2024-01-18T18:19:47.450Z] 2024-01-18 18:19:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] wait for mysql-ibm app to start running [2024-01-18T18:19:47.450Z] 2024-01-18 18:19:46 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T18:19:47.479Z] 2024-01-18 18:19:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T18:19:48.042Z] 2024-01-18 18:19:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T18:19:48.969Z] 2024-01-18 18:19:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating specs for the namespace (restoredAppContext) [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] in restore [tp-backup-1705598479-1705601844] [2024-01-18T18:19:48.969Z] 2024-01-18 18:19:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s]) was found in restore [tp-backup-1705598479-1705601844], as expected by presence in expectedRestoredAppContext [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T18:19:48.969Z] 2024-01-18 18:19:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s]) found in the restore [tp-backup-1705598479-1705601844] was also present on the cluster/namespace [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T18:19:48.969Z] 2024-01-18 18:19:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s]) was found in restore [tp-backup-1705598479-1705601844], as expected by presence in expectedRestoredAppContext [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T18:19:48.969Z] 2024-01-18 18:19:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s]) found in the restore [tp-backup-1705598479-1705601844] was also present on the cluster/namespace [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T18:19:48.969Z] 2024-01-18 18:19:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s]) was found in restore [tp-backup-1705598479-1705601844], as expected by presence in expectedRestoredAppContext [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T18:19:48.969Z] 2024-01-18 18:19:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s]) found in the restore [tp-backup-1705598479-1705601844] was also present on the cluster/namespace [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T18:19:48.970Z] 2024-01-18 18:19:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s]) was found in restore [tp-backup-1705598479-1705601844], as expected by presence in expectedRestoredAppContext [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T18:19:48.970Z] 2024-01-18 18:19:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s]) found in the restore [tp-backup-1705598479-1705601844] was also present on the cluster/namespace [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T18:19:48.970Z] 2024-01-18 18:19:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating Restored Volumes for the namespace (restoredAppContext) [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] in restore [tp-backup-1705598479-1705601844] [2024-01-18T18:19:48.970Z] 2024-01-18 18:19:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] in restore [tp-backup-1705598479-1705601844] [2024-01-18T18:19:48.970Z] 2024-01-18 18:19:48 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T18:19:48.970Z] 2024-01-18 18:19:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T18:19:48.970Z] 2024-01-18 18:19:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] are [[mysql-data]] [2024-01-18T18:19:48.970Z] 2024-01-18 18:19:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.ValidateRestore:#2243] - in restore [tp-backup-1705598479-1705601844], the status of the restored volume [pvc-de275780-a0db-4c53-a8ee-2c26f42e1539] was not Success. It was [Retained] with reason [Skipped from volume restore as policy is set to Retain and pvc already exists] [2024-01-18T18:19:48.970Z] 2024-01-18 18:19:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validate applications in restored namespace [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] due to restore [tp-backup-1705598479-1705601844] [2024-01-18T18:19:48.970Z] 2024-01-18 18:19:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating mysql-ibm app [2024-01-18T18:19:48.970Z] 2024-01-18 18:19:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] wait for mysql-ibm app to start running [2024-01-18T18:19:49.227Z] 2024-01-18 18:19:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T18:19:49.790Z] 2024-01-18 18:19:49 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T18:19:50.350Z] 2024-01-18 18:19:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T18:19:51.277Z] 2024-01-18 18:19:51 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Validating CR cleanup [2024-01-18T18:19:51.277Z] 2024-01-18 18:19:51 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T18:19:51.536Z] 2024-01-18 18:19:51 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] All CRs in [kube-system] are [[]] [2024-01-18T18:19:51.536Z] 2024-01-18 18:19:51 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switching cluster context back to cluster path [/tmp/source-config] [2024-01-18T18:19:51.536Z] 2024-01-18 18:19:51 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switching context: The context is already [/tmp/source-config] [2024-01-18T18:19:51.536Z] INFO[2024-01-18 18:19:51] Verifying : Description : Verification of restoring scheduled backups for all application [tp-backup-1705598479-1705601844] in custom namespace [map[mysql-ibm-pxb-0-84853-01-18-17h11m43s:tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s mysql-ibm-pxb-1-84853-01-18-17h11m43s:tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s:tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s]] [2024-01-18T18:19:51.536Z] INFO[2024-01-18 18:19:51] Actual:true, Expected: true [2024-01-18T18:19:51.536Z] 2024-01-18 18:19:51 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Deleting test data generated during test execution [2024-01-18T18:19:51.799Z] 2024/01/18 18:19:51 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598306] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:20:01.745Z] 2024/01/18 18:20:01 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598306] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:20:11.685Z] 2024/01/18 18:20:11 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598306] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:20:23.875Z] 2024/01/18 18:20:21 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598306] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:20:31.958Z] 2024/01/18 18:20:31 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598306] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:20:41.900Z] 2024/01/18 18:20:41 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598306] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:20:51.863Z] 2024/01/18 18:20:51 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598306] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:21:01.798Z] 2024/01/18 18:21:01 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598306] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:21:13.977Z] 2024/01/18 18:21:11 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598306] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:21:22.065Z] 2024/01/18 18:21:21 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598306] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:21:32.009Z] 2024/01/18 18:21:31 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598306] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:21:41.959Z] INFO[2024-01-18 18:21:41] Verifying : Description : Verification of deleting backup schedule - tp-backup-1705598306 [2024-01-18T18:21:41.959Z] INFO[2024-01-18 18:21:41] Actual:true, Expected: true [2024-01-18T18:21:41.959Z] 2024/01/18 18:21:41 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598367] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:21:54.126Z] 2024/01/18 18:21:51 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598367] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:22:02.208Z] 2024/01/18 18:22:01 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598367] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:22:12.199Z] 2024/01/18 18:22:11 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598367] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:22:22.176Z] 2024/01/18 18:22:21 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598367] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:22:32.118Z] 2024/01/18 18:22:31 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598367] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:22:42.057Z] 2024/01/18 18:22:41 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598367] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:22:54.223Z] 2024/01/18 18:22:52 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598367] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:23:02.309Z] 2024/01/18 18:23:02 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598367] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:23:12.249Z] 2024/01/18 18:23:12 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598367] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:23:23.792Z] 2024/01/18 18:23:22 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598367] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:23:33.736Z] 2024/01/18 18:23:32 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598367] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:23:43.675Z] INFO[2024-01-18 18:23:42] Verifying : Description : Verification of deleting backup schedule - tp-backup-1705598367 [2024-01-18T18:23:43.675Z] INFO[2024-01-18 18:23:42] Actual:true, Expected: true [2024-01-18T18:23:43.675Z] 2024/01/18 18:23:42 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598479] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:23:53.699Z] 2024/01/18 18:23:52 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598479] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:24:03.640Z] 2024/01/18 18:24:02 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598479] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:24:13.586Z] 2024/01/18 18:24:12 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598479] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:24:13.586Z] INFO[2024-01-18 18:24:12] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T18:24:13.586Z] INFO[2024-01-18 18:24:12] sched stats: tick interval: 10212 < 1.2s (100%) [2024-01-18T18:24:13.586Z] INFO[2024-01-18 18:24:12] sched stats: tick process durations: 10211 < 500ms (100%) [2024-01-18T18:24:23.532Z] 2024/01/18 18:24:22 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598479] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:24:33.478Z] 2024/01/18 18:24:32 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598479] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:24:43.424Z] 2024/01/18 18:24:42 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598479] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:24:53.362Z] 2024/01/18 18:24:52 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598479] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:25:03.303Z] 2024/01/18 18:25:02 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598479] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:25:13.244Z] 2024/01/18 18:25:12 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598479] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:25:23.191Z] 2024/01/18 18:25:22 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598479] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:25:33.135Z] 2024/01/18 18:25:32 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-1705598479] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:25:43.095Z] INFO[2024-01-18 18:25:42] Verifying : Description : Verification of deleting backup schedule - tp-backup-1705598479 [2024-01-18T18:25:43.095Z] INFO[2024-01-18 18:25:42] Actual:true, Expected: true [2024-01-18T18:25:43.095Z] INFO[2024-01-18 18:25:42] Verifying : Description : Deleting backup schedule policies [periodic-1705598304] [2024-01-18T18:25:43.095Z] INFO[2024-01-18 18:25:42] Actual:true, Expected: true [2024-01-18T18:25:43.095Z] INFO[2024-01-18 18:25:42] Verifying : Description : Getting the backup driver [2024-01-18T18:25:43.095Z] INFO[2024-01-18 18:25:42] Actual:true, Expected: true [2024-01-18T18:25:43.095Z] INFO[2024-01-18 18:25:42] Verifying : Description : Verifying the deletion of the restore named [single-namespace-backup-1705598025-1705598571] [2024-01-18T18:25:43.095Z] INFO[2024-01-18 18:25:42] Actual:true, Expected: true [2024-01-18T18:25:43.095Z] INFO[2024-01-18 18:25:42] Verifying : Description : Getting the backup driver [2024-01-18T18:25:43.095Z] INFO[2024-01-18 18:25:42] Actual:true, Expected: true [2024-01-18T18:25:43.095Z] INFO[2024-01-18 18:25:42] Verifying : Description : Verifying the deletion of the restore named [multiple-namespace-backup-1705598116-1705598699] [2024-01-18T18:25:43.095Z] INFO[2024-01-18 18:25:42] Actual:true, Expected: true [2024-01-18T18:25:43.095Z] INFO[2024-01-18 18:25:42] Verifying : Description : Getting the backup driver [2024-01-18T18:25:43.095Z] INFO[2024-01-18 18:25:42] Actual:true, Expected: true [2024-01-18T18:25:43.095Z] INFO[2024-01-18 18:25:42] Verifying : Description : Verifying the deletion of the restore named [all-namespace-backup-1705598207-1705598829] [2024-01-18T18:25:43.095Z] INFO[2024-01-18 18:25:42] Actual:true, Expected: true [2024-01-18T18:25:43.095Z] INFO[2024-01-18 18:25:42] Verifying : Description : Getting the backup driver [2024-01-18T18:25:43.095Z] INFO[2024-01-18 18:25:42] Actual:true, Expected: true [2024-01-18T18:25:43.095Z] INFO[2024-01-18 18:25:43] Verifying : Description : Verifying the deletion of the restore named [tp-backup-1705598306-1705599772] [2024-01-18T18:25:43.095Z] INFO[2024-01-18 18:25:43] Actual:true, Expected: true [2024-01-18T18:25:43.095Z] INFO[2024-01-18 18:25:43] Verifying : Description : Getting the backup driver [2024-01-18T18:25:43.095Z] INFO[2024-01-18 18:25:43] Actual:true, Expected: true [2024-01-18T18:25:43.351Z] INFO[2024-01-18 18:25:43] Verifying : Description : Verifying the deletion of the restore named [tp-backup-1705598367-1705600810] [2024-01-18T18:25:43.351Z] INFO[2024-01-18 18:25:43] Actual:true, Expected: true [2024-01-18T18:25:43.351Z] INFO[2024-01-18 18:25:43] Verifying : Description : Getting the backup driver [2024-01-18T18:25:43.351Z] INFO[2024-01-18 18:25:43] Actual:true, Expected: true [2024-01-18T18:25:43.351Z] INFO[2024-01-18 18:25:43] Verifying : Description : Verifying the deletion of the restore named [tp-backup-1705598479-1705601844] [2024-01-18T18:25:43.351Z] INFO[2024-01-18 18:25:43] Actual:true, Expected: true [2024-01-18T18:25:43.351Z] 2024-01-18 18:25:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Deleting labels from namespaces- [mysql-ibm-pxb-0-84853-01-18-17h11m43s mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T18:25:43.351Z] INFO[2024-01-18 18:25:43] Verifying : Description : Deleting labels [map[jcscxcmokfqjigdhdauujdccwnvaobakkvwqlltpoeesjygjfuoqvhovwgswoor:68f8b453-b3cf-4ffa-a7ad-1537518cf8ff kcvuvkfgekhphvfawturizbjgpfprhugkdjnfcqmvmiltrwcppbbuevnzswppph:ad9b899a-faee-4ea4-97f3-d0c839228f51 tyffzmgvkruasitedxdahjkjaihjewomwkwwxqkbhdbmghhnduauyfjbmlgvsbu:05b20d94-ef28-480f-be7d-3465fb05d059]] from namespaces [[mysql-ibm-pxb-0-84853-01-18-17h11m43s mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s]] [2024-01-18T18:25:43.351Z] INFO[2024-01-18 18:25:43] Actual:true, Expected: true [2024-01-18T18:25:43.351Z] 2024-01-18 18:25:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Deleting deployed namespaces - [mysql-ibm-pxb-0-84853-01-18-17h11m43s mysql-ibm-pxb-1-84853-01-18-17h11m43s mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T18:25:43.351Z] 2024-01-18 18:25:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Destroying apps [2024-01-18T18:25:43.351Z] 2024-01-18 18:25:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T18:25:43.609Z] 2024-01-18 18:25:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T18:25:43.609Z] 2024-01-18 18:25:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] start destroying mysql-ibm app [2024-01-18T18:25:43.865Z] 2024-01-18 18:25:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T18:25:43.865Z] 2024-01-18 18:25:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T18:25:43.865Z] 2024-01-18 18:25:43 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T18:25:43.865Z] INFO[2024-01-18 18:25:43] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T18:25:43.865Z] INFO[2024-01-18 18:25:43] Actual:true, Expected: true [2024-01-18T18:25:43.865Z] 2024-01-18 18:25:43 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T18:25:44.121Z] 2024-01-18 18:25:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T18:25:44.121Z] 2024-01-18 18:25:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] start destroying mysql-ibm app [2024-01-18T18:25:44.378Z] 2024-01-18 18:25:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T18:25:44.378Z] 2024-01-18 18:25:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T18:25:44.378Z] 2024-01-18 18:25:44 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T18:25:44.378Z] INFO[2024-01-18 18:25:44] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T18:25:44.378Z] INFO[2024-01-18 18:25:44] Actual:true, Expected: true [2024-01-18T18:25:44.378Z] 2024-01-18 18:25:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T18:25:44.635Z] 2024-01-18 18:25:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T18:25:44.635Z] 2024-01-18 18:25:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] start destroying mysql-ibm app [2024-01-18T18:25:44.892Z] 2024-01-18 18:25:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T18:25:45.148Z] 2024-01-18 18:25:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T18:25:45.148Z] 2024-01-18 18:25:44 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T18:25:45.148Z] INFO[2024-01-18 18:25:44] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T18:25:45.148Z] INFO[2024-01-18 18:25:44] Actual:true, Expected: true [2024-01-18T18:25:45.148Z] 2024-01-18 18:25:44 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Cleaning backup locations in map [map[653ae50e-f82a-4bd1-a867-9cd45d18bd8a:autogenerated-backup-location-1705598018]], cloud credential [autogenerated-cred-1705598018], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T18:25:45.148Z] INFO[2024-01-18 18:25:44] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-1705598018] [2024-01-18T18:25:45.148Z] INFO[2024-01-18 18:25:44] Actual:true, Expected: true [2024-01-18T18:25:45.148Z] 2024-01-18 18:25:45 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705598018] is present [2024-01-18T18:25:45.148Z] 2024/01/18 18:25:45 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705598018 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T18:26:17.228Z] 2024-01-18 18:26:15 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705598018] is present [2024-01-18T18:26:17.228Z] 2024/01/18 18:26:15 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705598018 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T18:26:49.253Z] 2024-01-18 18:26:45 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705598018] is present [2024-01-18T18:26:49.253Z] 2024/01/18 18:26:45 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705598018 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T18:27:15.767Z] 2024-01-18 18:27:15 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705598018] is present [2024-01-18T18:27:15.768Z] 2024/01/18 18:27:15 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705598018 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T18:27:49.380Z] 2024-01-18 18:27:45 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu] [2024-01-18T18:27:49.380Z] INFO[2024-01-18 18:27:45] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-1705598018 [2024-01-18T18:27:49.380Z] INFO[2024-01-18 18:27:45] Actual:true, Expected: true [2024-01-18T18:27:49.380Z] 2024-01-18 18:27:45 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-1705598018] is present [2024-01-18T18:27:49.380Z] INFO[2024-01-18 18:27:45] Verifying : Description : Verifying if cloud cred [autogenerated-cred-1705598018] is present [2024-01-18T18:27:49.380Z] INFO[2024-01-18 18:27:45] Actual:true, Expected: true [2024-01-18T18:27:49.380Z] INFO[2024-01-18 18:27:45] Verifying : Description : Verifying deletion of cloud cred [autogenerated-cred-1705598018] [2024-01-18T18:27:49.380Z] INFO[2024-01-18 18:27:45] Actual:true, Expected: true [2024-01-18T18:27:49.380Z] INFO[2024-01-18 18:27:45] Verifying : Description : Deleting cloud cred autogenerated-cred-1705598018 [2024-01-18T18:27:49.380Z] INFO[2024-01-18 18:27:45] Actual:true, Expected: true [2024-01-18T18:27:49.380Z] INFO[2024-01-18 18:27:45] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T18:27:49.381Z] INFO[2024-01-18 18:27:45] Actual:true, Expected: true [2024-01-18T18:27:49.381Z] INFO[2024-01-18 18:27:45] Verifying : Description : Deleting cluster source-cluster [2024-01-18T18:27:49.381Z] INFO[2024-01-18 18:27:45] Actual:true, Expected: true [2024-01-18T18:27:49.381Z] INFO[2024-01-18 18:27:46] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-eopef] [2024-01-18T18:27:49.381Z] INFO[2024-01-18 18:27:46] Actual:true, Expected: true [2024-01-18T18:27:49.381Z] INFO[2024-01-18 18:27:46] Verifying : Description : Deleting cluster destination-cluster [2024-01-18T18:27:49.381Z] INFO[2024-01-18 18:27:46] Actual:true, Expected: true [2024-01-18T18:27:49.381Z] INFO[2024-01-18 18:27:46] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-vmbmv] [2024-01-18T18:27:49.381Z] INFO[2024-01-18 18:27:46] Actual:true, Expected: true [2024-01-18T18:27:49.381Z] INFO[2024-01-18 18:27:46] --------Test End------ [2024-01-18T18:27:49.381Z] INFO[2024-01-18 18:27:46] #Test: [2024-01-18T18:27:49.381Z] INFO[2024-01-18 18:27:46] #Description: [2024-01-18T18:27:49.381Z] INFO[2024-01-18 18:27:46] ------------------------ [2024-01-18T18:27:49.381Z] 2024-01-18 18:27:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T18:27:49.381Z] 2024-01-18 18:27:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T18:27:49.381Z] 2024-01-18 18:27:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T18:27:49.381Z] 2024-01-18 18:27:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [tp-restore-mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T18:27:49.381Z] 2024-01-18 18:27:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [tp-restore-mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T18:27:49.381Z] 2024-01-18 18:27:47 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [tp-restore-mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T18:27:49.381Z] 2024-01-18 18:27:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T18:27:49.381Z] 2024-01-18 18:27:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switching context to [/tmp/destination-config] [2024-01-18T18:27:49.381Z] 2024-01-18 18:27:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:27:49.381Z] 2024-01-18 18:27:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:27:49.381Z] 2024-01-18 18:27:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:27:49.381Z] 2024-01-18 18:27:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:27:49.381Z] 2024-01-18 18:27:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:27:49.382Z] 2024-01-18 18:27:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:27:49.382Z] 2024-01-18 18:27:48 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:27:49.382Z] 2024-01-18 18:27:49 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:27:49.943Z] 2024-01-18 18:27:49 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:27:50.202Z] 2024-01-18 18:27:49 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T18:27:50.202Z] 2024-01-18 18:27:50 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T18:27:50.202Z] 2024-01-18 18:27:50 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T18:27:50.202Z] 2024-01-18 18:27:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T18:27:50.202Z] 2024-01-18 18:27:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switched context to [/tmp/destination-config] [2024-01-18T18:27:50.461Z] 2024-01-18 18:27:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-84853-01-18-17h11m43s] [2024-01-18T18:27:50.717Z] 2024-01-18 18:27:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-84853-01-18-17h11m43s-1705599772] [2024-01-18T18:27:50.973Z] 2024-01-18 18:27:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-1-84853-01-18-17h11m43s] [2024-01-18T18:27:51.229Z] 2024-01-18 18:27:50 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-2-84853-01-18-17h11m43s] [2024-01-18T18:27:51.229Z] 2024-01-18 18:27:51 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit} Manual, schedule backup and restore of single, multiple and all namespaces with namespace labeled as max character limit [2024-01-18T18:27:51.229Z] 2024-01-18 18:27:51 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-18T18:27:51.229Z] 2024/01/18 18:27:51 DoRetryWithTimeout - Error: {dial tcp 10.240.128.5:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T18:27:56.474Z] 2024/01/18 18:27:56 DoRetryWithTimeout - Error: {dial tcp 10.240.128.5:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T18:28:01.718Z] 2024/01/18 18:28:01 DoRetryWithTimeout - Error: {dial tcp 10.240.128.5:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T18:28:06.967Z] 2024/01/18 18:28:06 DoRetryWithTimeout - Error: {dial tcp 10.240.128.5:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T18:28:12.222Z] 2024/01/18 18:28:11 DoRetryWithTimeout - Error: {dial tcp 10.240.128.5:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T18:28:16.448Z] 2024/01/18 18:28:16 DoRetryWithTimeout - Error: {dial tcp 10.240.128.5:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T18:28:21.695Z] 2024/01/18 18:28:21 DoRetryWithTimeout - Error: {dial tcp 10.240.128.5:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T18:28:26.940Z] 2024/01/18 18:28:26 DoRetryWithTimeout - Error: {dial tcp 10.240.128.5:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T18:28:32.188Z] 2024/01/18 18:28:31 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.128.5. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.5 10.240.128.5]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.5:22: connect: connection refused} [2024-01-18T18:28:32.188Z] {dial tcp 10.240.128.5:22: connect: connection refused} [2024-01-18T18:28:32.188Z] {dial tcp 10.240.128.5:22: connect: connection refused} [2024-01-18T18:28:32.188Z] {dial tcp 10.240.128.5:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-18T18:28:36.357Z] 2024-01-18 18:28:36 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.5. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.5 10.240.128.5]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.5:22: connect: connection refused} [2024-01-18T18:28:36.357Z] {dial tcp 10.240.128.5:22: connect: connection refused} [2024-01-18T18:28:36.357Z] {dial tcp 10.240.128.5:22: connect: connection refused} [2024-01-18T18:28:36.357Z] {dial tcp 10.240.128.5:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T18:28:36.357Z] 2024-01-18 18:28:36 +0000:[ERROR] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.5. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.5 10.240.128.5]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.5:22: connect: connection refused} [2024-01-18T18:28:36.357Z] {dial tcp 10.240.128.5:22: connect: connection refused} [2024-01-18T18:28:36.357Z] {dial tcp 10.240.128.5:22: connect: connection refused} [2024-01-18T18:28:36.357Z] {dial tcp 10.240.128.5:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T18:28:36.357Z] 2024-01-18 18:28:36 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T18:28:36.357Z] 2024-01-18 18:28:36 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switching context to [/tmp/source-config] [2024-01-18T18:28:36.357Z] 2024-01-18 18:28:36 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:28:36.357Z] 2024-01-18 18:28:36 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:28:36.357Z] 2024-01-18 18:28:36 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:28:36.613Z] 2024-01-18 18:28:36 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:28:36.613Z] 2024-01-18 18:28:36 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:28:36.868Z] 2024-01-18 18:28:36 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:28:37.124Z] 2024-01-18 18:28:36 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:28:37.686Z] 2024-01-18 18:28:37 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:28:37.942Z] 2024-01-18 18:28:37 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:28:38.202Z] 2024-01-18 18:28:37 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T18:28:38.459Z] 2024-01-18 18:28:38 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T18:28:38.459Z] 2024-01-18 18:28:38 +0000:[WARNING] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T18:28:38.459Z] 2024-01-18 18:28:38 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T18:28:38.459Z] 2024-01-18 18:28:38 +0000:[INFO] [{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}] Switched context to [/tmp/source-config] [2024-01-18T18:28:38.459Z] • [SLOW TEST:4614.607 seconds] [2024-01-18T18:28:38.459Z] {ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit} [2024-01-18T18:28:38.459Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_namespace_labelled_test.go:858 [2024-01-18T18:28:38.459Z] Manual, schedule backup and restore of single, multiple and all namespaces with namespace labeled as max character limit [2024-01-18T18:28:38.459Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_namespace_labelled_test.go:917 [2024-01-18T18:28:38.459Z] ------------------------------ [2024-01-18T18:28:38.459Z] 2024-01-18 18:28:38 +0000:[INFO] [{DifferentAccessSameUser}] Log Dir: /testresults//DifferentAccessSameUser.log [2024-01-18T18:28:38.459Z] INFO[2024-01-18 18:28:38] --------Test Start------ [2024-01-18T18:28:38.459Z] INFO[2024-01-18 18:28:38] #Test: DifferentAccessSameUser [2024-01-18T18:28:38.459Z] INFO[2024-01-18 18:28:38] #Description: Take a backup and add user with readonly access and the group with full access [2024-01-18T18:28:38.460Z] INFO[2024-01-18 18:28:38] ------------------------ [2024-01-18T18:28:38.460Z] 2024-01-18 18:28:38 +0000:[INFO] [{DifferentAccessSameUser}] Deploy applications [2024-01-18T18:28:38.460Z] 2024-01-18 18:28:38 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T18:28:38.460Z] 2024-01-18 18:28:38 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T18:28:38.460Z] 2024-01-18 18:28:38 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T18:28:38.718Z] 2024-01-18 18:28:38 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T18:28:38.718Z] 2024-01-18 18:28:38 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T18:28:38.974Z] 2024-01-18 18:28:38 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T18:28:38.975Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T18:28:38.975Z] 2024-01-18 18:28:38 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T18:28:38.975Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T18:28:38.975Z] 2024-01-18 18:28:38 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T18:28:40.340Z] 2024-01-18 18:28:40 +0000:[INFO] [{DifferentAccessSameUser}] Validate applications [2024-01-18T18:28:40.340Z] 2024-01-18 18:28:40 +0000:[INFO] [{DifferentAccessSameUser}] Validate applications [2024-01-18T18:28:40.340Z] 2024-01-18 18:28:40 +0000:[INFO] [{DifferentAccessSameUser}] Validating mysql-ibm app [2024-01-18T18:28:40.340Z] 2024-01-18 18:28:40 +0000:[INFO] [{DifferentAccessSameUser}] Validating mysql-ibm app's volumes [2024-01-18T18:28:40.596Z] 2024-01-18 18:28:40 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T18:28:40.851Z] 2024/01/18 18:28:40 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T18:28:50.793Z] 2024-01-18 18:28:50 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-82938-01-18-18h28m38s [2024-01-18T18:28:50.793Z] 2024-01-18 18:28:50 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T18:28:50.793Z] 2024-01-18 18:28:50 +0000:[WARNING] [{DifferentAccessSameUser}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T18:28:50.793Z] 2024-01-18 18:28:50 +0000:[INFO] [{DifferentAccessSameUser}] wait for mysql-ibm app to start running [2024-01-18T18:28:50.793Z] 2024-01-18 18:28:50 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T18:28:51.049Z] 2024/01/18 18:28:50 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:28:51.049Z] pod name:mysql-7d766cd65c-h42xt namespace:mysql-ibm-pxb-0-82938-01-18-18h28m38s running:true ready:false node: [2024-01-18T18:28:51.049Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:29:03.219Z] 2024/01/18 18:29:00 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:29:03.219Z] pod name:mysql-7d766cd65c-h42xt namespace:mysql-ibm-pxb-0-82938-01-18-18h28m38s running:false ready:false node:10.240.64.9 [2024-01-18T18:29:03.219Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:29:11.307Z] 2024/01/18 18:29:11 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:29:11.307Z] pod name:mysql-7d766cd65c-h42xt namespace:mysql-ibm-pxb-0-82938-01-18-18h28m38s running:true ready:false node:10.240.64.9 [2024-01-18T18:29:11.307Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:29:21.242Z] 2024/01/18 18:29:21 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:29:21.242Z] pod name:mysql-7d766cd65c-h42xt namespace:mysql-ibm-pxb-0-82938-01-18-18h28m38s running:true ready:false node:10.240.64.9 [2024-01-18T18:29:21.243Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:29:33.409Z] 2024/01/18 18:29:31 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:29:33.409Z] pod name:mysql-7d766cd65c-h42xt namespace:mysql-ibm-pxb-0-82938-01-18-18h28m38s running:true ready:false node:10.240.64.9 [2024-01-18T18:29:33.409Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:29:41.492Z] 2024/01/18 18:29:41 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:29:41.492Z] pod name:mysql-7d766cd65c-h42xt namespace:mysql-ibm-pxb-0-82938-01-18-18h28m38s running:true ready:false node:10.240.64.9 [2024-01-18T18:29:41.492Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:29:53.676Z] 2024/01/18 18:29:51 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:29:53.676Z] pod name:mysql-7d766cd65c-h42xt namespace:mysql-ibm-pxb-0-82938-01-18-18h28m38s running:true ready:false node:10.240.64.9 [2024-01-18T18:29:53.676Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:30:01.760Z] 2024/01/18 18:30:01 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:30:01.760Z] pod name:mysql-7d766cd65c-h42xt namespace:mysql-ibm-pxb-0-82938-01-18-18h28m38s running:true ready:false node:10.240.64.9 [2024-01-18T18:30:01.760Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:30:11.706Z] 2024-01-18 18:30:11 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T18:30:11.962Z] 2024-01-18 18:30:11 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T18:30:11.962Z] 2024-01-18 18:30:11 +0000:[INFO] [{DifferentAccessSameUser}] validate if mysql-ibm app's volumes are setup [2024-01-18T18:30:11.962Z] 2024-01-18 18:30:11 +0000:[WARNING] [{DifferentAccessSameUser}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T18:30:11.962Z] 2024-01-18 18:30:11 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T18:30:11.962Z] 2024-01-18 18:30:11 +0000:[INFO] [{DifferentAccessSameUser}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T18:30:11.962Z] 2024-01-18 18:30:11 +0000:[WARNING] [{DifferentAccessSameUser}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T18:30:12.220Z] 2024-01-18 18:30:11 +0000:[INFO] [{DifferentAccessSameUser}] Creating users testuser [2024-01-18T18:30:12.220Z] 2024-01-18 18:30:11 +0000:[INFO] [{DifferentAccessSameUser}] Creating 1 users [2024-01-18T18:30:12.477Z] 2024-01-18 18:30:12 +0000:[INFO] [{DifferentAccessSameUser}] [backup.AddUser:#728] - Response for user [testuser1-1705602611] creation - [2024-01-18T18:30:12.477Z] 2024-01-18 18:30:12 +0000:[INFO] [{DifferentAccessSameUser}] [backup.AddUser:#733] - User [testuser1-1705602611] created [2024-01-18T18:30:12.477Z] INFO[2024-01-18 18:30:12] Verifying : Description : Creating user - testuser1-1705602611 [2024-01-18T18:30:12.477Z] INFO[2024-01-18 18:30:12] Actual:true, Expected: true [2024-01-18T18:30:12.477Z] 2024-01-18 18:30:12 +0000:[INFO] [{DifferentAccessSameUser}] [backup.glob..func82.2.2:#328] - Created 1 users and users list is [testuser1-1705602611] [2024-01-18T18:30:12.477Z] 2024-01-18 18:30:12 +0000:[INFO] [{DifferentAccessSameUser}] Creating group testGroup [2024-01-18T18:30:12.477Z] 2024-01-18 18:30:12 +0000:[INFO] [{DifferentAccessSameUser}] [backup.AddGroup:#989] - Creating group response - [2024-01-18T18:30:12.477Z] 2024-01-18 18:30:12 +0000:[INFO] [{DifferentAccessSameUser}] [backup.AddGroup:#994] - Group [testGroup] created [2024-01-18T18:30:12.477Z] 2024-01-18 18:30:12 +0000:[INFO] [{DifferentAccessSameUser}] Adding user to groups [2024-01-18T18:30:13.041Z] 2024-01-18 18:30:12 +0000:[INFO] [{DifferentAccessSameUser}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup - 8496e08b-68c6-47e3-8afd-253bc717d817 [2024-01-18T18:30:13.967Z] 2024-01-18 18:30:13 +0000:[INFO] [{DifferentAccessSameUser}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705602611 - 39e06e70-2f13-41dd-8777-7ef786f7532a [2024-01-18T18:30:14.937Z] 2024-01-18 18:30:14 +0000:[INFO] [{DifferentAccessSameUser}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-18T18:30:14.937Z] 2024-01-18 18:30:14 +0000:[INFO] [{DifferentAccessSameUser}] [backup.AddGroupToUser:#1119] - Added User [testuser1-1705602611] to Group [testGroup] [2024-01-18T18:30:14.937Z] INFO[2024-01-18 18:30:14] Verifying : Description : Adding user testuser1-1705602611 to group testGroup [2024-01-18T18:30:14.937Z] INFO[2024-01-18 18:30:14] Actual:true, Expected: true [2024-01-18T18:30:15.864Z] 2024-01-18 18:30:15 +0000:[INFO] [{DifferentAccessSameUser}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup - 8496e08b-68c6-47e3-8afd-253bc717d817 [2024-01-18T18:30:16.426Z] 2024-01-18 18:30:16 +0000:[DEBUG] [{DifferentAccessSameUser}] [backup.GetMembersOfGroup:#961] - list of members : [testuser1-1705602611] [2024-01-18T18:30:16.426Z] 2024-01-18 18:30:16 +0000:[INFO] [{DifferentAccessSameUser}] [backup.glob..func82.2.4:#343] - Group [testGroup] contains the following users: [2024-01-18T18:30:16.426Z] [testuser1-1705602611] [2024-01-18T18:30:16.426Z] 2024-01-18 18:30:16 +0000:[INFO] [{DifferentAccessSameUser}] Creating backup location and cloud setting [2024-01-18T18:30:16.426Z] 2024-01-18 18:30:16 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [cloudcred-aws-1705602616] for org [default] with [aws] as provider [2024-01-18T18:30:16.426Z] 2024-01-18 18:30:16 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-18T18:30:16.426Z] 2024-01-18 18:30:16 +0000:[INFO] [{DifferentAccessSameUser}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cloudcred-aws-1705602616] is present [2024-01-18T18:30:16.426Z] INFO[2024-01-18 18:30:16] Verifying : Description : Verifying creation of cloud credential named [cloudcred-aws-1705602616] for org [default] with [aws] as provider [2024-01-18T18:30:16.426Z] INFO[2024-01-18 18:30:16] Actual:true, Expected: true [2024-01-18T18:30:19.690Z] INFO[2024-01-18 18:30:19] Verifying : Description : Creating backup location aws-global-aws-px-backup-system-test-4cac76dc-1705602616-bl [2024-01-18T18:30:19.690Z] INFO[2024-01-18 18:30:19] Actual:true, Expected: true [2024-01-18T18:30:19.946Z] INFO[2024-01-18 18:30:19] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T18:30:19.946Z] INFO[2024-01-18 18:30:19] Actual:true, Expected: true [2024-01-18T18:30:19.946Z] 2024-01-18 18:30:19 +0000:[INFO] [{DifferentAccessSameUser}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T18:30:19.946Z] 2024-01-18 18:30:19 +0000:[INFO] [{DifferentAccessSameUser}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T18:30:19.946Z] 2024-01-18 18:30:19 +0000:[INFO] [{DifferentAccessSameUser}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T18:30:19.946Z] 2024-01-18 18:30:19 +0000:[INFO] [{DifferentAccessSameUser}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T18:30:19.946Z] 2024-01-18 18:30:19 +0000:[INFO] [{DifferentAccessSameUser}] Create cluster [source-cluster] in org [default] [2024-01-18T18:30:19.946Z] 2024-01-18 18:30:19 +0000:[INFO] [{DifferentAccessSameUser}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T18:30:19.946Z] 2024-01-18 18:30:19 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T18:30:19.946Z] 2024-01-18 18:30:19 +0000:[INFO] [{DifferentAccessSameUser}] Create cluster [destination-cluster] in org [default] [2024-01-18T18:30:19.946Z] 2024-01-18 18:30:19 +0000:[INFO] [{DifferentAccessSameUser}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T18:30:19.946Z] 2024-01-18 18:30:19 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T18:30:19.947Z] 2024-01-18 18:30:19 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-wmdak] for IBM [2024-01-18T18:30:19.947Z] 2024-01-18 18:30:19 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-wmdak] for org [default] with [ibm] as provider [2024-01-18T18:30:19.947Z] 2024-01-18 18:30:19 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T18:30:19.947Z] 2024-01-18 18:30:19 +0000:[INFO] [{DifferentAccessSameUser}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T18:30:19.947Z] 2024-01-18 18:30:19 +0000:[INFO] [{DifferentAccessSameUser}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-wmdak] is present [2024-01-18T18:30:19.947Z] 2024-01-18 18:30:19 +0000:[INFO] [{DifferentAccessSameUser}] Create cluster [source-cluster] in org [default] [2024-01-18T18:30:20.507Z] 2024-01-18 18:30:20 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-rsvcv] for IBM [2024-01-18T18:30:20.507Z] 2024-01-18 18:30:20 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-rsvcv] for org [default] with [ibm] as provider [2024-01-18T18:30:20.507Z] 2024-01-18 18:30:20 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T18:30:20.507Z] 2024-01-18 18:30:20 +0000:[INFO] [{DifferentAccessSameUser}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T18:30:20.507Z] 2024-01-18 18:30:20 +0000:[INFO] [{DifferentAccessSameUser}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-rsvcv] is present [2024-01-18T18:30:20.507Z] 2024-01-18 18:30:20 +0000:[INFO] [{DifferentAccessSameUser}] Create cluster [destination-cluster] in org [default] [2024-01-18T18:30:21.071Z] INFO[2024-01-18 18:30:21] Verifying : Description : Creating source and destination cluster [2024-01-18T18:30:21.071Z] INFO[2024-01-18 18:30:21] Actual:true, Expected: true [2024-01-18T18:30:21.329Z] INFO[2024-01-18 18:30:21] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-18T18:30:21.329Z] INFO[2024-01-18 18:30:21] Actual:Online, Expected: Online [2024-01-18T18:30:21.329Z] INFO[2024-01-18 18:30:21] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-18T18:30:21.329Z] INFO[2024-01-18 18:30:21] Actual:true, Expected: true [2024-01-18T18:30:21.329Z] 2024-01-18 18:30:21 +0000:[INFO] [{DifferentAccessSameUser}] Uid of [source-cluster] cluster is ab606618-ab9a-4007-b4fa-4c392a24a27a [2024-01-18T18:30:21.329Z] 2024-01-18 18:30:21 +0000:[INFO] [{DifferentAccessSameUser}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T18:30:21.329Z] 2024-01-18 18:30:21 +0000:[INFO] [{DifferentAccessSameUser}] Volume snapshot classes found in the cluster - [2024-01-18T18:30:21.329Z] 2024-01-18 18:30:21 +0000:[INFO] [{DifferentAccessSameUser}] ibmc-vpcblock-snapshot [2024-01-18T18:30:21.329Z] 2024-01-18 18:30:21 +0000:[INFO] [{DifferentAccessSameUser}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T18:30:21.890Z] 2024/01/18 18:30:21 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-82938-01-18-18h28m38s-1705602621] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T18:30:53.941Z] 2024/01/18 18:30:51 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-82938-01-18-18h28m38s-1705602621] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T18:31:25.968Z] 2024-01-18 18:31:21 +0000:[INFO] [{DifferentAccessSameUser}] [backup.CreateBackup:#313] - Backup [tp-backup-mysql-ibm-pxb-0-82938-01-18-18h28m38s-1705602621] created successfully [2024-01-18T18:31:25.968Z] 2024-01-18 18:31:21 +0000:[INFO] [{DifferentAccessSameUser}] Validating backup [tp-backup-mysql-ibm-pxb-0-82938-01-18-18h28m38s-1705602621] in org [default] [2024-01-18T18:31:25.968Z] 2024-01-18 18:31:21 +0000:[INFO] [{DifferentAccessSameUser}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-mysql-ibm-pxb-0-82938-01-18-18h28m38s-1705602621] [2024-01-18T18:31:25.968Z] 2024-01-18 18:31:21 +0000:[INFO] [{DifferentAccessSameUser}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82938-01-18-18h28m38s] in backup [tp-backup-mysql-ibm-pxb-0-82938-01-18-18h28m38s-1705602621] [2024-01-18T18:31:25.968Z] 2024-01-18 18:31:21 +0000:[INFO] [{DifferentAccessSameUser}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82938-01-18-18h28m38s] in backup [tp-backup-mysql-ibm-pxb-0-82938-01-18-18h28m38s-1705602621] [2024-01-18T18:31:25.968Z] 2024-01-18 18:31:21 +0000:[INFO] [{DifferentAccessSameUser}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82938-01-18-18h28m38s] [2024-01-18T18:31:25.968Z] 2024-01-18 18:31:21 +0000:[WARNING] [{DifferentAccessSameUser}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T18:31:25.968Z] 2024-01-18 18:31:21 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T18:31:25.968Z] 2024-01-18 18:31:21 +0000:[INFO] [{DifferentAccessSameUser}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82938-01-18-18h28m38s] are [[mysql-data]] [2024-01-18T18:31:25.968Z] 2024-01-18 18:31:21 +0000:[INFO] [{DifferentAccessSameUser}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T18:31:25.968Z] 2024-01-18 18:31:21 +0000:[INFO] [{DifferentAccessSameUser}] Validating CR cleanup [2024-01-18T18:31:25.968Z] 2024-01-18 18:31:21 +0000:[INFO] [{DifferentAccessSameUser}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T18:31:25.968Z] 2024-01-18 18:31:21 +0000:[INFO] [{DifferentAccessSameUser}] All CRs in [mysql-ibm-pxb-0-82938-01-18-18h28m38s] are [[tp-backup-mysql-ibm-pxb-0-82938-01-18-18h28m38s-1705602621-6e1bdb3]] [2024-01-18T18:31:25.968Z] 2024-01-18 18:31:21 +0000:[INFO] [{DifferentAccessSameUser}] CR found for [[tp-backup-mysql-ibm-pxb-0-82938-01-18-18h28m38s-1705602621-6e1bdb3]] under [mysql-ibm-pxb-0-82938-01-18-18h28m38s] namespace [2024-01-18T18:31:25.968Z] 2024/01/18 18:31:21 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-mysql-ibm-pxb-0-82938-01-18-18h28m38s-1705602621]}, Next try in [5s], timeout [5m0s] [2024-01-18T18:31:27.334Z] 2024-01-18 18:31:26 +0000:[INFO] [{DifferentAccessSameUser}] All CRs in [mysql-ibm-pxb-0-82938-01-18-18h28m38s] are [[]] [2024-01-18T18:31:27.334Z] INFO[2024-01-18 18:31:26] Verifying : Description : Creation and Validation of backup [tp-backup-mysql-ibm-pxb-0-82938-01-18-18h28m38s-1705602621] [2024-01-18T18:31:27.334Z] INFO[2024-01-18 18:31:26] Actual:true, Expected: true [2024-01-18T18:31:27.334Z] 2024-01-18 18:31:26 +0000:[INFO] [{DifferentAccessSameUser}] Share backup with user having viewonly access [2024-01-18T18:31:27.334Z] 2024-01-18 18:31:26 +0000:[INFO] [{DifferentAccessSameUser}] [backup.ShareBackup:#715] - Backup UID for tp-backup-mysql-ibm-pxb-0-82938-01-18-18h28m38s-1705602621 - 6e1bdb3b-f820-4ed6-b077-8e6ef41e79ad [2024-01-18T18:31:27.334Z] 2024-01-18 18:31:27 +0000:[INFO] [{DifferentAccessSameUser}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705602611 - 39e06e70-2f13-41dd-8777-7ef786f7532a [2024-01-18T18:31:27.334Z] 2024-01-18 18:31:27 +0000:[INFO] [{DifferentAccessSameUser}] Share backup with group having full access [2024-01-18T18:31:27.334Z] 2024-01-18 18:31:27 +0000:[INFO] [{DifferentAccessSameUser}] [backup.ShareBackup:#715] - Backup UID for tp-backup-mysql-ibm-pxb-0-82938-01-18-18h28m38s-1705602621 - 6e1bdb3b-f820-4ed6-b077-8e6ef41e79ad [2024-01-18T18:31:27.591Z] 2024-01-18 18:31:27 +0000:[INFO] [{DifferentAccessSameUser}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup - 8496e08b-68c6-47e3-8afd-253bc717d817 [2024-01-18T18:31:27.591Z] 2024-01-18 18:31:27 +0000:[INFO] [{DifferentAccessSameUser}] Backup is shared with Group having FullAccess after it is shared with user having ViewOnlyAccess, therefore user should have FullAccess [2024-01-18T18:31:27.847Z] 2024-01-18 18:31:27 +0000:[INFO] [{DifferentAccessSameUser}] Registering Source and Destination clusters from user context [2024-01-18T18:31:27.847Z] INFO[2024-01-18 18:31:27] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T18:31:27.847Z] INFO[2024-01-18 18:31:27] Actual:true, Expected: true [2024-01-18T18:31:27.847Z] 2024-01-18 18:31:27 +0000:[INFO] [{DifferentAccessSameUser}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T18:31:27.847Z] 2024-01-18 18:31:27 +0000:[INFO] [{DifferentAccessSameUser}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T18:31:27.847Z] 2024-01-18 18:31:27 +0000:[INFO] [{DifferentAccessSameUser}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T18:31:27.847Z] 2024-01-18 18:31:27 +0000:[INFO] [{DifferentAccessSameUser}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T18:31:27.847Z] 2024-01-18 18:31:27 +0000:[INFO] [{DifferentAccessSameUser}] Create cluster [source-cluster] in org [default] [2024-01-18T18:31:27.847Z] 2024-01-18 18:31:27 +0000:[INFO] [{DifferentAccessSameUser}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T18:31:27.847Z] 2024-01-18 18:31:27 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T18:31:27.847Z] 2024-01-18 18:31:27 +0000:[INFO] [{DifferentAccessSameUser}] Create cluster [destination-cluster] in org [default] [2024-01-18T18:31:27.847Z] 2024-01-18 18:31:27 +0000:[INFO] [{DifferentAccessSameUser}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T18:31:27.847Z] 2024-01-18 18:31:27 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T18:31:27.847Z] 2024-01-18 18:31:27 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-srwnm] for IBM [2024-01-18T18:31:27.847Z] 2024-01-18 18:31:27 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-srwnm] for org [default] with [ibm] as provider [2024-01-18T18:31:27.847Z] 2024-01-18 18:31:27 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T18:31:27.847Z] 2024-01-18 18:31:27 +0000:[INFO] [{DifferentAccessSameUser}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T18:31:27.848Z] 2024-01-18 18:31:27 +0000:[WARNING] [{DifferentAccessSameUser}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-srwnm] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user]] [2024-01-18T18:31:27.848Z] 2024-01-18 18:31:27 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user] [2024-01-18T18:31:28.778Z] 2024-01-18 18:31:28 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-srwnm from admin context and sharing with all the users [2024-01-18T18:31:28.778Z] 2024-01-18 18:31:28 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-srwnm] for org [default] with [ibm] as provider [2024-01-18T18:31:28.778Z] 2024-01-18 18:31:28 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T18:31:28.778Z] 2024-01-18 18:31:28 +0000:[INFO] [{DifferentAccessSameUser}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T18:31:28.778Z] 2024-01-18 18:31:28 +0000:[INFO] [{DifferentAccessSameUser}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-srwnm] is present [2024-01-18T18:31:29.039Z] 2024-01-18 18:31:28 +0000:[INFO] [{DifferentAccessSameUser}] Create cluster [source-cluster] in org [default] [2024-01-18T18:31:29.603Z] 2024-01-18 18:31:29 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-lifts] for IBM [2024-01-18T18:31:29.603Z] 2024-01-18 18:31:29 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-lifts] for org [default] with [ibm] as provider [2024-01-18T18:31:29.603Z] 2024-01-18 18:31:29 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T18:31:29.603Z] 2024-01-18 18:31:29 +0000:[INFO] [{DifferentAccessSameUser}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T18:31:29.603Z] 2024-01-18 18:31:29 +0000:[WARNING] [{DifferentAccessSameUser}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-lifts] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user]] [2024-01-18T18:31:29.603Z] 2024-01-18 18:31:29 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user] [2024-01-18T18:31:30.165Z] 2024-01-18 18:31:29 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-lifts from admin context and sharing with all the users [2024-01-18T18:31:30.165Z] 2024-01-18 18:31:29 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-lifts] for org [default] with [ibm] as provider [2024-01-18T18:31:30.165Z] 2024-01-18 18:31:29 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T18:31:30.166Z] 2024-01-18 18:31:29 +0000:[INFO] [{DifferentAccessSameUser}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T18:31:30.166Z] 2024-01-18 18:31:30 +0000:[INFO] [{DifferentAccessSameUser}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-lifts] is present [2024-01-18T18:31:30.423Z] 2024-01-18 18:31:30 +0000:[INFO] [{DifferentAccessSameUser}] Create cluster [destination-cluster] in org [default] [2024-01-18T18:31:30.679Z] INFO[2024-01-18 18:31:30] Verifying : Description : Creating source and destination cluster [2024-01-18T18:31:30.679Z] INFO[2024-01-18 18:31:30] Actual:true, Expected: true [2024-01-18T18:31:31.609Z] 2024/01/18 18:31:31 DoRetryWithTimeout - Error: {restore status for [tp-restore-1705602690] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T18:32:03.634Z] 2024/01/18 18:32:01 DoRetryWithTimeout - Error: {restore status for [tp-restore-1705602690] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T18:32:35.653Z] 2024-01-18 18:32:31 +0000:[INFO] [{DifferentAccessSameUser}] [backup.CreateRestore:#936] - Restore [tp-restore-1705602690] created successfully [2024-01-18T18:32:35.653Z] 2024-01-18 18:32:31 +0000:[INFO] [{DifferentAccessSameUser}] Switching cluster context to cluster [destination-cluster] [2024-01-18T18:32:35.654Z] 2024-01-18 18:32:31 +0000:[INFO] [{DifferentAccessSameUser}] Switching context to [/tmp/destination-config] [2024-01-18T18:32:35.664Z] 2024-01-18 18:32:31 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:32:35.664Z] 2024-01-18 18:32:31 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:32:35.664Z] 2024-01-18 18:32:31 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:32:35.664Z] 2024-01-18 18:32:31 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:32:35.664Z] 2024-01-18 18:32:31 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:32:35.664Z] 2024-01-18 18:32:31 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:32:35.664Z] 2024-01-18 18:32:32 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:32:35.664Z] 2024-01-18 18:32:32 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:32:35.664Z] 2024-01-18 18:32:32 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:32:35.664Z] 2024-01-18 18:32:33 +0000:[WARNING] [{DifferentAccessSameUser}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T18:32:35.664Z] 2024-01-18 18:32:33 +0000:[WARNING] [{DifferentAccessSameUser}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T18:32:35.664Z] 2024-01-18 18:32:33 +0000:[WARNING] [{DifferentAccessSameUser}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T18:32:35.664Z] 2024-01-18 18:32:33 +0000:[INFO] [{DifferentAccessSameUser}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T18:32:35.664Z] 2024-01-18 18:32:33 +0000:[INFO] [{DifferentAccessSameUser}] Switched context to [/tmp/destination-config] [2024-01-18T18:32:35.664Z] 2024-01-18 18:32:33 +0000:[INFO] [{DifferentAccessSameUser}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-82938-01-18-18h28m38s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-18T18:32:35.664Z] 2024-01-18 18:32:33 +0000:[INFO] [{DifferentAccessSameUser}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-82938-01-18-18h28m38s] [2024-01-18T18:32:35.664Z] 2024-01-18 18:32:33 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T18:32:35.664Z] 2024-01-18 18:32:33 +0000:[INFO] [{DifferentAccessSameUser}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T18:32:35.664Z] 2024-01-18 18:32:33 +0000:[INFO] [{DifferentAccessSameUser}] Validating restore [tp-restore-1705602690] in org [default] [2024-01-18T18:32:35.664Z] 2024-01-18 18:32:33 +0000:[INFO] [{DifferentAccessSameUser}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-restore-1705602690] [2024-01-18T18:32:35.664Z] 2024-01-18 18:32:33 +0000:[INFO] [{DifferentAccessSameUser}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-restore-1705602690] [2024-01-18T18:32:35.664Z] 2024-01-18 18:32:33 +0000:[INFO] [{DifferentAccessSameUser}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-82938-01-18-18h28m38s] in restore [tp-restore-1705602690] [2024-01-18T18:32:35.665Z] 2024-01-18 18:32:33 +0000:[INFO] [{DifferentAccessSameUser}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-82938-01-18-18h28m38s]) was found in restore [tp-restore-1705602690], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82938-01-18-18h28m38s] [2024-01-18T18:32:35.665Z] 2024-01-18 18:32:33 +0000:[INFO] [{DifferentAccessSameUser}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-82938-01-18-18h28m38s]) found in the restore [tp-restore-1705602690] was also present on the cluster/namespace [mysql-ibm-pxb-0-82938-01-18-18h28m38s] [2024-01-18T18:32:35.665Z] 2024-01-18 18:32:33 +0000:[INFO] [{DifferentAccessSameUser}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-82938-01-18-18h28m38s]) was found in restore [tp-restore-1705602690], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82938-01-18-18h28m38s] [2024-01-18T18:32:35.665Z] 2024-01-18 18:32:33 +0000:[INFO] [{DifferentAccessSameUser}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-82938-01-18-18h28m38s]) found in the restore [tp-restore-1705602690] was also present on the cluster/namespace [mysql-ibm-pxb-0-82938-01-18-18h28m38s] [2024-01-18T18:32:35.665Z] 2024-01-18 18:32:33 +0000:[INFO] [{DifferentAccessSameUser}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-82938-01-18-18h28m38s]) was found in restore [tp-restore-1705602690], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82938-01-18-18h28m38s] [2024-01-18T18:32:35.665Z] 2024-01-18 18:32:33 +0000:[INFO] [{DifferentAccessSameUser}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-82938-01-18-18h28m38s]) found in the restore [tp-restore-1705602690] was also present on the cluster/namespace [mysql-ibm-pxb-0-82938-01-18-18h28m38s] [2024-01-18T18:32:35.665Z] 2024-01-18 18:32:33 +0000:[INFO] [{DifferentAccessSameUser}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-82938-01-18-18h28m38s]) was found in restore [tp-restore-1705602690], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82938-01-18-18h28m38s] [2024-01-18T18:32:35.665Z] 2024-01-18 18:32:33 +0000:[INFO] [{DifferentAccessSameUser}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-82938-01-18-18h28m38s]) found in the restore [tp-restore-1705602690] was also present on the cluster/namespace [mysql-ibm-pxb-0-82938-01-18-18h28m38s] [2024-01-18T18:32:35.665Z] 2024-01-18 18:32:33 +0000:[INFO] [{DifferentAccessSameUser}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-82938-01-18-18h28m38s] in restore [tp-restore-1705602690] [2024-01-18T18:32:35.665Z] 2024-01-18 18:32:33 +0000:[INFO] [{DifferentAccessSameUser}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-82938-01-18-18h28m38s] in restore [tp-restore-1705602690] [2024-01-18T18:32:35.665Z] 2024-01-18 18:32:33 +0000:[WARNING] [{DifferentAccessSameUser}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T18:32:35.665Z] 2024-01-18 18:32:33 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T18:32:35.665Z] 2024-01-18 18:32:33 +0000:[INFO] [{DifferentAccessSameUser}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82938-01-18-18h28m38s] are [[mysql-data]] [2024-01-18T18:32:35.665Z] 2024-01-18 18:32:33 +0000:[INFO] [{DifferentAccessSameUser}] [backup.ValidateRestore:#2240] - in restore [tp-restore-1705602690], the status of the restored volume [pvc-7d57d036-c072-4f0f-a231-7da93be75d46] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-18T18:32:35.666Z] 2024-01-18 18:32:33 +0000:[INFO] [{DifferentAccessSameUser}] Validate applications in restored namespace [mysql-ibm-pxb-0-82938-01-18-18h28m38s] due to restore [tp-restore-1705602690] [2024-01-18T18:32:35.666Z] 2024-01-18 18:32:33 +0000:[INFO] [{DifferentAccessSameUser}] Validating mysql-ibm app [2024-01-18T18:32:35.666Z] 2024-01-18 18:32:33 +0000:[INFO] [{DifferentAccessSameUser}] wait for mysql-ibm app to start running [2024-01-18T18:32:35.666Z] 2024-01-18 18:32:34 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T18:32:35.666Z] 2024/01/18 18:32:34 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:32:35.666Z] pod name:mysql-7d766cd65c-47lfx namespace:mysql-ibm-pxb-0-82938-01-18-18h28m38s running:true ready:false node:10.240.64.7 [2024-01-18T18:32:35.666Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:32:45.620Z] 2024/01/18 18:32:44 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:32:45.620Z] pod name:mysql-7d766cd65c-47lfx namespace:mysql-ibm-pxb-0-82938-01-18-18h28m38s running:true ready:false node:10.240.64.7 [2024-01-18T18:32:45.620Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:32:55.559Z] 2024/01/18 18:32:54 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:32:55.559Z] pod name:mysql-7d766cd65c-47lfx namespace:mysql-ibm-pxb-0-82938-01-18-18h28m38s running:true ready:false node:10.240.64.7 [2024-01-18T18:32:55.559Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:33:05.500Z] 2024/01/18 18:33:04 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:33:05.500Z] pod name:mysql-7d766cd65c-47lfx namespace:mysql-ibm-pxb-0-82938-01-18-18h28m38s running:true ready:false node:10.240.64.7 [2024-01-18T18:33:05.500Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:33:15.436Z] 2024/01/18 18:33:14 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:33:15.436Z] pod name:mysql-7d766cd65c-47lfx namespace:mysql-ibm-pxb-0-82938-01-18-18h28m38s running:true ready:false node:10.240.64.7 [2024-01-18T18:33:15.436Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:33:25.372Z] 2024/01/18 18:33:24 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:33:25.372Z] pod name:mysql-7d766cd65c-47lfx namespace:mysql-ibm-pxb-0-82938-01-18-18h28m38s running:true ready:false node:10.240.64.7 [2024-01-18T18:33:25.372Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:33:35.314Z] 2024-01-18 18:33:34 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T18:33:35.314Z] 2024-01-18 18:33:34 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T18:33:35.314Z] 2024-01-18 18:33:35 +0000:[INFO] [{DifferentAccessSameUser}] Validating CR cleanup [2024-01-18T18:33:35.314Z] 2024-01-18 18:33:35 +0000:[INFO] [{DifferentAccessSameUser}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T18:33:35.314Z] 2024-01-18 18:33:35 +0000:[INFO] [{DifferentAccessSameUser}] All CRs in [mysql-ibm-pxb-0-82938-01-18-18h28m38s] are [[]] [2024-01-18T18:33:35.314Z] 2024-01-18 18:33:35 +0000:[INFO] [{DifferentAccessSameUser}] Switching cluster context back to cluster path [/tmp/source-config] [2024-01-18T18:33:35.314Z] 2024-01-18 18:33:35 +0000:[INFO] [{DifferentAccessSameUser}] Switching context to [/tmp/source-config] [2024-01-18T18:33:35.570Z] 2024-01-18 18:33:35 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:33:35.570Z] 2024-01-18 18:33:35 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:33:35.571Z] 2024-01-18 18:33:35 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:33:35.571Z] 2024-01-18 18:33:35 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:33:35.571Z] 2024-01-18 18:33:35 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:33:35.826Z] 2024-01-18 18:33:35 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:33:36.081Z] 2024-01-18 18:33:36 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:33:36.642Z] 2024-01-18 18:33:36 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:33:36.897Z] 2024-01-18 18:33:36 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:33:37.163Z] 2024-01-18 18:33:37 +0000:[WARNING] [{DifferentAccessSameUser}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T18:33:37.419Z] 2024-01-18 18:33:37 +0000:[WARNING] [{DifferentAccessSameUser}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T18:33:37.419Z] 2024-01-18 18:33:37 +0000:[WARNING] [{DifferentAccessSameUser}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T18:33:37.419Z] 2024-01-18 18:33:37 +0000:[INFO] [{DifferentAccessSameUser}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T18:33:37.419Z] 2024-01-18 18:33:37 +0000:[INFO] [{DifferentAccessSameUser}] Switched context to [/tmp/source-config] [2024-01-18T18:33:37.419Z] 2024-01-18 18:33:37 +0000:[INFO] [{DifferentAccessSameUser}] Restoring of backup [tp-backup-mysql-ibm-pxb-0-82938-01-18-18h28m38s-1705602621] was successful with name - [tp-restore-1705602690] [2024-01-18T18:33:37.419Z] 2024-01-18 18:33:37 +0000:[INFO] [{DifferentAccessSameUser}] [backup.glob..func82.2.10:#402] - About to delete restore - tp-restore-1705602690 to validate user can delete restore [2024-01-18T18:33:37.419Z] INFO[2024-01-18 18:33:37] Verifying : Description : Getting the backup driver [2024-01-18T18:33:37.419Z] INFO[2024-01-18 18:33:37] Actual:true, Expected: true [2024-01-18T18:33:37.419Z] INFO[2024-01-18 18:33:37] Verifying : Description : Deleting Restore tp-restore-1705602690 [2024-01-18T18:33:37.419Z] INFO[2024-01-18 18:33:37] Actual:true, Expected: true [2024-01-18T18:33:37.674Z] INFO[2024-01-18 18:33:37] Verifying : Description : Verifying backup tp-backup-mysql-ibm-pxb-0-82938-01-18-18h28m38s-1705602621 deletion [2024-01-18T18:33:37.674Z] INFO[2024-01-18 18:33:37] Actual:, Expected: [2024-01-18T18:33:39.037Z] 2024-01-18 18:33:38 +0000:[INFO] [{DifferentAccessSameUser}] Destroying apps [2024-01-18T18:33:39.037Z] 2024-01-18 18:33:38 +0000:[INFO] [{DifferentAccessSameUser}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T18:33:39.292Z] 2024-01-18 18:33:39 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T18:33:39.292Z] 2024-01-18 18:33:39 +0000:[INFO] [{DifferentAccessSameUser}] start destroying mysql-ibm app [2024-01-18T18:33:39.292Z] 2024-01-18 18:33:39 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T18:33:39.549Z] 2024-01-18 18:33:39 +0000:[INFO] [{DifferentAccessSameUser}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T18:33:39.549Z] 2024-01-18 18:33:39 +0000:[WARNING] [{DifferentAccessSameUser}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T18:33:39.549Z] INFO[2024-01-18 18:33:39] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T18:33:39.549Z] INFO[2024-01-18 18:33:39] Actual:true, Expected: true [2024-01-18T18:33:39.549Z] 2024-01-18 18:33:39 +0000:[INFO] [{DifferentAccessSameUser}] [backup.glob..func82.3:#422] - Generating user context [2024-01-18T18:33:39.805Z] 2024-01-18 18:33:39 +0000:[INFO] [{DifferentAccessSameUser}] [backup.glob..func82.3:#425] - Deleting registered clusters for non-admin context [2024-01-18T18:33:39.805Z] 2024-01-18 18:33:39 +0000:[INFO] [{DifferentAccessSameUser}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T18:33:39.805Z] INFO[2024-01-18 18:33:39] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T18:33:39.805Z] INFO[2024-01-18 18:33:39] Actual:true, Expected: true [2024-01-18T18:33:40.063Z] INFO[2024-01-18 18:33:39] Verifying : Description : Deleting cluster source-cluster [2024-01-18T18:33:40.063Z] INFO[2024-01-18 18:33:39] Actual:true, Expected: true [2024-01-18T18:33:41.426Z] INFO[2024-01-18 18:33:41] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-srwnm] [2024-01-18T18:33:41.426Z] INFO[2024-01-18 18:33:41] Actual:true, Expected: true [2024-01-18T18:33:41.426Z] INFO[2024-01-18 18:33:41] Verifying : Description : Deleting cluster destination-cluster [2024-01-18T18:33:41.426Z] INFO[2024-01-18 18:33:41] Actual:true, Expected: true [2024-01-18T18:33:42.794Z] INFO[2024-01-18 18:33:42] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-lifts] [2024-01-18T18:33:42.794Z] INFO[2024-01-18 18:33:42] Actual:true, Expected: true [2024-01-18T18:33:43.721Z] 2024-01-18 18:33:43 +0000:[INFO] [{DifferentAccessSameUser}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705602611 - 39e06e70-2f13-41dd-8777-7ef786f7532a [2024-01-18T18:33:44.288Z] 2024-01-18 18:33:44 +0000:[INFO] [{DifferentAccessSameUser}] [backup.DeleteUser:#755] - Response for user [testuser1-1705602611] deletion - [2024-01-18T18:33:44.288Z] 2024-01-18 18:33:44 +0000:[INFO] [{DifferentAccessSameUser}] [backup.DeleteUser:#759] - Deleted User - testuser1-1705602611 [2024-01-18T18:33:44.288Z] INFO[2024-01-18 18:33:44] Verifying : Description : Deleting user testuser1-1705602611 [2024-01-18T18:33:44.288Z] INFO[2024-01-18 18:33:44] Actual:true, Expected: true [2024-01-18T18:33:45.652Z] 2024-01-18 18:33:45 +0000:[INFO] [{DifferentAccessSameUser}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup - 8496e08b-68c6-47e3-8afd-253bc717d817 [2024-01-18T18:33:45.911Z] 2024-01-18 18:33:45 +0000:[INFO] [{DifferentAccessSameUser}] [backup.DeleteGroup:#1018] - Deleted Group - testGroup [2024-01-18T18:33:45.911Z] INFO[2024-01-18 18:33:45] Verifying : Description : Deleting group testGroup [2024-01-18T18:33:45.911Z] INFO[2024-01-18 18:33:45] Actual:true, Expected: true [2024-01-18T18:33:45.912Z] 2024-01-18 18:33:45 +0000:[INFO] [{DifferentAccessSameUser}] Cleaning backup locations in map [map[20e54b9e-93e5-4b15-9a15-fad34283e411:aws-global-aws-px-backup-system-test-4cac76dc-1705602616-bl]], cloud credential [cloudcred-aws-1705602616], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T18:33:45.912Z] INFO[2024-01-18 18:33:45] Verifying : Description : Verifying deletion of backup location [aws-global-aws-px-backup-system-test-4cac76dc-1705602616-bl] [2024-01-18T18:33:45.912Z] INFO[2024-01-18 18:33:45] Actual:true, Expected: true [2024-01-18T18:33:46.168Z] 2024-01-18 18:33:45 +0000:[INFO] [{DifferentAccessSameUser}] [backup.IsBackupLocationPresent:#2442] - Backup location [aws-global-aws-px-backup-system-test-4cac76dc-1705602616-bl] is present [2024-01-18T18:33:46.168Z] 2024/01/18 18:33:45 DoRetryWithTimeout - Error: {backup location aws-global-aws-px-backup-system-test-4cac76dc-1705602616-bl is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T18:34:18.195Z] INFO[2024-01-18 18:34:13] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T18:34:18.195Z] INFO[2024-01-18 18:34:13] sched stats: tick interval: 10813 < 1.2s (100%) [2024-01-18T18:34:18.195Z] INFO[2024-01-18 18:34:13] sched stats: tick process durations: 10812 < 500ms (100%) [2024-01-18T18:34:18.195Z] 2024-01-18 18:34:16 +0000:[INFO] [{DifferentAccessSameUser}] [backup.IsBackupLocationPresent:#2442] - Backup location [aws-global-aws-px-backup-system-test-4cac76dc-1705602616-bl] is present [2024-01-18T18:34:18.195Z] 2024/01/18 18:34:16 DoRetryWithTimeout - Error: {backup location aws-global-aws-px-backup-system-test-4cac76dc-1705602616-bl is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T18:34:50.215Z] 2024-01-18 18:34:46 +0000:[INFO] [{DifferentAccessSameUser}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu] [2024-01-18T18:34:50.215Z] INFO[2024-01-18 18:34:46] Verifying : Description : Verifying backup location deletion status aws-global-aws-px-backup-system-test-4cac76dc-1705602616-bl [2024-01-18T18:34:50.215Z] INFO[2024-01-18 18:34:46] Actual:true, Expected: true [2024-01-18T18:34:50.215Z] 2024-01-18 18:34:46 +0000:[INFO] [{DifferentAccessSameUser}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cloudcred-aws-1705602616] is present [2024-01-18T18:34:50.215Z] INFO[2024-01-18 18:34:46] Verifying : Description : Verifying if cloud cred [cloudcred-aws-1705602616] is present [2024-01-18T18:34:50.215Z] INFO[2024-01-18 18:34:46] Actual:true, Expected: true [2024-01-18T18:34:50.215Z] INFO[2024-01-18 18:34:46] Verifying : Description : Verifying deletion of cloud cred [cloudcred-aws-1705602616] [2024-01-18T18:34:50.215Z] INFO[2024-01-18 18:34:46] Actual:true, Expected: true [2024-01-18T18:34:50.215Z] INFO[2024-01-18 18:34:46] Verifying : Description : Deleting cloud cred cloudcred-aws-1705602616 [2024-01-18T18:34:50.215Z] INFO[2024-01-18 18:34:46] Actual:true, Expected: true [2024-01-18T18:34:50.215Z] INFO[2024-01-18 18:34:46] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T18:34:50.216Z] INFO[2024-01-18 18:34:46] Actual:true, Expected: true [2024-01-18T18:34:50.216Z] INFO[2024-01-18 18:34:46] Verifying : Description : Deleting cluster source-cluster [2024-01-18T18:34:50.216Z] INFO[2024-01-18 18:34:46] Actual:true, Expected: true [2024-01-18T18:34:50.216Z] INFO[2024-01-18 18:34:46] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-wmdak] [2024-01-18T18:34:50.216Z] INFO[2024-01-18 18:34:46] Actual:true, Expected: true [2024-01-18T18:34:50.216Z] INFO[2024-01-18 18:34:47] Verifying : Description : Deleting cluster destination-cluster [2024-01-18T18:34:50.216Z] INFO[2024-01-18 18:34:47] Actual:true, Expected: true [2024-01-18T18:34:50.216Z] INFO[2024-01-18 18:34:47] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-rsvcv] [2024-01-18T18:34:50.216Z] INFO[2024-01-18 18:34:47] Actual:true, Expected: true [2024-01-18T18:34:50.216Z] INFO[2024-01-18 18:34:47] --------Test End------ [2024-01-18T18:34:50.216Z] INFO[2024-01-18 18:34:47] #Test: [2024-01-18T18:34:50.216Z] INFO[2024-01-18 18:34:47] #Description: [2024-01-18T18:34:50.216Z] INFO[2024-01-18 18:34:47] ------------------------ [2024-01-18T18:34:50.216Z] 2024-01-18 18:34:47 +0000:[INFO] [{DifferentAccessSameUser}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-82938-01-18-18h28m38s] [2024-01-18T18:34:50.216Z] 2024-01-18 18:34:47 +0000:[INFO] [{DifferentAccessSameUser}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T18:34:50.216Z] 2024-01-18 18:34:47 +0000:[INFO] [{DifferentAccessSameUser}] Switching context to [/tmp/destination-config] [2024-01-18T18:34:50.216Z] 2024-01-18 18:34:47 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:34:50.216Z] 2024-01-18 18:34:47 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:34:50.216Z] 2024-01-18 18:34:47 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:34:50.216Z] 2024-01-18 18:34:47 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:34:50.216Z] 2024-01-18 18:34:47 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:34:50.216Z] 2024-01-18 18:34:48 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:34:50.216Z] 2024-01-18 18:34:48 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:34:50.216Z] 2024-01-18 18:34:48 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:34:50.216Z] 2024-01-18 18:34:49 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:34:50.216Z] 2024-01-18 18:34:49 +0000:[WARNING] [{DifferentAccessSameUser}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T18:34:50.216Z] 2024-01-18 18:34:49 +0000:[WARNING] [{DifferentAccessSameUser}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T18:34:50.216Z] 2024-01-18 18:34:49 +0000:[WARNING] [{DifferentAccessSameUser}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T18:34:50.216Z] 2024-01-18 18:34:49 +0000:[INFO] [{DifferentAccessSameUser}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T18:34:50.217Z] 2024-01-18 18:34:49 +0000:[INFO] [{DifferentAccessSameUser}] Switched context to [/tmp/destination-config] [2024-01-18T18:34:50.217Z] 2024-01-18 18:34:49 +0000:[INFO] [{DifferentAccessSameUser}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-82938-01-18-18h28m38s] [2024-01-18T18:34:50.217Z] 2024-01-18 18:34:50 +0000:[INFO] [{DifferentAccessSameUser}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {DifferentAccessSameUser} Different Access Same User [2024-01-18T18:34:50.217Z] 2024-01-18 18:34:50 +0000:[INFO] [{DifferentAccessSameUser}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-18T18:34:50.217Z] 2024/01/18 18:34:50 DoRetryWithTimeout - Error: {dial tcp 10.240.0.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T18:34:55.469Z] 2024/01/18 18:34:55 DoRetryWithTimeout - Error: {dial tcp 10.240.0.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T18:35:00.719Z] 2024/01/18 18:35:00 DoRetryWithTimeout - Error: {dial tcp 10.240.0.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T18:35:06.065Z] 2024/01/18 18:35:05 DoRetryWithTimeout - Error: {dial tcp 10.240.0.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T18:35:10.228Z] 2024/01/18 18:35:10 DoRetryWithTimeout - Error: {dial tcp 10.240.0.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T18:35:15.467Z] 2024/01/18 18:35:15 DoRetryWithTimeout - Error: {dial tcp 10.240.0.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T18:35:20.732Z] 2024/01/18 18:35:20 DoRetryWithTimeout - Error: {dial tcp 10.240.0.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T18:35:25.981Z] 2024/01/18 18:35:25 DoRetryWithTimeout - Error: {dial tcp 10.240.0.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T18:35:31.231Z] 2024/01/18 18:35:30 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.0.6. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.6 10.240.0.6]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.6:22: connect: connection refused} [2024-01-18T18:35:31.231Z] {dial tcp 10.240.0.6:22: connect: connection refused} [2024-01-18T18:35:31.231Z] {dial tcp 10.240.0.6:22: connect: connection refused} [2024-01-18T18:35:31.231Z] {dial tcp 10.240.0.6:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-18T18:35:35.401Z] 2024-01-18 18:35:35 +0000:[WARNING] [{DifferentAccessSameUser}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.6. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.6 10.240.0.6]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.6:22: connect: connection refused} [2024-01-18T18:35:35.401Z] {dial tcp 10.240.0.6:22: connect: connection refused} [2024-01-18T18:35:35.401Z] {dial tcp 10.240.0.6:22: connect: connection refused} [2024-01-18T18:35:35.401Z] {dial tcp 10.240.0.6:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T18:35:35.401Z] 2024-01-18 18:35:35 +0000:[ERROR] [{DifferentAccessSameUser}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.6. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.6 10.240.0.6]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.6:22: connect: connection refused} [2024-01-18T18:35:35.401Z] {dial tcp 10.240.0.6:22: connect: connection refused} [2024-01-18T18:35:35.401Z] {dial tcp 10.240.0.6:22: connect: connection refused} [2024-01-18T18:35:35.401Z] {dial tcp 10.240.0.6:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T18:35:35.401Z] 2024-01-18 18:35:35 +0000:[INFO] [{DifferentAccessSameUser}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T18:35:35.401Z] 2024-01-18 18:35:35 +0000:[INFO] [{DifferentAccessSameUser}] Switching context to [/tmp/source-config] [2024-01-18T18:35:35.401Z] 2024-01-18 18:35:35 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:35:35.401Z] 2024-01-18 18:35:35 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:35:35.401Z] 2024-01-18 18:35:35 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:35:35.658Z] 2024-01-18 18:35:35 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:35:35.658Z] 2024-01-18 18:35:35 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:35:35.658Z] 2024-01-18 18:35:35 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:35:36.219Z] 2024-01-18 18:35:36 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:35:36.474Z] 2024-01-18 18:35:36 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:35:37.036Z] 2024-01-18 18:35:36 +0000:[INFO] [{DifferentAccessSameUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:35:37.294Z] 2024-01-18 18:35:37 +0000:[WARNING] [{DifferentAccessSameUser}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T18:35:37.294Z] 2024-01-18 18:35:37 +0000:[WARNING] [{DifferentAccessSameUser}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T18:35:37.294Z] 2024-01-18 18:35:37 +0000:[WARNING] [{DifferentAccessSameUser}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T18:35:37.294Z] 2024-01-18 18:35:37 +0000:[INFO] [{DifferentAccessSameUser}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T18:35:37.294Z] 2024-01-18 18:35:37 +0000:[INFO] [{DifferentAccessSameUser}] Switched context to [/tmp/source-config] [2024-01-18T18:35:37.294Z] •SSSSS2024-01-18 18:35:37 +0000:[INFO] [{VerifyRBACForAppUser}] Log Dir: /testresults//VerifyRBACForAppUser.log [2024-01-18T18:35:37.294Z] INFO[2024-01-18 18:35:37] --------Test Start------ [2024-01-18T18:35:37.294Z] INFO[2024-01-18 18:35:37] #Test: VerifyRBACForAppUser [2024-01-18T18:35:37.294Z] INFO[2024-01-18 18:35:37] #Description: To verify all the RBAC operations for an app-user [2024-01-18T18:35:37.294Z] INFO[2024-01-18 18:35:37] ------------------------ [2024-01-18T18:35:37.294Z] 2024-01-18 18:35:37 +0000:[INFO] [{VerifyRBACForAppUser}] scheduling applications [2024-01-18T18:35:37.295Z] 2024-01-18 18:35:37 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T18:35:37.295Z] 2024-01-18 18:35:37 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T18:35:37.551Z] 2024-01-18 18:35:37 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T18:35:37.551Z] 2024-01-18 18:35:37 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T18:35:37.806Z] 2024-01-18 18:35:37 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T18:35:38.062Z] 2024-01-18 18:35:37 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T18:35:38.063Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T18:35:38.063Z] 2024-01-18 18:35:37 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T18:35:38.063Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T18:35:38.063Z] 2024-01-18 18:35:37 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T18:35:38.063Z] 2024-01-18 18:35:37 +0000:[INFO] [{VerifyRBACForAppUser}] Validating applications [2024-01-18T18:35:38.063Z] 2024-01-18 18:35:37 +0000:[INFO] [{VerifyRBACForAppUser}] Validate applications [2024-01-18T18:35:38.063Z] 2024-01-18 18:35:37 +0000:[INFO] [{VerifyRBACForAppUser}] Validating mysql-ibm app [2024-01-18T18:35:38.063Z] 2024-01-18 18:35:37 +0000:[INFO] [{VerifyRBACForAppUser}] Validating mysql-ibm app's volumes [2024-01-18T18:35:38.063Z] 2024-01-18 18:35:38 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T18:35:38.318Z] 2024/01/18 18:35:38 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T18:35:48.264Z] 2024/01/18 18:35:48 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T18:36:00.429Z] 2024-01-18 18:35:58 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-87889-01-18-18h35m37s [2024-01-18T18:36:00.429Z] 2024-01-18 18:35:58 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T18:36:00.429Z] 2024-01-18 18:35:58 +0000:[WARNING] [{VerifyRBACForAppUser}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T18:36:00.429Z] 2024-01-18 18:35:58 +0000:[INFO] [{VerifyRBACForAppUser}] wait for mysql-ibm app to start running [2024-01-18T18:36:00.429Z] 2024-01-18 18:35:58 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T18:36:00.429Z] 2024/01/18 18:35:58 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:36:00.429Z] pod name:mysql-7d766cd65c-d7twp namespace:mysql-ibm-pxb-0-87889-01-18-18h35m37s running:false ready:false node:10.240.64.9 [2024-01-18T18:36:00.429Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:36:10.369Z] 2024/01/18 18:36:08 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:36:10.370Z] pod name:mysql-7d766cd65c-d7twp namespace:mysql-ibm-pxb-0-87889-01-18-18h35m37s running:true ready:false node:10.240.64.9 [2024-01-18T18:36:10.370Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:36:20.307Z] 2024/01/18 18:36:18 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:36:20.307Z] pod name:mysql-7d766cd65c-d7twp namespace:mysql-ibm-pxb-0-87889-01-18-18h35m37s running:true ready:false node:10.240.64.9 [2024-01-18T18:36:20.307Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:36:30.342Z] 2024/01/18 18:36:28 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:36:30.342Z] pod name:mysql-7d766cd65c-d7twp namespace:mysql-ibm-pxb-0-87889-01-18-18h35m37s running:true ready:false node:10.240.64.9 [2024-01-18T18:36:30.342Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:36:40.303Z] 2024/01/18 18:36:38 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:36:40.303Z] pod name:mysql-7d766cd65c-d7twp namespace:mysql-ibm-pxb-0-87889-01-18-18h35m37s running:true ready:false node:10.240.64.9 [2024-01-18T18:36:40.303Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:36:50.276Z] 2024/01/18 18:36:48 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:36:50.276Z] pod name:mysql-7d766cd65c-d7twp namespace:mysql-ibm-pxb-0-87889-01-18-18h35m37s running:true ready:false node:10.240.64.9 [2024-01-18T18:36:50.276Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:37:00.220Z] 2024/01/18 18:36:59 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:37:00.220Z] pod name:mysql-7d766cd65c-d7twp namespace:mysql-ibm-pxb-0-87889-01-18-18h35m37s running:true ready:false node:10.240.64.9 [2024-01-18T18:37:00.220Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:37:10.174Z] 2024/01/18 18:37:09 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:37:10.174Z] pod name:mysql-7d766cd65c-d7twp namespace:mysql-ibm-pxb-0-87889-01-18-18h35m37s running:true ready:false node:10.240.64.9 [2024-01-18T18:37:10.174Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:37:20.135Z] 2024-01-18 18:37:19 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T18:37:20.135Z] 2024-01-18 18:37:19 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T18:37:20.135Z] 2024-01-18 18:37:19 +0000:[INFO] [{VerifyRBACForAppUser}] validate if mysql-ibm app's volumes are setup [2024-01-18T18:37:20.135Z] 2024-01-18 18:37:19 +0000:[WARNING] [{VerifyRBACForAppUser}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T18:37:20.135Z] 2024-01-18 18:37:19 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T18:37:20.135Z] 2024-01-18 18:37:19 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T18:37:20.135Z] 2024-01-18 18:37:19 +0000:[WARNING] [{VerifyRBACForAppUser}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T18:37:20.135Z] 2024-01-18 18:37:19 +0000:[INFO] [{VerifyRBACForAppUser}] Adding labels to namespaces [2024-01-18T18:37:20.135Z] INFO[2024-01-18 18:37:19] Verifying : Description : Adding labels [map[0-864abd34-8f71-49db-a527-f7eea2e4f188:864abd34-8f71-49db-a527-f7eea2e4f188 1-864abd34-8f71-49db-a527-f7eea2e4f188:864abd34-8f71-49db-a527-f7eea2e4f188]] to namespaces [[mysql-ibm-pxb-0-87889-01-18-18h35m37s]] [2024-01-18T18:37:20.135Z] INFO[2024-01-18 18:37:19] Actual:true, Expected: true [2024-01-18T18:37:20.135Z] 2024-01-18 18:37:19 +0000:[INFO] [{VerifyRBACForAppUser}] Generating namespace label string from label map for namespaces [2024-01-18T18:37:20.135Z] 2024-01-18 18:37:19 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.glob..func46.2.3:#1726] - label string for namespaces 0-864abd34-8f71-49db-a527-f7eea2e4f188=864abd34-8f71-49db-a527-f7eea2e4f188,1-864abd34-8f71-49db-a527-f7eea2e4f188=864abd34-8f71-49db-a527-f7eea2e4f188 [2024-01-18T18:37:20.135Z] 2024-01-18 18:37:19 +0000:[INFO] [{VerifyRBACForAppUser}] Creating a new user and assigning the role of App-user [2024-01-18T18:37:20.136Z] 2024-01-18 18:37:19 +0000:[INFO] [{VerifyRBACForAppUser}] Creating 1 users [2024-01-18T18:37:20.136Z] 2024-01-18 18:37:19 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.AddUser:#728] - Response for user [testuser1-1705603039] creation - [2024-01-18T18:37:20.136Z] 2024-01-18 18:37:19 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.AddUser:#733] - User [testuser1-1705603039] created [2024-01-18T18:37:20.136Z] INFO[2024-01-18 18:37:19] Verifying : Description : Creating user - testuser1-1705603039 [2024-01-18T18:37:20.136Z] INFO[2024-01-18 18:37:19] Actual:true, Expected: true [2024-01-18T18:37:20.392Z] 2024-01-18 18:37:20 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705603039 - 46d3e5a9-a13d-4004-a271-a2c0cfd5258d [2024-01-18T18:37:22.276Z] 2024-01-18 18:37:21 +0000:[INFO] [{VerifyRBACForAppUser}] Verify if the App-User doesn't have permission to create cloud credentials and backup location [2024-01-18T18:37:22.276Z] 2024-01-18 18:37:22 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [cred-aws-qwldcc] for org [default] with [aws] as provider [2024-01-18T18:37:22.276Z] 2024-01-18 18:37:22 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-18T18:37:22.276Z] 2024-01-18 18:37:22 +0000:[WARNING] [{VerifyRBACForAppUser}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [cred-aws-qwldcc] in org [default] with [aws] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user]] [2024-01-18T18:37:22.276Z] INFO[2024-01-18 18:37:22] Verifying : Description : Verifying if App-User [testuser1-1705603039] doesn't have permission for creating cloud credentials for provider [aws] [2024-01-18T18:37:22.276Z] INFO[2024-01-18 18:37:22] Actual:true, Expected: true [2024-01-18T18:37:22.276Z] INFO[2024-01-18 18:37:22] Verifying : Description : Verifying if App-User [testuser1-1705603039] doesn't have permission for creating backup location [2024-01-18T18:37:22.277Z] INFO[2024-01-18 18:37:22] Actual:true, Expected: true [2024-01-18T18:37:22.277Z] 2024-01-18 18:37:22 +0000:[INFO] [{VerifyRBACForAppUser}] Verify if App-User doesn't have permission to create a schedule policy [2024-01-18T18:37:22.838Z] INFO[2024-01-18 18:37:22] Verifying : Description : Verifying if App-User [testuser1-1705603039] doesn't have permission for creating schedule policy for user [2024-01-18T18:37:22.838Z] INFO[2024-01-18 18:37:22] Actual:true, Expected: true [2024-01-18T18:37:22.838Z] 2024-01-18 18:37:22 +0000:[INFO] [{VerifyRBACForAppUser}] Verify if the App-User doesn't have permission to create roles [2024-01-18T18:37:25.634Z] INFO[2024-01-18 18:37:24] Verifying : Description : Verifying if App-User [testuser1-1705603039] doesn't have permission for creating role [2024-01-18T18:37:25.634Z] INFO[2024-01-18 18:37:24] Actual:true, Expected: true [2024-01-18T18:37:25.634Z] 2024-01-18 18:37:24 +0000:[INFO] [{VerifyRBACForAppUser}] Verify if App-User doesn't have permission to create pre and post exec rules for applications [2024-01-18T18:37:25.634Z] INFO[2024-01-18 18:37:24] Verifying : Description : Verifying if App-User [testuser1-1705603039] doesn't have permission for creating rules [2024-01-18T18:37:25.634Z] INFO[2024-01-18 18:37:24] Actual:true, Expected: true [2024-01-18T18:37:25.634Z] 2024-01-18 18:37:24 +0000:[INFO] [{VerifyRBACForAppUser}] Validate creation of cloud credentials and backup locations for Px-Admin user [2024-01-18T18:37:26.562Z] 2024-01-18 18:37:26 +0000:[INFO] [{VerifyRBACForAppUser}] Creating cloud credential named [cred-aws-bkac] and uid [6dd1a1a8-db02-412f-99fc-eb02938a0d38] using [aws] as provider [2024-01-18T18:37:26.563Z] 2024-01-18 18:37:26 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [cred-aws-bkac] for org [default] with [aws] as provider [2024-01-18T18:37:26.563Z] 2024-01-18 18:37:26 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-18T18:37:26.563Z] 2024-01-18 18:37:26 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cred-aws-bkac] is present [2024-01-18T18:37:26.563Z] INFO[2024-01-18 18:37:26] Verifying : Description : Verifying creation of cloud credential named [cred-aws-bkac] for org [default] with [aws] as provider [2024-01-18T18:37:26.563Z] INFO[2024-01-18 18:37:26] Actual:true, Expected: true [2024-01-18T18:37:30.726Z] INFO[2024-01-18 18:37:29] Verifying : Description : Verifying creation of backup location named [aws-global-aws-px-backup-system-test-4cac76dc-qbwg] with uid [968d882d-b5f2-4fd7-8536-0b66237f1e4c] of [aws] as provider [2024-01-18T18:37:30.726Z] INFO[2024-01-18 18:37:29] Actual:true, Expected: true [2024-01-18T18:37:30.726Z] 2024-01-18 18:37:29 +0000:[INFO] [{VerifyRBACForAppUser}] Validate creation of schedule policy for px-admin user [2024-01-18T18:37:30.726Z] INFO[2024-01-18 18:37:30] Verifying : Description : Verifying creation of periodic schedule policy of interval [15] minutes named [periodic-igio] [2024-01-18T18:37:30.726Z] INFO[2024-01-18 18:37:30] Actual:true, Expected: true [2024-01-18T18:37:30.726Z] 2024-01-18 18:37:30 +0000:[INFO] [{VerifyRBACForAppUser}] Verify creation of pre and post exec rules for applications for Px-Admin user [2024-01-18T18:37:30.981Z] 2024-01-18 18:37:30 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.CreateRuleForBackupWithMultipleApplications:#4309] - Validate pre-rules for backup [2024-01-18T18:37:30.982Z] 2024-01-18 18:37:30 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.CreateRuleForBackupWithMultipleApplications:#4322] - Validate post-rules for backup [2024-01-18T18:37:31.237Z] INFO[2024-01-18 18:37:30] Verifying : Description : Verifying creation of pre and post exec rules for applications from px-admin [2024-01-18T18:37:31.237Z] INFO[2024-01-18 18:37:30] Actual:true, Expected: true [2024-01-18T18:37:31.237Z] 2024-01-18 18:37:31 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.glob..func46.2.11:#1828] - Pre backup rule [pre-rule-lwuiq] uid: [8f274cb1-cdfa-410e-b177-83ecffe5db97] [2024-01-18T18:37:31.237Z] 2024-01-18 18:37:31 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.glob..func46.2.11:#1833] - Post backup rule [post-rule-vhytm] uid: [58d028ef-41a0-4cd4-82ca-4cf93fc216a1] [2024-01-18T18:37:31.237Z] 2024-01-18 18:37:31 +0000:[INFO] [{VerifyRBACForAppUser}] Verify if the Px-Admin user has permission to share RBAC resources with the App-user [2024-01-18T18:37:32.166Z] 2024-01-18 18:37:32 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.glob..func46.2.12:#1843] - Updating CloudAccount - cred-aws-bkac ownership for user - [testuser1-1705603039] [2024-01-18T18:37:33.092Z] 2024-01-18 18:37:32 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705603039 - 46d3e5a9-a13d-4004-a271-a2c0cfd5258d [2024-01-18T18:37:33.092Z] INFO[2024-01-18 18:37:33] Verifying : Description : Verifying updation of ownership for CloudCredential- cred-aws-bkac [2024-01-18T18:37:33.092Z] INFO[2024-01-18 18:37:33] Actual:true, Expected: true [2024-01-18T18:37:33.092Z] 2024-01-18 18:37:33 +0000:[INFO] [{VerifyRBACForAppUser}] Updating BackupLocation - aws-global-aws-px-backup-system-test-4cac76dc-qbwg ownership for users - [testuser1-1705603039] [2024-01-18T18:37:34.022Z] 2024-01-18 18:37:33 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705603039 - 46d3e5a9-a13d-4004-a271-a2c0cfd5258d [2024-01-18T18:37:34.022Z] INFO[2024-01-18 18:37:33] Verifying : Description : Verifying updation of ownership for backuplocation - aws-global-aws-px-backup-system-test-4cac76dc-qbwg [2024-01-18T18:37:34.022Z] INFO[2024-01-18 18:37:33] Actual:true, Expected: true [2024-01-18T18:37:34.022Z] 2024-01-18 18:37:33 +0000:[INFO] [{VerifyRBACForAppUser}] Updating SchedulePolicy - periodic-igio ownership for users - [testuser1-1705603039] [2024-01-18T18:37:34.583Z] 2024-01-18 18:37:34 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705603039 - 46d3e5a9-a13d-4004-a271-a2c0cfd5258d [2024-01-18T18:37:34.840Z] INFO[2024-01-18 18:37:34] Verifying : Description : Verifying updation of ownership for schedulepolicy - periodic-igio [2024-01-18T18:37:34.840Z] INFO[2024-01-18 18:37:34] Actual:true, Expected: true [2024-01-18T18:37:34.840Z] 2024-01-18 18:37:34 +0000:[INFO] [{VerifyRBACForAppUser}] Updating Application Rules ownership for users - [testuser1-1705603039] [2024-01-18T18:37:35.401Z] 2024-01-18 18:37:35 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705603039 - 46d3e5a9-a13d-4004-a271-a2c0cfd5258d [2024-01-18T18:37:35.656Z] INFO[2024-01-18 18:37:35] Verifying : Description : Verifying updation of ownership for pre-rule of application [2024-01-18T18:37:35.656Z] INFO[2024-01-18 18:37:35] Actual:true, Expected: true [2024-01-18T18:37:36.217Z] 2024-01-18 18:37:36 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705603039 - 46d3e5a9-a13d-4004-a271-a2c0cfd5258d [2024-01-18T18:37:36.472Z] INFO[2024-01-18 18:37:36] Verifying : Description : Verifying updation of ownership for post-rule of application [2024-01-18T18:37:36.472Z] INFO[2024-01-18 18:37:36] Actual:true, Expected: true [2024-01-18T18:37:36.473Z] 2024-01-18 18:37:36 +0000:[INFO] [{VerifyRBACForAppUser}] Validate adding of source and destination clusters with App-User ctx [2024-01-18T18:37:36.729Z] 2024-01-18 18:37:36 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.glob..func46.2.13:#1869] - Creating source [source-cluster] and destination [destination-cluster] clusters [2024-01-18T18:37:36.729Z] INFO[2024-01-18 18:37:36] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T18:37:36.729Z] INFO[2024-01-18 18:37:36] Actual:true, Expected: true [2024-01-18T18:37:36.729Z] 2024-01-18 18:37:36 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T18:37:36.730Z] 2024-01-18 18:37:36 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T18:37:36.730Z] 2024-01-18 18:37:36 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T18:37:36.730Z] 2024-01-18 18:37:36 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T18:37:36.730Z] 2024-01-18 18:37:36 +0000:[INFO] [{VerifyRBACForAppUser}] Create cluster [source-cluster] in org [default] [2024-01-18T18:37:36.730Z] 2024-01-18 18:37:36 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T18:37:36.730Z] 2024-01-18 18:37:36 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T18:37:36.730Z] 2024-01-18 18:37:36 +0000:[INFO] [{VerifyRBACForAppUser}] Create cluster [destination-cluster] in org [default] [2024-01-18T18:37:36.730Z] 2024-01-18 18:37:36 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T18:37:36.730Z] 2024-01-18 18:37:36 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T18:37:36.730Z] 2024-01-18 18:37:36 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-dldli] for IBM [2024-01-18T18:37:36.730Z] 2024-01-18 18:37:36 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-dldli] for org [default] with [ibm] as provider [2024-01-18T18:37:36.730Z] 2024-01-18 18:37:36 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T18:37:36.730Z] 2024-01-18 18:37:36 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T18:37:36.986Z] 2024-01-18 18:37:36 +0000:[WARNING] [{VerifyRBACForAppUser}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-dldli] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user]] [2024-01-18T18:37:36.986Z] 2024-01-18 18:37:36 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user] [2024-01-18T18:37:38.350Z] 2024-01-18 18:37:38 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-dldli from admin context and sharing with all the users [2024-01-18T18:37:38.350Z] 2024-01-18 18:37:38 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-dldli] for org [default] with [ibm] as provider [2024-01-18T18:37:38.350Z] 2024-01-18 18:37:38 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T18:37:38.350Z] 2024-01-18 18:37:38 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T18:37:38.350Z] 2024-01-18 18:37:38 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-dldli] is present [2024-01-18T18:37:38.605Z] 2024-01-18 18:37:38 +0000:[INFO] [{VerifyRBACForAppUser}] Create cluster [source-cluster] in org [default] [2024-01-18T18:37:39.166Z] 2024-01-18 18:37:38 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-exwrq] for IBM [2024-01-18T18:37:39.166Z] 2024-01-18 18:37:38 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-exwrq] for org [default] with [ibm] as provider [2024-01-18T18:37:39.166Z] 2024-01-18 18:37:38 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T18:37:39.166Z] 2024-01-18 18:37:38 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T18:37:39.166Z] 2024-01-18 18:37:38 +0000:[WARNING] [{VerifyRBACForAppUser}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-exwrq] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user]] [2024-01-18T18:37:39.166Z] 2024-01-18 18:37:38 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user] [2024-01-18T18:37:39.726Z] 2024-01-18 18:37:39 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-exwrq from admin context and sharing with all the users [2024-01-18T18:37:39.726Z] 2024-01-18 18:37:39 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-exwrq] for org [default] with [ibm] as provider [2024-01-18T18:37:39.726Z] 2024-01-18 18:37:39 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T18:37:39.726Z] 2024-01-18 18:37:39 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T18:37:39.726Z] 2024-01-18 18:37:39 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-exwrq] is present [2024-01-18T18:37:39.982Z] 2024-01-18 18:37:39 +0000:[INFO] [{VerifyRBACForAppUser}] Create cluster [destination-cluster] in org [default] [2024-01-18T18:37:40.544Z] INFO[2024-01-18 18:37:40] Verifying : Description : Verifying creation of source [source-cluster] and destination [destination-cluster] clusters with App-User ctx [2024-01-18T18:37:40.544Z] INFO[2024-01-18 18:37:40] Actual:true, Expected: true [2024-01-18T18:37:40.544Z] INFO[2024-01-18 18:37:40] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-18T18:37:40.544Z] INFO[2024-01-18 18:37:40] Actual:Online, Expected: Online [2024-01-18T18:37:40.544Z] INFO[2024-01-18 18:37:40] Verifying : Description : Verifying if [destination-cluster] cluster is online [2024-01-18T18:37:40.544Z] INFO[2024-01-18 18:37:40] Actual:Online, Expected: Online [2024-01-18T18:37:40.544Z] 2024-01-18 18:37:40 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.glob..func46.2.13:#1880] - Cluster [source-cluster] uid: [e569b38d-0af9-4019-afa6-bf33943bc2c1] [2024-01-18T18:37:40.544Z] 2024-01-18 18:37:40 +0000:[INFO] [{VerifyRBACForAppUser}] Validate taking a scheduled backup of applications from the App-User [testuser1-1705603039] [2024-01-18T18:37:40.545Z] 2024-01-18 18:37:40 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-18T18:37:40.545Z] 2024-01-18 18:37:40 +0000:[INFO] [{VerifyRBACForAppUser}] Volume snapshot classes found in the cluster - [2024-01-18T18:37:40.545Z] 2024-01-18 18:37:40 +0000:[INFO] [{VerifyRBACForAppUser}] ibmc-vpcblock-snapshot [2024-01-18T18:37:40.545Z] 2024-01-18 18:37:40 +0000:[INFO] [{VerifyRBACForAppUser}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T18:38:48.164Z] 2024-01-18 18:38:41 +0000:[INFO] [{VerifyRBACForAppUser}] first schedule backup for schedule name [backup-schedule-vqju] is [backup-schedule-vqju-f1b8514-interval-2024-01-18-183740] [2024-01-18T18:38:48.164Z] 2024/01/18 18:38:41 DoRetryWithTimeout - Error: {backup status for [backup-schedule-vqju-f1b8514-interval-2024-01-18-183740] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T18:39:14.656Z] 2024/01/18 18:39:11 DoRetryWithTimeout - Error: {backup status for [backup-schedule-vqju-f1b8514-interval-2024-01-18-183740] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T18:39:41.215Z] 2024-01-18 18:39:41 +0000:[INFO] [{VerifyRBACForAppUser}] Validating backup [backup-schedule-vqju-f1b8514-interval-2024-01-18-183740] in org [default] [2024-01-18T18:39:41.215Z] 2024-01-18 18:39:41 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [backup-schedule-vqju-f1b8514-interval-2024-01-18-183740] [2024-01-18T18:39:41.215Z] 2024-01-18 18:39:41 +0000:[INFO] [{VerifyRBACForAppUser}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87889-01-18-18h35m37s] in backup [backup-schedule-vqju-f1b8514-interval-2024-01-18-183740] [2024-01-18T18:39:41.215Z] 2024-01-18 18:39:41 +0000:[INFO] [{VerifyRBACForAppUser}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87889-01-18-18h35m37s] in backup [backup-schedule-vqju-f1b8514-interval-2024-01-18-183740] [2024-01-18T18:39:41.215Z] 2024-01-18 18:39:41 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:39:41.480Z] 2024-01-18 18:39:41 +0000:[WARNING] [{VerifyRBACForAppUser}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T18:39:41.480Z] 2024-01-18 18:39:41 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T18:39:41.480Z] 2024-01-18 18:39:41 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87889-01-18-18h35m37s] are [[mysql-data]] [2024-01-18T18:39:41.480Z] 2024-01-18 18:39:41 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T18:39:41.480Z] 2024-01-18 18:39:41 +0000:[INFO] [{VerifyRBACForAppUser}] Validating CR cleanup [2024-01-18T18:39:41.480Z] 2024-01-18 18:39:41 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T18:39:41.480Z] 2024-01-18 18:39:41 +0000:[INFO] [{VerifyRBACForAppUser}] All CRs in [mysql-ibm-pxb-0-87889-01-18-18h35m37s] are [[backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]] [2024-01-18T18:39:41.480Z] 2024-01-18 18:39:41 +0000:[INFO] [{VerifyRBACForAppUser}] CR found for [[backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]] under [mysql-ibm-pxb-0-87889-01-18-18h35m37s] namespace [2024-01-18T18:39:41.480Z] 2024/01/18 18:39:41 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]}, Next try in [5s], timeout [5m0s] [2024-01-18T18:39:46.723Z] 2024-01-18 18:39:46 +0000:[INFO] [{VerifyRBACForAppUser}] All CRs in [mysql-ibm-pxb-0-87889-01-18-18h35m37s] are [[backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]] [2024-01-18T18:39:46.723Z] 2024-01-18 18:39:46 +0000:[INFO] [{VerifyRBACForAppUser}] CR found for [[backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]] under [mysql-ibm-pxb-0-87889-01-18-18h35m37s] namespace [2024-01-18T18:39:46.723Z] 2024/01/18 18:39:46 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]}, Next try in [5s], timeout [5m0s] [2024-01-18T18:39:51.971Z] 2024-01-18 18:39:51 +0000:[INFO] [{VerifyRBACForAppUser}] All CRs in [mysql-ibm-pxb-0-87889-01-18-18h35m37s] are [[backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]] [2024-01-18T18:39:51.971Z] 2024-01-18 18:39:51 +0000:[INFO] [{VerifyRBACForAppUser}] CR found for [[backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]] under [mysql-ibm-pxb-0-87889-01-18-18h35m37s] namespace [2024-01-18T18:39:51.971Z] 2024/01/18 18:39:51 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]}, Next try in [5s], timeout [5m0s] [2024-01-18T18:39:57.218Z] 2024-01-18 18:39:56 +0000:[INFO] [{VerifyRBACForAppUser}] All CRs in [mysql-ibm-pxb-0-87889-01-18-18h35m37s] are [[backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]] [2024-01-18T18:39:57.219Z] 2024-01-18 18:39:56 +0000:[INFO] [{VerifyRBACForAppUser}] CR found for [[backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]] under [mysql-ibm-pxb-0-87889-01-18-18h35m37s] namespace [2024-01-18T18:39:57.219Z] 2024/01/18 18:39:56 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]}, Next try in [5s], timeout [5m0s] [2024-01-18T18:40:02.458Z] 2024-01-18 18:40:01 +0000:[INFO] [{VerifyRBACForAppUser}] All CRs in [mysql-ibm-pxb-0-87889-01-18-18h35m37s] are [[backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]] [2024-01-18T18:40:02.458Z] 2024-01-18 18:40:01 +0000:[INFO] [{VerifyRBACForAppUser}] CR found for [[backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]] under [mysql-ibm-pxb-0-87889-01-18-18h35m37s] namespace [2024-01-18T18:40:02.458Z] 2024/01/18 18:40:01 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]}, Next try in [5s], timeout [5m0s] [2024-01-18T18:40:06.628Z] 2024-01-18 18:40:06 +0000:[INFO] [{VerifyRBACForAppUser}] All CRs in [mysql-ibm-pxb-0-87889-01-18-18h35m37s] are [[backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]] [2024-01-18T18:40:06.629Z] 2024-01-18 18:40:06 +0000:[INFO] [{VerifyRBACForAppUser}] CR found for [[backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]] under [mysql-ibm-pxb-0-87889-01-18-18h35m37s] namespace [2024-01-18T18:40:06.629Z] 2024/01/18 18:40:06 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]}, Next try in [5s], timeout [5m0s] [2024-01-18T18:40:11.881Z] 2024-01-18 18:40:11 +0000:[INFO] [{VerifyRBACForAppUser}] All CRs in [mysql-ibm-pxb-0-87889-01-18-18h35m37s] are [[backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]] [2024-01-18T18:40:11.881Z] 2024-01-18 18:40:11 +0000:[INFO] [{VerifyRBACForAppUser}] CR found for [[backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]] under [mysql-ibm-pxb-0-87889-01-18-18h35m37s] namespace [2024-01-18T18:40:11.881Z] 2024/01/18 18:40:11 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]}, Next try in [5s], timeout [5m0s] [2024-01-18T18:40:17.121Z] 2024-01-18 18:40:16 +0000:[INFO] [{VerifyRBACForAppUser}] All CRs in [mysql-ibm-pxb-0-87889-01-18-18h35m37s] are [[backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]] [2024-01-18T18:40:17.122Z] 2024-01-18 18:40:16 +0000:[INFO] [{VerifyRBACForAppUser}] CR found for [[backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]] under [mysql-ibm-pxb-0-87889-01-18-18h35m37s] namespace [2024-01-18T18:40:17.122Z] 2024/01/18 18:40:16 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]}, Next try in [5s], timeout [5m0s] [2024-01-18T18:40:22.418Z] 2024-01-18 18:40:21 +0000:[INFO] [{VerifyRBACForAppUser}] All CRs in [mysql-ibm-pxb-0-87889-01-18-18h35m37s] are [[backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]] [2024-01-18T18:40:22.418Z] 2024-01-18 18:40:21 +0000:[INFO] [{VerifyRBACForAppUser}] CR found for [[backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]] under [mysql-ibm-pxb-0-87889-01-18-18h35m37s] namespace [2024-01-18T18:40:22.418Z] 2024/01/18 18:40:21 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]}, Next try in [5s], timeout [5m0s] [2024-01-18T18:40:27.677Z] 2024-01-18 18:40:26 +0000:[INFO] [{VerifyRBACForAppUser}] All CRs in [mysql-ibm-pxb-0-87889-01-18-18h35m37s] are [[backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]] [2024-01-18T18:40:27.677Z] 2024-01-18 18:40:26 +0000:[INFO] [{VerifyRBACForAppUser}] CR found for [[backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]] under [mysql-ibm-pxb-0-87889-01-18-18h35m37s] namespace [2024-01-18T18:40:27.677Z] 2024/01/18 18:40:26 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]}, Next try in [5s], timeout [5m0s] [2024-01-18T18:40:31.845Z] 2024-01-18 18:40:31 +0000:[INFO] [{VerifyRBACForAppUser}] All CRs in [mysql-ibm-pxb-0-87889-01-18-18h35m37s] are [[backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]] [2024-01-18T18:40:31.845Z] 2024-01-18 18:40:31 +0000:[INFO] [{VerifyRBACForAppUser}] CR found for [[backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]] under [mysql-ibm-pxb-0-87889-01-18-18h35m37s] namespace [2024-01-18T18:40:31.845Z] 2024/01/18 18:40:31 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]}, Next try in [5s], timeout [5m0s] [2024-01-18T18:40:37.135Z] 2024-01-18 18:40:36 +0000:[INFO] [{VerifyRBACForAppUser}] All CRs in [mysql-ibm-pxb-0-87889-01-18-18h35m37s] are [[backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]] [2024-01-18T18:40:37.135Z] 2024-01-18 18:40:36 +0000:[INFO] [{VerifyRBACForAppUser}] CR found for [[backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]] under [mysql-ibm-pxb-0-87889-01-18-18h35m37s] namespace [2024-01-18T18:40:37.135Z] 2024/01/18 18:40:36 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]}, Next try in [5s], timeout [5m0s] [2024-01-18T18:40:42.376Z] 2024-01-18 18:40:41 +0000:[INFO] [{VerifyRBACForAppUser}] All CRs in [mysql-ibm-pxb-0-87889-01-18-18h35m37s] are [[backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]] [2024-01-18T18:40:42.376Z] 2024-01-18 18:40:41 +0000:[INFO] [{VerifyRBACForAppUser}] CR found for [[backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]] under [mysql-ibm-pxb-0-87889-01-18-18h35m37s] namespace [2024-01-18T18:40:42.376Z] 2024/01/18 18:40:41 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-vqju-f1b8514-interval-2024-01-18-183740]}, Next try in [5s], timeout [5m0s] [2024-01-18T18:40:47.618Z] 2024-01-18 18:40:46 +0000:[INFO] [{VerifyRBACForAppUser}] All CRs in [mysql-ibm-pxb-0-87889-01-18-18h35m37s] are [[]] [2024-01-18T18:40:47.618Z] INFO[2024-01-18 18:40:46] Verifying : Description : Verifying creation and validation of schedule backup with schedule name [backup-schedule-vqju] [2024-01-18T18:40:47.618Z] INFO[2024-01-18 18:40:46] Actual:true, Expected: true [2024-01-18T18:40:47.618Z] INFO[2024-01-18 18:40:46] Verifying : Description : Suspending Backup Schedule [backup-schedule-vqju] for user [testuser1-1705603039] [2024-01-18T18:40:47.618Z] INFO[2024-01-18 18:40:46] Actual:true, Expected: true [2024-01-18T18:40:47.618Z] 2024-01-18 18:40:46 +0000:[INFO] [{VerifyRBACForAppUser}] Validate restoring backups on destination cluster for the App-User [testuser1-1705603039] [2024-01-18T18:40:47.876Z] 2024/01/18 18:40:47 DoRetryWithTimeout - Error: {restore status for [tp-restore-backup-schedule-vqju-f1b8514-interval-2024-01-18-183740] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T18:41:19.907Z] 2024-01-18 18:41:17 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.CreateRestore:#936] - Restore [tp-restore-backup-schedule-vqju-f1b8514-interval-2024-01-18-183740] created successfully [2024-01-18T18:41:19.907Z] 2024-01-18 18:41:17 +0000:[INFO] [{VerifyRBACForAppUser}] Switching cluster context to cluster [destination-cluster] [2024-01-18T18:41:19.907Z] 2024-01-18 18:41:17 +0000:[INFO] [{VerifyRBACForAppUser}] Switching context to [/tmp/destination-config] [2024-01-18T18:41:19.907Z] 2024-01-18 18:41:17 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:41:19.907Z] 2024-01-18 18:41:18 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:41:19.907Z] 2024-01-18 18:41:18 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:41:19.907Z] 2024-01-18 18:41:18 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:41:19.907Z] 2024-01-18 18:41:18 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:41:19.907Z] 2024-01-18 18:41:18 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:41:19.907Z] 2024-01-18 18:41:18 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:41:19.907Z] 2024-01-18 18:41:19 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:41:19.907Z] 2024-01-18 18:41:19 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:41:19.907Z] 2024-01-18 18:41:19 +0000:[WARNING] [{VerifyRBACForAppUser}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T18:41:20.164Z] 2024-01-18 18:41:19 +0000:[WARNING] [{VerifyRBACForAppUser}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T18:41:20.164Z] 2024-01-18 18:41:19 +0000:[WARNING] [{VerifyRBACForAppUser}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T18:41:20.164Z] 2024-01-18 18:41:19 +0000:[INFO] [{VerifyRBACForAppUser}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T18:41:20.164Z] 2024-01-18 18:41:19 +0000:[INFO] [{VerifyRBACForAppUser}] Switched context to [/tmp/destination-config] [2024-01-18T18:41:20.164Z] 2024-01-18 18:41:19 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-87889-01-18-18h35m37s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-18T18:41:20.165Z] 2024-01-18 18:41:19 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:41:20.165Z] 2024-01-18 18:41:19 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T18:41:20.165Z] 2024-01-18 18:41:19 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T18:41:20.165Z] 2024-01-18 18:41:19 +0000:[INFO] [{VerifyRBACForAppUser}] Validating restore [tp-restore-backup-schedule-vqju-f1b8514-interval-2024-01-18-183740] in org [default] [2024-01-18T18:41:20.165Z] 2024-01-18 18:41:19 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-restore-backup-schedule-vqju-f1b8514-interval-2024-01-18-183740] [2024-01-18T18:41:20.165Z] 2024-01-18 18:41:19 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-restore-backup-schedule-vqju-f1b8514-interval-2024-01-18-183740] [2024-01-18T18:41:20.165Z] 2024-01-18 18:41:19 +0000:[INFO] [{VerifyRBACForAppUser}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-87889-01-18-18h35m37s] in restore [tp-restore-backup-schedule-vqju-f1b8514-interval-2024-01-18-183740] [2024-01-18T18:41:20.165Z] 2024-01-18 18:41:19 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-87889-01-18-18h35m37s]) was found in restore [tp-restore-backup-schedule-vqju-f1b8514-interval-2024-01-18-183740], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:41:20.165Z] 2024-01-18 18:41:20 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-87889-01-18-18h35m37s]) found in the restore [tp-restore-backup-schedule-vqju-f1b8514-interval-2024-01-18-183740] was also present on the cluster/namespace [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:41:20.166Z] 2024-01-18 18:41:20 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-87889-01-18-18h35m37s]) was found in restore [tp-restore-backup-schedule-vqju-f1b8514-interval-2024-01-18-183740], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:41:20.421Z] 2024-01-18 18:41:20 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-87889-01-18-18h35m37s]) found in the restore [tp-restore-backup-schedule-vqju-f1b8514-interval-2024-01-18-183740] was also present on the cluster/namespace [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:41:20.421Z] 2024-01-18 18:41:20 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-87889-01-18-18h35m37s]) was found in restore [tp-restore-backup-schedule-vqju-f1b8514-interval-2024-01-18-183740], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:41:20.421Z] 2024-01-18 18:41:20 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-87889-01-18-18h35m37s]) found in the restore [tp-restore-backup-schedule-vqju-f1b8514-interval-2024-01-18-183740] was also present on the cluster/namespace [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:41:20.421Z] 2024-01-18 18:41:20 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-87889-01-18-18h35m37s]) was found in restore [tp-restore-backup-schedule-vqju-f1b8514-interval-2024-01-18-183740], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:41:20.422Z] 2024-01-18 18:41:20 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-87889-01-18-18h35m37s]) found in the restore [tp-restore-backup-schedule-vqju-f1b8514-interval-2024-01-18-183740] was also present on the cluster/namespace [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:41:20.422Z] 2024-01-18 18:41:20 +0000:[INFO] [{VerifyRBACForAppUser}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-87889-01-18-18h35m37s] in restore [tp-restore-backup-schedule-vqju-f1b8514-interval-2024-01-18-183740] [2024-01-18T18:41:20.422Z] 2024-01-18 18:41:20 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-87889-01-18-18h35m37s] in restore [tp-restore-backup-schedule-vqju-f1b8514-interval-2024-01-18-183740] [2024-01-18T18:41:20.678Z] 2024-01-18 18:41:20 +0000:[WARNING] [{VerifyRBACForAppUser}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T18:41:20.678Z] 2024-01-18 18:41:20 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T18:41:20.678Z] 2024-01-18 18:41:20 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87889-01-18-18h35m37s] are [[mysql-data]] [2024-01-18T18:41:20.678Z] 2024-01-18 18:41:20 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2240] - in restore [tp-restore-backup-schedule-vqju-f1b8514-interval-2024-01-18-183740], the status of the restored volume [pvc-32a85f6b-1b18-4370-8967-fd791a2042b3] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-18T18:41:20.678Z] 2024-01-18 18:41:20 +0000:[INFO] [{VerifyRBACForAppUser}] Validate applications in restored namespace [mysql-ibm-pxb-0-87889-01-18-18h35m37s] due to restore [tp-restore-backup-schedule-vqju-f1b8514-interval-2024-01-18-183740] [2024-01-18T18:41:20.679Z] 2024-01-18 18:41:20 +0000:[INFO] [{VerifyRBACForAppUser}] Validating mysql-ibm app [2024-01-18T18:41:20.679Z] 2024-01-18 18:41:20 +0000:[INFO] [{VerifyRBACForAppUser}] wait for mysql-ibm app to start running [2024-01-18T18:41:20.976Z] 2024-01-18 18:41:20 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T18:41:20.976Z] 2024/01/18 18:41:20 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:41:20.976Z] pod name:mysql-7d766cd65c-h7srd namespace:mysql-ibm-pxb-0-87889-01-18-18h35m37s running:false ready:false node:10.240.64.7 [2024-01-18T18:41:20.976Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:41:30.940Z] 2024/01/18 18:41:30 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:41:30.940Z] pod name:mysql-7d766cd65c-h7srd namespace:mysql-ibm-pxb-0-87889-01-18-18h35m37s running:false ready:false node:10.240.64.7 [2024-01-18T18:41:30.940Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:41:43.110Z] 2024/01/18 18:41:40 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:41:43.110Z] pod name:mysql-7d766cd65c-h7srd namespace:mysql-ibm-pxb-0-87889-01-18-18h35m37s running:true ready:false node:10.240.64.7 [2024-01-18T18:41:43.110Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:41:51.243Z] 2024/01/18 18:41:51 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:41:51.244Z] pod name:mysql-7d766cd65c-h7srd namespace:mysql-ibm-pxb-0-87889-01-18-18h35m37s running:true ready:false node:10.240.64.7 [2024-01-18T18:41:51.244Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:42:01.188Z] 2024/01/18 18:42:01 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:42:01.188Z] pod name:mysql-7d766cd65c-h7srd namespace:mysql-ibm-pxb-0-87889-01-18-18h35m37s running:true ready:false node:10.240.64.7 [2024-01-18T18:42:01.188Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:42:13.356Z] 2024/01/18 18:42:11 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:42:13.356Z] pod name:mysql-7d766cd65c-h7srd namespace:mysql-ibm-pxb-0-87889-01-18-18h35m37s running:true ready:false node:10.240.64.7 [2024-01-18T18:42:13.356Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:42:21.492Z] 2024/01/18 18:42:21 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:42:21.492Z] pod name:mysql-7d766cd65c-h7srd namespace:mysql-ibm-pxb-0-87889-01-18-18h35m37s running:true ready:false node:10.240.64.7 [2024-01-18T18:42:21.492Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:42:33.664Z] 2024/01/18 18:42:31 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:42:33.664Z] pod name:mysql-7d766cd65c-h7srd namespace:mysql-ibm-pxb-0-87889-01-18-18h35m37s running:true ready:false node:10.240.64.7 [2024-01-18T18:42:33.664Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:42:41.791Z] 2024/01/18 18:42:41 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:42:41.791Z] pod name:mysql-7d766cd65c-h7srd namespace:mysql-ibm-pxb-0-87889-01-18-18h35m37s running:true ready:false node:10.240.64.7 [2024-01-18T18:42:41.791Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:42:51.742Z] 2024-01-18 18:42:51 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T18:42:51.998Z] 2024-01-18 18:42:51 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T18:42:51.998Z] 2024-01-18 18:42:51 +0000:[INFO] [{VerifyRBACForAppUser}] Validating CR cleanup [2024-01-18T18:42:52.254Z] 2024-01-18 18:42:52 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T18:42:52.254Z] 2024-01-18 18:42:52 +0000:[INFO] [{VerifyRBACForAppUser}] All CRs in [mysql-ibm-pxb-0-87889-01-18-18h35m37s] are [[]] [2024-01-18T18:42:52.254Z] 2024-01-18 18:42:52 +0000:[INFO] [{VerifyRBACForAppUser}] Switching cluster context back to cluster path [/tmp/source-config] [2024-01-18T18:42:52.254Z] 2024-01-18 18:42:52 +0000:[INFO] [{VerifyRBACForAppUser}] Switching context to [/tmp/source-config] [2024-01-18T18:42:52.511Z] 2024-01-18 18:42:52 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:42:52.511Z] 2024-01-18 18:42:52 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:42:52.511Z] 2024-01-18 18:42:52 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:42:52.511Z] 2024-01-18 18:42:52 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:42:52.511Z] 2024-01-18 18:42:52 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:42:52.766Z] 2024-01-18 18:42:52 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:42:53.023Z] 2024-01-18 18:42:52 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:42:53.584Z] 2024-01-18 18:42:53 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:42:53.840Z] 2024-01-18 18:42:53 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:42:54.099Z] 2024-01-18 18:42:53 +0000:[WARNING] [{VerifyRBACForAppUser}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T18:42:54.356Z] 2024-01-18 18:42:54 +0000:[WARNING] [{VerifyRBACForAppUser}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T18:42:54.356Z] 2024-01-18 18:42:54 +0000:[WARNING] [{VerifyRBACForAppUser}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T18:42:54.356Z] 2024-01-18 18:42:54 +0000:[INFO] [{VerifyRBACForAppUser}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T18:42:54.356Z] 2024-01-18 18:42:54 +0000:[INFO] [{VerifyRBACForAppUser}] Switched context to [/tmp/source-config] [2024-01-18T18:42:54.356Z] INFO[2024-01-18 18:42:54] Verifying : Description : Verifying creation of restore tp-restore-backup-schedule-vqju-f1b8514-interval-2024-01-18-183740 of backup backup-schedule-vqju-f1b8514-interval-2024-01-18-183740 [2024-01-18T18:42:54.356Z] INFO[2024-01-18 18:42:54] Actual:true, Expected: true [2024-01-18T18:42:54.356Z] 2024-01-18 18:42:54 +0000:[INFO] [{VerifyRBACForAppUser}] Validate taking manual backup of applications with namespace label [2024-01-18T18:42:54.918Z] 2024-01-18 18:42:54 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T18:42:54.918Z] 2024-01-18 18:42:54 +0000:[INFO] [{VerifyRBACForAppUser}] Volume snapshot classes found in the cluster - [2024-01-18T18:42:54.918Z] 2024-01-18 18:42:54 +0000:[INFO] [{VerifyRBACForAppUser}] ibmc-vpcblock-snapshot [2024-01-18T18:42:54.918Z] 2024-01-18 18:42:54 +0000:[INFO] [{VerifyRBACForAppUser}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T18:42:55.173Z] 2024/01/18 18:42:55 DoRetryWithTimeout - Error: {backup status for [backup-avmo] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T18:43:27.218Z] 2024/01/18 18:43:25 DoRetryWithTimeout - Error: {backup status for [backup-avmo] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T18:43:59.261Z] 2024/01/18 18:43:55 DoRetryWithTimeout - Error: {backup status for [backup-avmo] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T18:44:17.345Z] INFO[2024-01-18 18:44:14] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T18:44:17.345Z] INFO[2024-01-18 18:44:14] sched stats: tick interval: 11414 < 1.2s (100%) [2024-01-18T18:44:17.345Z] INFO[2024-01-18 18:44:14] sched stats: tick process durations: 11413 < 500ms (100%) [2024-01-18T18:44:25.435Z] 2024-01-18 18:44:25 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.CreateBackupWithNamespaceLabel:#3030] - Successfully created backup [backup-avmo] with namespace label [0-864abd34-8f71-49db-a527-f7eea2e4f188=864abd34-8f71-49db-a527-f7eea2e4f188,1-864abd34-8f71-49db-a527-f7eea2e4f188=864abd34-8f71-49db-a527-f7eea2e4f188] [2024-01-18T18:44:25.435Z] 2024-01-18 18:44:25 +0000:[INFO] [{VerifyRBACForAppUser}] Validating backup [backup-avmo] in org [default] [2024-01-18T18:44:25.435Z] 2024-01-18 18:44:25 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [backup-avmo] [2024-01-18T18:44:25.435Z] 2024-01-18 18:44:25 +0000:[INFO] [{VerifyRBACForAppUser}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87889-01-18-18h35m37s] in backup [backup-avmo] [2024-01-18T18:44:25.435Z] 2024-01-18 18:44:25 +0000:[INFO] [{VerifyRBACForAppUser}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87889-01-18-18h35m37s] in backup [backup-avmo] [2024-01-18T18:44:25.435Z] 2024-01-18 18:44:25 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:44:25.435Z] 2024-01-18 18:44:25 +0000:[WARNING] [{VerifyRBACForAppUser}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T18:44:25.435Z] 2024-01-18 18:44:25 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T18:44:25.435Z] 2024-01-18 18:44:25 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87889-01-18-18h35m37s] are [[mysql-data]] [2024-01-18T18:44:25.435Z] 2024-01-18 18:44:25 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T18:44:25.435Z] 2024-01-18 18:44:25 +0000:[INFO] [{VerifyRBACForAppUser}] Validating CR cleanup [2024-01-18T18:44:25.694Z] 2024-01-18 18:44:25 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T18:44:25.694Z] 2024-01-18 18:44:25 +0000:[INFO] [{VerifyRBACForAppUser}] All CRs in [mysql-ibm-pxb-0-87889-01-18-18h35m37s] are [[]] [2024-01-18T18:44:25.694Z] INFO[2024-01-18 18:44:25] Verifying : Description : Verifying backup [backup-avmo] creation with labels [0-864abd34-8f71-49db-a527-f7eea2e4f188=864abd34-8f71-49db-a527-f7eea2e4f188,1-864abd34-8f71-49db-a527-f7eea2e4f188=864abd34-8f71-49db-a527-f7eea2e4f188] [2024-01-18T18:44:25.694Z] INFO[2024-01-18 18:44:25] Actual:true, Expected: true [2024-01-18T18:44:25.694Z] 2024-01-18 18:44:25 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup backup-avmo [2024-01-18T18:44:25.694Z] 2024-01-18 18:44:25 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:44:25.694Z] 2024-01-18 18:44:25 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are 0-864abd34-8f71-49db-a527-f7eea2e4f188=864abd34-8f71-49db-a527-f7eea2e4f188,1-864abd34-8f71-49db-a527-f7eea2e4f188=864abd34-8f71-49db-a527-f7eea2e4f188 [2024-01-18T18:44:25.695Z] INFO[2024-01-18 18:44:25] Verifying : Description : Verifying if the labeled namespaces [[mysql-ibm-pxb-0-87889-01-18-18h35m37s]] are backed up, and check if labels [0-864abd34-8f71-49db-a527-f7eea2e4f188=864abd34-8f71-49db-a527-f7eea2e4f188,1-864abd34-8f71-49db-a527-f7eea2e4f188=864abd34-8f71-49db-a527-f7eea2e4f188] are applied to backups [backup-avmo] [2024-01-18T18:44:25.695Z] INFO[2024-01-18 18:44:25] Actual:true, Expected: true [2024-01-18T18:44:25.695Z] 2024-01-18 18:44:25 +0000:[INFO] [{VerifyRBACForAppUser}] Validate restoring manual backup of applications with namespace label [2024-01-18T18:44:26.256Z] 2024/01/18 18:44:26 DoRetryWithTimeout - Error: {restore status for [tp-restore-backup-avmo] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T18:44:58.284Z] 2024-01-18 18:44:56 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.CreateRestore:#936] - Restore [tp-restore-backup-avmo] created successfully [2024-01-18T18:44:58.284Z] 2024-01-18 18:44:56 +0000:[INFO] [{VerifyRBACForAppUser}] Switching cluster context to cluster [destination-cluster] [2024-01-18T18:44:58.284Z] 2024-01-18 18:44:56 +0000:[INFO] [{VerifyRBACForAppUser}] Switching context to [/tmp/destination-config] [2024-01-18T18:44:58.284Z] 2024-01-18 18:44:56 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:44:58.284Z] 2024-01-18 18:44:56 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:44:58.284Z] 2024-01-18 18:44:56 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:44:58.284Z] 2024-01-18 18:44:56 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:44:58.284Z] 2024-01-18 18:44:56 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:44:58.284Z] 2024-01-18 18:44:56 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:44:58.284Z] 2024-01-18 18:44:56 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:44:58.284Z] 2024-01-18 18:44:57 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:44:58.284Z] 2024-01-18 18:44:57 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:44:58.284Z] 2024-01-18 18:44:57 +0000:[WARNING] [{VerifyRBACForAppUser}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T18:44:58.284Z] 2024-01-18 18:44:58 +0000:[WARNING] [{VerifyRBACForAppUser}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T18:44:58.284Z] 2024-01-18 18:44:58 +0000:[WARNING] [{VerifyRBACForAppUser}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T18:44:58.284Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T18:44:58.284Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] Switched context to [/tmp/destination-config] [2024-01-18T18:44:58.284Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-87889-01-18-18h35m37s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-18T18:44:58.284Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:44:58.284Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T18:44:58.285Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T18:44:58.285Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] Validating restore [tp-restore-backup-avmo] in org [default] [2024-01-18T18:44:58.285Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-restore-backup-avmo] [2024-01-18T18:44:58.285Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-restore-backup-avmo] [2024-01-18T18:44:58.285Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-87889-01-18-18h35m37s] in restore [tp-restore-backup-avmo] [2024-01-18T18:44:58.285Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-87889-01-18-18h35m37s]) was found in restore [tp-restore-backup-avmo], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:44:58.542Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-87889-01-18-18h35m37s]) found in the restore [tp-restore-backup-avmo] was also present on the cluster/namespace [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:44:58.542Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-87889-01-18-18h35m37s]) was found in restore [tp-restore-backup-avmo], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:44:58.798Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-87889-01-18-18h35m37s]) found in the restore [tp-restore-backup-avmo] was also present on the cluster/namespace [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:44:58.798Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-87889-01-18-18h35m37s]) was found in restore [tp-restore-backup-avmo], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:44:58.798Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-87889-01-18-18h35m37s]) found in the restore [tp-restore-backup-avmo] was also present on the cluster/namespace [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:44:58.798Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-87889-01-18-18h35m37s]) was found in restore [tp-restore-backup-avmo], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:44:58.798Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-87889-01-18-18h35m37s]) found in the restore [tp-restore-backup-avmo] was also present on the cluster/namespace [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:44:58.798Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-87889-01-18-18h35m37s] in restore [tp-restore-backup-avmo] [2024-01-18T18:44:58.798Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-87889-01-18-18h35m37s] in restore [tp-restore-backup-avmo] [2024-01-18T18:44:58.798Z] 2024-01-18 18:44:58 +0000:[WARNING] [{VerifyRBACForAppUser}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T18:44:58.798Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T18:44:58.798Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87889-01-18-18h35m37s] are [[mysql-data]] [2024-01-18T18:44:58.798Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2243] - in restore [tp-restore-backup-avmo], the status of the restored volume [pvc-32a85f6b-1b18-4370-8967-fd791a2042b3] was not Success. It was [Retained] with reason [Skipped from volume restore as policy is set to Retain and pvc already exists] [2024-01-18T18:44:58.798Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] Validate applications in restored namespace [mysql-ibm-pxb-0-87889-01-18-18h35m37s] due to restore [tp-restore-backup-avmo] [2024-01-18T18:44:58.799Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] Validating mysql-ibm app [2024-01-18T18:44:58.799Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] wait for mysql-ibm app to start running [2024-01-18T18:44:59.056Z] 2024-01-18 18:44:58 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T18:44:59.056Z] 2024-01-18 18:44:59 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T18:44:59.312Z] 2024-01-18 18:44:59 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T18:44:59.568Z] 2024-01-18 18:44:59 +0000:[INFO] [{VerifyRBACForAppUser}] Validating CR cleanup [2024-01-18T18:44:59.824Z] 2024-01-18 18:44:59 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T18:44:59.824Z] 2024-01-18 18:44:59 +0000:[INFO] [{VerifyRBACForAppUser}] All CRs in [mysql-ibm-pxb-0-87889-01-18-18h35m37s] are [[]] [2024-01-18T18:44:59.824Z] 2024-01-18 18:44:59 +0000:[INFO] [{VerifyRBACForAppUser}] Switching cluster context back to cluster path [/tmp/source-config] [2024-01-18T18:44:59.824Z] 2024-01-18 18:44:59 +0000:[INFO] [{VerifyRBACForAppUser}] Switching context to [/tmp/source-config] [2024-01-18T18:44:59.824Z] 2024-01-18 18:44:59 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:44:59.824Z] 2024-01-18 18:44:59 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:44:59.824Z] 2024-01-18 18:44:59 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:45:00.080Z] 2024-01-18 18:44:59 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:45:00.080Z] 2024-01-18 18:44:59 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:45:00.080Z] 2024-01-18 18:45:00 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:45:00.643Z] 2024-01-18 18:45:00 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:45:00.900Z] 2024-01-18 18:45:00 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:45:01.462Z] 2024-01-18 18:45:01 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:45:01.719Z] 2024-01-18 18:45:01 +0000:[WARNING] [{VerifyRBACForAppUser}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T18:45:01.719Z] 2024-01-18 18:45:01 +0000:[WARNING] [{VerifyRBACForAppUser}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T18:45:01.719Z] 2024-01-18 18:45:01 +0000:[WARNING] [{VerifyRBACForAppUser}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T18:45:01.719Z] 2024-01-18 18:45:01 +0000:[INFO] [{VerifyRBACForAppUser}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T18:45:01.719Z] 2024-01-18 18:45:01 +0000:[INFO] [{VerifyRBACForAppUser}] Switched context to [/tmp/source-config] [2024-01-18T18:45:01.719Z] INFO[2024-01-18 18:45:01] Verifying : Description : Verifying restoration of backup backup-avmo [2024-01-18T18:45:01.719Z] INFO[2024-01-18 18:45:01] Actual:true, Expected: true [2024-01-18T18:45:01.719Z] 2024-01-18 18:45:01 +0000:[INFO] [{VerifyRBACForAppUser}] Validate creating scheduled backup with namespace label [2024-01-18T18:45:02.284Z] 2024-01-18 18:45:02 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-18T18:45:02.284Z] 2024-01-18 18:45:02 +0000:[INFO] [{VerifyRBACForAppUser}] Volume snapshot classes found in the cluster - [2024-01-18T18:45:02.285Z] 2024-01-18 18:45:02 +0000:[INFO] [{VerifyRBACForAppUser}] ibmc-vpcblock-snapshot [2024-01-18T18:45:02.285Z] 2024-01-18 18:45:02 +0000:[INFO] [{VerifyRBACForAppUser}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T18:46:09.909Z] 2024-01-18 18:46:02 +0000:[INFO] [{VerifyRBACForAppUser}] first schedule backup for schedule name [tp-backup-ylri] is [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502] [2024-01-18T18:46:09.909Z] 2024-01-18 18:46:02 +0000:[INFO] [{VerifyRBACForAppUser}] Validating backup [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502] in org [default] [2024-01-18T18:46:09.909Z] 2024-01-18 18:46:02 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502] [2024-01-18T18:46:09.909Z] 2024-01-18 18:46:02 +0000:[INFO] [{VerifyRBACForAppUser}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87889-01-18-18h35m37s] in backup [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502] [2024-01-18T18:46:09.909Z] 2024-01-18 18:46:02 +0000:[INFO] [{VerifyRBACForAppUser}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87889-01-18-18h35m37s] in backup [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502] [2024-01-18T18:46:09.909Z] 2024-01-18 18:46:02 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:46:09.909Z] 2024-01-18 18:46:02 +0000:[WARNING] [{VerifyRBACForAppUser}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T18:46:09.909Z] 2024-01-18 18:46:02 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T18:46:09.910Z] 2024-01-18 18:46:02 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87889-01-18-18h35m37s] are [[mysql-data]] [2024-01-18T18:46:09.910Z] 2024-01-18 18:46:02 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T18:46:09.910Z] 2024-01-18 18:46:02 +0000:[INFO] [{VerifyRBACForAppUser}] Validating CR cleanup [2024-01-18T18:46:09.910Z] 2024-01-18 18:46:03 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T18:46:09.910Z] 2024-01-18 18:46:03 +0000:[INFO] [{VerifyRBACForAppUser}] All CRs in [mysql-ibm-pxb-0-87889-01-18-18h35m37s] are [[]] [2024-01-18T18:46:09.910Z] INFO[2024-01-18 18:46:03] Verifying : Description : Verification of creating first schedule backup tp-backup-ylri with labels [0-864abd34-8f71-49db-a527-f7eea2e4f188=864abd34-8f71-49db-a527-f7eea2e4f188,1-864abd34-8f71-49db-a527-f7eea2e4f188=864abd34-8f71-49db-a527-f7eea2e4f188] [2024-01-18T18:46:09.910Z] INFO[2024-01-18 18:46:03] Actual:true, Expected: true [2024-01-18T18:46:09.910Z] INFO[2024-01-18 18:46:03] Verifying : Description : Suspending Backup Schedule [tp-backup-ylri] for user [testuser1-1705603039] [2024-01-18T18:46:09.910Z] INFO[2024-01-18 18:46:03] Actual:true, Expected: true [2024-01-18T18:46:09.910Z] 2024-01-18 18:46:03 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502 [2024-01-18T18:46:09.910Z] 2024-01-18 18:46:03 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:46:09.910Z] 2024-01-18 18:46:03 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are 0-864abd34-8f71-49db-a527-f7eea2e4f188=864abd34-8f71-49db-a527-f7eea2e4f188,1-864abd34-8f71-49db-a527-f7eea2e4f188=864abd34-8f71-49db-a527-f7eea2e4f188 [2024-01-18T18:46:09.910Z] INFO[2024-01-18 18:46:03] Verifying : Description : Verifying if the labeled namespaces [[mysql-ibm-pxb-0-87889-01-18-18h35m37s]] are backed up, and check if labels [0-864abd34-8f71-49db-a527-f7eea2e4f188=864abd34-8f71-49db-a527-f7eea2e4f188,1-864abd34-8f71-49db-a527-f7eea2e4f188=864abd34-8f71-49db-a527-f7eea2e4f188] are applied to backups [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502] [2024-01-18T18:46:09.910Z] INFO[2024-01-18 18:46:03] Actual:true, Expected: true [2024-01-18T18:46:09.910Z] 2024-01-18 18:46:03 +0000:[INFO] [{VerifyRBACForAppUser}] Validate restoring the scheduled backup with namespace label [2024-01-18T18:46:09.910Z] INFO[2024-01-18 18:46:03] Verifying : Description : Fetching namespaces [mysql-ibm-pxb-0-87889-01-18-18h35m37s] from backup tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502 [2024-01-18T18:46:09.910Z] INFO[2024-01-18 18:46:03] Actual:true, Expected: true [2024-01-18T18:46:09.910Z] 2024/01/18 18:46:03 DoRetryWithTimeout - Error: {restore status for [customrestore-rewv] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T18:46:36.469Z] 2024-01-18 18:46:34 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.CreateRestore:#936] - Restore [customrestore-rewv] created successfully [2024-01-18T18:46:36.469Z] 2024-01-18 18:46:34 +0000:[INFO] [{VerifyRBACForAppUser}] Switching cluster context to cluster [destination-cluster] [2024-01-18T18:46:36.469Z] 2024-01-18 18:46:34 +0000:[INFO] [{VerifyRBACForAppUser}] Switching context to [/tmp/destination-config] [2024-01-18T18:46:36.469Z] 2024-01-18 18:46:34 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:46:36.469Z] 2024-01-18 18:46:34 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:46:36.469Z] 2024-01-18 18:46:34 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:46:36.469Z] 2024-01-18 18:46:34 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:46:36.469Z] 2024-01-18 18:46:34 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:46:36.469Z] 2024-01-18 18:46:34 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:46:36.469Z] 2024-01-18 18:46:34 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:46:36.469Z] 2024-01-18 18:46:35 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:46:36.469Z] 2024-01-18 18:46:35 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:46:36.470Z] 2024-01-18 18:46:35 +0000:[WARNING] [{VerifyRBACForAppUser}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T18:46:36.470Z] 2024-01-18 18:46:36 +0000:[WARNING] [{VerifyRBACForAppUser}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T18:46:36.470Z] 2024-01-18 18:46:36 +0000:[WARNING] [{VerifyRBACForAppUser}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T18:46:36.470Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T18:46:36.470Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] Switched context to [/tmp/destination-config] [2024-01-18T18:46:36.470Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-87889-01-18-18h35m37s] with namespace mapping [map[mysql-ibm-pxb-0-87889-01-18-18h35m37s:tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap]] and storage Class Mapping [map[]] [2024-01-18T18:46:36.470Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:46:36.470Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T18:46:36.470Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T18:46:36.470Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] Validating restore [customrestore-rewv] in org [default] [2024-01-18T18:46:36.470Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [customrestore-rewv] [2024-01-18T18:46:36.470Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2110] - Checking status of restore [customrestore-rewv] [2024-01-18T18:46:36.470Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] Validating specs for the namespace (restoredAppContext) [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap] in restore [customrestore-rewv] [2024-01-18T18:46:36.470Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap]) was found in restore [customrestore-rewv], as expected by presence in expectedRestoredAppContext [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap] [2024-01-18T18:46:36.470Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap]) found in the restore [customrestore-rewv] was also present on the cluster/namespace [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap] [2024-01-18T18:46:36.470Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap]) was found in restore [customrestore-rewv], as expected by presence in expectedRestoredAppContext [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap] [2024-01-18T18:46:36.470Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap]) found in the restore [customrestore-rewv] was also present on the cluster/namespace [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap] [2024-01-18T18:46:36.471Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap]) was found in restore [customrestore-rewv], as expected by presence in expectedRestoredAppContext [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap] [2024-01-18T18:46:36.728Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap]) found in the restore [customrestore-rewv] was also present on the cluster/namespace [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap] [2024-01-18T18:46:36.728Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap]) was found in restore [customrestore-rewv], as expected by presence in expectedRestoredAppContext [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap] [2024-01-18T18:46:36.728Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap]) found in the restore [customrestore-rewv] was also present on the cluster/namespace [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap] [2024-01-18T18:46:36.728Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] Validating Restored Volumes for the namespace (restoredAppContext) [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap] in restore [customrestore-rewv] [2024-01-18T18:46:36.728Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap] in restore [customrestore-rewv] [2024-01-18T18:46:36.728Z] 2024-01-18 18:46:36 +0000:[WARNING] [{VerifyRBACForAppUser}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T18:46:36.728Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T18:46:36.728Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap] are [[mysql-data]] [2024-01-18T18:46:36.729Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.ValidateRestore:#2240] - in restore [customrestore-rewv], the status of the restored volume [pvc-ddde003a-4407-4f78-bbe9-1457892e6a8b] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-18T18:46:36.729Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] Validate applications in restored namespace [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap] due to restore [customrestore-rewv] [2024-01-18T18:46:36.729Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] Validating mysql-ibm app [2024-01-18T18:46:36.729Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] wait for mysql-ibm app to start running [2024-01-18T18:46:36.985Z] 2024-01-18 18:46:36 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T18:46:36.985Z] 2024/01/18 18:46:36 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:46:36.985Z] pod name:mysql-7d766cd65c-f4ddj namespace:tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap running:false ready:false node:10.240.64.7 [2024-01-18T18:46:36.985Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:46:49.156Z] 2024/01/18 18:46:47 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:46:49.156Z] pod name:mysql-7d766cd65c-f4ddj namespace:tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap running:false ready:false node:10.240.64.7 [2024-01-18T18:46:49.156Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:46:57.246Z] 2024/01/18 18:46:57 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:46:57.246Z] pod name:mysql-7d766cd65c-f4ddj namespace:tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap running:false ready:false node:10.240.64.7 [2024-01-18T18:46:57.246Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:47:09.423Z] 2024/01/18 18:47:07 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:47:09.423Z] pod name:mysql-7d766cd65c-f4ddj namespace:tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap running:true ready:false node:10.240.64.7 [2024-01-18T18:47:09.423Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:47:17.537Z] 2024/01/18 18:47:17 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:47:17.537Z] pod name:mysql-7d766cd65c-f4ddj namespace:tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap running:true ready:false node:10.240.64.7 [2024-01-18T18:47:17.537Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:47:28.941Z] 2024/01/18 18:47:27 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:47:28.941Z] pod name:mysql-7d766cd65c-f4ddj namespace:tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap running:true ready:false node:10.240.64.7 [2024-01-18T18:47:28.941Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:47:38.887Z] 2024/01/18 18:47:37 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:47:38.887Z] pod name:mysql-7d766cd65c-f4ddj namespace:tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap running:true ready:false node:10.240.64.7 [2024-01-18T18:47:38.887Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:47:48.843Z] 2024/01/18 18:47:47 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:47:48.843Z] pod name:mysql-7d766cd65c-f4ddj namespace:tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap running:true ready:false node:10.240.64.7 [2024-01-18T18:47:48.843Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:47:58.788Z] 2024/01/18 18:47:57 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:47:58.788Z] pod name:mysql-7d766cd65c-f4ddj namespace:tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap running:true ready:false node:10.240.64.7 [2024-01-18T18:47:58.788Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:48:08.741Z] 2024-01-18 18:48:07 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T18:48:08.741Z] 2024-01-18 18:48:07 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T18:48:08.741Z] 2024-01-18 18:48:07 +0000:[INFO] [{VerifyRBACForAppUser}] Validating CR cleanup [2024-01-18T18:48:08.741Z] 2024-01-18 18:48:08 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T18:48:08.741Z] 2024-01-18 18:48:08 +0000:[INFO] [{VerifyRBACForAppUser}] All CRs in [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap] are [[]] [2024-01-18T18:48:08.741Z] 2024-01-18 18:48:08 +0000:[INFO] [{VerifyRBACForAppUser}] Switching cluster context back to cluster path [/tmp/source-config] [2024-01-18T18:48:08.741Z] 2024-01-18 18:48:08 +0000:[INFO] [{VerifyRBACForAppUser}] Switching context to [/tmp/source-config] [2024-01-18T18:48:08.741Z] 2024-01-18 18:48:08 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:48:08.741Z] 2024-01-18 18:48:08 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:48:08.741Z] 2024-01-18 18:48:08 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:48:08.742Z] 2024-01-18 18:48:08 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:48:08.742Z] 2024-01-18 18:48:08 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:48:08.742Z] 2024-01-18 18:48:08 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:48:08.998Z] 2024-01-18 18:48:08 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:48:09.560Z] 2024-01-18 18:48:09 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:48:09.817Z] 2024-01-18 18:48:09 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:48:10.073Z] 2024-01-18 18:48:09 +0000:[WARNING] [{VerifyRBACForAppUser}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T18:48:10.331Z] 2024-01-18 18:48:10 +0000:[WARNING] [{VerifyRBACForAppUser}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T18:48:10.331Z] 2024-01-18 18:48:10 +0000:[WARNING] [{VerifyRBACForAppUser}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T18:48:10.331Z] 2024-01-18 18:48:10 +0000:[INFO] [{VerifyRBACForAppUser}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T18:48:10.331Z] 2024-01-18 18:48:10 +0000:[INFO] [{VerifyRBACForAppUser}] Switched context to [/tmp/source-config] [2024-01-18T18:48:10.331Z] INFO[2024-01-18 18:48:10] Verifying : Description : Verifying multiple backup restore [customrestore-rewv] in custom namespace [map[mysql-ibm-pxb-0-87889-01-18-18h35m37s:tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502-drkap]] [2024-01-18T18:48:10.331Z] INFO[2024-01-18 18:48:10] Actual:true, Expected: true [2024-01-18T18:48:10.331Z] 2024-01-18 18:48:10 +0000:[INFO] [{VerifyRBACForAppUser}] Validate deleting of backups for the context of App-User [testuser1-1705603039] [2024-01-18T18:48:10.896Z] 2024/01/18 18:48:10 DoRetryWithTimeout - Error: {backupObject [backup-avmo] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-18T18:48:10.896Z] 2024/01/18 18:48:10 DoRetryWithTimeout - Error: {backupObject [backup-schedule-vqju-f1b8514-interval-2024-01-18-183740] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-18T18:48:10.896Z] 2024/01/18 18:48:10 DoRetryWithTimeout - Error: {backupObject [tp-backup-ylri-cb8ca6d-interval-2024-01-18-184502] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-18T18:48:42.938Z] 2024-01-18 18:48:40 +0000:[INFO] [{VerifyRBACForAppUser}] Validate deleting of restores for the context of App-User [testuser1-1705603039] [2024-01-18T18:48:42.938Z] INFO[2024-01-18 18:48:40] Verifying : Description : Getting the backup driver [2024-01-18T18:48:42.938Z] INFO[2024-01-18 18:48:40] Actual:true, Expected: true [2024-01-18T18:48:42.938Z] INFO[2024-01-18 18:48:40] Verifying : Description : Verifying the deletion of the restore named [tp-restore-backup-schedule-vqju-f1b8514-interval-2024-01-18-183740] [2024-01-18T18:48:42.939Z] INFO[2024-01-18 18:48:40] Actual:true, Expected: true [2024-01-18T18:48:42.939Z] INFO[2024-01-18 18:48:40] Verifying : Description : Getting the backup driver [2024-01-18T18:48:42.939Z] INFO[2024-01-18 18:48:40] Actual:true, Expected: true [2024-01-18T18:48:42.939Z] INFO[2024-01-18 18:48:40] Verifying : Description : Verifying the deletion of the restore named [tp-restore-backup-avmo] [2024-01-18T18:48:42.939Z] INFO[2024-01-18 18:48:40] Actual:true, Expected: true [2024-01-18T18:48:42.939Z] INFO[2024-01-18 18:48:40] Verifying : Description : Getting the backup driver [2024-01-18T18:48:42.939Z] INFO[2024-01-18 18:48:40] Actual:true, Expected: true [2024-01-18T18:48:42.939Z] INFO[2024-01-18 18:48:41] Verifying : Description : Verifying the deletion of the restore named [customrestore-rewv] [2024-01-18T18:48:42.939Z] INFO[2024-01-18 18:48:41] Actual:true, Expected: true [2024-01-18T18:48:42.939Z] 2024-01-18 18:48:41 +0000:[INFO] [{VerifyRBACForAppUser}] Validate deleting of backup schedules for the App-User [testuser1-1705603039] [2024-01-18T18:48:42.939Z] 2024/01/18 18:48:41 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-vqju] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:48:52.892Z] 2024/01/18 18:48:51 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-vqju] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:49:02.833Z] 2024/01/18 18:49:01 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-vqju] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:49:12.776Z] 2024/01/18 18:49:11 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-vqju] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:49:22.756Z] 2024/01/18 18:49:21 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-vqju] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:49:32.692Z] 2024/01/18 18:49:31 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-vqju] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:49:42.690Z] INFO[2024-01-18 18:49:41] Verifying : Description : Deleting Backup Schedule [backup-schedule-vqju] for user [testuser1-1705603039] [2024-01-18T18:49:42.690Z] INFO[2024-01-18 18:49:41] Actual:true, Expected: true [2024-01-18T18:49:42.690Z] 2024/01/18 18:49:41 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-ylri] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:49:52.630Z] 2024/01/18 18:49:51 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-ylri] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:50:02.574Z] 2024/01/18 18:50:01 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-ylri] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:50:12.523Z] 2024/01/18 18:50:11 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-ylri] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:50:22.519Z] 2024/01/18 18:50:21 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-ylri] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:50:32.473Z] 2024/01/18 18:50:31 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-ylri] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T18:50:42.420Z] INFO[2024-01-18 18:50:41] Verifying : Description : Deleting Backup Schedule [tp-backup-ylri] for user [testuser1-1705603039] [2024-01-18T18:50:42.421Z] INFO[2024-01-18 18:50:41] Actual:true, Expected: true [2024-01-18T18:50:42.421Z] 2024-01-18 18:50:41 +0000:[INFO] [{VerifyRBACForAppUser}] Validate deleting of source and destination cluster for the App-user [testuser1-1705603039] [2024-01-18T18:50:42.421Z] INFO[2024-01-18 18:50:41] Verifying : Description : Verifying deletion of cluster [source-cluster] of the user testuser1-1705603039 [2024-01-18T18:50:42.421Z] INFO[2024-01-18 18:50:41] Actual:true, Expected: true [2024-01-18T18:50:42.421Z] INFO[2024-01-18 18:50:41] Verifying : Description : Verifying deletion of cluster [destination-cluster] of the user testuser1-1705603039 [2024-01-18T18:50:42.421Z] INFO[2024-01-18 18:50:41] Actual:true, Expected: true [2024-01-18T18:50:42.421Z] 2024-01-18 18:50:42 +0000:[INFO] [{VerifyRBACForAppUser}] Deleting the deployed apps after the testcase [2024-01-18T18:50:42.421Z] 2024-01-18 18:50:42 +0000:[INFO] [{VerifyRBACForAppUser}] Destroying apps [2024-01-18T18:50:42.421Z] 2024-01-18 18:50:42 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T18:50:42.679Z] 2024-01-18 18:50:42 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T18:50:42.679Z] 2024-01-18 18:50:42 +0000:[INFO] [{VerifyRBACForAppUser}] start destroying mysql-ibm app [2024-01-18T18:50:42.679Z] 2024-01-18 18:50:42 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T18:50:42.679Z] 2024-01-18 18:50:42 +0000:[INFO] [{VerifyRBACForAppUser}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T18:50:42.679Z] 2024-01-18 18:50:42 +0000:[WARNING] [{VerifyRBACForAppUser}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T18:50:42.679Z] INFO[2024-01-18 18:50:42] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T18:50:42.679Z] INFO[2024-01-18 18:50:42] Actual:true, Expected: true [2024-01-18T18:50:42.679Z] 2024-01-18 18:50:42 +0000:[INFO] [{VerifyRBACForAppUser}] Deleting labels from namespaces - [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:50:42.935Z] INFO[2024-01-18 18:50:42] Verifying : Description : Deleting labels [map[0-864abd34-8f71-49db-a527-f7eea2e4f188:864abd34-8f71-49db-a527-f7eea2e4f188 1-864abd34-8f71-49db-a527-f7eea2e4f188:864abd34-8f71-49db-a527-f7eea2e4f188]] from namespaces [[mysql-ibm-pxb-0-87889-01-18-18h35m37s]] [2024-01-18T18:50:42.935Z] INFO[2024-01-18 18:50:42] Actual:true, Expected: true [2024-01-18T18:50:42.935Z] 2024-01-18 18:50:42 +0000:[INFO] [{VerifyRBACForAppUser}] Deleting the px-backup objects [2024-01-18T18:50:42.936Z] 2024-01-18 18:50:42 +0000:[INFO] [{VerifyRBACForAppUser}] Cleaning backup locations in map [map[968d882d-b5f2-4fd7-8536-0b66237f1e4c:aws-global-aws-px-backup-system-test-4cac76dc-qbwg]], cloud credential [cred-aws-bkac], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T18:50:42.936Z] INFO[2024-01-18 18:50:42] Verifying : Description : Verifying deletion of backup location [aws-global-aws-px-backup-system-test-4cac76dc-qbwg] [2024-01-18T18:50:42.936Z] INFO[2024-01-18 18:50:42] Actual:true, Expected: true [2024-01-18T18:50:43.195Z] 2024-01-18 18:50:42 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.IsBackupLocationPresent:#2442] - Backup location [aws-global-aws-px-backup-system-test-4cac76dc-qbwg] is present [2024-01-18T18:50:43.195Z] 2024/01/18 18:50:42 DoRetryWithTimeout - Error: {backup location aws-global-aws-px-backup-system-test-4cac76dc-qbwg is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T18:51:15.221Z] 2024-01-18 18:51:13 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.IsBackupLocationPresent:#2442] - Backup location [aws-global-aws-px-backup-system-test-4cac76dc-qbwg] is present [2024-01-18T18:51:15.221Z] 2024/01/18 18:51:13 DoRetryWithTimeout - Error: {backup location aws-global-aws-px-backup-system-test-4cac76dc-qbwg is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T18:51:47.275Z] 2024-01-18 18:51:43 +0000:[INFO] [{VerifyRBACForAppUser}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu] [2024-01-18T18:51:47.275Z] INFO[2024-01-18 18:51:43] Verifying : Description : Verifying backup location deletion status aws-global-aws-px-backup-system-test-4cac76dc-qbwg [2024-01-18T18:51:47.275Z] INFO[2024-01-18 18:51:43] Actual:true, Expected: true [2024-01-18T18:51:47.275Z] 2024-01-18 18:51:43 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cred-aws-bkac] is present [2024-01-18T18:51:47.275Z] INFO[2024-01-18 18:51:43] Verifying : Description : Verifying if cloud cred [cred-aws-bkac] is present [2024-01-18T18:51:47.275Z] INFO[2024-01-18 18:51:43] Actual:true, Expected: true [2024-01-18T18:51:47.275Z] INFO[2024-01-18 18:51:43] Verifying : Description : Verifying deletion of cloud cred [cred-aws-bkac] [2024-01-18T18:51:47.275Z] INFO[2024-01-18 18:51:43] Actual:true, Expected: true [2024-01-18T18:51:47.275Z] INFO[2024-01-18 18:51:43] Verifying : Description : Deleting cloud cred cred-aws-bkac [2024-01-18T18:51:47.275Z] INFO[2024-01-18 18:51:43] Actual:true, Expected: true [2024-01-18T18:51:47.275Z] INFO[2024-01-18 18:51:43] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T18:51:47.275Z] INFO[2024-01-18 18:51:43] Actual:true, Expected: true [2024-01-18T18:51:47.275Z] INFO[2024-01-18 18:51:43] Verifying : Description : Verifying deletion of rule [pre-rule-lwuiq] for the Px-Admin user [2024-01-18T18:51:47.275Z] INFO[2024-01-18 18:51:43] Actual:true, Expected: true [2024-01-18T18:51:47.275Z] INFO[2024-01-18 18:51:43] Verifying : Description : Verifying deletion of rule [post-rule-vhytm] for the Px-Admin user [2024-01-18T18:51:47.275Z] INFO[2024-01-18 18:51:43] Actual:true, Expected: true [2024-01-18T18:51:47.275Z] 2024-01-18 18:51:43 +0000:[INFO] [{VerifyRBACForAppUser}] Switching context to destination cluster for clean up [2024-01-18T18:51:47.275Z] 2024-01-18 18:51:43 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T18:51:47.275Z] 2024-01-18 18:51:43 +0000:[INFO] [{VerifyRBACForAppUser}] Switching context to [/tmp/destination-config] [2024-01-18T18:51:47.275Z] 2024-01-18 18:51:43 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:51:47.275Z] 2024-01-18 18:51:43 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:51:47.275Z] 2024-01-18 18:51:43 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:51:47.276Z] 2024-01-18 18:51:43 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:51:47.276Z] 2024-01-18 18:51:44 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:51:47.276Z] 2024-01-18 18:51:44 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:51:47.276Z] 2024-01-18 18:51:44 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:51:47.276Z] 2024-01-18 18:51:44 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:51:47.276Z] 2024-01-18 18:51:45 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:51:47.276Z] 2024-01-18 18:51:45 +0000:[WARNING] [{VerifyRBACForAppUser}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T18:51:47.276Z] 2024-01-18 18:51:45 +0000:[WARNING] [{VerifyRBACForAppUser}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T18:51:47.276Z] 2024-01-18 18:51:45 +0000:[WARNING] [{VerifyRBACForAppUser}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T18:51:47.276Z] 2024-01-18 18:51:45 +0000:[INFO] [{VerifyRBACForAppUser}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T18:51:47.276Z] 2024-01-18 18:51:45 +0000:[INFO] [{VerifyRBACForAppUser}] Switched context to [/tmp/destination-config] [2024-01-18T18:51:47.276Z] 2024-01-18 18:51:45 +0000:[INFO] [{VerifyRBACForAppUser}] Destroying apps [2024-01-18T18:51:47.276Z] 2024-01-18 18:51:45 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T18:51:47.276Z] 2024-01-18 18:51:46 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T18:51:47.276Z] 2024-01-18 18:51:46 +0000:[INFO] [{VerifyRBACForAppUser}] start destroying mysql-ibm app [2024-01-18T18:51:47.276Z] 2024-01-18 18:51:46 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T18:51:47.276Z] 2024-01-18 18:51:46 +0000:[INFO] [{VerifyRBACForAppUser}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T18:51:47.276Z] 2024-01-18 18:51:46 +0000:[WARNING] [{VerifyRBACForAppUser}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T18:51:47.276Z] INFO[2024-01-18 18:51:46] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T18:51:47.276Z] INFO[2024-01-18 18:51:46] Actual:true, Expected: true [2024-01-18T18:51:47.276Z] 2024-01-18 18:51:46 +0000:[INFO] [{VerifyRBACForAppUser}] Switching back context to Source cluster [2024-01-18T18:51:47.276Z] 2024-01-18 18:51:46 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T18:51:47.276Z] 2024-01-18 18:51:46 +0000:[INFO] [{VerifyRBACForAppUser}] Switching context to [/tmp/source-config] [2024-01-18T18:51:47.276Z] 2024-01-18 18:51:46 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:51:47.276Z] 2024-01-18 18:51:46 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:51:47.276Z] 2024-01-18 18:51:46 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:51:47.276Z] 2024-01-18 18:51:46 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:51:47.277Z] 2024-01-18 18:51:46 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:51:47.277Z] 2024-01-18 18:51:46 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:51:47.533Z] 2024-01-18 18:51:47 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:51:48.097Z] 2024-01-18 18:51:47 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:51:48.353Z] 2024-01-18 18:51:48 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:51:48.608Z] 2024-01-18 18:51:48 +0000:[WARNING] [{VerifyRBACForAppUser}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T18:51:48.864Z] 2024-01-18 18:51:48 +0000:[WARNING] [{VerifyRBACForAppUser}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T18:51:48.864Z] 2024-01-18 18:51:48 +0000:[WARNING] [{VerifyRBACForAppUser}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T18:51:48.864Z] 2024-01-18 18:51:48 +0000:[INFO] [{VerifyRBACForAppUser}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T18:51:48.864Z] 2024-01-18 18:51:48 +0000:[INFO] [{VerifyRBACForAppUser}] Switched context to [/tmp/source-config] [2024-01-18T18:51:48.864Z] 2024-01-18 18:51:48 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T18:51:48.864Z] 2024-01-18 18:51:48 +0000:[INFO] [{VerifyRBACForAppUser}] Switching context: The context is already [/tmp/source-config] [2024-01-18T18:51:48.864Z] INFO[2024-01-18 18:51:48] --------Test End------ [2024-01-18T18:51:48.864Z] INFO[2024-01-18 18:51:48] #Test: [2024-01-18T18:51:48.864Z] INFO[2024-01-18 18:51:48] #Description: [2024-01-18T18:51:48.864Z] INFO[2024-01-18 18:51:48] ------------------------ [2024-01-18T18:51:49.120Z] 2024-01-18 18:51:48 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:51:49.120Z] 2024-01-18 18:51:48 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T18:51:49.120Z] 2024-01-18 18:51:48 +0000:[INFO] [{VerifyRBACForAppUser}] Switching context to [/tmp/destination-config] [2024-01-18T18:51:49.120Z] 2024-01-18 18:51:49 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:51:49.375Z] 2024-01-18 18:51:49 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:51:49.375Z] 2024-01-18 18:51:49 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:51:49.375Z] 2024-01-18 18:51:49 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:51:49.375Z] 2024-01-18 18:51:49 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:51:49.630Z] 2024-01-18 18:51:49 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:51:49.886Z] 2024-01-18 18:51:49 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:51:50.449Z] 2024-01-18 18:51:50 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:51:50.705Z] 2024-01-18 18:51:50 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:51:50.959Z] 2024-01-18 18:51:50 +0000:[WARNING] [{VerifyRBACForAppUser}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T18:51:51.215Z] 2024-01-18 18:51:51 +0000:[WARNING] [{VerifyRBACForAppUser}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T18:51:51.215Z] 2024-01-18 18:51:51 +0000:[WARNING] [{VerifyRBACForAppUser}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T18:51:51.215Z] 2024-01-18 18:51:51 +0000:[INFO] [{VerifyRBACForAppUser}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T18:51:51.215Z] 2024-01-18 18:51:51 +0000:[INFO] [{VerifyRBACForAppUser}] Switched context to [/tmp/destination-config] [2024-01-18T18:51:51.470Z] 2024-01-18 18:51:51 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-87889-01-18-18h35m37s] [2024-01-18T18:51:51.470Z] 2024-01-18 18:51:51 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {VerifyRBACForAppUser} To verify all the RBAC operations for an App-user [2024-01-18T18:51:51.470Z] 2024-01-18 18:51:51 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-18T18:51:51.470Z] 2024/01/18 18:51:51 DoRetryWithTimeout - Error: {dial tcp 10.240.0.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T18:51:56.721Z] 2024/01/18 18:51:56 DoRetryWithTimeout - Error: {dial tcp 10.240.0.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T18:52:01.974Z] 2024/01/18 18:52:01 DoRetryWithTimeout - Error: {dial tcp 10.240.0.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T18:52:07.218Z] 2024/01/18 18:52:06 DoRetryWithTimeout - Error: {dial tcp 10.240.0.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T18:52:12.466Z] 2024/01/18 18:52:11 DoRetryWithTimeout - Error: {dial tcp 10.240.0.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T18:52:16.631Z] 2024/01/18 18:52:16 DoRetryWithTimeout - Error: {dial tcp 10.240.0.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T18:52:21.877Z] 2024/01/18 18:52:21 DoRetryWithTimeout - Error: {dial tcp 10.240.0.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T18:52:27.121Z] 2024/01/18 18:52:26 DoRetryWithTimeout - Error: {dial tcp 10.240.0.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T18:52:32.361Z] 2024/01/18 18:52:31 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.0.6. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.6 10.240.0.6]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.6:22: connect: connection refused} [2024-01-18T18:52:32.361Z] {dial tcp 10.240.0.6:22: connect: connection refused} [2024-01-18T18:52:32.361Z] {dial tcp 10.240.0.6:22: connect: connection refused} [2024-01-18T18:52:32.361Z] {dial tcp 10.240.0.6:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-18T18:52:36.527Z] 2024-01-18 18:52:36 +0000:[WARNING] [{VerifyRBACForAppUser}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.6. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.6 10.240.0.6]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.6:22: connect: connection refused} [2024-01-18T18:52:36.527Z] {dial tcp 10.240.0.6:22: connect: connection refused} [2024-01-18T18:52:36.527Z] {dial tcp 10.240.0.6:22: connect: connection refused} [2024-01-18T18:52:36.528Z] {dial tcp 10.240.0.6:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T18:52:36.528Z] 2024-01-18 18:52:36 +0000:[ERROR] [{VerifyRBACForAppUser}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.6. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.6 10.240.0.6]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.6:22: connect: connection refused} [2024-01-18T18:52:36.528Z] {dial tcp 10.240.0.6:22: connect: connection refused} [2024-01-18T18:52:36.528Z] {dial tcp 10.240.0.6:22: connect: connection refused} [2024-01-18T18:52:36.528Z] {dial tcp 10.240.0.6:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T18:52:36.528Z] 2024-01-18 18:52:36 +0000:[INFO] [{VerifyRBACForAppUser}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T18:52:36.528Z] 2024-01-18 18:52:36 +0000:[INFO] [{VerifyRBACForAppUser}] Switching context to [/tmp/source-config] [2024-01-18T18:52:36.790Z] 2024-01-18 18:52:36 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:52:36.790Z] 2024-01-18 18:52:36 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:52:36.790Z] 2024-01-18 18:52:36 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T18:52:36.790Z] 2024-01-18 18:52:36 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:52:36.790Z] 2024-01-18 18:52:36 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:52:37.048Z] 2024-01-18 18:52:36 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T18:52:37.612Z] 2024-01-18 18:52:37 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:52:37.869Z] 2024-01-18 18:52:37 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:52:38.124Z] 2024-01-18 18:52:38 +0000:[INFO] [{VerifyRBACForAppUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T18:52:38.379Z] 2024-01-18 18:52:38 +0000:[WARNING] [{VerifyRBACForAppUser}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T18:52:38.634Z] 2024-01-18 18:52:38 +0000:[WARNING] [{VerifyRBACForAppUser}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T18:52:38.634Z] 2024-01-18 18:52:38 +0000:[WARNING] [{VerifyRBACForAppUser}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T18:52:38.634Z] 2024-01-18 18:52:38 +0000:[INFO] [{VerifyRBACForAppUser}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T18:52:38.634Z] 2024-01-18 18:52:38 +0000:[INFO] [{VerifyRBACForAppUser}] Switched context to [/tmp/source-config] [2024-01-18T18:52:38.634Z] [2024-01-18T18:52:38.634Z] ------------------------------ [2024-01-18T18:52:38.634Z] • [SLOW TEST:1021.256 seconds] [2024-01-18T18:52:38.634Z] {VerifyRBACForAppUser} [2024-01-18T18:52:38.635Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_rbac_test.go:1658 [2024-01-18T18:52:38.635Z] To verify all the RBAC operations for an App-user [2024-01-18T18:52:38.635Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_rbac_test.go:1710 [2024-01-18T18:52:38.635Z] ------------------------------ [2024-01-18T18:52:38.635Z] 2024-01-18 18:52:38 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Log Dir: /testresults//DeleteNSDeleteClusterRestore.log [2024-01-18T18:52:38.635Z] INFO[2024-01-18 18:52:38] --------Test Start------ [2024-01-18T18:52:38.635Z] INFO[2024-01-18 18:52:38] #Test: DeleteNSDeleteClusterRestore [2024-01-18T18:52:38.635Z] INFO[2024-01-18 18:52:38] #Description: Delete namespace from application cluster and delete cluster and add it back then restore for last backup [2024-01-18T18:52:38.635Z] INFO[2024-01-18 18:52:38] ------------------------ [2024-01-18T18:52:38.635Z] 2024-01-18 18:52:38 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Scheduling applications [2024-01-18T18:52:38.635Z] 2024-01-18 18:52:38 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T18:52:38.635Z] 2024-01-18 18:52:38 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T18:52:39.195Z] 2024-01-18 18:52:38 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T18:52:39.195Z] 2024-01-18 18:52:38 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T18:52:39.196Z] 2024-01-18 18:52:39 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T18:52:39.451Z] 2024-01-18 18:52:39 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T18:52:39.452Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T18:52:39.452Z] 2024-01-18 18:52:39 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T18:52:39.452Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T18:52:39.452Z] 2024-01-18 18:52:39 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T18:52:39.452Z] 2024-01-18 18:52:39 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Scheduled application [mysql-ibm] in source cluster in namespace [test-namespace-src-pxb-0] [2024-01-18T18:52:39.452Z] 2024-01-18 18:52:39 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T18:52:39.453Z] 2024-01-18 18:52:39 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T18:52:39.707Z] 2024-01-18 18:52:39 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T18:52:39.963Z] 2024-01-18 18:52:39 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T18:52:39.963Z] 2024-01-18 18:52:39 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T18:52:40.219Z] 2024-01-18 18:52:40 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T18:52:40.220Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T18:52:40.220Z] 2024-01-18 18:52:40 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T18:52:40.220Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T18:52:40.221Z] 2024-01-18 18:52:40 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T18:52:40.221Z] 2024-01-18 18:52:40 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Scheduled application [mysql-ibm] in source cluster in namespace [test-namespace-src-pxb-1] [2024-01-18T18:52:40.221Z] 2024-01-18 18:52:40 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Validating applications [2024-01-18T18:52:40.221Z] 2024-01-18 18:52:40 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Validate applications [2024-01-18T18:52:40.221Z] 2024-01-18 18:52:40 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Validating mysql-ibm app [2024-01-18T18:52:40.221Z] 2024-01-18 18:52:40 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Validating mysql-ibm app's volumes [2024-01-18T18:52:40.478Z] 2024-01-18 18:52:40 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T18:52:40.738Z] 2024/01/18 18:52:40 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T18:52:50.683Z] 2024-01-18 18:52:50 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: test-namespace-src-pxb-0 [2024-01-18T18:52:50.683Z] 2024-01-18 18:52:50 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T18:52:50.938Z] 2024-01-18 18:52:50 +0000:[WARNING] [{DeleteNSDeleteClusterRestore}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T18:52:50.938Z] 2024-01-18 18:52:50 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] wait for mysql-ibm app to start running [2024-01-18T18:52:50.939Z] 2024-01-18 18:52:50 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T18:52:50.939Z] 2024/01/18 18:52:50 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:52:50.939Z] pod name:mysql-7d766cd65c-99hb6 namespace:test-namespace-src-pxb-0 running:true ready:false node: [2024-01-18T18:52:50.939Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:53:00.883Z] 2024/01/18 18:53:00 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:53:00.883Z] pod name:mysql-7d766cd65c-99hb6 namespace:test-namespace-src-pxb-0 running:false ready:false node:10.240.64.8 [2024-01-18T18:53:00.883Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:53:13.052Z] 2024/01/18 18:53:10 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:53:13.052Z] pod name:mysql-7d766cd65c-99hb6 namespace:test-namespace-src-pxb-0 running:true ready:false node:10.240.64.8 [2024-01-18T18:53:13.052Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:53:21.133Z] 2024/01/18 18:53:21 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:53:21.134Z] pod name:mysql-7d766cd65c-99hb6 namespace:test-namespace-src-pxb-0 running:true ready:false node:10.240.64.8 [2024-01-18T18:53:21.134Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:53:33.314Z] 2024/01/18 18:53:31 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:53:33.314Z] pod name:mysql-7d766cd65c-99hb6 namespace:test-namespace-src-pxb-0 running:true ready:false node:10.240.64.8 [2024-01-18T18:53:33.314Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:53:41.405Z] 2024/01/18 18:53:41 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:53:41.405Z] pod name:mysql-7d766cd65c-99hb6 namespace:test-namespace-src-pxb-0 running:true ready:false node:10.240.64.8 [2024-01-18T18:53:41.405Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:53:51.352Z] 2024/01/18 18:53:51 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:53:51.353Z] pod name:mysql-7d766cd65c-99hb6 namespace:test-namespace-src-pxb-0 running:true ready:false node:10.240.64.8 [2024-01-18T18:53:51.353Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:54:03.524Z] 2024/01/18 18:54:01 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:54:03.524Z] pod name:mysql-7d766cd65c-99hb6 namespace:test-namespace-src-pxb-0 running:true ready:false node:10.240.64.8 [2024-01-18T18:54:03.524Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:54:11.614Z] 2024/01/18 18:54:11 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T18:54:11.614Z] pod name:mysql-7d766cd65c-99hb6 namespace:test-namespace-src-pxb-0 running:true ready:false node:10.240.64.8 [2024-01-18T18:54:11.614Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T18:54:14.881Z] INFO[2024-01-18 18:54:14] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T18:54:14.881Z] INFO[2024-01-18 18:54:14] sched stats: tick interval: 12014 < 1.2s (100%) [2024-01-18T18:54:14.881Z] INFO[2024-01-18 18:54:14] sched stats: tick process durations: 12013 < 500ms (100%) [2024-01-18T18:54:23.002Z] 2024-01-18 18:54:21 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T18:54:23.002Z] 2024-01-18 18:54:21 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T18:54:23.002Z] 2024-01-18 18:54:21 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] validate if mysql-ibm app's volumes are setup [2024-01-18T18:54:23.002Z] 2024-01-18 18:54:21 +0000:[WARNING] [{DeleteNSDeleteClusterRestore}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T18:54:23.002Z] 2024-01-18 18:54:21 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T18:54:23.002Z] 2024-01-18 18:54:21 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T18:54:23.002Z] 2024-01-18 18:54:21 +0000:[WARNING] [{DeleteNSDeleteClusterRestore}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T18:54:23.002Z] 2024-01-18 18:54:22 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Validating mysql-ibm app [2024-01-18T18:54:23.002Z] 2024-01-18 18:54:22 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Validating mysql-ibm app's volumes [2024-01-18T18:54:23.002Z] 2024-01-18 18:54:22 +0000:[WARNING] [{DeleteNSDeleteClusterRestore}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T18:54:23.002Z] 2024-01-18 18:54:22 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T18:54:23.002Z] 2024-01-18 18:54:22 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: test-namespace-src-pxb-1 [2024-01-18T18:54:23.002Z] 2024-01-18 18:54:22 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T18:54:23.002Z] 2024-01-18 18:54:22 +0000:[WARNING] [{DeleteNSDeleteClusterRestore}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T18:54:23.002Z] 2024-01-18 18:54:22 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] wait for mysql-ibm app to start running [2024-01-18T18:54:23.002Z] 2024-01-18 18:54:22 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T18:54:23.002Z] 2024-01-18 18:54:22 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T18:54:23.002Z] 2024-01-18 18:54:22 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T18:54:23.930Z] 2024-01-18 18:54:23 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] validate if mysql-ibm app's volumes are setup [2024-01-18T18:54:23.930Z] 2024-01-18 18:54:23 +0000:[WARNING] [{DeleteNSDeleteClusterRestore}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T18:54:23.930Z] 2024-01-18 18:54:23 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T18:54:23.930Z] 2024-01-18 18:54:23 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T18:54:23.930Z] 2024-01-18 18:54:23 +0000:[WARNING] [{DeleteNSDeleteClusterRestore}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T18:54:23.930Z] 2024-01-18 18:54:23 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Creating cloud credentials and backup locations [2024-01-18T18:54:24.185Z] 2024-01-18 18:54:23 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Creating cloud credential named [2ac7e749-6633-4ecb-8dc8-2ee69aed4eba] and uid [cred-aws-1705604063] using [aws] as provider [2024-01-18T18:54:24.185Z] 2024-01-18 18:54:23 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [cred-aws-1705604063] for org [default] with [aws] as provider [2024-01-18T18:54:24.185Z] 2024-01-18 18:54:23 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-18T18:54:24.185Z] 2024-01-18 18:54:24 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cred-aws-1705604063] is present [2024-01-18T18:54:24.185Z] INFO[2024-01-18 18:54:24] Verifying : Description : Verifying creation of cloud credential named [cred-aws-1705604063] for org [default] with [aws] as provider [2024-01-18T18:54:24.185Z] INFO[2024-01-18 18:54:24] Actual:true, Expected: true [2024-01-18T18:54:28.374Z] INFO[2024-01-18 18:54:28] Verifying : Description : Verifying creation of backup location named [aws-global-aws-px-backup-system-test-4cac76dc-bl-1705604064] with uid [2ac9e7f6-fe46-4e8f-b031-67023aec1477] of [aws] as provider [2024-01-18T18:54:28.374Z] INFO[2024-01-18 18:54:28] Actual:true, Expected: true [2024-01-18T18:54:28.374Z] 2024-01-18 18:54:28 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Adding source and destination clusters with px-central-admin ctx [2024-01-18T18:54:28.374Z] 2024-01-18 18:54:28 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [backup.glob..func71.2.3:#3239] - Creating source [source-cluster] and destination [destination-cluster] clusters [2024-01-18T18:54:28.374Z] INFO[2024-01-18 18:54:28] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T18:54:28.375Z] INFO[2024-01-18 18:54:28] Actual:true, Expected: true [2024-01-18T18:54:28.375Z] 2024-01-18 18:54:28 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T18:54:28.375Z] 2024-01-18 18:54:28 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T18:54:28.375Z] 2024-01-18 18:54:28 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T18:54:28.375Z] 2024-01-18 18:54:28 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T18:54:28.375Z] 2024-01-18 18:54:28 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Create cluster [source-cluster] in org [default] [2024-01-18T18:54:28.375Z] 2024-01-18 18:54:28 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T18:54:28.375Z] 2024-01-18 18:54:28 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T18:54:28.375Z] 2024-01-18 18:54:28 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Create cluster [destination-cluster] in org [default] [2024-01-18T18:54:28.375Z] 2024-01-18 18:54:28 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T18:54:28.375Z] 2024-01-18 18:54:28 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T18:54:28.375Z] 2024-01-18 18:54:28 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-kopvx] for IBM [2024-01-18T18:54:28.375Z] 2024-01-18 18:54:28 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-kopvx] for org [default] with [ibm] as provider [2024-01-18T18:54:28.375Z] 2024-01-18 18:54:28 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T18:54:28.375Z] 2024-01-18 18:54:28 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T18:54:28.631Z] 2024-01-18 18:54:28 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-kopvx] is present [2024-01-18T18:54:28.631Z] 2024-01-18 18:54:28 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Create cluster [source-cluster] in org [default] [2024-01-18T18:54:29.193Z] 2024-01-18 18:54:29 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-srwym] for IBM [2024-01-18T18:54:29.193Z] 2024-01-18 18:54:29 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-srwym] for org [default] with [ibm] as provider [2024-01-18T18:54:29.193Z] 2024-01-18 18:54:29 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T18:54:29.194Z] 2024-01-18 18:54:29 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T18:54:29.449Z] 2024-01-18 18:54:29 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-srwym] is present [2024-01-18T18:54:29.449Z] 2024-01-18 18:54:29 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Create cluster [destination-cluster] in org [default] [2024-01-18T18:54:29.705Z] INFO[2024-01-18 18:54:29] Verifying : Description : Verifying creation of source [source-cluster] and destination [destination-cluster] clusters with px-central-admin ctx [2024-01-18T18:54:29.705Z] INFO[2024-01-18 18:54:29] Actual:true, Expected: true [2024-01-18T18:54:29.705Z] INFO[2024-01-18 18:54:29] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-18T18:54:29.705Z] INFO[2024-01-18 18:54:29] Actual:Online, Expected: Online [2024-01-18T18:54:29.705Z] 2024-01-18 18:54:29 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [backup.glob..func71.2.3:#3247] - Cluster [source-cluster] uid: [cdadc6de-00b7-4793-922c-c6e192086fe5] [2024-01-18T18:54:29.961Z] INFO[2024-01-18 18:54:29] Verifying : Description : Verifying if [destination-cluster] cluster is online [2024-01-18T18:54:29.961Z] INFO[2024-01-18 18:54:29] Actual:Online, Expected: Online [2024-01-18T18:54:29.961Z] 2024-01-18 18:54:29 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [backup.glob..func71.2.3:#3253] - Cluster [destination-cluster] uid: [cd34700c-55eb-4275-a1a0-90d92f616bb1] [2024-01-18T18:54:29.961Z] 2024-01-18 18:54:29 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Taking backup of applications [2024-01-18T18:54:29.961Z] 2024-01-18 18:54:29 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T18:54:30.216Z] 2024-01-18 18:54:29 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Volume snapshot classes found in the cluster - [2024-01-18T18:54:30.216Z] 2024-01-18 18:54:29 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] ibmc-vpcblock-snapshot [2024-01-18T18:54:30.216Z] 2024-01-18 18:54:29 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T18:54:30.778Z] 2024/01/18 18:54:30 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705604069] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T18:55:02.809Z] 2024/01/18 18:55:00 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705604069] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T18:55:34.837Z] 2024/01/18 18:55:30 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705604069] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T18:56:02.812Z] 2024-01-18 18:56:00 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [backup.CreateBackup:#313] - Backup [tp-backup-1705604069] created successfully [2024-01-18T18:56:02.812Z] 2024-01-18 18:56:00 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Validating backup [tp-backup-1705604069] in org [default] [2024-01-18T18:56:02.812Z] 2024-01-18 18:56:00 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705604069] [2024-01-18T18:56:02.812Z] 2024-01-18 18:56:00 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Validating specs for the namespace (scheduledAppContext) [test-namespace-src-pxb-0] in backup [tp-backup-1705604069] [2024-01-18T18:56:02.812Z] 2024-01-18 18:56:00 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Validating backed up volumes for the namespace (scheduledAppContext) [test-namespace-src-pxb-0] in backup [tp-backup-1705604069] [2024-01-18T18:56:02.812Z] 2024-01-18 18:56:00 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [test-namespace-src-pxb-0] [2024-01-18T18:56:02.812Z] 2024-01-18 18:56:00 +0000:[WARNING] [{DeleteNSDeleteClusterRestore}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T18:56:02.812Z] 2024-01-18 18:56:00 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T18:56:02.812Z] 2024-01-18 18:56:00 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [test-namespace-src-pxb-0] are [[mysql-data]] [2024-01-18T18:56:02.812Z] 2024-01-18 18:56:00 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T18:56:02.812Z] 2024-01-18 18:56:00 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Validating CR cleanup [2024-01-18T18:56:02.812Z] 2024-01-18 18:56:00 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T18:56:02.812Z] 2024-01-18 18:56:00 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] All CRs in [test-namespace-src-pxb-0] are [[]] [2024-01-18T18:56:02.813Z] INFO[2024-01-18 18:56:00] Verifying : Description : Verifying backup tp-backup-1705604069 creation [2024-01-18T18:56:02.813Z] INFO[2024-01-18 18:56:00] Actual:true, Expected: true [2024-01-18T18:56:02.813Z] 2024-01-18 18:56:00 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T18:56:02.813Z] 2024-01-18 18:56:00 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Volume snapshot classes found in the cluster - [2024-01-18T18:56:02.813Z] 2024-01-18 18:56:00 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] ibmc-vpcblock-snapshot [2024-01-18T18:56:02.813Z] 2024-01-18 18:56:00 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T18:56:02.813Z] 2024/01/18 18:56:01 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705604160] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T18:56:34.842Z] 2024/01/18 18:56:31 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705604160] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T18:57:06.865Z] 2024/01/18 18:57:01 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705604160] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T18:57:33.375Z] 2024-01-18 18:57:31 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [backup.CreateBackup:#313] - Backup [tp-backup-1705604160] created successfully [2024-01-18T18:57:33.375Z] 2024-01-18 18:57:31 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Validating backup [tp-backup-1705604160] in org [default] [2024-01-18T18:57:33.375Z] 2024-01-18 18:57:31 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705604160] [2024-01-18T18:57:33.375Z] 2024-01-18 18:57:31 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Validating specs for the namespace (scheduledAppContext) [test-namespace-src-pxb-1] in backup [tp-backup-1705604160] [2024-01-18T18:57:33.376Z] 2024-01-18 18:57:31 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Validating backed up volumes for the namespace (scheduledAppContext) [test-namespace-src-pxb-1] in backup [tp-backup-1705604160] [2024-01-18T18:57:33.376Z] 2024-01-18 18:57:31 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [test-namespace-src-pxb-1] [2024-01-18T18:57:33.376Z] 2024-01-18 18:57:31 +0000:[WARNING] [{DeleteNSDeleteClusterRestore}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T18:57:33.376Z] 2024-01-18 18:57:31 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T18:57:33.376Z] 2024-01-18 18:57:31 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [test-namespace-src-pxb-1] are [[mysql-data]] [2024-01-18T18:57:33.376Z] 2024-01-18 18:57:31 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T18:57:33.376Z] 2024-01-18 18:57:31 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Validating CR cleanup [2024-01-18T18:57:33.376Z] 2024-01-18 18:57:31 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T18:57:33.376Z] 2024-01-18 18:57:31 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] All CRs in [test-namespace-src-pxb-1] are [[]] [2024-01-18T18:57:33.376Z] INFO[2024-01-18 18:57:31] Verifying : Description : Verifying backup tp-backup-1705604160 creation [2024-01-18T18:57:33.376Z] INFO[2024-01-18 18:57:31] Actual:true, Expected: true [2024-01-18T18:57:33.376Z] 2024-01-18 18:57:31 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Delete the App namespaces created [2024-01-18T18:57:33.376Z] 2024/01/18 18:57:31 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T18:57:43.320Z] 2024/01/18 18:57:41 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T18:57:53.262Z] 2024/01/18 18:57:51 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T18:58:03.209Z] 2024/01/18 18:58:01 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T18:58:13.163Z] 2024/01/18 18:58:11 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T18:58:23.158Z] 2024/01/18 18:58:22 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T18:58:33.095Z] 2024/01/18 18:58:32 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T18:58:43.041Z] 2024/01/18 18:58:42 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T18:58:52.987Z] 2024/01/18 18:58:52 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T18:59:02.930Z] 2024/01/18 18:59:02 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T18:59:12.867Z] 2024/01/18 18:59:12 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T18:59:22.811Z] 2024/01/18 18:59:22 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T18:59:32.779Z] 2024/01/18 18:59:32 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T18:59:42.778Z] 2024/01/18 18:59:42 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T18:59:52.725Z] 2024/01/18 18:59:52 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:00:02.678Z] 2024/01/18 19:00:02 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:00:12.692Z] 2024/01/18 19:00:12 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:00:22.636Z] 2024/01/18 19:00:22 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:00:32.579Z] 2024/01/18 19:00:32 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:00:42.528Z] 2024/01/18 19:00:42 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:00:52.473Z] 2024/01/18 19:00:52 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:01:04.684Z] 2024/01/18 19:01:02 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:01:12.823Z] 2024/01/18 19:01:12 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:01:22.807Z] 2024/01/18 19:01:22 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:01:32.775Z] 2024/01/18 19:01:32 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:01:42.732Z] 2024/01/18 19:01:42 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:01:52.709Z] 2024/01/18 19:01:52 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:02:02.655Z] 2024/01/18 19:02:02 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:02:12.607Z] 2024/01/18 19:02:12 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:02:24.815Z] 2024/01/18 19:02:22 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:02:32.925Z] 2024/01/18 19:02:32 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:02:42.869Z] 2024/01/18 19:02:42 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:02:52.817Z] 2024/01/18 19:02:52 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:03:02.768Z] 2024/01/18 19:03:02 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:03:14.935Z] 2024/01/18 19:03:12 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:03:23.049Z] 2024/01/18 19:03:22 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:03:32.997Z] 2024/01/18 19:03:32 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:03:42.963Z] 2024/01/18 19:03:42 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:03:52.933Z] 2024/01/18 19:03:52 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:04:02.882Z] 2024/01/18 19:04:02 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:04:12.863Z] 2024/01/18 19:04:12 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:04:16.130Z] INFO[2024-01-18 19:04:15] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T19:04:16.130Z] INFO[2024-01-18 19:04:15] sched stats: tick interval: 12615 < 1.2s (100%) [2024-01-18T19:04:16.130Z] INFO[2024-01-18 19:04:15] sched stats: tick process durations: 12614 < 500ms (100%) [2024-01-18T19:04:24.227Z] 2024/01/18 19:04:22 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:04:34.197Z] 2024/01/18 19:04:32 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:04:44.146Z] 2024/01/18 19:04:42 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:04:54.104Z] 2024/01/18 19:04:52 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:05:04.052Z] 2024/01/18 19:05:02 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:05:14.002Z] 2024/01/18 19:05:12 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:05:23.948Z] 2024/01/18 19:05:23 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:05:33.907Z] 2024/01/18 19:05:33 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:05:43.850Z] 2024/01/18 19:05:43 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:05:53.791Z] 2024/01/18 19:05:53 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:06:03.736Z] 2024/01/18 19:06:03 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:06:13.682Z] 2024/01/18 19:06:13 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:06:25.069Z] 2024/01/18 19:06:23 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:06:35.019Z] 2024/01/18 19:06:33 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:06:44.963Z] 2024/01/18 19:06:43 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:06:54.908Z] 2024/01/18 19:06:53 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:07:04.852Z] 2024/01/18 19:07:03 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:07:14.799Z] 2024/01/18 19:07:13 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:07:24.746Z] 2024/01/18 19:07:23 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:07:34.698Z] 2024-01-18 19:07:33 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [backup.DeleteAppNamespace:#3765] - Namespace - test-namespace-src-pxb-0 is in Terminating phase [2024-01-18T19:07:34.698Z] 2024-01-18 19:07:33 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [backup.DeleteTerminatingNamespace:#3781] - Namespace - test-namespace-src-pxb-0 is in Terminating phase [2024-01-18T19:07:34.698Z] 2024-01-18 19:07:33 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [backup.DeleteTerminatingNamespace:#3782] - Finalizers to be set to nil - [kubernetes] [2024-01-18T19:07:34.698Z] 2024/01/18 19:07:33 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:07:44.653Z] 2024/01/18 19:07:43 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:07:54.596Z] 2024/01/18 19:07:53 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:08:04.539Z] 2024/01/18 19:08:03 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:08:14.481Z] 2024/01/18 19:08:13 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:08:24.422Z] 2024/01/18 19:08:23 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:08:34.375Z] 2024/01/18 19:08:33 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:08:44.326Z] 2024/01/18 19:08:43 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:08:54.282Z] 2024/01/18 19:08:53 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:09:04.225Z] 2024/01/18 19:09:03 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:09:14.208Z] 2024/01/18 19:09:13 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:09:24.150Z] 2024/01/18 19:09:23 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:09:34.095Z] 2024/01/18 19:09:33 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:09:44.038Z] 2024/01/18 19:09:43 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:09:54.000Z] 2024/01/18 19:09:53 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:10:03.989Z] 2024/01/18 19:10:03 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:10:13.982Z] 2024/01/18 19:10:13 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:10:23.924Z] 2024/01/18 19:10:23 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:10:33.882Z] 2024/01/18 19:10:33 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:10:46.051Z] 2024/01/18 19:10:43 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:10:54.136Z] 2024/01/18 19:10:53 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:11:04.087Z] 2024/01/18 19:11:03 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:11:14.033Z] 2024/01/18 19:11:13 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:11:23.977Z] 2024/01/18 19:11:23 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:11:36.144Z] 2024/01/18 19:11:33 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:11:44.233Z] 2024/01/18 19:11:43 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:11:54.190Z] 2024/01/18 19:11:54 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:12:04.140Z] 2024/01/18 19:12:04 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:12:14.089Z] 2024/01/18 19:12:14 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:12:26.259Z] 2024/01/18 19:12:24 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:12:34.371Z] 2024/01/18 19:12:34 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:12:44.317Z] 2024/01/18 19:12:44 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:12:54.255Z] 2024/01/18 19:12:54 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:13:04.256Z] 2024/01/18 19:13:04 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:13:16.430Z] 2024/01/18 19:13:14 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:13:24.531Z] 2024/01/18 19:13:24 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:13:34.475Z] 2024/01/18 19:13:34 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:13:44.434Z] 2024/01/18 19:13:44 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:13:54.374Z] 2024/01/18 19:13:54 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:14:06.541Z] 2024/01/18 19:14:04 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:14:14.622Z] 2024/01/18 19:14:14 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:14:17.139Z] INFO[2024-01-18 19:14:16] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T19:14:17.139Z] INFO[2024-01-18 19:14:16] sched stats: tick interval: 13216 < 1.2s (100%) [2024-01-18T19:14:17.139Z] INFO[2024-01-18 19:14:16] sched stats: tick process durations: 13215 < 500ms (100%) [2024-01-18T19:14:25.238Z] 2024/01/18 19:14:24 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:14:35.185Z] 2024/01/18 19:14:34 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:14:45.136Z] 2024/01/18 19:14:44 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:14:55.077Z] 2024/01/18 19:14:54 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:15:05.020Z] 2024/01/18 19:15:04 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:15:14.977Z] 2024/01/18 19:15:14 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:15:24.925Z] 2024/01/18 19:15:24 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:15:34.881Z] 2024/01/18 19:15:34 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:15:44.823Z] 2024/01/18 19:15:44 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:15:54.801Z] 2024/01/18 19:15:54 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:16:04.710Z] 2024/01/18 19:16:04 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:16:14.647Z] 2024/01/18 19:16:14 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:16:26.817Z] 2024/01/18 19:16:24 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:16:34.902Z] 2024/01/18 19:16:34 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:16:44.844Z] 2024/01/18 19:16:44 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:16:54.787Z] 2024/01/18 19:16:54 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:17:06.965Z] 2024/01/18 19:17:04 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:17:15.054Z] 2024/01/18 19:17:14 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:17:24.996Z] 2024/01/18 19:17:24 DoRetryWithTimeout - Error: {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Next try in [10s], timeout [10m0s] [2024-01-18T19:17:34.983Z] INFO[2024-01-18 19:17:34] Verifying : Description : Verifiying the deletion of namespace [test-namespace-src-pxb-0] [2024-01-18T19:17:34.983Z] ERRO[2024-01-18 19:17:34] Actual:timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.983Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.983Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.983Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.983Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.983Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.983Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.983Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.983Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.983Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.983Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.983Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.983Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.983Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.983Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.983Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.983Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:17:34.984Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Expected:  [2024-01-18T19:17:35.248Z] INFO[2024-01-18 19:17:35] Verifying : Description : Deleting backup [tp-backup-1705604069] [2024-01-18T19:17:35.248Z] INFO[2024-01-18 19:17:35] Actual:true, Expected: true [2024-01-18T19:17:35.248Z] INFO[2024-01-18 19:17:35] Verifying : Description : Deleting backup [tp-backup-1705604160] [2024-01-18T19:17:35.248Z] INFO[2024-01-18 19:17:35] Actual:true, Expected: true [2024-01-18T19:17:35.248Z] 2024-01-18 19:17:35 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Destroying apps [2024-01-18T19:17:35.248Z] 2024-01-18 19:17:35 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T19:17:35.503Z] 2024-01-18 19:17:35 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).DeleteVolumes:#4024] - [mysql-ibm] PVC is not found: mysql-data, skipping deletion [2024-01-18T19:17:35.503Z] 2024-01-18 19:17:35 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] start destroying mysql-ibm app [2024-01-18T19:17:35.503Z] 2024-01-18 19:17:35 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T19:17:35.503Z] 2024-01-18 19:17:35 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T19:17:35.503Z] 2024-01-18 19:17:35 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] start destroying mysql-ibm app [2024-01-18T19:17:35.503Z] 2024-01-18 19:17:35 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T19:17:35.503Z] 2024-01-18 19:17:35 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T19:17:35.503Z] 2024-01-18 19:17:35 +0000:[WARNING] [{DeleteNSDeleteClusterRestore}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T19:17:35.503Z] INFO[2024-01-18 19:17:35] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T19:17:35.503Z] INFO[2024-01-18 19:17:35] Actual:true, Expected: true [2024-01-18T19:17:35.504Z] 2024-01-18 19:17:35 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Cleaning backup locations in map [map[2ac9e7f6-fe46-4e8f-b031-67023aec1477:aws-global-aws-px-backup-system-test-4cac76dc-bl-1705604064]], cloud credential [cred-aws-1705604063], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T19:17:35.758Z] INFO[2024-01-18 19:17:35] Verifying : Description : Verifying deletion of backup location [aws-global-aws-px-backup-system-test-4cac76dc-bl-1705604064] [2024-01-18T19:17:35.758Z] INFO[2024-01-18 19:17:35] Actual:true, Expected: true [2024-01-18T19:17:35.758Z] 2024-01-18 19:17:35 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [backup.IsBackupLocationPresent:#2442] - Backup location [aws-global-aws-px-backup-system-test-4cac76dc-bl-1705604064] is present [2024-01-18T19:17:35.758Z] 2024/01/18 19:17:35 DoRetryWithTimeout - Error: {backup location aws-global-aws-px-backup-system-test-4cac76dc-bl-1705604064 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T19:18:07.793Z] 2024-01-18 19:18:05 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [backup.IsBackupLocationPresent:#2442] - Backup location [aws-global-aws-px-backup-system-test-4cac76dc-bl-1705604064] is present [2024-01-18T19:18:07.793Z] 2024/01/18 19:18:05 DoRetryWithTimeout - Error: {backup location aws-global-aws-px-backup-system-test-4cac76dc-bl-1705604064 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T19:18:39.831Z] 2024-01-18 19:18:35 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [backup.IsBackupLocationPresent:#2442] - Backup location [aws-global-aws-px-backup-system-test-4cac76dc-bl-1705604064] is present [2024-01-18T19:18:39.831Z] 2024/01/18 19:18:35 DoRetryWithTimeout - Error: {backup location aws-global-aws-px-backup-system-test-4cac76dc-bl-1705604064 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T19:19:06.330Z] 2024-01-18 19:19:05 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu] [2024-01-18T19:19:06.330Z] INFO[2024-01-18 19:19:05] Verifying : Description : Verifying backup location deletion status aws-global-aws-px-backup-system-test-4cac76dc-bl-1705604064 [2024-01-18T19:19:06.330Z] INFO[2024-01-18 19:19:05] Actual:true, Expected: true [2024-01-18T19:19:06.330Z] 2024-01-18 19:19:06 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cred-aws-1705604063] is present [2024-01-18T19:19:06.330Z] INFO[2024-01-18 19:19:06] Verifying : Description : Verifying if cloud cred [cred-aws-1705604063] is present [2024-01-18T19:19:06.330Z] INFO[2024-01-18 19:19:06] Actual:true, Expected: true [2024-01-18T19:19:06.330Z] INFO[2024-01-18 19:19:06] Verifying : Description : Verifying deletion of cloud cred [cred-aws-1705604063] [2024-01-18T19:19:06.330Z] INFO[2024-01-18 19:19:06] Actual:true, Expected: true [2024-01-18T19:19:06.330Z] INFO[2024-01-18 19:19:06] Verifying : Description : Deleting cloud cred cred-aws-1705604063 [2024-01-18T19:19:06.330Z] INFO[2024-01-18 19:19:06] Actual:true, Expected: true [2024-01-18T19:19:06.330Z] INFO[2024-01-18 19:19:06] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T19:19:06.330Z] INFO[2024-01-18 19:19:06] Actual:true, Expected: true [2024-01-18T19:19:06.589Z] INFO[2024-01-18 19:19:06] Verifying : Description : Deleting cluster source-cluster [2024-01-18T19:19:06.589Z] INFO[2024-01-18 19:19:06] Actual:true, Expected: true [2024-01-18T19:19:06.844Z] INFO[2024-01-18 19:19:06] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-kopvx] [2024-01-18T19:19:06.844Z] INFO[2024-01-18 19:19:06] Actual:true, Expected: true [2024-01-18T19:19:07.101Z] INFO[2024-01-18 19:19:06] Verifying : Description : Deleting cluster destination-cluster [2024-01-18T19:19:07.101Z] INFO[2024-01-18 19:19:06] Actual:true, Expected: true [2024-01-18T19:19:07.364Z] INFO[2024-01-18 19:19:07] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-srwym] [2024-01-18T19:19:07.364Z] INFO[2024-01-18 19:19:07] Actual:true, Expected: true [2024-01-18T19:19:07.364Z] INFO[2024-01-18 19:19:07] --------Test End------ [2024-01-18T19:19:07.364Z] INFO[2024-01-18 19:19:07] #Test: [2024-01-18T19:19:07.364Z] INFO[2024-01-18 19:19:07] #Description: [2024-01-18T19:19:07.364Z] INFO[2024-01-18 19:19:07] ------------------------ [2024-01-18T19:19:07.364Z] 2024-01-18 19:19:07 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.EndPxBackupTorpedoTest:#7261] - >>>> FAILED TEST: {DeleteNSDeleteClusterRestore} Validates deleted namespace is restored when the application cluster is removed and re-added [2024-01-18T19:19:07.621Z] 2024-01-18 19:19:07 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T19:19:07.621Z] 2024-01-18 19:19:07 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Switching context to [/tmp/destination-config] [2024-01-18T19:19:07.621Z] 2024-01-18 19:19:07 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T19:19:07.621Z] 2024-01-18 19:19:07 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T19:19:07.879Z] 2024-01-18 19:19:07 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T19:19:07.879Z] 2024-01-18 19:19:07 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T19:19:07.879Z] 2024-01-18 19:19:07 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T19:19:08.136Z] 2024-01-18 19:19:07 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T19:19:08.393Z] 2024-01-18 19:19:08 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T19:19:08.958Z] 2024-01-18 19:19:08 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T19:19:09.218Z] 2024-01-18 19:19:09 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T19:19:09.475Z] 2024-01-18 19:19:09 +0000:[WARNING] [{DeleteNSDeleteClusterRestore}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T19:19:09.731Z] 2024-01-18 19:19:09 +0000:[WARNING] [{DeleteNSDeleteClusterRestore}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T19:19:09.731Z] 2024-01-18 19:19:09 +0000:[WARNING] [{DeleteNSDeleteClusterRestore}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T19:19:09.731Z] 2024-01-18 19:19:09 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T19:19:09.731Z] 2024-01-18 19:19:09 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Switched context to [/tmp/destination-config] [2024-01-18T19:19:09.995Z] 2024-01-18 19:19:09 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {DeleteNSDeleteClusterRestore} Validates deleted namespace is restored when the application cluster is removed and re-added [2024-01-18T19:19:09.995Z] 2024-01-18 19:19:09 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-18T19:19:09.995Z] 2024/01/18 19:19:09 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T19:19:15.243Z] 2024/01/18 19:19:14 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T19:19:20.490Z] 2024/01/18 19:19:19 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T19:19:25.739Z] 2024/01/18 19:19:24 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T19:19:29.907Z] 2024/01/18 19:19:29 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T19:19:35.182Z] 2024/01/18 19:19:34 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T19:19:40.448Z] 2024/01/18 19:19:39 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T19:19:45.696Z] 2024/01/18 19:19:44 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T19:19:49.863Z] 2024/01/18 19:19:49 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.0.4. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.4 10.240.0.4]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T19:19:49.863Z] {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T19:19:49.863Z] {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T19:19:49.863Z] {dial tcp 10.240.0.4:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-18T19:19:55.109Z] 2024-01-18 19:19:54 +0000:[WARNING] [{DeleteNSDeleteClusterRestore}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.4. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.4 10.240.0.4]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T19:19:55.109Z] {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T19:19:55.109Z] {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T19:19:55.110Z] {dial tcp 10.240.0.4:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T19:19:55.110Z] 2024-01-18 19:19:54 +0000:[ERROR] [{DeleteNSDeleteClusterRestore}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.4. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.4 10.240.0.4]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T19:19:55.110Z] {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T19:19:55.110Z] {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T19:19:55.110Z] {dial tcp 10.240.0.4:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T19:19:55.110Z] 2024-01-18 19:19:54 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T19:19:55.110Z] 2024-01-18 19:19:54 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Switching context to [/tmp/source-config] [2024-01-18T19:19:55.110Z] 2024-01-18 19:19:54 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T19:19:55.110Z] 2024-01-18 19:19:54 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T19:19:55.110Z] 2024-01-18 19:19:54 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T19:19:55.110Z] 2024-01-18 19:19:54 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T19:19:55.110Z] 2024-01-18 19:19:54 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T19:19:55.366Z] 2024-01-18 19:19:55 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T19:19:55.622Z] 2024-01-18 19:19:55 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T19:19:56.225Z] 2024-01-18 19:19:55 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T19:19:56.480Z] 2024-01-18 19:19:56 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T19:19:56.736Z] 2024-01-18 19:19:56 +0000:[WARNING] [{DeleteNSDeleteClusterRestore}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T19:19:56.992Z] 2024-01-18 19:19:56 +0000:[WARNING] [{DeleteNSDeleteClusterRestore}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T19:19:56.992Z] 2024-01-18 19:19:56 +0000:[WARNING] [{DeleteNSDeleteClusterRestore}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T19:19:56.992Z] 2024-01-18 19:19:56 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T19:19:56.992Z] 2024-01-18 19:19:56 +0000:[INFO] [{DeleteNSDeleteClusterRestore}] Switched context to [/tmp/source-config] [2024-01-18T19:19:56.992Z] • Failure [1638.300 seconds] [2024-01-18T19:19:56.992Z] {DeleteNSDeleteClusterRestore} [2024-01-18T19:19:56.992Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:3174 [2024-01-18T19:19:56.992Z] Validates deleted namespace is restored when the application cluster is removed and re-added [It] [2024-01-18T19:19:56.992Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:3210 [2024-01-18T19:19:56.992Z] [2024-01-18T19:19:56.992Z] Unexpected error: [2024-01-18T19:19:56.992Z] <*errors.errorString | 0xc0013a06c0>: { [2024-01-18T19:19:56.994Z] s: "error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Terminating phase }\n{namespace - test-namespace-src-pxb-0 is in Te... [2024-01-18T19:19:56.994Z] [2024-01-18T19:19:56.994Z] Gomega truncated this representation as it exceeds 'format.MaxLength'. [2024-01-18T19:19:56.994Z] Consider having the object provide a custom 'GomegaStringer' representation [2024-01-18T19:19:56.994Z] or adjust the parameters in Gomega's 'format' package. [2024-01-18T19:19:56.994Z] [2024-01-18T19:19:56.994Z] Learn more here: https://onsi.github.io/gomega/#adjusting-output [2024-01-18T19:19:56.994Z] [2024-01-18T19:19:56.994Z] error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.994Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.994Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.994Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.994Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.994Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.994Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.994Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.994Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.994Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.994Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.995Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.996Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.996Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.996Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.996Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.996Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.996Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.996Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.996Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.996Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.996Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.996Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.996Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.996Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.996Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.996Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.996Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase } [2024-01-18T19:19:56.996Z] {namespace - test-namespace-src-pxb-0 is in Terminating phase }, Description: Verifiying the deletion of namespace [test-namespace-src-pxb-0] [2024-01-18T19:19:56.996Z] occurred [2024-01-18T19:19:56.996Z] [2024-01-18T19:19:56.996Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-18T19:19:56.996Z] [2024-01-18T19:19:56.996Z] Full Stack Trace [2024-01-18T19:19:56.996Z] github.com/portworx/torpedo/pkg/aetosutil.(*Dashboard).VerifyFatal(0x7f6bebb?, {0x7038360?, 0xc0013a0560}, {0x0?, 0x0}, {0xc0013ca6c0, 0x3f}) [2024-01-18T19:19:56.996Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 +0x143 [2024-01-18T19:19:56.996Z] github.com/portworx/torpedo/tests/backup.glob..func71.2.5() [2024-01-18T19:19:56.996Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:3271 +0x12f [2024-01-18T19:19:56.996Z] github.com/portworx/torpedo/tests/backup.glob..func71.2() [2024-01-18T19:19:56.996Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:3267 +0x41e [2024-01-18T19:19:56.996Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-18T19:19:56.996Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-18T19:19:56.996Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-18T19:19:56.996Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-18T19:19:56.996Z] created by testing.(*T).Run [2024-01-18T19:19:56.996Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-18T19:19:56.996Z] ------------------------------ [2024-01-18T19:19:56.996Z] STEP: schedule applications [2024-01-18T19:19:56.996Z] STEP: schedule applications [2024-01-18T19:19:56.996Z] STEP: Validate applications [2024-01-18T19:19:56.996Z] STEP: validate applications [2024-01-18T19:19:56.997Z] STEP: validate mysql-ibm app's volumes [2024-01-18T19:19:56.997Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T19:19:56.997Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T19:19:56.997Z] STEP: get mysql-ibm app's volume: pvc-4aad4541-041e-4274-8eab-6e5efac119d1 inspected by the volume driver [2024-01-18T19:19:56.997Z] STEP: wait for mysql-ibm app to start running [2024-01-18T19:19:56.997Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T19:19:56.997Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T19:19:56.997Z] STEP: Validate Px pod restart count [2024-01-18T19:19:56.997Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T19:19:56.997Z] STEP: validate mysql-ibm app's volumes [2024-01-18T19:19:56.997Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T19:19:56.997Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T19:19:56.997Z] STEP: get mysql-ibm app's volume: pvc-478ae7f7-4348-4ed7-960f-e9cc2b32c469 inspected by the volume driver [2024-01-18T19:19:56.997Z] STEP: wait for mysql-ibm app to start running [2024-01-18T19:19:56.997Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T19:19:56.997Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T19:19:56.997Z] STEP: Validate Px pod restart count [2024-01-18T19:19:56.997Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T19:19:56.997Z] STEP: Create cloud credentials and backup locations [2024-01-18T19:19:56.997Z] STEP: Add source and destination clusters with px-central-admin ctx [2024-01-18T19:19:56.997Z] STEP: Taking backup of applications [2024-01-18T19:19:56.997Z] STEP: Delete the App namespaces created [2024-01-18T19:19:56.997Z] STEP: destroy apps [2024-01-18T19:19:56.997Z] STEP: destroy the mysql-ibm app's volumes [2024-01-18T19:19:56.997Z] STEP: start destroying mysql-ibm app [2024-01-18T19:19:56.997Z] STEP: destroy the mysql-ibm app's volumes [2024-01-18T19:19:56.997Z] STEP: start destroying mysql-ibm app [2024-01-18T19:19:56.997Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T19:19:56.997Z] SS2024-01-18 19:19:56 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Log Dir: /testresults//AllNSBackupWithIncludeNewNSOption.log [2024-01-18T19:19:56.997Z] INFO[2024-01-18 19:19:56] --------Test Start------ [2024-01-18T19:19:56.997Z] INFO[2024-01-18 19:19:56] #Test: AllNSBackupWithIncludeNewNSOption [2024-01-18T19:19:56.997Z] INFO[2024-01-18 19:19:56] #Description: Verification of schedule backups created with include new namespaces option [2024-01-18T19:19:56.997Z] INFO[2024-01-18 19:19:56] ------------------------ [2024-01-18T19:19:58.364Z] 2024-01-18 19:19:58 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Creating cloud credentials and backup locations [2024-01-18T19:19:58.364Z] 2024-01-18 19:19:58 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Creating cloud credential named [48f219bc-3c87-4b44-bb04-a629bc2026fc] and uid [cred-aws-1705605598] using [aws] as provider [2024-01-18T19:19:58.364Z] 2024-01-18 19:19:58 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [cred-aws-1705605598] for org [default] with [aws] as provider [2024-01-18T19:19:58.364Z] 2024-01-18 19:19:58 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-18T19:19:58.619Z] 2024-01-18 19:19:58 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cred-aws-1705605598] is present [2024-01-18T19:19:58.619Z] INFO[2024-01-18 19:19:58] Verifying : Description : Verifying creation of cloud credential named [cred-aws-1705605598] for org [default] with [aws] as provider [2024-01-18T19:19:58.619Z] INFO[2024-01-18 19:19:58] Actual:true, Expected: true [2024-01-18T19:20:01.886Z] INFO[2024-01-18 19:20:01] Verifying : Description : Verifying creation of backup location named [aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598] with uid [4c14da25-9552-4acb-a4e8-a02da0fa1688] of [aws] as provider [2024-01-18T19:20:01.886Z] INFO[2024-01-18 19:20:01] Actual:true, Expected: true [2024-01-18T19:20:01.886Z] 2024-01-18 19:20:01 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Adding source and destination clusters with px-central-admin ctx [2024-01-18T19:20:01.886Z] 2024-01-18 19:20:01 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.glob..func60.2.2:#979] - Creating source [source-cluster] and destination [destination-cluster] clusters [2024-01-18T19:20:01.886Z] INFO[2024-01-18 19:20:01] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T19:20:01.886Z] INFO[2024-01-18 19:20:01] Actual:true, Expected: true [2024-01-18T19:20:01.886Z] 2024-01-18 19:20:01 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T19:20:01.886Z] 2024-01-18 19:20:01 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T19:20:01.886Z] 2024-01-18 19:20:01 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T19:20:01.886Z] 2024-01-18 19:20:01 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T19:20:01.886Z] 2024-01-18 19:20:01 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Create cluster [source-cluster] in org [default] [2024-01-18T19:20:01.886Z] 2024-01-18 19:20:01 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T19:20:01.886Z] 2024-01-18 19:20:01 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T19:20:01.886Z] 2024-01-18 19:20:01 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Create cluster [destination-cluster] in org [default] [2024-01-18T19:20:01.886Z] 2024-01-18 19:20:01 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T19:20:01.886Z] 2024-01-18 19:20:01 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T19:20:01.886Z] 2024-01-18 19:20:01 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-seowm] for IBM [2024-01-18T19:20:01.887Z] 2024-01-18 19:20:01 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-seowm] for org [default] with [ibm] as provider [2024-01-18T19:20:01.887Z] 2024-01-18 19:20:01 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T19:20:01.887Z] 2024-01-18 19:20:01 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T19:20:02.144Z] 2024-01-18 19:20:01 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-seowm] is present [2024-01-18T19:20:02.144Z] 2024-01-18 19:20:01 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Create cluster [source-cluster] in org [default] [2024-01-18T19:20:02.715Z] 2024-01-18 19:20:02 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-zbbvt] for IBM [2024-01-18T19:20:02.715Z] 2024-01-18 19:20:02 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-zbbvt] for org [default] with [ibm] as provider [2024-01-18T19:20:02.715Z] 2024-01-18 19:20:02 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T19:20:02.715Z] 2024-01-18 19:20:02 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T19:20:02.715Z] 2024-01-18 19:20:02 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-zbbvt] is present [2024-01-18T19:20:02.715Z] 2024-01-18 19:20:02 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Create cluster [destination-cluster] in org [default] [2024-01-18T19:20:03.280Z] INFO[2024-01-18 19:20:03] Verifying : Description : Verifying creation of source [source-cluster] and destination [destination-cluster] clusters with px-central-admin ctx [2024-01-18T19:20:03.280Z] INFO[2024-01-18 19:20:03] Actual:true, Expected: true [2024-01-18T19:20:03.280Z] INFO[2024-01-18 19:20:03] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-18T19:20:03.280Z] INFO[2024-01-18 19:20:03] Actual:Online, Expected: Online [2024-01-18T19:20:03.280Z] 2024-01-18 19:20:03 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.glob..func60.2.2:#987] - Cluster [source-cluster] uid: [db13d75d-31aa-465d-9e0c-963efb810f31] [2024-01-18T19:20:03.280Z] INFO[2024-01-18 19:20:03] Verifying : Description : Verifying if [destination-cluster] cluster is online [2024-01-18T19:20:03.280Z] INFO[2024-01-18 19:20:03] Actual:Online, Expected: Online [2024-01-18T19:20:03.280Z] 2024-01-18 19:20:03 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.glob..func60.2.2:#993] - Cluster [destination-cluster] uid: [5f787597-2bbc-4af4-8e4a-caba411c5430] [2024-01-18T19:20:03.280Z] 2024-01-18 19:20:03 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Creating a schedule policy with interval [15] mins [2024-01-18T19:20:03.280Z] 2024-01-18 19:20:03 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Create Backup Schedule Policy [2024-01-18T19:20:03.539Z] INFO[2024-01-18 19:20:03] Verifying : Description : Verifying creation of schedule policy [interval-15-1705605603] with interval [15] mins [2024-01-18T19:20:03.539Z] INFO[2024-01-18 19:20:03] Actual:true, Expected: true [2024-01-18T19:20:03.539Z] 2024-01-18 19:20:03 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.glob..func60.2.3:#1004] - Schedule policy [interval-15-1705605603] uid: [8565a4ad-e89f-4b9d-ba88-bcfba5866aa5] [2024-01-18T19:20:03.539Z] 2024-01-18 19:20:03 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Scheduling applications in destination cluster [2024-01-18T19:20:03.540Z] 2024-01-18 19:20:03 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T19:20:03.540Z] 2024-01-18 19:20:03 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Switching context to [/tmp/destination-config] [2024-01-18T19:20:03.540Z] 2024-01-18 19:20:03 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T19:20:03.540Z] 2024-01-18 19:20:03 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T19:20:03.799Z] 2024-01-18 19:20:03 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T19:20:03.799Z] 2024-01-18 19:20:03 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T19:20:03.799Z] 2024-01-18 19:20:03 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T19:20:04.057Z] 2024-01-18 19:20:03 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T19:20:04.314Z] 2024-01-18 19:20:04 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T19:20:04.877Z] 2024-01-18 19:20:04 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T19:20:05.133Z] 2024-01-18 19:20:05 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T19:20:05.390Z] 2024-01-18 19:20:05 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T19:20:05.647Z] 2024-01-18 19:20:05 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T19:20:05.647Z] 2024-01-18 19:20:05 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T19:20:05.647Z] 2024-01-18 19:20:05 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T19:20:05.647Z] 2024-01-18 19:20:05 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Switched context to [/tmp/destination-config] [2024-01-18T19:20:05.647Z] 2024-01-18 19:20:05 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T19:20:05.648Z] 2024-01-18 19:20:05 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T19:20:05.903Z] 2024-01-18 19:20:05 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T19:20:05.903Z] 2024-01-18 19:20:05 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T19:20:05.903Z] 2024-01-18 19:20:05 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T19:20:06.158Z] 2024-01-18 19:20:06 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T19:20:06.159Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T19:20:06.159Z] 2024-01-18 19:20:06 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T19:20:06.160Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T19:20:06.160Z] 2024-01-18 19:20:06 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T19:20:06.160Z] 2024-01-18 19:20:06 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Scheduled application [mysql-ibm] in destination cluster in namespace [mysql-ibm-dst-pxb-0-84760-01-18-19h19m56s] [2024-01-18T19:20:06.160Z] 2024-01-18 19:20:06 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T19:20:06.160Z] 2024-01-18 19:20:06 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T19:20:06.416Z] 2024-01-18 19:20:06 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T19:20:06.416Z] 2024-01-18 19:20:06 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T19:20:06.672Z] 2024-01-18 19:20:06 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T19:20:06.928Z] 2024-01-18 19:20:06 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T19:20:06.929Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T19:20:06.929Z] 2024-01-18 19:20:06 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T19:20:06.930Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T19:20:06.930Z] 2024-01-18 19:20:06 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T19:20:06.930Z] 2024-01-18 19:20:06 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Scheduled application [mysql-ibm] in destination cluster in namespace [mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s] [2024-01-18T19:20:06.930Z] 2024-01-18 19:20:06 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating app namespaces in destination cluster [2024-01-18T19:20:06.930Z] 2024-01-18 19:20:06 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validate applications [2024-01-18T19:20:06.930Z] 2024-01-18 19:20:06 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating mysql-ibm app [2024-01-18T19:20:06.930Z] 2024-01-18 19:20:06 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating mysql-ibm app's volumes [2024-01-18T19:20:06.930Z] 2024-01-18 19:20:06 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T19:20:07.188Z] 2024/01/18 19:20:07 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T19:20:17.140Z] 2024/01/18 19:20:17 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T19:20:29.344Z] 2024-01-18 19:20:27 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-dst-pxb-0-84760-01-18-19h19m56s [2024-01-18T19:20:29.344Z] 2024-01-18 19:20:27 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T19:20:29.344Z] 2024-01-18 19:20:27 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T19:20:29.344Z] 2024-01-18 19:20:27 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] wait for mysql-ibm app to start running [2024-01-18T19:20:29.344Z] 2024-01-18 19:20:27 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T19:20:29.344Z] 2024/01/18 19:20:27 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:20:29.345Z] pod name:mysql-7d766cd65c-sgr7x namespace:mysql-ibm-dst-pxb-0-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T19:20:29.345Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T19:20:37.455Z] 2024/01/18 19:20:37 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:20:37.455Z] pod name:mysql-7d766cd65c-sgr7x namespace:mysql-ibm-dst-pxb-0-84760-01-18-19h19m56s running:true ready:false node:10.240.64.29 [2024-01-18T19:20:37.455Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T19:20:49.633Z] 2024/01/18 19:20:47 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:20:49.633Z] pod name:mysql-7d766cd65c-sgr7x namespace:mysql-ibm-dst-pxb-0-84760-01-18-19h19m56s running:true ready:false node:10.240.64.29 [2024-01-18T19:20:49.633Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T19:20:57.730Z] 2024/01/18 19:20:57 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:20:57.730Z] pod name:mysql-7d766cd65c-sgr7x namespace:mysql-ibm-dst-pxb-0-84760-01-18-19h19m56s running:true ready:false node:10.240.64.29 [2024-01-18T19:20:57.730Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T19:21:07.744Z] 2024/01/18 19:21:07 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:21:07.744Z] pod name:mysql-7d766cd65c-sgr7x namespace:mysql-ibm-dst-pxb-0-84760-01-18-19h19m56s running:true ready:false node:10.240.64.29 [2024-01-18T19:21:07.744Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T19:21:17.724Z] 2024/01/18 19:21:17 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:21:17.724Z] pod name:mysql-7d766cd65c-sgr7x namespace:mysql-ibm-dst-pxb-0-84760-01-18-19h19m56s running:true ready:false node:10.240.64.29 [2024-01-18T19:21:17.724Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T19:21:29.904Z] 2024/01/18 19:21:27 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:21:29.904Z] pod name:mysql-7d766cd65c-sgr7x namespace:mysql-ibm-dst-pxb-0-84760-01-18-19h19m56s running:true ready:false node:10.240.64.29 [2024-01-18T19:21:29.904Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T19:21:38.020Z] 2024/01/18 19:21:37 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:21:38.020Z] pod name:mysql-7d766cd65c-sgr7x namespace:mysql-ibm-dst-pxb-0-84760-01-18-19h19m56s running:true ready:false node:10.240.64.29 [2024-01-18T19:21:38.020Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T19:21:47.975Z] 2024-01-18 19:21:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T19:21:47.975Z] 2024-01-18 19:21:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T19:21:47.975Z] 2024-01-18 19:21:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] validate if mysql-ibm app's volumes are setup [2024-01-18T19:21:48.231Z] 2024-01-18 19:21:47 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T19:21:48.231Z] 2024-01-18 19:21:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T19:21:48.231Z] 2024-01-18 19:21:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T19:21:48.231Z] 2024-01-18 19:21:47 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T19:21:48.231Z] 2024-01-18 19:21:48 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating mysql-ibm app [2024-01-18T19:21:48.231Z] 2024-01-18 19:21:48 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating mysql-ibm app's volumes [2024-01-18T19:21:48.231Z] 2024-01-18 19:21:48 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T19:21:48.231Z] 2024-01-18 19:21:48 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T19:21:48.231Z] 2024-01-18 19:21:48 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s [2024-01-18T19:21:48.231Z] 2024-01-18 19:21:48 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T19:21:48.231Z] 2024-01-18 19:21:48 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T19:21:48.231Z] 2024-01-18 19:21:48 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] wait for mysql-ibm app to start running [2024-01-18T19:21:48.231Z] 2024-01-18 19:21:48 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T19:21:48.488Z] 2024-01-18 19:21:48 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T19:21:49.050Z] 2024-01-18 19:21:48 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T19:21:49.981Z] 2024-01-18 19:21:49 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] validate if mysql-ibm app's volumes are setup [2024-01-18T19:21:49.981Z] 2024-01-18 19:21:49 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T19:21:49.981Z] 2024-01-18 19:21:49 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T19:21:49.981Z] 2024-01-18 19:21:49 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T19:21:49.981Z] 2024-01-18 19:21:49 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T19:21:49.981Z] 2024-01-18 19:21:49 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Creating a schedule backup [2024-01-18T19:21:49.981Z] 2024-01-18 19:21:49 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-18T19:21:49.981Z] 2024-01-18 19:21:49 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Volume snapshot classes found in the cluster - [2024-01-18T19:21:49.981Z] 2024-01-18 19:21:49 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] ibmc-vpcblock-snapshot [2024-01-18T19:21:49.981Z] 2024-01-18 19:21:49 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T19:22:57.643Z] 2024/01/18 19:22:50 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T19:23:25.172Z] 2024/01/18 19:23:20 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T19:23:51.678Z] 2024/01/18 19:23:50 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T19:24:18.196Z] INFO[2024-01-18 19:24:17] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T19:24:18.196Z] INFO[2024-01-18 19:24:17] sched stats: tick interval: 13817 < 1.2s (100%) [2024-01-18T19:24:18.196Z] INFO[2024-01-18 19:24:17] sched stats: tick process durations: 13816 < 500ms (100%) [2024-01-18T19:24:20.712Z] 2024/01/18 19:24:20 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T19:24:52.745Z] 2024/01/18 19:24:50 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T19:25:24.797Z] 2024/01/18 19:25:20 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T19:25:51.318Z] 2024/01/18 19:25:50 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T19:26:23.402Z] 2024/01/18 19:26:20 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T19:26:55.430Z] 2024/01/18 19:26:50 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T19:27:21.935Z] 2024/01/18 19:27:20 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T19:27:53.969Z] 2024/01/18 19:27:50 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T19:28:26.010Z] 2024/01/18 19:28:20 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T19:28:52.510Z] 2024/01/18 19:28:50 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T19:29:24.536Z] 2024/01/18 19:29:20 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T19:29:51.050Z] 2024/01/18 19:29:50 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T19:30:23.160Z] 2024/01/18 19:30:21 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T19:30:55.196Z] 2024/01/18 19:30:51 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T19:31:21.699Z] 2024/01/18 19:31:21 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T19:31:53.721Z] 2024/01/18 19:31:51 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T19:32:25.757Z] 2024/01/18 19:32:21 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T19:32:52.254Z] 2024-01-18 19:32:51 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.CreateScheduleBackup:#508] - Schedule backup [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] created successfully [2024-01-18T19:32:52.254Z] INFO[2024-01-18 19:32:51] Verifying : Description : Verifying creation of schedule backup with schedule name [tp-backup-schedule-1705605709] [2024-01-18T19:32:52.254Z] INFO[2024-01-18 19:32:51] Actual:true, Expected: true [2024-01-18T19:32:52.254Z] 2024-01-18 19:32:51 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating backup [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] in org [default] [2024-01-18T19:32:52.254Z] 2024-01-18 19:32:51 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] [2024-01-18T19:32:52.254Z] 2024-01-18 19:32:51 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-dst-pxb-0-84760-01-18-19h19m56s] in backup [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] [2024-01-18T19:32:52.254Z] 2024-01-18 19:32:51 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-dst-pxb-0-84760-01-18-19h19m56s] in backup [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] [2024-01-18T19:32:52.254Z] 2024-01-18 19:32:51 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-dst-pxb-0-84760-01-18-19h19m56s] [2024-01-18T19:32:52.254Z] 2024-01-18 19:32:51 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T19:32:52.254Z] 2024-01-18 19:32:51 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T19:32:52.254Z] 2024-01-18 19:32:51 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-dst-pxb-0-84760-01-18-19h19m56s] are [[mysql-data]] [2024-01-18T19:32:52.254Z] 2024-01-18 19:32:51 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T19:32:52.254Z] 2024-01-18 19:32:51 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s] in backup [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] [2024-01-18T19:32:52.254Z] 2024-01-18 19:32:51 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s] in backup [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] [2024-01-18T19:32:52.255Z] 2024-01-18 19:32:51 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s] [2024-01-18T19:32:52.255Z] 2024-01-18 19:32:51 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T19:32:52.255Z] 2024-01-18 19:32:51 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T19:32:52.255Z] 2024-01-18 19:32:51 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s] are [[mysql-data]] [2024-01-18T19:32:52.255Z] 2024-01-18 19:32:51 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T19:32:52.255Z] 2024-01-18 19:32:51 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating CR cleanup [2024-01-18T19:32:52.255Z] 2024-01-18 19:32:51 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T19:32:52.255Z] 2024-01-18 19:32:51 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:32:52.255Z] 2024-01-18 19:32:51 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:32:52.255Z] 2024/01/18 19:32:51 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:32:57.533Z] 2024-01-18 19:32:57 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:32:57.533Z] 2024-01-18 19:32:57 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:32:57.533Z] 2024/01/18 19:32:57 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:33:02.784Z] 2024-01-18 19:33:02 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:33:02.784Z] 2024-01-18 19:33:02 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:33:02.784Z] 2024/01/18 19:33:02 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:33:08.030Z] 2024-01-18 19:33:07 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:33:08.030Z] 2024-01-18 19:33:07 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:33:08.030Z] 2024/01/18 19:33:07 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:33:13.272Z] 2024-01-18 19:33:12 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:33:13.272Z] 2024-01-18 19:33:12 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:33:13.272Z] 2024/01/18 19:33:12 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:33:17.456Z] 2024-01-18 19:33:17 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:33:17.456Z] 2024-01-18 19:33:17 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:33:17.456Z] 2024/01/18 19:33:17 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:33:23.237Z] 2024-01-18 19:33:22 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:33:23.237Z] 2024-01-18 19:33:22 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:33:23.237Z] 2024/01/18 19:33:22 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:33:28.484Z] 2024-01-18 19:33:27 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:33:28.484Z] 2024-01-18 19:33:27 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:33:28.484Z] 2024/01/18 19:33:27 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:33:32.656Z] 2024-01-18 19:33:32 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:33:32.656Z] 2024-01-18 19:33:32 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:33:32.656Z] 2024/01/18 19:33:32 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:33:37.898Z] 2024-01-18 19:33:37 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:33:37.898Z] 2024-01-18 19:33:37 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:33:37.898Z] 2024/01/18 19:33:37 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:33:43.142Z] 2024-01-18 19:33:42 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:33:43.142Z] 2024-01-18 19:33:42 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:33:43.142Z] 2024/01/18 19:33:42 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:33:48.388Z] 2024-01-18 19:33:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:33:48.388Z] 2024-01-18 19:33:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:33:48.388Z] 2024/01/18 19:33:47 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:33:53.631Z] 2024-01-18 19:33:52 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:33:53.631Z] 2024-01-18 19:33:52 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:33:53.631Z] 2024/01/18 19:33:52 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:33:58.877Z] 2024-01-18 19:33:57 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:33:58.877Z] 2024-01-18 19:33:57 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:33:58.877Z] 2024/01/18 19:33:57 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:34:03.048Z] 2024-01-18 19:34:02 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:34:03.048Z] 2024-01-18 19:34:02 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:34:03.048Z] 2024/01/18 19:34:02 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:34:04.414Z] Iteration: 60 [2024-01-18T19:34:05.780Z] Logs from iteration: 60 [2024-01-18T19:34:06.351Z] 2024-01-18 19:33:07 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:34:06.351Z] 2024-01-18 19:33:07 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:34:06.351Z] 2024/01/18 19:33:07 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:34:06.351Z] 2024-01-18 19:33:12 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:34:06.351Z] 2024-01-18 19:33:12 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:34:06.351Z] 2024/01/18 19:33:12 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:34:06.351Z] 2024-01-18 19:33:17 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:34:06.351Z] 2024-01-18 19:33:17 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:34:06.351Z] 2024/01/18 19:33:17 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:34:06.351Z] 2024-01-18 19:33:22 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:34:06.351Z] 2024-01-18 19:33:22 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:34:06.351Z] 2024/01/18 19:33:22 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:34:06.351Z] 2024-01-18 19:33:27 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:34:06.352Z] 2024-01-18 19:33:27 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:34:06.352Z] 2024/01/18 19:33:27 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:34:06.352Z] 2024-01-18 19:33:32 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:34:06.352Z] 2024-01-18 19:33:32 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:34:06.352Z] 2024/01/18 19:33:32 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:34:06.352Z] 2024-01-18 19:33:37 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:34:06.352Z] 2024-01-18 19:33:37 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:34:06.352Z] 2024/01/18 19:33:37 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:34:06.352Z] 2024-01-18 19:33:42 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:34:06.352Z] 2024-01-18 19:33:42 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:34:06.352Z] 2024/01/18 19:33:42 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:34:06.352Z] 2024-01-18 19:33:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:34:06.352Z] 2024-01-18 19:33:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:34:06.352Z] 2024/01/18 19:33:47 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:34:06.352Z] 2024-01-18 19:33:52 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:34:06.352Z] 2024-01-18 19:33:52 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:34:06.352Z] 2024/01/18 19:33:52 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:34:06.352Z] 2024-01-18 19:33:57 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:34:06.352Z] 2024-01-18 19:33:57 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:34:06.352Z] 2024/01/18 19:33:57 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:34:06.352Z] 2024-01-18 19:34:02 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:34:06.352Z] 2024-01-18 19:34:02 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:34:06.353Z] 2024/01/18 19:34:02 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:34:08.243Z] 2024-01-18 19:34:08 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:34:08.243Z] 2024-01-18 19:34:08 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:34:08.243Z] 2024/01/18 19:34:08 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:34:13.507Z] 2024-01-18 19:34:13 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:34:13.507Z] 2024-01-18 19:34:13 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:34:13.507Z] 2024/01/18 19:34:13 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:34:18.754Z] 2024-01-18 19:34:18 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:34:18.754Z] 2024-01-18 19:34:18 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:34:18.754Z] 2024/01/18 19:34:18 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:34:18.754Z] INFO[2024-01-18 19:34:18] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T19:34:18.754Z] INFO[2024-01-18 19:34:18] sched stats: tick interval: 14418 < 1.2s (100%) [2024-01-18T19:34:18.754Z] INFO[2024-01-18 19:34:18] sched stats: tick process durations: 14417 < 500ms (100%) [2024-01-18T19:34:23.992Z] 2024-01-18 19:34:23 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] [2024-01-18T19:34:23.992Z] 2024-01-18 19:34:23 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] CR found for [[tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]] under [kube-system] namespace [2024-01-18T19:34:23.993Z] 2024/01/18 19:34:23 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150]}, Next try in [5s], timeout [5m0s] [2024-01-18T19:34:29.237Z] 2024-01-18 19:34:28 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[]] [2024-01-18T19:34:29.237Z] INFO[2024-01-18 19:34:28] Verifying : Description : Validation of the first schedule backup [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150] [2024-01-18T19:34:29.237Z] INFO[2024-01-18 19:34:28] Actual:true, Expected: true [2024-01-18T19:34:29.237Z] 2024-01-18 19:34:28 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Scheduling applications in destination cluster to create new namespaces [2024-01-18T19:34:29.237Z] 2024-01-18 19:34:28 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T19:34:29.237Z] 2024-01-18 19:34:28 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T19:34:29.237Z] 2024-01-18 19:34:28 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T19:34:29.237Z] 2024-01-18 19:34:28 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T19:34:29.237Z] 2024-01-18 19:34:28 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T19:34:29.237Z] 2024-01-18 19:34:28 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T19:34:29.238Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T19:34:29.238Z] 2024-01-18 19:34:28 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T19:34:29.238Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T19:34:29.238Z] 2024-01-18 19:34:28 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T19:34:29.238Z] 2024-01-18 19:34:28 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Scheduled application [mysql-ibm] in destination cluster in namespace [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s] [2024-01-18T19:34:29.238Z] 2024-01-18 19:34:28 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating new app namespaces in destination cluster [2024-01-18T19:34:29.238Z] 2024-01-18 19:34:28 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validate applications [2024-01-18T19:34:29.238Z] 2024-01-18 19:34:28 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating mysql-ibm app [2024-01-18T19:34:29.238Z] 2024-01-18 19:34:28 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating mysql-ibm app's volumes [2024-01-18T19:34:29.238Z] 2024-01-18 19:34:28 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T19:34:29.238Z] 2024/01/18 19:34:28 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T19:34:39.193Z] 2024/01/18 19:34:38 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T19:34:49.131Z] 2024-01-18 19:34:48 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s [2024-01-18T19:34:49.131Z] 2024-01-18 19:34:48 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T19:34:49.131Z] 2024-01-18 19:34:48 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T19:34:49.131Z] 2024-01-18 19:34:48 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] wait for mysql-ibm app to start running [2024-01-18T19:34:49.131Z] 2024-01-18 19:34:48 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T19:34:49.131Z] 2024/01/18 19:34:48 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:34:49.131Z] pod name:mysql-7d766cd65c-drvc5 namespace:mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s running:false ready:false node:10.240.64.6 [2024-01-18T19:34:49.131Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T19:34:59.071Z] 2024/01/18 19:34:58 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:34:59.071Z] pod name:mysql-7d766cd65c-drvc5 namespace:mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s running:true ready:false node:10.240.64.6 [2024-01-18T19:34:59.071Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T19:35:11.241Z] 2024/01/18 19:35:09 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:35:11.241Z] pod name:mysql-7d766cd65c-drvc5 namespace:mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s running:true ready:false node:10.240.64.6 [2024-01-18T19:35:11.241Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T19:35:19.324Z] 2024/01/18 19:35:19 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:35:19.324Z] pod name:mysql-7d766cd65c-drvc5 namespace:mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s running:true ready:false node:10.240.64.6 [2024-01-18T19:35:19.324Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T19:35:29.271Z] 2024/01/18 19:35:29 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:35:29.271Z] pod name:mysql-7d766cd65c-drvc5 namespace:mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s running:true ready:false node:10.240.64.6 [2024-01-18T19:35:29.271Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T19:35:41.443Z] 2024/01/18 19:35:39 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:35:41.443Z] pod name:mysql-7d766cd65c-drvc5 namespace:mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s running:true ready:false node:10.240.64.6 [2024-01-18T19:35:41.443Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T19:35:49.529Z] 2024/01/18 19:35:49 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:35:49.529Z] pod name:mysql-7d766cd65c-drvc5 namespace:mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s running:true ready:false node:10.240.64.6 [2024-01-18T19:35:49.529Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T19:35:59.502Z] 2024/01/18 19:35:59 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:35:59.502Z] pod name:mysql-7d766cd65c-drvc5 namespace:mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s running:true ready:false node:10.240.64.6 [2024-01-18T19:35:59.502Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T19:36:11.670Z] 2024-01-18 19:36:09 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T19:36:11.670Z] 2024-01-18 19:36:09 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T19:36:11.670Z] 2024-01-18 19:36:09 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] validate if mysql-ibm app's volumes are setup [2024-01-18T19:36:11.670Z] 2024-01-18 19:36:09 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T19:36:11.670Z] 2024-01-18 19:36:09 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T19:36:11.670Z] 2024-01-18 19:36:09 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T19:36:11.670Z] 2024-01-18 19:36:09 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T19:36:11.670Z] 2024-01-18 19:36:09 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Verifying new application namespaces inclusion in next schedule backup [2024-01-18T19:36:11.670Z] 2024-01-18 19:36:09 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Waiting for [15] minutes for the next schedule backup to be triggered [2024-01-18T19:44:33.289Z] INFO[2024-01-18 19:44:19] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T19:44:33.289Z] INFO[2024-01-18 19:44:19] sched stats: tick interval: 15019 < 1.2s (100%) [2024-01-18T19:44:33.289Z] INFO[2024-01-18 19:44:19] sched stats: tick process durations: 15018 < 500ms (100%) [2024-01-18T19:51:09.831Z] 2024-01-18 19:51:09 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] schedule name tp-backup-schedule-1705605709, Next schedule backup name: tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-193657 [2024-01-18T19:51:10.442Z] 2024-01-18 19:51:09 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Inspect obj backup: last_update_time: create_time_in_sec:1705606617 ownership: > > backup_info: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"6c224a5f-c7b0-4339-80a5-78ff3b57619c" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"6b9abe1a-85ec-4fbc-b413-fac7c1f6a4cd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"3f21e1f0-3c1c-4079-922d-b3a77ee64891" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"72c57770-1e12-4da4-91b7-967895866f13" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bfbe9aa1-d48e-4284-b4ad-7c5d202e7250" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"ddde003a-4407-4f78-bbe9-1457892e6a8b" > backup_path:"kube-system/tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-193657/6960808a-8e71-4567-b270-241c6153a9d7" stage:Final backup_schedule: cr_name:"tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-193657" total_size:85899345920 resource_count:1044 backup_location_ref: cloud_credential_ref: cr_uid:"6960808a-8e71-4567-b270-241c6153a9d7" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > [2024-01-18T19:51:10.442Z] 2024-01-18 19:51:09 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Next schedule backup name [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-193657] [2024-01-18T19:51:10.442Z] 2024-01-18 19:51:09 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating backup [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-193657] in org [default] [2024-01-18T19:51:10.442Z] 2024-01-18 19:51:09 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-193657] [2024-01-18T19:51:10.442Z] 2024-01-18 19:51:10 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-dst-pxb-0-84760-01-18-19h19m56s] in backup [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-193657] [2024-01-18T19:51:10.442Z] 2024-01-18 19:51:10 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-dst-pxb-0-84760-01-18-19h19m56s] in backup [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-193657] [2024-01-18T19:51:10.442Z] 2024-01-18 19:51:10 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-dst-pxb-0-84760-01-18-19h19m56s] [2024-01-18T19:51:10.442Z] 2024-01-18 19:51:10 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T19:51:10.442Z] 2024-01-18 19:51:10 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T19:51:10.443Z] 2024-01-18 19:51:10 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-dst-pxb-0-84760-01-18-19h19m56s] are [[mysql-data]] [2024-01-18T19:51:10.443Z] 2024-01-18 19:51:10 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T19:51:10.443Z] 2024-01-18 19:51:10 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s] in backup [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-193657] [2024-01-18T19:51:10.443Z] 2024-01-18 19:51:10 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s] in backup [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-193657] [2024-01-18T19:51:10.443Z] 2024-01-18 19:51:10 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s] [2024-01-18T19:51:10.443Z] 2024-01-18 19:51:10 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T19:51:10.443Z] 2024-01-18 19:51:10 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T19:51:10.443Z] 2024-01-18 19:51:10 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s] are [[mysql-data]] [2024-01-18T19:51:10.443Z] 2024-01-18 19:51:10 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T19:51:10.443Z] 2024-01-18 19:51:10 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s] in backup [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-193657] [2024-01-18T19:51:10.443Z] 2024-01-18 19:51:10 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s] in backup [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-193657] [2024-01-18T19:51:10.443Z] 2024-01-18 19:51:10 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s] [2024-01-18T19:51:10.443Z] 2024-01-18 19:51:10 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T19:51:10.443Z] 2024-01-18 19:51:10 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T19:51:10.443Z] 2024-01-18 19:51:10 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s] are [[mysql-data]] [2024-01-18T19:51:10.443Z] 2024-01-18 19:51:10 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T19:51:10.443Z] 2024-01-18 19:51:10 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating CR cleanup [2024-01-18T19:51:10.700Z] 2024-01-18 19:51:10 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T19:51:10.700Z] 2024-01-18 19:51:10 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [kube-system] are [[]] [2024-01-18T19:51:10.700Z] INFO[2024-01-18 19:51:10] Verifying : Description : Validation of next schedule backup [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-193657] of schedule [tp-backup-schedule-1705605709] after getting triggered and completing, and inclusion of new app namespaces [[mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s]] [2024-01-18T19:51:10.700Z] INFO[2024-01-18 19:51:10] Actual:true, Expected: true [2024-01-18T19:51:10.700Z] 2024-01-18 19:51:10 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Restoring new application namespaces from next schedule backup in source cluster [2024-01-18T19:51:10.700Z] 2024-01-18 19:51:10 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Namespace mapping used for restoring - map[mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s:mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored] [2024-01-18T19:51:12.069Z] 2024/01/18 19:51:11 DoRetryWithTimeout - Error: {restore status for [test-restore-kjiy] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T19:51:44.100Z] 2024/01/18 19:51:41 DoRetryWithTimeout - Error: {restore status for [test-restore-kjiy] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T19:52:16.144Z] 2024/01/18 19:52:11 DoRetryWithTimeout - Error: {restore status for [test-restore-kjiy] expected was [[PartialSuccess Success]] but got [InProgress] because of [Application resources restore is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T19:52:42.657Z] 2024/01/18 19:52:41 DoRetryWithTimeout - Error: {restore status for [test-restore-kjiy] expected was [[PartialSuccess Success]] but got [InProgress] because of [Application resources restore is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T19:53:16.239Z] 2024-01-18 19:53:11 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.CreateRestore:#936] - Restore [test-restore-kjiy] created successfully [2024-01-18T19:53:16.239Z] 2024-01-18 19:53:11 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Switching cluster context to cluster [source-cluster] [2024-01-18T19:53:16.239Z] 2024-01-18 19:53:11 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Switching context to [/tmp/source-config] [2024-01-18T19:53:16.239Z] 2024-01-18 19:53:12 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T19:53:16.240Z] 2024-01-18 19:53:12 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T19:53:16.240Z] 2024-01-18 19:53:12 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T19:53:16.240Z] 2024-01-18 19:53:12 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T19:53:16.240Z] 2024-01-18 19:53:12 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T19:53:16.240Z] 2024-01-18 19:53:12 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T19:53:16.240Z] 2024-01-18 19:53:12 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T19:53:16.240Z] 2024-01-18 19:53:13 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T19:53:16.240Z] 2024-01-18 19:53:13 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T19:53:16.240Z] 2024-01-18 19:53:13 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T19:53:16.240Z] 2024-01-18 19:53:13 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T19:53:16.240Z] 2024-01-18 19:53:13 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T19:53:16.240Z] 2024-01-18 19:53:13 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T19:53:16.240Z] 2024-01-18 19:53:13 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Switched context to [/tmp/source-config] [2024-01-18T19:53:16.240Z] 2024-01-18 19:53:13 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s] with namespace mapping [map[mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s:mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored]] and storage Class Mapping [map[]] [2024-01-18T19:53:16.240Z] 2024-01-18 19:53:13 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s] [2024-01-18T19:53:16.241Z] 2024-01-18 19:53:13 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T19:53:16.241Z] 2024-01-18 19:53:13 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T19:53:16.241Z] 2024-01-18 19:53:13 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating restore [test-restore-kjiy] in org [default] [2024-01-18T19:53:16.241Z] 2024-01-18 19:53:13 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [test-restore-kjiy] [2024-01-18T19:53:16.241Z] 2024-01-18 19:53:14 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateRestore:#2110] - Checking status of restore [test-restore-kjiy] [2024-01-18T19:53:16.241Z] 2024-01-18 19:53:14 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored] in restore [test-restore-kjiy] [2024-01-18T19:53:16.241Z] 2024-01-18 19:53:14 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored]) was found in restore [test-restore-kjiy], as expected by presence in expectedRestoredAppContext [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored] [2024-01-18T19:53:16.241Z] 2024-01-18 19:53:14 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored]) found in the restore [test-restore-kjiy] was also present on the cluster/namespace [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored] [2024-01-18T19:53:16.241Z] 2024-01-18 19:53:14 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored]) was found in restore [test-restore-kjiy], as expected by presence in expectedRestoredAppContext [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored] [2024-01-18T19:53:16.241Z] 2024-01-18 19:53:14 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored]) found in the restore [test-restore-kjiy] was also present on the cluster/namespace [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored] [2024-01-18T19:53:16.241Z] 2024-01-18 19:53:14 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored]) was found in restore [test-restore-kjiy], as expected by presence in expectedRestoredAppContext [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored] [2024-01-18T19:53:16.241Z] 2024-01-18 19:53:14 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored]) found in the restore [test-restore-kjiy] was also present on the cluster/namespace [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored] [2024-01-18T19:53:16.241Z] 2024-01-18 19:53:14 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored]) was found in restore [test-restore-kjiy], as expected by presence in expectedRestoredAppContext [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored] [2024-01-18T19:53:16.241Z] 2024-01-18 19:53:14 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored]) found in the restore [test-restore-kjiy] was also present on the cluster/namespace [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored] [2024-01-18T19:53:16.242Z] 2024-01-18 19:53:14 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored] in restore [test-restore-kjiy] [2024-01-18T19:53:16.242Z] 2024-01-18 19:53:14 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored] in restore [test-restore-kjiy] [2024-01-18T19:53:16.242Z] 2024-01-18 19:53:14 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T19:53:16.242Z] 2024-01-18 19:53:14 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T19:53:16.242Z] 2024-01-18 19:53:14 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored] are [[mysql-data]] [2024-01-18T19:53:16.242Z] 2024-01-18 19:53:14 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.ValidateRestore:#2240] - in restore [test-restore-kjiy], the status of the restored volume [pvc-bbee4f98-d548-4abd-8499-0c12ef8128f3] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-18T19:53:16.242Z] 2024-01-18 19:53:14 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validate applications in restored namespace [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored] due to restore [test-restore-kjiy] [2024-01-18T19:53:16.242Z] 2024-01-18 19:53:14 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating mysql-ibm app [2024-01-18T19:53:16.242Z] 2024-01-18 19:53:14 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] wait for mysql-ibm app to start running [2024-01-18T19:53:16.242Z] 2024-01-18 19:53:14 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T19:53:16.242Z] 2024/01/18 19:53:14 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:53:16.242Z] pod name:mysql-7d766cd65c-kf2qj namespace:mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored running:true ready:false node:10.240.64.9 [2024-01-18T19:53:16.242Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T19:53:26.189Z] 2024/01/18 19:53:24 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:53:26.189Z] pod name:mysql-7d766cd65c-kf2qj namespace:mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored running:true ready:false node:10.240.64.9 [2024-01-18T19:53:26.189Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T19:53:36.136Z] 2024/01/18 19:53:34 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:53:36.136Z] pod name:mysql-7d766cd65c-kf2qj namespace:mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored running:true ready:false node:10.240.64.9 [2024-01-18T19:53:36.136Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T19:53:46.077Z] 2024-01-18 19:53:45 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T19:53:46.077Z] 2024-01-18 19:53:45 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T19:53:46.077Z] 2024-01-18 19:53:45 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating CR cleanup [2024-01-18T19:53:46.078Z] 2024-01-18 19:53:45 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T19:53:46.078Z] 2024-01-18 19:53:45 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] All CRs in [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s-restored] are [[]] [2024-01-18T19:53:46.078Z] 2024-01-18 19:53:45 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Switching cluster context back to cluster path [/tmp/destination-config] [2024-01-18T19:53:46.078Z] 2024-01-18 19:53:45 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Switching context to [/tmp/destination-config] [2024-01-18T19:53:46.078Z] 2024-01-18 19:53:45 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T19:53:46.078Z] 2024-01-18 19:53:45 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T19:53:46.078Z] 2024-01-18 19:53:45 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T19:53:46.078Z] 2024-01-18 19:53:45 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T19:53:46.078Z] 2024-01-18 19:53:45 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T19:53:46.078Z] 2024-01-18 19:53:45 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T19:53:46.640Z] 2024-01-18 19:53:46 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T19:53:46.896Z] 2024-01-18 19:53:46 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T19:53:47.152Z] 2024/01/18 19:53:46 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:53:57.096Z] 2024/01/18 19:53:56 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:54:07.034Z] 2024/01/18 19:54:06 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:54:16.982Z] 2024/01/18 19:54:16 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:54:21.151Z] INFO[2024-01-18 19:54:20] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T19:54:21.151Z] INFO[2024-01-18 19:54:20] sched stats: tick interval: 15620 < 1.2s (100%) [2024-01-18T19:54:21.151Z] INFO[2024-01-18 19:54:20] sched stats: tick process durations: 15619 < 500ms (100%) [2024-01-18T19:54:27.686Z] 2024/01/18 19:54:26 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:54:37.644Z] 2024/01/18 19:54:37 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:54:47.587Z] 2024/01/18 19:54:47 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:54:57.532Z] 2024/01/18 19:54:57 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:55:07.590Z] 2024/01/18 19:55:07 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:55:17.535Z] 2024/01/18 19:55:17 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:55:27.476Z] 2024/01/18 19:55:27 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:55:37.430Z] 2024/01/18 19:55:37 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:55:47.375Z] 2024/01/18 19:55:47 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:55:57.313Z] 2024/01/18 19:55:57 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:56:07.262Z] 2024/01/18 19:56:07 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:56:17.205Z] 2024/01/18 19:56:17 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:56:29.377Z] 2024/01/18 19:56:27 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:56:37.464Z] 2024/01/18 19:56:37 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:56:47.409Z] 2024/01/18 19:56:47 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:56:57.352Z] 2024/01/18 19:56:57 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:57:07.296Z] 2024/01/18 19:57:07 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:57:19.464Z] 2024/01/18 19:57:17 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:57:27.561Z] 2024/01/18 19:57:27 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:57:37.502Z] 2024/01/18 19:57:37 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:57:47.440Z] 2024/01/18 19:57:47 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:57:57.383Z] 2024/01/18 19:57:57 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:58:09.570Z] 2024/01/18 19:58:07 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:58:17.704Z] 2024/01/18 19:58:17 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:58:27.661Z] 2024/01/18 19:58:27 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:58:37.602Z] 2024/01/18 19:58:37 DoRetryWithTimeout - Error: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T19:58:47.577Z] 2024-01-18 19:58:47 +0000:[ERROR] [{AllNSBackupWithIncludeNewNSOption}] [backup.CreateRestoreWithValidation.func1:#1149] - Failed switching cluster context back to cluster path [/tmp/destination-config]. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.578Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.578Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.578Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.578Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.578Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.578Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.578Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.578Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.578Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.578Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.578Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.578Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.578Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.578Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.578Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.578Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.578Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.578Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.578Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.578Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.578Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.579Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.579Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.579Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.579Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.579Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.579Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.579Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T19:58:47.579Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-18T19:58:47.579Z] 2024-01-18 19:58:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Deleting deployed namespaces - [mysql-ibm-dst-pxb-0-84760-01-18-19h19m56s mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s] [2024-01-18T19:58:47.579Z] 2024-01-18 19:58:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating apps [2024-01-18T19:58:47.579Z] 2024-01-18 19:58:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating mysql-ibm app [2024-01-18T19:58:47.579Z] 2024-01-18 19:58:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating mysql-ibm app's volumes [2024-01-18T19:58:47.579Z] 2024-01-18 19:58:47 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T19:58:47.579Z] 2024-01-18 19:58:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T19:58:47.579Z] 2024-01-18 19:58:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-dst-pxb-0-84760-01-18-19h19m56s [2024-01-18T19:58:47.579Z] 2024-01-18 19:58:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T19:58:47.836Z] 2024-01-18 19:58:47 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T19:58:47.836Z] 2024-01-18 19:58:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] wait for mysql-ibm app to start running [2024-01-18T19:58:47.836Z] 2024-01-18 19:58:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T19:58:47.836Z] 2024-01-18 19:58:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T19:58:47.836Z] 2024-01-18 19:58:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T19:58:47.836Z] 2024-01-18 19:58:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] validate if mysql-ibm app's volumes are setup [2024-01-18T19:58:47.836Z] 2024-01-18 19:58:47 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T19:58:47.836Z] 2024-01-18 19:58:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T19:58:47.836Z] 2024-01-18 19:58:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T19:58:47.836Z] 2024-01-18 19:58:47 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T19:58:48.091Z] 2024-01-18 19:58:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating mysql-ibm app [2024-01-18T19:58:48.091Z] 2024-01-18 19:58:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating mysql-ibm app's volumes [2024-01-18T19:58:48.091Z] 2024-01-18 19:58:47 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T19:58:48.091Z] 2024-01-18 19:58:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T19:58:48.091Z] 2024-01-18 19:58:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s [2024-01-18T19:58:48.346Z] 2024-01-18 19:58:48 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T19:58:48.909Z] 2024-01-18 19:58:48 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T19:58:48.909Z] 2024-01-18 19:58:48 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] wait for mysql-ibm app to start running [2024-01-18T19:58:48.909Z] 2024-01-18 19:58:48 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T19:58:49.174Z] 2024/01/18 19:58:48 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:58:49.174Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T19:58:49.174Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T19:58:49.174Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T19:58:59.128Z] 2024/01/18 19:58:58 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:58:59.129Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T19:58:59.129Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T19:58:59.129Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T19:59:09.065Z] 2024/01/18 19:59:09 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:59:09.065Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T19:59:09.065Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T19:59:09.065Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T19:59:21.248Z] 2024/01/18 19:59:19 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:59:21.248Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T19:59:21.248Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T19:59:21.248Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T19:59:29.329Z] 2024/01/18 19:59:29 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:59:29.329Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T19:59:29.329Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T19:59:29.329Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T19:59:39.270Z] 2024/01/18 19:59:39 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:59:39.270Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T19:59:39.270Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T19:59:39.270Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T19:59:51.454Z] 2024/01/18 19:59:49 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:59:51.454Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T19:59:51.454Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T19:59:51.454Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T19:59:59.543Z] 2024/01/18 19:59:59 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T19:59:59.543Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T19:59:59.543Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T19:59:59.543Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:00:09.493Z] 2024/01/18 20:00:09 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:00:09.493Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:00:09.493Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T20:00:09.493Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:00:21.661Z] 2024/01/18 20:00:19 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:00:21.661Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:00:21.661Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T20:00:21.661Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:00:29.752Z] 2024/01/18 20:00:29 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:00:29.752Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:00:29.752Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T20:00:29.752Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:00:39.710Z] 2024/01/18 20:00:39 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:00:39.710Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:00:39.710Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T20:00:39.710Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:00:51.876Z] 2024/01/18 20:00:49 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:00:51.876Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:00:51.876Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T20:00:51.876Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:00:59.969Z] 2024/01/18 20:00:59 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:00:59.970Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:00:59.970Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T20:00:59.970Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:01:09.933Z] 2024/01/18 20:01:09 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:01:09.933Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:01:09.933Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T20:01:09.933Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:01:22.107Z] 2024/01/18 20:01:19 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:01:22.107Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:01:22.108Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T20:01:22.108Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:01:30.194Z] 2024/01/18 20:01:29 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:01:30.194Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:01:30.194Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T20:01:30.194Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:01:40.148Z] 2024/01/18 20:01:40 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:01:40.148Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:01:40.148Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T20:01:40.148Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:01:52.317Z] 2024/01/18 20:01:50 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:01:52.317Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:01:52.317Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T20:01:52.317Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:02:00.402Z] 2024/01/18 20:02:00 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:02:00.402Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:02:00.402Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T20:02:00.402Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:02:10.339Z] 2024/01/18 20:02:10 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:02:10.339Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:02:10.339Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T20:02:10.339Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:02:22.509Z] 2024/01/18 20:02:20 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:02:22.509Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:02:22.509Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T20:02:22.509Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:02:30.593Z] 2024/01/18 20:02:30 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:02:30.593Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:02:30.593Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T20:02:30.593Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:02:40.545Z] 2024/01/18 20:02:40 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:02:40.545Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:02:40.545Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T20:02:40.545Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:02:52.714Z] 2024/01/18 20:02:50 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:02:52.714Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:02:52.714Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T20:02:52.714Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:03:00.828Z] 2024/01/18 20:03:00 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:03:00.828Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:03:00.828Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T20:03:00.828Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:03:10.789Z] 2024/01/18 20:03:10 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:03:10.789Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:03:10.789Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T20:03:10.789Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:03:22.979Z] 2024/01/18 20:03:20 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:03:22.979Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:03:22.979Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T20:03:22.979Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:03:31.058Z] 2024/01/18 20:03:30 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:03:31.059Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:03:31.059Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T20:03:31.059Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:03:41.007Z] 2024/01/18 20:03:40 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:03:41.007Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:03:41.007Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T20:03:41.007Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:03:53.174Z] 2024/01/18 20:03:50 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:03:53.174Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:03:53.174Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T20:03:53.174Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:04:01.252Z] 2024/01/18 20:04:01 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:04:01.252Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:04:01.252Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T20:04:01.252Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:04:11.192Z] 2024/01/18 20:04:11 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:04:11.192Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:04:11.192Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T20:04:11.192Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:04:23.367Z] 2024/01/18 20:04:21 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:04:23.367Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:04:23.367Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:true node:10.240.64.7 [2024-01-18T20:04:23.367Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:04:23.367Z] INFO[2024-01-18 20:04:21] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T20:04:23.367Z] INFO[2024-01-18 20:04:21] sched stats: tick interval: 16221 < 1.2s (100%) [2024-01-18T20:04:23.367Z] INFO[2024-01-18 20:04:21] sched stats: tick process durations: 16220 < 500ms (100%) [2024-01-18T20:04:31.451Z] 2024/01/18 20:04:31 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:04:31.451Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:04:31.451Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.7 [2024-01-18T20:04:31.451Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:04:41.386Z] 2024/01/18 20:04:41 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:04:41.386Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:04:41.386Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.7 [2024-01-18T20:04:41.386Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:04:53.558Z] 2024/01/18 20:04:51 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:04:53.558Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:04:53.558Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.7 [2024-01-18T20:04:53.558Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:05:01.649Z] 2024/01/18 20:05:01 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:05:01.649Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:05:01.649Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.7 [2024-01-18T20:05:01.649Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:05:11.648Z] 2024/01/18 20:05:11 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:05:11.648Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:05:11.648Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.7 [2024-01-18T20:05:11.648Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:05:23.811Z] 2024/01/18 20:05:21 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:05:23.811Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:05:23.811Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.7 [2024-01-18T20:05:23.811Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:05:31.892Z] 2024/01/18 20:05:31 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:05:31.892Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:05:31.892Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.7 [2024-01-18T20:05:31.892Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:05:41.831Z] 2024/01/18 20:05:41 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:05:41.831Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:05:41.831Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.7 [2024-01-18T20:05:41.831Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:05:54.000Z] 2024/01/18 20:05:51 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:05:54.000Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:05:54.000Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.7 [2024-01-18T20:05:54.000Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:06:02.087Z] 2024/01/18 20:06:01 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:06:02.087Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:06:02.087Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.7 [2024-01-18T20:06:02.087Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:06:12.026Z] 2024/01/18 20:06:11 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:06:12.026Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:06:12.026Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.7 [2024-01-18T20:06:12.026Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:06:24.198Z] 2024/01/18 20:06:22 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:06:24.198Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:06:24.198Z] pod name:mysql-7d766cd65c-vpxgk namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.7 [2024-01-18T20:06:24.198Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:06:32.284Z] 2024/01/18 20:06:32 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:06:32.284Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:06:32.284Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:06:44.450Z] 2024/01/18 20:06:42 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:06:44.450Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:06:44.450Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:06:52.536Z] 2024/01/18 20:06:52 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:06:52.536Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:false ready:false node:10.240.64.29 [2024-01-18T20:06:52.536Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:07:02.475Z] 2024/01/18 20:07:02 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:07:02.475Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:false node:10.240.64.29 [2024-01-18T20:07:02.475Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:07:12.450Z] 2024/01/18 20:07:12 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:07:12.450Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:false node:10.240.64.29 [2024-01-18T20:07:12.450Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:07:24.639Z] 2024/01/18 20:07:22 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:07:24.639Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:false node:10.240.64.29 [2024-01-18T20:07:24.639Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:07:32.723Z] 2024/01/18 20:07:32 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:07:32.723Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:false node:10.240.64.29 [2024-01-18T20:07:32.723Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:07:42.678Z] 2024/01/18 20:07:42 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:07:42.678Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:false node:10.240.64.29 [2024-01-18T20:07:42.678Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:07:54.935Z] 2024/01/18 20:07:52 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:07:54.936Z] pod name:mysql-7d766cd65c-jxkxj namespace:mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s running:true ready:false node:10.240.64.29 [2024-01-18T20:07:54.936Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:08:03.038Z] 2024-01-18 20:08:02 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T20:08:03.038Z] 2024-01-18 20:08:02 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T20:08:03.038Z] 2024-01-18 20:08:02 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] validate if mysql-ibm app's volumes are setup [2024-01-18T20:08:03.038Z] 2024-01-18 20:08:02 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T20:08:03.038Z] 2024-01-18 20:08:02 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:08:03.038Z] 2024-01-18 20:08:02 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T20:08:03.039Z] 2024-01-18 20:08:02 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T20:08:03.039Z] 2024-01-18 20:08:02 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating mysql-ibm app [2024-01-18T20:08:03.039Z] 2024-01-18 20:08:02 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Validating mysql-ibm app's volumes [2024-01-18T20:08:03.039Z] 2024-01-18 20:08:02 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T20:08:03.039Z] 2024-01-18 20:08:02 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:08:03.039Z] 2024-01-18 20:08:02 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s [2024-01-18T20:08:03.296Z] 2024-01-18 20:08:03 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T20:08:03.296Z] 2024-01-18 20:08:03 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T20:08:03.296Z] 2024-01-18 20:08:03 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] wait for mysql-ibm app to start running [2024-01-18T20:08:03.296Z] 2024-01-18 20:08:03 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T20:08:03.554Z] 2024/01/18 20:08:03 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:08:03.555Z] pod name:mysql-7d766cd65c-jsq98 namespace:mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s running:true ready:false node:10.240.64.29 [2024-01-18T20:08:03.555Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:08:13.501Z] 2024/01/18 20:08:13 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:08:13.501Z] pod name:mysql-7d766cd65c-jsq98 namespace:mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s running:true ready:false node:10.240.64.29 [2024-01-18T20:08:13.501Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:08:25.662Z] 2024/01/18 20:08:23 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:08:25.662Z] pod name:mysql-7d766cd65c-jsq98 namespace:mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s running:true ready:false node:10.240.64.29 [2024-01-18T20:08:25.662Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:08:33.746Z] 2024/01/18 20:08:33 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:08:33.746Z] pod name:mysql-7d766cd65c-jsq98 namespace:mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s running:true ready:false node:10.240.64.29 [2024-01-18T20:08:33.746Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:08:43.697Z] 2024-01-18 20:08:43 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T20:08:43.952Z] 2024-01-18 20:08:43 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T20:08:43.952Z] 2024-01-18 20:08:43 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] validate if mysql-ibm app's volumes are setup [2024-01-18T20:08:43.952Z] 2024-01-18 20:08:43 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T20:08:43.952Z] 2024-01-18 20:08:43 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:08:43.952Z] 2024-01-18 20:08:43 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T20:08:43.952Z] 2024-01-18 20:08:43 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T20:08:43.952Z] 2024-01-18 20:08:43 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Destroying apps [2024-01-18T20:08:43.953Z] 2024-01-18 20:08:43 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T20:08:44.210Z] 2024-01-18 20:08:43 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T20:08:44.210Z] 2024-01-18 20:08:43 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] start destroying mysql-ibm app [2024-01-18T20:08:44.210Z] 2024-01-18 20:08:43 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T20:08:44.210Z] 2024-01-18 20:08:44 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T20:08:44.210Z] 2024-01-18 20:08:44 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T20:08:44.210Z] INFO[2024-01-18 20:08:44] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T20:08:44.210Z] INFO[2024-01-18 20:08:44] Actual:true, Expected: true [2024-01-18T20:08:44.210Z] 2024-01-18 20:08:44 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T20:08:44.211Z] 2024-01-18 20:08:44 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T20:08:44.211Z] 2024-01-18 20:08:44 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] start destroying mysql-ibm app [2024-01-18T20:08:44.211Z] 2024-01-18 20:08:44 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T20:08:44.467Z] 2024-01-18 20:08:44 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T20:08:44.467Z] 2024-01-18 20:08:44 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T20:08:44.467Z] INFO[2024-01-18 20:08:44] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T20:08:44.467Z] INFO[2024-01-18 20:08:44] Actual:true, Expected: true [2024-01-18T20:08:44.467Z] 2024-01-18 20:08:44 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T20:08:44.467Z] 2024-01-18 20:08:44 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T20:08:44.468Z] 2024-01-18 20:08:44 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] start destroying mysql-ibm app [2024-01-18T20:08:44.468Z] 2024-01-18 20:08:44 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T20:08:45.029Z] 2024-01-18 20:08:44 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T20:08:45.029Z] 2024-01-18 20:08:44 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T20:08:45.029Z] INFO[2024-01-18 20:08:44] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T20:08:45.029Z] INFO[2024-01-18 20:08:44] Actual:true, Expected: true [2024-01-18T20:08:45.029Z] 2024-01-18 20:08:44 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T20:08:45.029Z] 2024-01-18 20:08:44 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Switching context: The context is already [/tmp/source-config] [2024-01-18T20:08:45.029Z] 2024-01-18 20:08:44 +0000:[ERROR] [{AllNSBackupWithIncludeNewNSOption}] [backup.glob..func60.3:#1100] - Fetching px-central-admin ctx. Err: can't find PxBackup service [px-backup] from list of services [2024-01-18T20:08:45.029Z] 2024-01-18 20:08:44 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T20:08:45.029Z] 2024-01-18 20:08:44 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Switching context: The context is already [/tmp/source-config] [2024-01-18T20:08:45.029Z] INFO[2024-01-18 20:08:44] --------Test End------ [2024-01-18T20:08:45.029Z] INFO[2024-01-18 20:08:44] #Test: [2024-01-18T20:08:45.029Z] INFO[2024-01-18 20:08:44] #Description: [2024-01-18T20:08:45.029Z] INFO[2024-01-18 20:08:44] ------------------------ [2024-01-18T20:08:45.030Z] 2024-01-18 20:08:44 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.EndPxBackupTorpedoTest:#7261] - >>>> FAILED TEST: {AllNSBackupWithIncludeNewNSOption} Validates schedule backups created with include new namespaces option includes newly created namespaces [2024-01-18T20:08:45.030Z] 2024-01-18 20:08:44 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-dst-pxb-0-84760-01-18-19h19m56s] [2024-01-18T20:08:45.030Z] 2024-01-18 20:08:44 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s] [2024-01-18T20:08:45.030Z] 2024-01-18 20:08:45 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s] [2024-01-18T20:08:45.285Z] 2024-01-18 20:08:45 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T20:08:45.285Z] 2024-01-18 20:08:45 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Switching context to [/tmp/destination-config] [2024-01-18T20:08:45.285Z] 2024-01-18 20:08:45 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T20:08:45.548Z] 2024-01-18 20:08:45 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T20:08:45.548Z] 2024-01-18 20:08:45 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T20:08:45.548Z] 2024-01-18 20:08:45 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T20:08:45.548Z] 2024-01-18 20:08:45 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T20:08:45.803Z] 2024-01-18 20:08:45 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T20:08:46.059Z] 2024-01-18 20:08:46 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T20:08:46.619Z] 2024-01-18 20:08:46 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T20:08:46.875Z] 2024-01-18 20:08:46 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T20:08:47.131Z] 2024-01-18 20:08:47 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T20:08:47.425Z] 2024-01-18 20:08:47 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T20:08:47.425Z] 2024-01-18 20:08:47 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T20:08:47.426Z] 2024-01-18 20:08:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T20:08:47.426Z] 2024-01-18 20:08:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Switched context to [/tmp/destination-config] [2024-01-18T20:08:47.685Z] 2024-01-18 20:08:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-dst-pxb-0-84760-01-18-19h19m56s] [2024-01-18T20:08:47.685Z] 2024-01-18 20:08:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-dst-pxb-1-84760-01-18-19h19m56s] [2024-01-18T20:08:47.940Z] 2024-01-18 20:08:47 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-dst-pxb-new-2-84760-01-18-19h19m56s] [2024-01-18T20:08:48.196Z] 2024-01-18 20:08:48 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {AllNSBackupWithIncludeNewNSOption} Validates schedule backups created with include new namespaces option includes newly created namespaces [2024-01-18T20:08:48.196Z] 2024-01-18 20:08:48 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-18T20:08:48.196Z] 2024/01/18 20:08:48 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:08:53.445Z] 2024/01/18 20:08:53 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:08:58.695Z] 2024/01/18 20:08:58 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:09:03.936Z] 2024/01/18 20:09:03 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:09:08.104Z] 2024/01/18 20:09:08 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:09:13.348Z] 2024/01/18 20:09:13 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:09:18.601Z] 2024/01/18 20:09:18 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:09:23.888Z] 2024/01/18 20:09:23 DoRetryWithTimeout - Error: {dial tcp 10.240.0.4:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:09:29.134Z] 2024/01/18 20:09:28 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.0.4. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.4 10.240.0.4]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T20:09:29.134Z] {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T20:09:29.134Z] {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T20:09:29.134Z] {dial tcp 10.240.0.4:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-18T20:09:33.301Z] 2024-01-18 20:09:33 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.4. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.4 10.240.0.4]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T20:09:33.301Z] {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T20:09:33.301Z] {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T20:09:33.301Z] {dial tcp 10.240.0.4:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T20:09:33.301Z] 2024-01-18 20:09:33 +0000:[ERROR] [{AllNSBackupWithIncludeNewNSOption}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.4. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.4 10.240.0.4]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T20:09:33.301Z] {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T20:09:33.301Z] {dial tcp 10.240.0.4:22: connect: connection refused} [2024-01-18T20:09:33.301Z] {dial tcp 10.240.0.4:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T20:09:33.301Z] 2024-01-18 20:09:33 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T20:09:33.301Z] 2024-01-18 20:09:33 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Switching context to [/tmp/source-config] [2024-01-18T20:09:33.301Z] 2024-01-18 20:09:33 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T20:09:33.301Z] 2024-01-18 20:09:33 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T20:09:33.301Z] 2024-01-18 20:09:33 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T20:09:33.557Z] 2024-01-18 20:09:33 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T20:09:33.557Z] 2024-01-18 20:09:33 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T20:09:33.557Z] 2024-01-18 20:09:33 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T20:09:34.119Z] 2024-01-18 20:09:33 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T20:09:34.374Z] 2024-01-18 20:09:34 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T20:09:34.935Z] 2024-01-18 20:09:34 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T20:09:34.935Z] 2024-01-18 20:09:34 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T20:09:35.193Z] 2024-01-18 20:09:35 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T20:09:35.193Z] 2024-01-18 20:09:35 +0000:[WARNING] [{AllNSBackupWithIncludeNewNSOption}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T20:09:35.193Z] 2024-01-18 20:09:35 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T20:09:35.193Z] 2024-01-18 20:09:35 +0000:[INFO] [{AllNSBackupWithIncludeNewNSOption}] Switched context to [/tmp/source-config] [2024-01-18T20:09:35.193Z] [2024-01-18T20:09:35.193Z] ------------------------------ [2024-01-18T20:09:35.193Z] • Failure [2978.315 seconds] [2024-01-18T20:09:35.193Z] {AllNSBackupWithIncludeNewNSOption} [2024-01-18T20:09:35.193Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:924 [2024-01-18T20:09:35.193Z] Validates schedule backups created with include new namespaces option includes newly created namespaces [It] [2024-01-18T20:09:35.193Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:958 [2024-01-18T20:09:35.193Z] [2024-01-18T20:09:35.193Z] Unexpected error: [2024-01-18T20:09:35.193Z] <*errors.errorString | 0xc001930b30>: { [2024-01-18T20:09:35.194Z] s: "failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{No... [2024-01-18T20:09:35.194Z] [2024-01-18T20:09:35.194Z] Gomega truncated this representation as it exceeds 'format.MaxLength'. [2024-01-18T20:09:35.194Z] Consider having the object provide a custom 'GomegaStringer' representation [2024-01-18T20:09:35.195Z] or adjust the parameters in Gomega's 'format' package. [2024-01-18T20:09:35.195Z] [2024-01-18T20:09:35.195Z] Learn more here: https://onsi.github.io/gomega/#adjusting-output [2024-01-18T20:09:35.195Z] [2024-01-18T20:09:35.195Z] failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.195Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.195Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.195Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.195Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.195Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.195Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.195Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.195Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.195Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.195Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.195Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.195Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.195Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.195Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.195Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.195Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.195Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.195Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.195Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.196Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.196Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.196Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.196Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.196Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.196Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.196Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.196Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.196Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:09:35.196Z] {Node: 10.240.64.6 is not ready due to err: node: 10.240.64.6 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-18T20:09:35.196Z] occurred [2024-01-18T20:09:35.196Z] [2024-01-18T20:09:35.196Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-18T20:09:35.196Z] [2024-01-18T20:09:35.196Z] Full Stack Trace [2024-01-18T20:09:35.196Z] github.com/portworx/torpedo/pkg/log.FailOnError({0x88a5060, 0xc001930b30}, {0x7fd2f06?, 0x1?}, {0xc0011f2de8?, 0x0?, 0x1?}) [2024-01-18T20:09:35.196Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 +0x290 [2024-01-18T20:09:35.196Z] github.com/portworx/torpedo/tests/backup.CreateRestoreWithValidation.func1() [2024-01-18T20:09:35.196Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_helper.go:1149 +0xd2 [2024-01-18T20:09:35.196Z] github.com/portworx/torpedo/tests/backup.CreateRestoreWithValidation({0x88eb700, 0xc00184a4b0}, {0xc001acf2d8, 0x11}, {0xc002014200?, 0xc001ab6480?}, 0x268cb17?, 0x5c?, {0x7eb9fc4, 0xe}, ...) [2024-01-18T20:09:35.196Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_helper.go:1162 +0x4a9 [2024-01-18T20:09:35.196Z] github.com/portworx/torpedo/tests/backup.glob..func60.2.10() [2024-01-18T20:09:35.196Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:1079 +0x3b5 [2024-01-18T20:09:35.196Z] github.com/portworx/torpedo/tests/backup.glob..func60.2() [2024-01-18T20:09:35.196Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:1069 +0x969 [2024-01-18T20:09:35.196Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-18T20:09:35.196Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-18T20:09:35.196Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-18T20:09:35.196Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-18T20:09:35.196Z] created by testing.(*T).Run [2024-01-18T20:09:35.196Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-18T20:09:35.196Z] ------------------------------ [2024-01-18T20:09:35.196Z] STEP: Create cloud credentials and backup locations [2024-01-18T20:09:35.196Z] STEP: Add source and destination clusters with px-central-admin ctx [2024-01-18T20:09:35.196Z] STEP: Create a schedule policy [2024-01-18T20:09:35.196Z] STEP: Schedule applications in destination cluster [2024-01-18T20:09:35.196Z] STEP: schedule applications [2024-01-18T20:09:35.196Z] STEP: schedule applications [2024-01-18T20:09:35.196Z] STEP: Validate app namespaces in destination cluster [2024-01-18T20:09:35.196Z] STEP: validate applications [2024-01-18T20:09:35.196Z] STEP: validate mysql-ibm app's volumes [2024-01-18T20:09:35.196Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T20:09:35.196Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T20:09:35.197Z] STEP: get mysql-ibm app's volume: pvc-6c224a5f-c7b0-4339-80a5-78ff3b57619c inspected by the volume driver [2024-01-18T20:09:35.197Z] STEP: wait for mysql-ibm app to start running [2024-01-18T20:09:35.197Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T20:09:35.197Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T20:09:35.197Z] STEP: Validate Px pod restart count [2024-01-18T20:09:35.197Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T20:09:35.197Z] STEP: validate mysql-ibm app's volumes [2024-01-18T20:09:35.197Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T20:09:35.197Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T20:09:35.197Z] STEP: get mysql-ibm app's volume: pvc-6b9abe1a-85ec-4fbc-b413-fac7c1f6a4cd inspected by the volume driver [2024-01-18T20:09:35.197Z] STEP: wait for mysql-ibm app to start running [2024-01-18T20:09:35.197Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T20:09:35.197Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T20:09:35.197Z] STEP: Validate Px pod restart count [2024-01-18T20:09:35.197Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T20:09:35.197Z] STEP: Create schedule backup [2024-01-18T20:09:35.197Z] STEP: Schedule applications in destination cluster to create new namespaces [2024-01-18T20:09:35.197Z] STEP: schedule applications [2024-01-18T20:09:35.197Z] STEP: Validate new app namespaces in destination cluster [2024-01-18T20:09:35.197Z] STEP: validate applications [2024-01-18T20:09:35.197Z] STEP: validate mysql-ibm app's volumes [2024-01-18T20:09:35.197Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T20:09:35.197Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T20:09:35.197Z] STEP: get mysql-ibm app's volume: pvc-3f21e1f0-3c1c-4079-922d-b3a77ee64891 inspected by the volume driver [2024-01-18T20:09:35.197Z] STEP: wait for mysql-ibm app to start running [2024-01-18T20:09:35.197Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T20:09:35.197Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T20:09:35.197Z] STEP: Validate Px pod restart count [2024-01-18T20:09:35.197Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T20:09:35.197Z] STEP: Verify new application namespaces inclusion in next schedule backup [2024-01-18T20:09:35.197Z] STEP: Restore new application namespaces from next schedule backup in source cluster [2024-01-18T20:09:35.197Z] STEP: validate mysql-ibm app's volumes [2024-01-18T20:09:35.197Z] STEP: wait for mysql-ibm app to start running [2024-01-18T20:09:35.197Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T20:09:35.197Z] STEP: Validate Px pod restart count [2024-01-18T20:09:35.197Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T20:09:35.197Z] STEP: validate apps [2024-01-18T20:09:35.197Z] STEP: validate mysql-ibm app's volumes [2024-01-18T20:09:35.197Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T20:09:35.197Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T20:09:35.197Z] STEP: get mysql-ibm app's volume: pvc-6c224a5f-c7b0-4339-80a5-78ff3b57619c inspected by the volume driver [2024-01-18T20:09:35.197Z] STEP: wait for mysql-ibm app to start running [2024-01-18T20:09:35.197Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T20:09:35.197Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T20:09:35.197Z] STEP: Validate Px pod restart count [2024-01-18T20:09:35.197Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T20:09:35.197Z] STEP: validate mysql-ibm app's volumes [2024-01-18T20:09:35.197Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T20:09:35.197Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T20:09:35.197Z] STEP: get mysql-ibm app's volume: pvc-6b9abe1a-85ec-4fbc-b413-fac7c1f6a4cd inspected by the volume driver [2024-01-18T20:09:35.197Z] STEP: wait for mysql-ibm app to start running [2024-01-18T20:09:35.197Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T20:09:35.197Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T20:09:35.197Z] STEP: Validate Px pod restart count [2024-01-18T20:09:35.197Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T20:09:35.197Z] STEP: validate mysql-ibm app's volumes [2024-01-18T20:09:35.197Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T20:09:35.197Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T20:09:35.198Z] STEP: get mysql-ibm app's volume: pvc-3f21e1f0-3c1c-4079-922d-b3a77ee64891 inspected by the volume driver [2024-01-18T20:09:35.198Z] STEP: wait for mysql-ibm app to start running [2024-01-18T20:09:35.198Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T20:09:35.198Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T20:09:35.198Z] STEP: Validate Px pod restart count [2024-01-18T20:09:35.198Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T20:09:35.198Z] STEP: destroy apps [2024-01-18T20:09:35.198Z] STEP: destroy the mysql-ibm app's volumes [2024-01-18T20:09:35.198Z] STEP: start destroying mysql-ibm app [2024-01-18T20:09:35.198Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T20:09:35.198Z] STEP: destroy the mysql-ibm app's volumes [2024-01-18T20:09:35.198Z] STEP: start destroying mysql-ibm app [2024-01-18T20:09:35.198Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T20:09:35.198Z] STEP: destroy the mysql-ibm app's volumes [2024-01-18T20:09:35.198Z] STEP: start destroying mysql-ibm app [2024-01-18T20:09:35.198Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T20:09:35.198Z] 2024-01-18 20:09:35 +0000:[INFO] [{BasicSelectiveRestore}] Log Dir: /testresults//BasicSelectiveRestore.log [2024-01-18T20:09:35.198Z] INFO[2024-01-18 20:09:35] --------Test Start------ [2024-01-18T20:09:35.198Z] INFO[2024-01-18 20:09:35] #Test: BasicSelectiveRestore [2024-01-18T20:09:35.198Z] INFO[2024-01-18 20:09:35] #Description: All namespace backup and restore selective namespaces [2024-01-18T20:09:35.198Z] INFO[2024-01-18 20:09:35] ------------------------ [2024-01-18T20:09:35.198Z] 2024-01-18 20:09:35 +0000:[INFO] [{BasicSelectiveRestore}] App list [mysql-ibm] [2024-01-18T20:09:35.198Z] 2024-01-18 20:09:35 +0000:[INFO] [{BasicSelectiveRestore}] Starting to deploy applications [2024-01-18T20:09:35.198Z] 2024-01-18 20:09:35 +0000:[INFO] [{BasicSelectiveRestore}] Iteration 0 of deploying applications [2024-01-18T20:09:35.198Z] 2024-01-18 20:09:35 +0000:[INFO] [{BasicSelectiveRestore}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T20:09:35.198Z] 2024-01-18 20:09:35 +0000:[INFO] [{BasicSelectiveRestore}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T20:09:35.454Z] 2024-01-18 20:09:35 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T20:09:35.454Z] 2024-01-18 20:09:35 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T20:09:35.712Z] 2024-01-18 20:09:35 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T20:09:35.967Z] 2024-01-18 20:09:35 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T20:09:35.968Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T20:09:35.968Z] 2024-01-18 20:09:35 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T20:09:35.969Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T20:09:35.969Z] 2024-01-18 20:09:35 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T20:09:35.969Z] 2024-01-18 20:09:35 +0000:[INFO] [{BasicSelectiveRestore}] Iteration 1 of deploying applications [2024-01-18T20:09:35.969Z] 2024-01-18 20:09:35 +0000:[INFO] [{BasicSelectiveRestore}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T20:09:35.969Z] 2024-01-18 20:09:35 +0000:[INFO] [{BasicSelectiveRestore}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T20:09:35.969Z] 2024-01-18 20:09:35 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T20:09:36.226Z] 2024-01-18 20:09:35 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T20:09:36.226Z] 2024-01-18 20:09:36 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T20:09:36.483Z] 2024-01-18 20:09:36 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T20:09:36.484Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T20:09:36.484Z] 2024-01-18 20:09:36 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T20:09:36.484Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T20:09:36.484Z] 2024-01-18 20:09:36 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T20:09:36.484Z] 2024-01-18 20:09:36 +0000:[INFO] [{BasicSelectiveRestore}] Iteration 2 of deploying applications [2024-01-18T20:09:36.484Z] 2024-01-18 20:09:36 +0000:[INFO] [{BasicSelectiveRestore}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T20:09:36.484Z] 2024-01-18 20:09:36 +0000:[INFO] [{BasicSelectiveRestore}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T20:09:36.739Z] 2024-01-18 20:09:36 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T20:09:36.740Z] 2024-01-18 20:09:36 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T20:09:36.740Z] 2024-01-18 20:09:36 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T20:09:36.995Z] 2024-01-18 20:09:36 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T20:09:36.996Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T20:09:36.996Z] 2024-01-18 20:09:36 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T20:09:36.996Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T20:09:36.996Z] 2024-01-18 20:09:36 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T20:09:36.996Z] 2024-01-18 20:09:36 +0000:[INFO] [{BasicSelectiveRestore}] Iteration 3 of deploying applications [2024-01-18T20:09:36.996Z] 2024-01-18 20:09:36 +0000:[INFO] [{BasicSelectiveRestore}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T20:09:36.996Z] 2024-01-18 20:09:36 +0000:[INFO] [{BasicSelectiveRestore}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T20:09:37.252Z] 2024-01-18 20:09:37 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T20:09:37.252Z] 2024-01-18 20:09:37 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T20:09:37.510Z] 2024-01-18 20:09:37 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T20:09:37.765Z] 2024-01-18 20:09:37 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T20:09:37.766Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T20:09:37.766Z] 2024-01-18 20:09:37 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T20:09:37.767Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T20:09:37.767Z] 2024-01-18 20:09:37 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T20:09:37.767Z] 2024-01-18 20:09:37 +0000:[INFO] [{BasicSelectiveRestore}] Iteration 4 of deploying applications [2024-01-18T20:09:37.767Z] 2024-01-18 20:09:37 +0000:[INFO] [{BasicSelectiveRestore}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T20:09:37.767Z] 2024-01-18 20:09:37 +0000:[INFO] [{BasicSelectiveRestore}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T20:09:37.767Z] 2024-01-18 20:09:37 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T20:09:38.022Z] 2024-01-18 20:09:37 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T20:09:38.022Z] 2024-01-18 20:09:37 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T20:09:38.278Z] 2024-01-18 20:09:38 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T20:09:38.279Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T20:09:38.279Z] 2024-01-18 20:09:38 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T20:09:38.280Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T20:09:38.280Z] 2024-01-18 20:09:38 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T20:09:38.280Z] 2024-01-18 20:09:38 +0000:[INFO] [{BasicSelectiveRestore}] Iteration 5 of deploying applications [2024-01-18T20:09:38.280Z] 2024-01-18 20:09:38 +0000:[INFO] [{BasicSelectiveRestore}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T20:09:38.280Z] 2024-01-18 20:09:38 +0000:[INFO] [{BasicSelectiveRestore}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T20:09:38.535Z] 2024-01-18 20:09:38 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T20:09:38.535Z] 2024-01-18 20:09:38 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T20:09:38.791Z] 2024-01-18 20:09:38 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T20:09:38.791Z] 2024-01-18 20:09:38 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T20:09:38.791Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T20:09:38.792Z] 2024-01-18 20:09:38 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T20:09:38.792Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T20:09:39.047Z] 2024-01-18 20:09:38 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T20:09:39.047Z] 2024-01-18 20:09:38 +0000:[INFO] [{BasicSelectiveRestore}] Validating deployed applications [2024-01-18T20:09:39.047Z] 2024-01-18 20:09:38 +0000:[INFO] [{BasicSelectiveRestore}] Validate applications [2024-01-18T20:09:39.047Z] 2024-01-18 20:09:38 +0000:[INFO] [{BasicSelectiveRestore}] Validating mysql-ibm app [2024-01-18T20:09:39.047Z] 2024-01-18 20:09:38 +0000:[INFO] [{BasicSelectiveRestore}] Validating mysql-ibm app's volumes [2024-01-18T20:09:39.047Z] 2024-01-18 20:09:38 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:09:39.303Z] 2024/01/18 20:09:39 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T20:09:49.267Z] 2024-01-18 20:09:49 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-83717-01-18-20h09m35s [2024-01-18T20:09:49.267Z] 2024-01-18 20:09:49 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T20:09:49.267Z] 2024-01-18 20:09:49 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T20:09:49.267Z] 2024-01-18 20:09:49 +0000:[INFO] [{BasicSelectiveRestore}] wait for mysql-ibm app to start running [2024-01-18T20:09:49.522Z] 2024-01-18 20:09:49 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T20:09:49.522Z] 2024/01/18 20:09:49 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:09:49.522Z] pod name:mysql-7d766cd65c-dvj5m namespace:mysql-ibm-pxb-0-83717-01-18-20h09m35s running:true ready:false node: [2024-01-18T20:09:49.522Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:10:01.699Z] 2024/01/18 20:09:59 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:10:01.699Z] pod name:mysql-7d766cd65c-dvj5m namespace:mysql-ibm-pxb-0-83717-01-18-20h09m35s running:false ready:false node:10.240.64.8 [2024-01-18T20:10:01.699Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:10:09.782Z] 2024/01/18 20:10:09 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:10:09.782Z] pod name:mysql-7d766cd65c-dvj5m namespace:mysql-ibm-pxb-0-83717-01-18-20h09m35s running:false ready:false node:10.240.64.8 [2024-01-18T20:10:09.782Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:10:19.731Z] 2024/01/18 20:10:19 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:10:19.731Z] pod name:mysql-7d766cd65c-dvj5m namespace:mysql-ibm-pxb-0-83717-01-18-20h09m35s running:true ready:false node:10.240.64.8 [2024-01-18T20:10:19.731Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:10:31.905Z] 2024/01/18 20:10:29 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:10:31.905Z] pod name:mysql-7d766cd65c-dvj5m namespace:mysql-ibm-pxb-0-83717-01-18-20h09m35s running:true ready:false node:10.240.64.8 [2024-01-18T20:10:31.905Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:10:39.994Z] 2024/01/18 20:10:39 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:10:39.994Z] pod name:mysql-7d766cd65c-dvj5m namespace:mysql-ibm-pxb-0-83717-01-18-20h09m35s running:true ready:false node:10.240.64.8 [2024-01-18T20:10:39.994Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:10:52.182Z] 2024/01/18 20:10:49 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:10:52.182Z] pod name:mysql-7d766cd65c-dvj5m namespace:mysql-ibm-pxb-0-83717-01-18-20h09m35s running:true ready:false node:10.240.64.8 [2024-01-18T20:10:52.182Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:11:00.270Z] 2024/01/18 20:11:00 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:11:00.270Z] pod name:mysql-7d766cd65c-dvj5m namespace:mysql-ibm-pxb-0-83717-01-18-20h09m35s running:true ready:false node:10.240.64.8 [2024-01-18T20:11:00.270Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:11:10.249Z] 2024/01/18 20:11:10 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:11:10.249Z] pod name:mysql-7d766cd65c-dvj5m namespace:mysql-ibm-pxb-0-83717-01-18-20h09m35s running:true ready:false node:10.240.64.8 [2024-01-18T20:11:10.249Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:11:22.426Z] 2024-01-18 20:11:20 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T20:11:22.426Z] 2024-01-18 20:11:20 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T20:11:22.426Z] 2024-01-18 20:11:20 +0000:[INFO] [{BasicSelectiveRestore}] validate if mysql-ibm app's volumes are setup [2024-01-18T20:11:22.426Z] 2024-01-18 20:11:20 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T20:11:22.426Z] 2024-01-18 20:11:20 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:11:22.426Z] 2024-01-18 20:11:20 +0000:[INFO] [{BasicSelectiveRestore}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T20:11:22.426Z] 2024-01-18 20:11:20 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T20:11:22.426Z] 2024-01-18 20:11:20 +0000:[INFO] [{BasicSelectiveRestore}] Validating mysql-ibm app [2024-01-18T20:11:22.426Z] 2024-01-18 20:11:20 +0000:[INFO] [{BasicSelectiveRestore}] Validating mysql-ibm app's volumes [2024-01-18T20:11:22.426Z] 2024-01-18 20:11:20 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T20:11:22.426Z] 2024-01-18 20:11:20 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:11:22.426Z] 2024-01-18 20:11:20 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-1-83717-01-18-20h09m35s [2024-01-18T20:11:22.426Z] 2024-01-18 20:11:20 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T20:11:22.426Z] 2024-01-18 20:11:20 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T20:11:22.426Z] 2024-01-18 20:11:20 +0000:[INFO] [{BasicSelectiveRestore}] wait for mysql-ibm app to start running [2024-01-18T20:11:22.426Z] 2024-01-18 20:11:20 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T20:11:22.427Z] 2024-01-18 20:11:20 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T20:11:22.427Z] 2024-01-18 20:11:21 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T20:11:22.427Z] 2024-01-18 20:11:22 +0000:[INFO] [{BasicSelectiveRestore}] validate if mysql-ibm app's volumes are setup [2024-01-18T20:11:22.427Z] 2024-01-18 20:11:22 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T20:11:22.427Z] 2024-01-18 20:11:22 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:11:22.427Z] 2024-01-18 20:11:22 +0000:[INFO] [{BasicSelectiveRestore}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T20:11:22.427Z] 2024-01-18 20:11:22 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T20:11:22.427Z] 2024-01-18 20:11:22 +0000:[INFO] [{BasicSelectiveRestore}] Validating mysql-ibm app [2024-01-18T20:11:22.427Z] 2024-01-18 20:11:22 +0000:[INFO] [{BasicSelectiveRestore}] Validating mysql-ibm app's volumes [2024-01-18T20:11:22.427Z] 2024-01-18 20:11:22 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T20:11:22.427Z] 2024-01-18 20:11:22 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:11:22.427Z] 2024-01-18 20:11:22 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-2-83717-01-18-20h09m35s [2024-01-18T20:11:22.427Z] 2024-01-18 20:11:22 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T20:11:22.704Z] 2024-01-18 20:11:22 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T20:11:22.704Z] 2024-01-18 20:11:22 +0000:[INFO] [{BasicSelectiveRestore}] wait for mysql-ibm app to start running [2024-01-18T20:11:22.704Z] 2024-01-18 20:11:22 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T20:11:22.959Z] 2024-01-18 20:11:22 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T20:11:23.523Z] 2024-01-18 20:11:23 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T20:11:24.450Z] 2024-01-18 20:11:24 +0000:[INFO] [{BasicSelectiveRestore}] validate if mysql-ibm app's volumes are setup [2024-01-18T20:11:24.450Z] 2024-01-18 20:11:24 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T20:11:24.450Z] 2024-01-18 20:11:24 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:11:24.450Z] 2024-01-18 20:11:24 +0000:[INFO] [{BasicSelectiveRestore}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T20:11:24.450Z] 2024-01-18 20:11:24 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T20:11:24.450Z] 2024-01-18 20:11:24 +0000:[INFO] [{BasicSelectiveRestore}] Validating mysql-ibm app [2024-01-18T20:11:24.450Z] 2024-01-18 20:11:24 +0000:[INFO] [{BasicSelectiveRestore}] Validating mysql-ibm app's volumes [2024-01-18T20:11:24.450Z] 2024-01-18 20:11:24 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T20:11:24.450Z] 2024-01-18 20:11:24 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:11:24.450Z] 2024-01-18 20:11:24 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-3-83717-01-18-20h09m35s [2024-01-18T20:11:24.706Z] 2024-01-18 20:11:24 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T20:11:24.706Z] 2024-01-18 20:11:24 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T20:11:24.706Z] 2024-01-18 20:11:24 +0000:[INFO] [{BasicSelectiveRestore}] wait for mysql-ibm app to start running [2024-01-18T20:11:24.706Z] 2024-01-18 20:11:24 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T20:11:24.961Z] 2024-01-18 20:11:24 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T20:11:25.522Z] 2024-01-18 20:11:25 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T20:11:26.449Z] 2024-01-18 20:11:26 +0000:[INFO] [{BasicSelectiveRestore}] validate if mysql-ibm app's volumes are setup [2024-01-18T20:11:26.449Z] 2024-01-18 20:11:26 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T20:11:26.449Z] 2024-01-18 20:11:26 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:11:26.449Z] 2024-01-18 20:11:26 +0000:[INFO] [{BasicSelectiveRestore}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T20:11:26.449Z] 2024-01-18 20:11:26 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T20:11:26.449Z] 2024-01-18 20:11:26 +0000:[INFO] [{BasicSelectiveRestore}] Validating mysql-ibm app [2024-01-18T20:11:26.449Z] 2024-01-18 20:11:26 +0000:[INFO] [{BasicSelectiveRestore}] Validating mysql-ibm app's volumes [2024-01-18T20:11:26.449Z] 2024-01-18 20:11:26 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T20:11:26.449Z] 2024-01-18 20:11:26 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:11:26.449Z] 2024-01-18 20:11:26 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-4-83717-01-18-20h09m35s [2024-01-18T20:11:26.449Z] 2024-01-18 20:11:26 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T20:11:26.708Z] 2024-01-18 20:11:26 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T20:11:26.708Z] 2024-01-18 20:11:26 +0000:[INFO] [{BasicSelectiveRestore}] wait for mysql-ibm app to start running [2024-01-18T20:11:26.708Z] 2024-01-18 20:11:26 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T20:11:26.969Z] 2024-01-18 20:11:26 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T20:11:27.532Z] 2024-01-18 20:11:27 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T20:11:28.460Z] 2024-01-18 20:11:28 +0000:[INFO] [{BasicSelectiveRestore}] validate if mysql-ibm app's volumes are setup [2024-01-18T20:11:28.461Z] 2024-01-18 20:11:28 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T20:11:28.461Z] 2024-01-18 20:11:28 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:11:28.461Z] 2024-01-18 20:11:28 +0000:[INFO] [{BasicSelectiveRestore}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T20:11:28.461Z] 2024-01-18 20:11:28 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T20:11:28.461Z] 2024-01-18 20:11:28 +0000:[INFO] [{BasicSelectiveRestore}] Validating mysql-ibm app [2024-01-18T20:11:28.461Z] 2024-01-18 20:11:28 +0000:[INFO] [{BasicSelectiveRestore}] Validating mysql-ibm app's volumes [2024-01-18T20:11:28.461Z] 2024-01-18 20:11:28 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T20:11:28.461Z] 2024-01-18 20:11:28 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:11:28.461Z] 2024-01-18 20:11:28 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-5-83717-01-18-20h09m35s [2024-01-18T20:11:28.461Z] 2024-01-18 20:11:28 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T20:11:28.716Z] 2024-01-18 20:11:28 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T20:11:28.716Z] 2024-01-18 20:11:28 +0000:[INFO] [{BasicSelectiveRestore}] wait for mysql-ibm app to start running [2024-01-18T20:11:28.716Z] 2024-01-18 20:11:28 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T20:11:28.973Z] 2024-01-18 20:11:28 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T20:11:29.540Z] 2024-01-18 20:11:29 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T20:11:30.473Z] 2024-01-18 20:11:30 +0000:[INFO] [{BasicSelectiveRestore}] validate if mysql-ibm app's volumes are setup [2024-01-18T20:11:30.473Z] 2024-01-18 20:11:30 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T20:11:30.473Z] 2024-01-18 20:11:30 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:11:30.473Z] 2024-01-18 20:11:30 +0000:[INFO] [{BasicSelectiveRestore}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T20:11:30.473Z] 2024-01-18 20:11:30 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T20:11:30.473Z] 2024-01-18 20:11:30 +0000:[INFO] [{BasicSelectiveRestore}] Creating backup location and cloud setting [2024-01-18T20:11:30.732Z] 2024-01-18 20:11:30 +0000:[INFO] [{BasicSelectiveRestore}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [cred-aws-1705608690] for org [default] with [aws] as provider [2024-01-18T20:11:30.732Z] 2024-01-18 20:11:30 +0000:[INFO] [{BasicSelectiveRestore}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-18T20:11:30.732Z] 2024-01-18 20:11:30 +0000:[INFO] [{BasicSelectiveRestore}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cred-aws-1705608690] is present [2024-01-18T20:11:30.732Z] INFO[2024-01-18 20:11:30] Verifying : Description : Verifying creation of cloud credential named [cred-aws-1705608690] for org [default] with [aws] as provider [2024-01-18T20:11:30.732Z] INFO[2024-01-18 20:11:30] Actual:true, Expected: true [2024-01-18T20:11:34.895Z] INFO[2024-01-18 20:11:34] Verifying : Description : Creating backup location aws-global-aws-px-backup-system-test-4cac76dc-bl [2024-01-18T20:11:34.895Z] INFO[2024-01-18 20:11:34] Actual:true, Expected: true [2024-01-18T20:11:34.895Z] 2024-01-18 20:11:34 +0000:[INFO] [{BasicSelectiveRestore}] Registering cluster for backup [2024-01-18T20:11:34.895Z] INFO[2024-01-18 20:11:34] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T20:11:34.895Z] INFO[2024-01-18 20:11:34] Actual:true, Expected: true [2024-01-18T20:11:34.895Z] 2024-01-18 20:11:34 +0000:[INFO] [{BasicSelectiveRestore}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T20:11:34.895Z] 2024-01-18 20:11:34 +0000:[INFO] [{BasicSelectiveRestore}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T20:11:34.895Z] 2024-01-18 20:11:34 +0000:[INFO] [{BasicSelectiveRestore}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T20:11:34.895Z] 2024-01-18 20:11:34 +0000:[INFO] [{BasicSelectiveRestore}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T20:11:34.896Z] 2024-01-18 20:11:34 +0000:[INFO] [{BasicSelectiveRestore}] Create cluster [source-cluster] in org [default] [2024-01-18T20:11:34.896Z] 2024-01-18 20:11:34 +0000:[INFO] [{BasicSelectiveRestore}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T20:11:34.896Z] 2024-01-18 20:11:34 +0000:[INFO] [{BasicSelectiveRestore}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T20:11:34.896Z] 2024-01-18 20:11:34 +0000:[INFO] [{BasicSelectiveRestore}] Create cluster [destination-cluster] in org [default] [2024-01-18T20:11:34.896Z] 2024-01-18 20:11:34 +0000:[INFO] [{BasicSelectiveRestore}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T20:11:34.896Z] 2024-01-18 20:11:34 +0000:[INFO] [{BasicSelectiveRestore}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T20:11:34.896Z] 2024-01-18 20:11:34 +0000:[INFO] [{BasicSelectiveRestore}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-xnwmi] for IBM [2024-01-18T20:11:34.896Z] 2024-01-18 20:11:34 +0000:[INFO] [{BasicSelectiveRestore}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-xnwmi] for org [default] with [ibm] as provider [2024-01-18T20:11:34.896Z] 2024-01-18 20:11:34 +0000:[INFO] [{BasicSelectiveRestore}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T20:11:34.896Z] 2024-01-18 20:11:34 +0000:[INFO] [{BasicSelectiveRestore}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T20:11:34.896Z] 2024-01-18 20:11:34 +0000:[INFO] [{BasicSelectiveRestore}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-xnwmi] is present [2024-01-18T20:11:34.896Z] 2024-01-18 20:11:34 +0000:[INFO] [{BasicSelectiveRestore}] Create cluster [source-cluster] in org [default] [2024-01-18T20:11:34.896Z] 2024-01-18 20:11:34 +0000:[INFO] [{BasicSelectiveRestore}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-rpaqk] for IBM [2024-01-18T20:11:34.896Z] 2024-01-18 20:11:34 +0000:[INFO] [{BasicSelectiveRestore}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-rpaqk] for org [default] with [ibm] as provider [2024-01-18T20:11:34.896Z] 2024-01-18 20:11:34 +0000:[INFO] [{BasicSelectiveRestore}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T20:11:34.896Z] 2024-01-18 20:11:34 +0000:[INFO] [{BasicSelectiveRestore}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T20:11:35.153Z] 2024-01-18 20:11:34 +0000:[INFO] [{BasicSelectiveRestore}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-rpaqk] is present [2024-01-18T20:11:35.153Z] 2024-01-18 20:11:34 +0000:[INFO] [{BasicSelectiveRestore}] Create cluster [destination-cluster] in org [default] [2024-01-18T20:11:35.153Z] INFO[2024-01-18 20:11:34] Verifying : Description : Creating source and destination cluster [2024-01-18T20:11:35.153Z] INFO[2024-01-18 20:11:34] Actual:true, Expected: true [2024-01-18T20:11:35.153Z] INFO[2024-01-18 20:11:34] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-18T20:11:35.153Z] INFO[2024-01-18 20:11:34] Actual:Online, Expected: Online [2024-01-18T20:11:35.153Z] INFO[2024-01-18 20:11:34] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-18T20:11:35.153Z] INFO[2024-01-18 20:11:34] Actual:true, Expected: true [2024-01-18T20:11:35.153Z] 2024-01-18 20:11:34 +0000:[INFO] [{BasicSelectiveRestore}] Taking backup of multiple namespaces [[mysql-ibm-pxb-0-83717-01-18-20h09m35s mysql-ibm-pxb-1-83717-01-18-20h09m35s mysql-ibm-pxb-2-83717-01-18-20h09m35s mysql-ibm-pxb-3-83717-01-18-20h09m35s mysql-ibm-pxb-4-83717-01-18-20h09m35s mysql-ibm-pxb-5-83717-01-18-20h09m35s]] [2024-01-18T20:11:35.411Z] 2024-01-18 20:11:35 +0000:[INFO] [{BasicSelectiveRestore}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T20:11:35.411Z] 2024-01-18 20:11:35 +0000:[INFO] [{BasicSelectiveRestore}] Volume snapshot classes found in the cluster - [2024-01-18T20:11:35.411Z] 2024-01-18 20:11:35 +0000:[INFO] [{BasicSelectiveRestore}] ibmc-vpcblock-snapshot [2024-01-18T20:11:35.411Z] 2024-01-18 20:11:35 +0000:[INFO] [{BasicSelectiveRestore}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T20:11:36.341Z] 2024/01/18 20:11:35 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705608575] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T20:12:08.391Z] 2024/01/18 20:12:06 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705608575] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T20:12:40.436Z] 2024/01/18 20:12:36 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705608575] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T20:13:06.932Z] 2024/01/18 20:13:06 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705608575] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T20:13:39.032Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [backup.CreateBackup:#313] - Backup [tp-backup-1705608575] created successfully [2024-01-18T20:13:39.032Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] Validating backup [tp-backup-1705608575] in org [default] [2024-01-18T20:13:39.032Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705608575] [2024-01-18T20:13:39.032Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-83717-01-18-20h09m35s] in backup [tp-backup-1705608575] [2024-01-18T20:13:39.032Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-83717-01-18-20h09m35s] in backup [tp-backup-1705608575] [2024-01-18T20:13:39.032Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-83717-01-18-20h09m35s] [2024-01-18T20:13:39.032Z] 2024-01-18 20:13:36 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T20:13:39.032Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:13:39.032Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-83717-01-18-20h09m35s] are [[mysql-data]] [2024-01-18T20:13:39.032Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T20:13:39.032Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-83717-01-18-20h09m35s] in backup [tp-backup-1705608575] [2024-01-18T20:13:39.033Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-83717-01-18-20h09m35s] in backup [tp-backup-1705608575] [2024-01-18T20:13:39.033Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-83717-01-18-20h09m35s] [2024-01-18T20:13:39.033Z] 2024-01-18 20:13:36 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T20:13:39.033Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:13:39.033Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-83717-01-18-20h09m35s] are [[mysql-data]] [2024-01-18T20:13:39.033Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T20:13:39.034Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-83717-01-18-20h09m35s] in backup [tp-backup-1705608575] [2024-01-18T20:13:39.034Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-83717-01-18-20h09m35s] in backup [tp-backup-1705608575] [2024-01-18T20:13:39.034Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-2-83717-01-18-20h09m35s] [2024-01-18T20:13:39.034Z] 2024-01-18 20:13:36 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T20:13:39.034Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:13:39.034Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-2-83717-01-18-20h09m35s] are [[mysql-data]] [2024-01-18T20:13:39.034Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T20:13:39.034Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-3-83717-01-18-20h09m35s] in backup [tp-backup-1705608575] [2024-01-18T20:13:39.034Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-3-83717-01-18-20h09m35s] in backup [tp-backup-1705608575] [2024-01-18T20:13:39.034Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-3-83717-01-18-20h09m35s] [2024-01-18T20:13:39.034Z] 2024-01-18 20:13:36 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T20:13:39.034Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:13:39.034Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-3-83717-01-18-20h09m35s] are [[mysql-data]] [2024-01-18T20:13:39.034Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T20:13:39.034Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-4-83717-01-18-20h09m35s] in backup [tp-backup-1705608575] [2024-01-18T20:13:39.034Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-4-83717-01-18-20h09m35s] in backup [tp-backup-1705608575] [2024-01-18T20:13:39.034Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-4-83717-01-18-20h09m35s] [2024-01-18T20:13:39.034Z] 2024-01-18 20:13:36 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T20:13:39.034Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:13:39.034Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-4-83717-01-18-20h09m35s] are [[mysql-data]] [2024-01-18T20:13:39.035Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T20:13:39.035Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-5-83717-01-18-20h09m35s] in backup [tp-backup-1705608575] [2024-01-18T20:13:39.035Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-5-83717-01-18-20h09m35s] in backup [tp-backup-1705608575] [2024-01-18T20:13:39.035Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-5-83717-01-18-20h09m35s] [2024-01-18T20:13:39.035Z] 2024-01-18 20:13:36 +0000:[WARNING] [{BasicSelectiveRestore}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T20:13:39.035Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:13:39.035Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-5-83717-01-18-20h09m35s] are [[mysql-data]] [2024-01-18T20:13:39.035Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T20:13:39.035Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] Validating CR cleanup [2024-01-18T20:13:39.035Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T20:13:39.035Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-1705608575-5486a15]] [2024-01-18T20:13:39.035Z] 2024-01-18 20:13:36 +0000:[INFO] [{BasicSelectiveRestore}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-1705608575-5486a15]] under [kube-system] namespace [2024-01-18T20:13:39.035Z] 2024/01/18 20:13:36 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-1705608575]}, Next try in [5s], timeout [5m0s] [2024-01-18T20:13:42.314Z] 2024-01-18 20:13:41 +0000:[INFO] [{BasicSelectiveRestore}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-1705608575-5486a15]] [2024-01-18T20:13:42.314Z] 2024-01-18 20:13:41 +0000:[INFO] [{BasicSelectiveRestore}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-1705608575-5486a15]] under [kube-system] namespace [2024-01-18T20:13:42.314Z] 2024/01/18 20:13:41 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-1705608575]}, Next try in [5s], timeout [5m0s] [2024-01-18T20:13:47.609Z] 2024-01-18 20:13:46 +0000:[INFO] [{BasicSelectiveRestore}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133]] [2024-01-18T20:13:47.609Z] INFO[2024-01-18 20:13:46] Verifying : Description : Creation and Validation of backup [tp-backup-1705608575] [2024-01-18T20:13:47.609Z] INFO[2024-01-18 20:13:46] Actual:true, Expected: true [2024-01-18T20:13:47.609Z] 2024-01-18 20:13:46 +0000:[INFO] [{BasicSelectiveRestore}] Selecting random backed-up apps and restoring them [2024-01-18T20:13:47.609Z] 2024-01-18 20:13:46 +0000:[INFO] [{BasicSelectiveRestore}] Selected application namespaces to restore: [[mysql-ibm-pxb-0-83717-01-18-20h09m35s mysql-ibm-pxb-5-83717-01-18-20h09m35s mysql-ibm-pxb-2-83717-01-18-20h09m35s]] [2024-01-18T20:13:48.170Z] 2024/01/18 20:13:47 DoRetryWithTimeout - Error: {restore status for [tp-restore-1705608575] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T20:14:20.212Z] 2024/01/18 20:14:17 DoRetryWithTimeout - Error: {restore status for [tp-restore-1705608575] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T20:14:22.738Z] INFO[2024-01-18 20:14:22] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T20:14:22.738Z] INFO[2024-01-18 20:14:22] sched stats: tick interval: 16822 < 1.2s (100%) [2024-01-18T20:14:22.738Z] INFO[2024-01-18 20:14:22] sched stats: tick process durations: 16821 < 500ms (100%) [2024-01-18T20:14:49.451Z] 2024/01/18 20:14:47 DoRetryWithTimeout - Error: {restore status for [tp-restore-1705608575] expected was [[PartialSuccess Success]] but got [InProgress] because of [Application resources restore is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T20:15:21.472Z] 2024-01-18 20:15:17 +0000:[INFO] [{BasicSelectiveRestore}] [backup.CreateRestore:#936] - Restore [tp-restore-1705608575] created successfully [2024-01-18T20:15:21.472Z] 2024-01-18 20:15:17 +0000:[INFO] [{BasicSelectiveRestore}] Switching cluster context to cluster [destination-cluster] [2024-01-18T20:15:21.472Z] 2024-01-18 20:15:17 +0000:[INFO] [{BasicSelectiveRestore}] Switching context to [/tmp/destination-config] [2024-01-18T20:15:21.472Z] 2024-01-18 20:15:18 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T20:15:21.472Z] 2024-01-18 20:15:18 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T20:15:21.472Z] 2024-01-18 20:15:18 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T20:15:21.472Z] 2024-01-18 20:15:18 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T20:15:21.472Z] 2024-01-18 20:15:18 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T20:15:21.472Z] 2024-01-18 20:15:18 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T20:15:21.472Z] 2024-01-18 20:15:18 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T20:15:21.472Z] 2024/01/18 20:15:18 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:15:29.563Z] 2024/01/18 20:15:29 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:15:39.504Z] 2024/01/18 20:15:39 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:15:49.445Z] 2024/01/18 20:15:49 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:15:59.390Z] 2024/01/18 20:15:59 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:16:09.336Z] 2024/01/18 20:16:09 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:16:19.278Z] 2024/01/18 20:16:19 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:16:29.241Z] 2024/01/18 20:16:29 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:16:41.409Z] 2024/01/18 20:16:39 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:16:49.500Z] 2024/01/18 20:16:49 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:16:59.452Z] 2024/01/18 20:16:59 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:17:09.402Z] 2024/01/18 20:17:09 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:17:21.582Z] 2024/01/18 20:17:19 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:17:29.666Z] 2024/01/18 20:17:29 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:17:39.607Z] 2024/01/18 20:17:39 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:17:49.549Z] 2024/01/18 20:17:49 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:18:01.730Z] 2024/01/18 20:17:59 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:18:09.832Z] 2024/01/18 20:18:09 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:18:19.775Z] 2024/01/18 20:18:19 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:18:29.726Z] 2024/01/18 20:18:29 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:18:41.901Z] 2024/01/18 20:18:39 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:18:49.998Z] 2024/01/18 20:18:49 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:18:59.937Z] 2024/01/18 20:18:59 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:19:09.883Z] 2024/01/18 20:19:09 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:19:22.070Z] 2024/01/18 20:19:19 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:19:30.153Z] 2024/01/18 20:19:29 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:19:40.102Z] 2024/01/18 20:19:39 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:19:50.069Z] 2024/01/18 20:19:49 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:20:00.007Z] 2024/01/18 20:19:59 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:20:12.177Z] 2024/01/18 20:20:09 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:20:20.265Z] INFO[2024-01-18 20:20:19] Verifying : Description : Creating restore [tp-restore-1705608575] from backup [tp-backup-1705608575] with selected namespaces [[mysql-ibm-pxb-0-83717-01-18-20h09m35s mysql-ibm-pxb-5-83717-01-18-20h09m35s mysql-ibm-pxb-2-83717-01-18-20h09m35s]] [2024-01-18T20:20:20.265Z] ERRO[2024-01-18 20:20:19] failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.265Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.265Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.265Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.265Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.265Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.265Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.265Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.265Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.265Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.265Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.265Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.265Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.265Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.265Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.265Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.265Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.266Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.266Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.266Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.266Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.266Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.266Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.266Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.266Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.266Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.266Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.266Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.266Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:20:20.266Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-18T20:20:20.266Z] 2024-01-18 20:20:20 +0000:[ERROR] [{BasicSelectiveRestore}] [backup.glob..func55.3:#138] - Fetching px-central-admin ctx. Err: can't find PxBackup service [px-backup] from list of services [2024-01-18T20:20:20.266Z] INFO[2024-01-18 20:20:20] --------Test End------ [2024-01-18T20:20:20.266Z] INFO[2024-01-18 20:20:20] #Test: [2024-01-18T20:20:20.266Z] INFO[2024-01-18 20:20:20] #Description: [2024-01-18T20:20:20.266Z] INFO[2024-01-18 20:20:20] ------------------------ [2024-01-18T20:20:20.266Z] 2024-01-18 20:20:20 +0000:[INFO] [{BasicSelectiveRestore}] [tests.EndPxBackupTorpedoTest:#7261] - >>>> FAILED TEST: {BasicSelectiveRestore} Selective Restore From a Basic Backup [2024-01-18T20:20:20.266Z] 2024-01-18 20:20:20 +0000:[INFO] [{BasicSelectiveRestore}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-83717-01-18-20h09m35s] [2024-01-18T20:20:20.266Z] 2024-01-18 20:20:20 +0000:[INFO] [{BasicSelectiveRestore}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-2-83717-01-18-20h09m35s] [2024-01-18T20:20:20.266Z] 2024-01-18 20:20:20 +0000:[INFO] [{BasicSelectiveRestore}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-5-83717-01-18-20h09m35s] [2024-01-18T20:20:20.266Z] 2024-01-18 20:20:20 +0000:[INFO] [{BasicSelectiveRestore}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T20:20:20.266Z] 2024-01-18 20:20:20 +0000:[INFO] [{BasicSelectiveRestore}] Switching context to [/tmp/destination-config] [2024-01-18T20:20:20.525Z] 2024-01-18 20:20:20 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T20:20:20.525Z] 2024-01-18 20:20:20 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T20:20:20.525Z] 2024-01-18 20:20:20 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T20:20:20.525Z] 2024-01-18 20:20:20 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T20:20:20.525Z] 2024-01-18 20:20:20 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T20:20:20.783Z] 2024-01-18 20:20:20 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T20:20:21.040Z] 2024-01-18 20:20:21 +0000:[INFO] [{BasicSelectiveRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T20:20:21.296Z] 2024/01/18 20:20:21 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:20:34.995Z] 2024/01/18 20:20:31 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:20:41.553Z] 2024/01/18 20:20:41 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:20:51.496Z] 2024/01/18 20:20:51 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:21:01.455Z] 2024/01/18 20:21:01 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:21:11.449Z] 2024/01/18 20:21:11 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:21:23.617Z] 2024/01/18 20:21:21 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:21:31.750Z] 2024/01/18 20:21:31 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:21:41.761Z] 2024/01/18 20:21:41 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:21:51.759Z] 2024/01/18 20:21:51 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:22:01.781Z] 2024/01/18 20:22:01 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:22:11.721Z] 2024/01/18 20:22:11 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:22:21.671Z] 2024/01/18 20:22:21 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:22:31.662Z] 2024/01/18 20:22:31 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:22:43.828Z] 2024/01/18 20:22:41 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:22:51.926Z] 2024/01/18 20:22:51 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:23:01.864Z] 2024/01/18 20:23:01 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:23:11.801Z] 2024/01/18 20:23:11 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:23:23.966Z] 2024/01/18 20:23:21 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:23:32.049Z] 2024/01/18 20:23:31 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:23:41.989Z] 2024/01/18 20:23:41 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:23:51.925Z] 2024/01/18 20:23:51 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:24:04.094Z] 2024/01/18 20:24:01 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:24:12.176Z] 2024/01/18 20:24:11 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:24:22.117Z] 2024/01/18 20:24:21 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:24:24.011Z] INFO[2024-01-18 20:24:23] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T20:24:24.011Z] INFO[2024-01-18 20:24:23] sched stats: tick interval: 17423 < 1.2s (100%) [2024-01-18T20:24:24.011Z] INFO[2024-01-18 20:24:23] sched stats: tick process durations: 17422 < 500ms (100%) [2024-01-18T20:24:32.099Z] 2024/01/18 20:24:31 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:24:42.039Z] 2024/01/18 20:24:42 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:24:54.202Z] 2024/01/18 20:24:52 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?) is False. Reason: KubeletNotReady}, Next try in [10s], timeout [5m0s] [2024-01-18T20:25:02.283Z] 2024/01/18 20:25:02 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?) is False. Reason: KubeletNotReady}, Next try in [10s], timeout [5m0s] [2024-01-18T20:25:12.223Z] 2024/01/18 20:25:12 DoRetryWithTimeout - Error: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?) is False. Reason: KubeletNotReady}, Next try in [10s], timeout [5m0s] [2024-01-18T20:25:22.184Z] 2024-01-18 20:25:22 +0000:[ERROR] [{BasicSelectiveRestore}] [tests.EndPxBackupTorpedoTest:#7272] - Error in setting destination kubeconfig. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.184Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.184Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.184Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.184Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.184Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.184Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.184Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.184Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.184Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.184Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.184Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.184Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.184Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.184Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.184Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.184Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.184Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.184Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.184Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.184Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.184Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.185Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.185Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.185Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.185Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.185Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.185Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?) is False. Reason: KubeletNotReady} [2024-01-18T20:25:22.185Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?) is False. Reason: KubeletNotReady} [2024-01-18T20:25:22.185Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?) is False. Reason: KubeletNotReady}] [2024-01-18T20:25:22.185Z] • Failure [947.009 seconds] [2024-01-18T20:25:22.185Z] {BasicSelectiveRestore} [2024-01-18T20:25:22.185Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:33 [2024-01-18T20:25:22.185Z] Selective Restore From a Basic Backup [It] [2024-01-18T20:25:22.185Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:76 [2024-01-18T20:25:22.185Z] [2024-01-18T20:25:22.185Z] Unexpected error: [2024-01-18T20:25:22.185Z] <*errors.errorString | 0xc0016ca750>: { [2024-01-18T20:25:22.186Z] s: "error: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node s... [2024-01-18T20:25:22.186Z] [2024-01-18T20:25:22.186Z] Gomega truncated this representation as it exceeds 'format.MaxLength'. [2024-01-18T20:25:22.186Z] Consider having the object provide a custom 'GomegaStringer' representation [2024-01-18T20:25:22.186Z] or adjust the parameters in Gomega's 'format' package. [2024-01-18T20:25:22.186Z] [2024-01-18T20:25:22.186Z] Learn more here: https://onsi.github.io/gomega/#adjusting-output [2024-01-18T20:25:22.186Z] [2024-01-18T20:25:22.186Z] error: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.186Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.186Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.186Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.186Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.187Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.187Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.187Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.187Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.187Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.187Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.187Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.187Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.187Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.187Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.187Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.187Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.187Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.187Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.187Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.187Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.187Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.187Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.187Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.187Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.187Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.187Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.188Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.188Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:25:22.188Z] {Node: 10.240.64.29 is not ready due to err: node: 10.240.64.29 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}], Description: Creating restore [tp-restore-1705608575] from backup [tp-backup-1705608575] with selected namespaces [[mysql-ibm-pxb-0-83717-01-18-20h09m35s mysql-ibm-pxb-5-83717-01-18-20h09m35s mysql-ibm-pxb-2-83717-01-18-20h09m35s]] [2024-01-18T20:25:22.188Z] occurred [2024-01-18T20:25:22.188Z] [2024-01-18T20:25:22.188Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-18T20:25:22.188Z] [2024-01-18T20:25:22.188Z] Full Stack Trace [2024-01-18T20:25:22.188Z] github.com/portworx/torpedo/pkg/aetosutil.(*Dashboard).VerifyFatal(0x7ffb600?, {0x702af20?, 0xc0016ca610}, {0x0?, 0x0}, {0xc001d6c1c0, 0xda}) [2024-01-18T20:25:22.188Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 +0x143 [2024-01-18T20:25:22.188Z] github.com/portworx/torpedo/tests/backup.glob..func55.2.5() [2024-01-18T20:25:22.188Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:132 +0x47f [2024-01-18T20:25:22.188Z] github.com/portworx/torpedo/tests/backup.glob..func55.2() [2024-01-18T20:25:22.188Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:119 +0x4d9 [2024-01-18T20:25:22.188Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-18T20:25:22.188Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-18T20:25:22.188Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-18T20:25:22.188Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-18T20:25:22.188Z] created by testing.(*T).Run [2024-01-18T20:25:22.188Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-18T20:25:22.188Z] ------------------------------ [2024-01-18T20:25:22.188Z] STEP: schedule applications [2024-01-18T20:25:22.188Z] STEP: schedule applications [2024-01-18T20:25:22.188Z] STEP: schedule applications [2024-01-18T20:25:22.188Z] STEP: schedule applications [2024-01-18T20:25:22.188Z] STEP: schedule applications [2024-01-18T20:25:22.188Z] STEP: schedule applications [2024-01-18T20:25:22.188Z] STEP: Validating deployed applications [2024-01-18T20:25:22.188Z] STEP: validate applications [2024-01-18T20:25:22.188Z] STEP: validate mysql-ibm app's volumes [2024-01-18T20:25:22.188Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T20:25:22.188Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T20:25:22.188Z] STEP: get mysql-ibm app's volume: pvc-3533cf38-8bcc-45fc-bee6-b8bb4d41c70f inspected by the volume driver [2024-01-18T20:25:22.188Z] STEP: wait for mysql-ibm app to start running [2024-01-18T20:25:22.188Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T20:25:22.188Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T20:25:22.188Z] STEP: Validate Px pod restart count [2024-01-18T20:25:22.188Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T20:25:22.188Z] STEP: validate mysql-ibm app's volumes [2024-01-18T20:25:22.188Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T20:25:22.188Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T20:25:22.188Z] STEP: get mysql-ibm app's volume: pvc-d7d7cb90-2508-46c8-807c-098ae6519ec2 inspected by the volume driver [2024-01-18T20:25:22.188Z] STEP: wait for mysql-ibm app to start running [2024-01-18T20:25:22.188Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T20:25:22.188Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T20:25:22.188Z] STEP: Validate Px pod restart count [2024-01-18T20:25:22.188Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T20:25:22.188Z] STEP: validate mysql-ibm app's volumes [2024-01-18T20:25:22.188Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T20:25:22.188Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T20:25:22.188Z] STEP: get mysql-ibm app's volume: pvc-257b09ac-f456-49a3-b554-d1a6b955a67a inspected by the volume driver [2024-01-18T20:25:22.188Z] STEP: wait for mysql-ibm app to start running [2024-01-18T20:25:22.188Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T20:25:22.188Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T20:25:22.188Z] STEP: Validate Px pod restart count [2024-01-18T20:25:22.188Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T20:25:22.189Z] STEP: validate mysql-ibm app's volumes [2024-01-18T20:25:22.189Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T20:25:22.189Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T20:25:22.189Z] STEP: get mysql-ibm app's volume: pvc-beacc472-7dad-4629-9976-00fb76efdb0c inspected by the volume driver [2024-01-18T20:25:22.189Z] STEP: wait for mysql-ibm app to start running [2024-01-18T20:25:22.189Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T20:25:22.189Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T20:25:22.189Z] STEP: Validate Px pod restart count [2024-01-18T20:25:22.189Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T20:25:22.189Z] STEP: validate mysql-ibm app's volumes [2024-01-18T20:25:22.189Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T20:25:22.189Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T20:25:22.189Z] STEP: get mysql-ibm app's volume: pvc-6a12597a-9251-44de-8b14-d45388db6303 inspected by the volume driver [2024-01-18T20:25:22.189Z] STEP: wait for mysql-ibm app to start running [2024-01-18T20:25:22.189Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T20:25:22.189Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T20:25:22.189Z] STEP: Validate Px pod restart count [2024-01-18T20:25:22.189Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T20:25:22.189Z] STEP: validate mysql-ibm app's volumes [2024-01-18T20:25:22.189Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T20:25:22.189Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T20:25:22.189Z] STEP: get mysql-ibm app's volume: pvc-3b2558ec-36dc-473d-bcd4-5a8ef6073298 inspected by the volume driver [2024-01-18T20:25:22.189Z] STEP: wait for mysql-ibm app to start running [2024-01-18T20:25:22.189Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T20:25:22.189Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T20:25:22.189Z] STEP: Validate Px pod restart count [2024-01-18T20:25:22.189Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T20:25:22.189Z] STEP: Creating backup location and cloud setting [2024-01-18T20:25:22.189Z] STEP: Registering cluster for backup [2024-01-18T20:25:22.189Z] STEP: Taking backup of multiple namespaces [2024-01-18T20:25:22.189Z] STEP: Selecting random backed-up apps and restoring them [2024-01-18T20:25:22.189Z] S2024-01-18 20:25:22 +0000:[INFO] [{DeleteAllBackupObjects}] Log Dir: /testresults//DeleteAllBackupObjects.log [2024-01-18T20:25:22.189Z] INFO[2024-01-18 20:25:22] --------Test Start------ [2024-01-18T20:25:22.189Z] INFO[2024-01-18 20:25:22] #Test: DeleteAllBackupObjects [2024-01-18T20:25:22.189Z] INFO[2024-01-18 20:25:22] #Description: Create the backup Objects and Delete [2024-01-18T20:25:22.189Z] INFO[2024-01-18 20:25:22] ------------------------ [2024-01-18T20:25:22.189Z] 2024-01-18 20:25:22 +0000:[INFO] [{DeleteAllBackupObjects}] Verifying if the pre/post rules for the required apps are present in the AppParameters or not [2024-01-18T20:25:22.189Z] 2024-01-18 20:25:22 +0000:[INFO] [{DeleteAllBackupObjects}] Deploy applications [2024-01-18T20:25:22.189Z] 2024-01-18 20:25:22 +0000:[INFO] [{DeleteAllBackupObjects}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T20:25:22.189Z] 2024-01-18 20:25:22 +0000:[INFO] [{DeleteAllBackupObjects}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T20:25:22.189Z] 2024-01-18 20:25:22 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T20:25:22.445Z] 2024-01-18 20:25:22 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T20:25:22.445Z] 2024-01-18 20:25:22 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T20:25:22.445Z] 2024-01-18 20:25:22 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T20:25:22.446Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T20:25:22.446Z] 2024-01-18 20:25:22 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T20:25:22.447Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T20:25:22.447Z] 2024-01-18 20:25:22 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T20:25:22.447Z] 2024-01-18 20:25:22 +0000:[INFO] [{DeleteAllBackupObjects}] Validate applications [2024-01-18T20:25:22.447Z] 2024-01-18 20:25:22 +0000:[INFO] [{DeleteAllBackupObjects}] Validating mysql-ibm app [2024-01-18T20:25:22.447Z] 2024-01-18 20:25:22 +0000:[INFO] [{DeleteAllBackupObjects}] Validating mysql-ibm app's volumes [2024-01-18T20:25:22.702Z] 2024-01-18 20:25:22 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:25:22.702Z] 2024/01/18 20:25:22 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T20:25:32.668Z] 2024/01/18 20:25:32 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T20:25:44.837Z] 2024-01-18 20:25:42 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-58088-01-18-20h25m22s [2024-01-18T20:25:44.837Z] 2024-01-18 20:25:42 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T20:25:44.837Z] 2024-01-18 20:25:42 +0000:[WARNING] [{DeleteAllBackupObjects}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T20:25:44.837Z] 2024-01-18 20:25:42 +0000:[INFO] [{DeleteAllBackupObjects}] wait for mysql-ibm app to start running [2024-01-18T20:25:44.837Z] 2024-01-18 20:25:42 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T20:25:44.837Z] 2024/01/18 20:25:42 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:25:44.837Z] pod name:mysql-7d766cd65c-9tkvn namespace:mysql-ibm-pxb-0-58088-01-18-20h25m22s running:false ready:false node:10.240.64.29 [2024-01-18T20:25:44.837Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:25:54.800Z] 2024/01/18 20:25:52 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:25:54.800Z] pod name:mysql-7d766cd65c-9tkvn namespace:mysql-ibm-pxb-0-58088-01-18-20h25m22s running:false ready:false node:10.240.64.29 [2024-01-18T20:25:54.800Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:26:04.755Z] 2024/01/18 20:26:03 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:26:04.755Z] pod name:mysql-7d766cd65c-9tkvn namespace:mysql-ibm-pxb-0-58088-01-18-20h25m22s running:false ready:false node:10.240.64.29 [2024-01-18T20:26:04.755Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:26:14.702Z] 2024/01/18 20:26:13 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:26:14.702Z] pod name:mysql-7d766cd65c-9tkvn namespace:mysql-ibm-pxb-0-58088-01-18-20h25m22s running:false ready:false node:10.240.64.29 [2024-01-18T20:26:14.702Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:26:24.646Z] 2024/01/18 20:26:23 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:26:24.646Z] pod name:mysql-7d766cd65c-9tkvn namespace:mysql-ibm-pxb-0-58088-01-18-20h25m22s running:true ready:false node:10.240.64.29 [2024-01-18T20:26:24.646Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:26:34.590Z] 2024/01/18 20:26:33 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:26:34.591Z] pod name:mysql-7d766cd65c-9tkvn namespace:mysql-ibm-pxb-0-58088-01-18-20h25m22s running:true ready:false node:10.240.64.29 [2024-01-18T20:26:34.591Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:26:44.535Z] 2024/01/18 20:26:43 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:26:44.535Z] pod name:mysql-7d766cd65c-9tkvn namespace:mysql-ibm-pxb-0-58088-01-18-20h25m22s running:true ready:false node:10.240.64.29 [2024-01-18T20:26:44.535Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:26:54.478Z] 2024/01/18 20:26:53 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:26:54.478Z] pod name:mysql-7d766cd65c-9tkvn namespace:mysql-ibm-pxb-0-58088-01-18-20h25m22s running:true ready:false node:10.240.64.29 [2024-01-18T20:26:54.478Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:27:04.447Z] 2024/01/18 20:27:03 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:27:04.447Z] pod name:mysql-7d766cd65c-9tkvn namespace:mysql-ibm-pxb-0-58088-01-18-20h25m22s running:true ready:false node:10.240.64.29 [2024-01-18T20:27:04.447Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:27:14.443Z] 2024/01/18 20:27:13 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:27:14.443Z] pod name:mysql-7d766cd65c-9tkvn namespace:mysql-ibm-pxb-0-58088-01-18-20h25m22s running:true ready:false node:10.240.64.29 [2024-01-18T20:27:14.443Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:27:24.411Z] 2024/01/18 20:27:23 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:27:24.411Z] pod name:mysql-7d766cd65c-9tkvn namespace:mysql-ibm-pxb-0-58088-01-18-20h25m22s running:true ready:false node:10.240.64.29 [2024-01-18T20:27:24.411Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:27:34.358Z] 2024-01-18 20:27:33 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T20:27:34.358Z] 2024-01-18 20:27:33 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T20:27:34.358Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] validate if mysql-ibm app's volumes are setup [2024-01-18T20:27:34.358Z] 2024-01-18 20:27:34 +0000:[WARNING] [{DeleteAllBackupObjects}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T20:27:34.358Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:27:34.358Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T20:27:34.358Z] 2024-01-18 20:27:34 +0000:[WARNING] [{DeleteAllBackupObjects}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T20:27:34.358Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] Creating pre rule for deployed apps [2024-01-18T20:27:34.358Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] [portworx.(*portworx).CreateRuleForBackup:#2137] - Pre rule not required for this application [2024-01-18T20:27:34.358Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] [portworx.(*portworx).CreateRuleForBackup:#2159] - Rules not required for the apps [2024-01-18T20:27:34.358Z] INFO[2024-01-18 20:27:34] Verifying : Description : Verifying pre rule for backup [2024-01-18T20:27:34.358Z] INFO[2024-01-18 20:27:34] Actual:true, Expected: true [2024-01-18T20:27:34.358Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] Creating post rule for deployed apps [2024-01-18T20:27:34.358Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] [portworx.(*portworx).CreateRuleForBackup:#2154] - Post rule not required for this application [2024-01-18T20:27:34.358Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] [portworx.(*portworx).CreateRuleForBackup:#2159] - Rules not required for the apps [2024-01-18T20:27:34.358Z] INFO[2024-01-18 20:27:34] Verifying : Description : Verifying Post rule for backup [2024-01-18T20:27:34.358Z] INFO[2024-01-18 20:27:34] Actual:true, Expected: true [2024-01-18T20:27:34.358Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] Creating cloud account and backup location [2024-01-18T20:27:34.358Z] 2024-01-18 20:27:34 +0000:[ERROR] [{DeleteAllBackupObjects}] [backup.glob..func57.2.3:#396] - Fetching px-central-admin ctx. Err: can't find PxBackup service [px-backup] from list of services [2024-01-18T20:27:34.358Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] [backup.glob..func57.3:#514] - Deleting deployed applications [2024-01-18T20:27:34.358Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] Destroying apps [2024-01-18T20:27:34.358Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T20:27:34.620Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T20:27:34.620Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] start destroying mysql-ibm app [2024-01-18T20:27:34.620Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T20:27:34.620Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T20:27:34.620Z] 2024-01-18 20:27:34 +0000:[WARNING] [{DeleteAllBackupObjects}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T20:27:34.620Z] INFO[2024-01-18 20:27:34] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T20:27:34.620Z] INFO[2024-01-18 20:27:34] Actual:true, Expected: true [2024-01-18T20:27:34.620Z] INFO[2024-01-18 20:27:34] --------Test End------ [2024-01-18T20:27:34.620Z] INFO[2024-01-18 20:27:34] #Test: [2024-01-18T20:27:34.620Z] INFO[2024-01-18 20:27:34] #Description: [2024-01-18T20:27:34.620Z] INFO[2024-01-18 20:27:34] ------------------------ [2024-01-18T20:27:34.620Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] [tests.EndPxBackupTorpedoTest:#7261] - >>>> FAILED TEST: {DeleteAllBackupObjects} Create backup objects and delete [2024-01-18T20:27:34.620Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-58088-01-18-20h25m22s] [2024-01-18T20:27:34.620Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T20:27:34.620Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] Switching context to [/tmp/destination-config] [2024-01-18T20:27:34.875Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.4] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T20:27:34.875Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.44] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T20:27:34.875Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.6] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T20:27:34.876Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.23] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T20:27:34.876Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.4] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T20:27:35.131Z] 2024-01-18 20:27:34 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.5] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T20:27:35.693Z] 2024-01-18 20:27:35 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.29] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T20:27:35.949Z] 2024-01-18 20:27:35 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.6] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T20:27:36.510Z] 2024-01-18 20:27:36 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.7] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T20:27:36.510Z] 2024-01-18 20:27:36 +0000:[WARNING] [{DeleteAllBackupObjects}] [ibm.(*ibm).RefreshDriverEndpoints:#75] - RefreshDriverEndpoints function has not been implemented for volume driver - ibm [2024-01-18T20:27:36.767Z] 2024-01-18 20:27:36 +0000:[WARNING] [{DeleteAllBackupObjects}] [ssh.GetExecPodNamespace:#841] - can't find portworx-service Portworx service from list of services. [2024-01-18T20:27:36.767Z] 2024-01-18 20:27:36 +0000:[WARNING] [{DeleteAllBackupObjects}] [ssh.(*SSH).updateDriver:#169] - Unable to find portworx namespace. Setting debug pod namespace to kube-system [2024-01-18T20:27:36.767Z] 2024-01-18 20:27:36 +0000:[INFO] [{DeleteAllBackupObjects}] UpdateDriver: ssh driver's namespace (the 'portworx-service' namespace) is updated to [kube-system] [2024-01-18T20:27:36.767Z] 2024-01-18 20:27:36 +0000:[INFO] [{DeleteAllBackupObjects}] Switched context to [/tmp/destination-config] [2024-01-18T20:27:37.022Z] 2024-01-18 20:27:36 +0000:[INFO] [{DeleteAllBackupObjects}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-58088-01-18-20h25m22s] [2024-01-18T20:27:37.278Z] 2024-01-18 20:27:37 +0000:[INFO] [{DeleteAllBackupObjects}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {DeleteAllBackupObjects} Create backup objects and delete [2024-01-18T20:27:37.278Z] 2024-01-18 20:27:37 +0000:[INFO] [{DeleteAllBackupObjects}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-18T20:27:37.278Z] 2024/01/18 20:27:37 DoRetryWithTimeout - Error: {dial tcp 10.240.64.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:27:43.555Z] 2024/01/18 20:27:42 DoRetryWithTimeout - Error: {dial tcp 10.240.64.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:27:47.723Z] 2024/01/18 20:27:47 DoRetryWithTimeout - Error: {dial tcp 10.240.64.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:27:52.966Z] 2024/01/18 20:27:52 DoRetryWithTimeout - Error: {dial tcp 10.240.64.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:27:57.130Z] 2024/01/18 20:27:57 DoRetryWithTimeout - Error: {dial tcp 10.240.64.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:28:02.375Z] 2024/01/18 20:28:02 DoRetryWithTimeout - Error: {dial tcp 10.240.64.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:28:07.628Z] 2024/01/18 20:28:07 DoRetryWithTimeout - Error: {dial tcp 10.240.64.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:28:12.870Z] 2024/01/18 20:28:12 DoRetryWithTimeout - Error: {dial tcp 10.240.64.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:28:18.112Z] 2024/01/18 20:28:17 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.64.6. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.6 10.240.64.6]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.6:22: connect: connection refused} [2024-01-18T20:28:18.112Z] {dial tcp 10.240.64.6:22: connect: connection refused} [2024-01-18T20:28:18.113Z] {dial tcp 10.240.64.6:22: connect: connection refused} [2024-01-18T20:28:18.113Z] {dial tcp 10.240.64.6:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-18T20:28:22.281Z] 2024-01-18 20:28:22 +0000:[WARNING] [{DeleteAllBackupObjects}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.64.6. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.6 10.240.64.6]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.6:22: connect: connection refused} [2024-01-18T20:28:22.281Z] {dial tcp 10.240.64.6:22: connect: connection refused} [2024-01-18T20:28:22.281Z] {dial tcp 10.240.64.6:22: connect: connection refused} [2024-01-18T20:28:22.281Z] {dial tcp 10.240.64.6:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T20:28:22.281Z] 2024-01-18 20:28:22 +0000:[ERROR] [{DeleteAllBackupObjects}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.64.6. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.6 10.240.64.6]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.6:22: connect: connection refused} [2024-01-18T20:28:22.281Z] {dial tcp 10.240.64.6:22: connect: connection refused} [2024-01-18T20:28:22.281Z] {dial tcp 10.240.64.6:22: connect: connection refused} [2024-01-18T20:28:22.281Z] {dial tcp 10.240.64.6:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T20:28:22.281Z] 2024-01-18 20:28:22 +0000:[INFO] [{DeleteAllBackupObjects}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T20:28:22.281Z] 2024-01-18 20:28:22 +0000:[INFO] [{DeleteAllBackupObjects}] Switching context to [/tmp/source-config] [2024-01-18T20:28:22.281Z] 2024-01-18 20:28:22 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T20:28:22.537Z] 2024-01-18 20:28:22 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T20:28:22.537Z] 2024-01-18 20:28:22 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T20:28:22.537Z] 2024-01-18 20:28:22 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T20:28:22.793Z] 2024-01-18 20:28:22 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T20:28:22.793Z] 2024-01-18 20:28:22 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T20:28:23.049Z] 2024-01-18 20:28:22 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T20:28:23.612Z] 2024-01-18 20:28:23 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T20:28:23.869Z] 2024-01-18 20:28:23 +0000:[INFO] [{DeleteAllBackupObjects}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T20:28:24.126Z] 2024/01/18 20:28:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:28:34.102Z] 2024/01/18 20:28:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:28:44.093Z] 2024/01/18 20:28:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:28:54.034Z] 2024/01/18 20:28:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:29:06.203Z] 2024/01/18 20:29:04 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:29:14.288Z] 2024/01/18 20:29:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:29:24.228Z] 2024/01/18 20:29:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:29:34.169Z] 2024/01/18 20:29:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:29:46.340Z] 2024/01/18 20:29:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:29:54.422Z] 2024/01/18 20:29:54 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:30:04.358Z] 2024/01/18 20:30:04 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:30:14.293Z] 2024/01/18 20:30:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:30:26.461Z] 2024/01/18 20:30:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:30:34.544Z] 2024/01/18 20:30:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:30:44.486Z] 2024/01/18 20:30:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:30:54.428Z] 2024/01/18 20:30:54 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:31:06.598Z] 2024/01/18 20:31:04 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:31:14.703Z] 2024/01/18 20:31:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:31:24.661Z] 2024/01/18 20:31:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:31:34.600Z] 2024/01/18 20:31:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:31:44.542Z] 2024/01/18 20:31:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:31:56.713Z] 2024/01/18 20:31:54 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:32:04.818Z] 2024/01/18 20:32:04 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:32:14.778Z] 2024/01/18 20:32:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:32:24.764Z] 2024/01/18 20:32:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:32:34.746Z] 2024/01/18 20:32:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:32:44.706Z] 2024/01/18 20:32:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:32:56.886Z] 2024/01/18 20:32:54 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:33:04.967Z] 2024/01/18 20:33:04 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:33:14.920Z] 2024/01/18 20:33:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:33:24.859Z] 2024-01-18 20:33:24 +0000:[ERROR] [{DeleteAllBackupObjects}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.859Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.859Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.859Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.859Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.859Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.859Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.859Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.859Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.859Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.859Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.860Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.860Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.860Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.860Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.860Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.860Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.860Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.860Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.860Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.860Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.860Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.860Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.860Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.860Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.860Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.860Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.860Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.860Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:33:24.860Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-18T20:33:24.860Z] [2024-01-18T20:33:24.860Z] ------------------------------ [2024-01-18T20:33:24.860Z] • Failure [482.631 seconds] [2024-01-18T20:33:24.860Z] {DeleteAllBackupObjects} [2024-01-18T20:33:24.860Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:311 [2024-01-18T20:33:24.860Z] Create backup objects and delete [It] [2024-01-18T20:33:24.860Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:366 [2024-01-18T20:33:24.860Z] [2024-01-18T20:33:24.860Z] Unexpected error: [2024-01-18T20:33:24.860Z] <*errors.errorString | 0xc001865700>: { [2024-01-18T20:33:24.860Z] s: "can't find PxBackup service [px-backup] from list of services", [2024-01-18T20:33:24.860Z] } [2024-01-18T20:33:24.860Z] can't find PxBackup service [px-backup] from list of services [2024-01-18T20:33:24.860Z] occurred [2024-01-18T20:33:24.860Z] [2024-01-18T20:33:24.861Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-18T20:33:24.861Z] [2024-01-18T20:33:24.861Z] Full Stack Trace [2024-01-18T20:33:24.861Z] github.com/portworx/torpedo/pkg/log.FailOnError({0x88a5060, 0xc001865700}, {0x7f14b70?, 0x0?}, {0x0?, 0x40?, 0xc000100000?}) [2024-01-18T20:33:24.861Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 +0x290 [2024-01-18T20:33:24.861Z] github.com/portworx/torpedo/tests/backup.glob..func57.2.3() [2024-01-18T20:33:24.861Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:396 +0xc9 [2024-01-18T20:33:24.861Z] github.com/portworx/torpedo/tests/backup.glob..func57.2() [2024-01-18T20:33:24.861Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:393 +0x409 [2024-01-18T20:33:24.861Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-18T20:33:24.861Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-18T20:33:24.861Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-18T20:33:24.861Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-18T20:33:24.861Z] created by testing.(*T).Run [2024-01-18T20:33:24.861Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-18T20:33:24.861Z] ------------------------------ [2024-01-18T20:33:24.861Z] STEP: schedule applications [2024-01-18T20:33:24.861Z] STEP: Validate applications [2024-01-18T20:33:24.861Z] STEP: validate applications [2024-01-18T20:33:24.861Z] STEP: validate mysql-ibm app's volumes [2024-01-18T20:33:24.861Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T20:33:24.861Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T20:33:24.861Z] STEP: get mysql-ibm app's volume: pvc-bb664f36-88b3-406f-97eb-79831bb7af59 inspected by the volume driver [2024-01-18T20:33:24.861Z] STEP: wait for mysql-ibm app to start running [2024-01-18T20:33:24.861Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T20:33:24.861Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T20:33:24.861Z] STEP: Validate Px pod restart count [2024-01-18T20:33:24.861Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T20:33:24.861Z] STEP: Creating rules for backup [2024-01-18T20:33:24.861Z] STEP: Creating cloud account and backup location [2024-01-18T20:33:24.861Z] STEP: destroy apps [2024-01-18T20:33:24.861Z] STEP: destroy the mysql-ibm app's volumes [2024-01-18T20:33:24.861Z] STEP: start destroying mysql-ibm app [2024-01-18T20:33:24.861Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T20:33:24.861Z] S2024-01-18 20:33:24 +0000:[INFO] [{DeleteUsersRole}] Log Dir: /testresults//DeleteUsersRole.log [2024-01-18T20:33:24.861Z] INFO[2024-01-18 20:33:24] --------Test Start------ [2024-01-18T20:33:24.861Z] INFO[2024-01-18 20:33:24] #Test: DeleteUsersRole [2024-01-18T20:33:24.861Z] INFO[2024-01-18 20:33:24] #Description: Delete role and users [2024-01-18T20:33:24.861Z] INFO[2024-01-18 20:33:24] ------------------------ [2024-01-18T20:33:24.861Z] 2024-01-18 20:33:24 +0000:[INFO] [{DeleteUsersRole}] Creating 80 users [2024-01-18T20:33:27.380Z] 2024-01-18 20:33:26 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610004] creation - [2024-01-18T20:33:27.380Z] 2024-01-18 20:33:26 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610004] created [2024-01-18T20:33:27.380Z] 2024-01-18 20:33:26 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610004 [2024-01-18T20:33:27.380Z] 2024-01-18 20:33:27 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610004 - 5c93ae4c-682e-4b77-92e8-36dcf764a6ab [2024-01-18T20:33:29.269Z] 2024-01-18 20:33:28 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610006] creation - [2024-01-18T20:33:29.269Z] 2024-01-18 20:33:28 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610006] created [2024-01-18T20:33:29.269Z] 2024-01-18 20:33:28 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610006 [2024-01-18T20:33:29.831Z] 2024-01-18 20:33:29 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610006 - e67bd13c-0032-4115-8a8e-fb7a4bb412ac [2024-01-18T20:33:32.349Z] 2024-01-18 20:33:32 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610008] creation - [2024-01-18T20:33:32.349Z] 2024-01-18 20:33:32 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610008] created [2024-01-18T20:33:32.349Z] 2024-01-18 20:33:32 +0000:[INFO] [{DeleteUsersRole}] Adding role default-roles-master to user testautouser1705610008 [2024-01-18T20:33:32.910Z] 2024-01-18 20:33:32 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610008 - fd9cfa2f-6ba4-40a1-a0f1-5c111aa0a5e8 [2024-01-18T20:33:34.798Z] 2024-01-18 20:33:34 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610010] creation - [2024-01-18T20:33:34.798Z] 2024-01-18 20:33:34 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610010] created [2024-01-18T20:33:34.798Z] 2024-01-18 20:33:34 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610010 [2024-01-18T20:33:36.685Z] 2024-01-18 20:33:36 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610010 - c78a9cc3-7668-40be-9f4a-71cf45ade350 [2024-01-18T20:33:37.246Z] 2024-01-18 20:33:37 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610012] creation - [2024-01-18T20:33:37.246Z] 2024-01-18 20:33:37 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610012] created [2024-01-18T20:33:37.246Z] 2024-01-18 20:33:37 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610012 [2024-01-18T20:33:39.146Z] 2024-01-18 20:33:39 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610014] creation - [2024-01-18T20:33:39.146Z] 2024-01-18 20:33:39 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610014] created [2024-01-18T20:33:39.146Z] 2024-01-18 20:33:39 +0000:[INFO] [{DeleteUsersRole}] Adding role default-roles-master to user testautouser1705610014 [2024-01-18T20:33:39.707Z] 2024-01-18 20:33:39 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610012 - 84b1f4be-bffc-4566-8592-4faeb1e00fd9 [2024-01-18T20:33:42.223Z] 2024-01-18 20:33:42 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610016] creation - [2024-01-18T20:33:42.223Z] 2024-01-18 20:33:42 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610016] created [2024-01-18T20:33:42.223Z] 2024-01-18 20:33:42 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610016 [2024-01-18T20:33:42.485Z] 2024-01-18 20:33:42 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610014 - 769b0e6a-d44f-4666-a9dd-c5820ce32ff1 [2024-01-18T20:33:45.000Z] 2024-01-18 20:33:44 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610018] creation - [2024-01-18T20:33:45.000Z] 2024-01-18 20:33:44 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610018] created [2024-01-18T20:33:45.000Z] 2024-01-18 20:33:44 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610018 [2024-01-18T20:33:46.364Z] I0118 20:33:45.942890 14 request.go:682] Waited for 1.154244014s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-18T20:33:46.364Z] 2024-01-18 20:33:45 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610016 - fbd35ec6-1a78-49ff-b7b0-74d1b338037c [2024-01-18T20:33:46.926Z] 2024-01-18 20:33:46 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610020] creation - [2024-01-18T20:33:46.926Z] 2024-01-18 20:33:46 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610020] created [2024-01-18T20:33:46.926Z] 2024-01-18 20:33:46 +0000:[INFO] [{DeleteUsersRole}] Adding role default-roles-master to user testautouser1705610020 [2024-01-18T20:33:49.443Z] 2024-01-18 20:33:49 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610018 - 48d8a9a5-648d-46d3-b1a6-3e27c16d3a2e [2024-01-18T20:33:50.004Z] 2024-01-18 20:33:49 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610022] creation - [2024-01-18T20:33:50.004Z] 2024-01-18 20:33:49 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610022] created [2024-01-18T20:33:50.004Z] 2024-01-18 20:33:49 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610022 [2024-01-18T20:33:52.534Z] 2024-01-18 20:33:52 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610024] creation - [2024-01-18T20:33:52.534Z] 2024-01-18 20:33:52 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610024] created [2024-01-18T20:33:52.534Z] 2024-01-18 20:33:52 +0000:[INFO] [{DeleteUsersRole}] Adding role default-roles-master to user testautouser1705610024 [2024-01-18T20:33:52.534Z] 2024-01-18 20:33:52 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610020 - 3d974540-f2fb-4df5-becb-1b26936765f6 [2024-01-18T20:33:55.058Z] 2024-01-18 20:33:54 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610026] creation - [2024-01-18T20:33:55.059Z] 2024-01-18 20:33:54 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610026] created [2024-01-18T20:33:55.059Z] 2024-01-18 20:33:54 +0000:[INFO] [{DeleteUsersRole}] Adding role default-roles-master to user testautouser1705610026 [2024-01-18T20:33:56.433Z] 2024-01-18 20:33:55 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610022 - c3b79415-4826-451a-be4d-707b5011e3f7 [2024-01-18T20:33:56.433Z] I0118 20:33:56.143151 14 request.go:682] Waited for 1.431195452s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/services [2024-01-18T20:33:57.366Z] 2024-01-18 20:33:57 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610028] creation - [2024-01-18T20:33:57.366Z] 2024-01-18 20:33:57 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610028] created [2024-01-18T20:33:57.366Z] 2024-01-18 20:33:57 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610028 [2024-01-18T20:33:59.258Z] 2024-01-18 20:33:58 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610024 - 878b64e4-b6cd-464c-888e-e6ffc9769335 [2024-01-18T20:33:59.819Z] 2024-01-18 20:33:59 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610030] creation - [2024-01-18T20:33:59.820Z] 2024-01-18 20:33:59 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610030] created [2024-01-18T20:33:59.820Z] 2024-01-18 20:33:59 +0000:[INFO] [{DeleteUsersRole}] Adding role default-roles-master to user testautouser1705610030 [2024-01-18T20:34:02.341Z] 2024-01-18 20:34:01 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610026 - a0235953-6fbc-4765-bf99-d076797582a4 [2024-01-18T20:34:02.341Z] 2024-01-18 20:34:02 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610032] creation - [2024-01-18T20:34:02.341Z] 2024-01-18 20:34:02 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610032] created [2024-01-18T20:34:02.341Z] 2024-01-18 20:34:02 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610032 [2024-01-18T20:34:04.858Z] 2024-01-18 20:34:04 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610034] creation - [2024-01-18T20:34:04.858Z] 2024-01-18 20:34:04 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610034] created [2024-01-18T20:34:04.858Z] 2024-01-18 20:34:04 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610034 [2024-01-18T20:34:05.787Z] 2024-01-18 20:34:05 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610028 - e08f60f9-2318-418d-a57f-bbf03280c55d [2024-01-18T20:34:06.349Z] I0118 20:34:06.143520 14 request.go:682] Waited for 2.156849761s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-18T20:34:06.910Z] 2024-01-18 20:34:06 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610036] creation - [2024-01-18T20:34:06.911Z] 2024-01-18 20:34:06 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610036] created [2024-01-18T20:34:06.911Z] 2024-01-18 20:34:06 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610036 [2024-01-18T20:34:08.278Z] 2024-01-18 20:34:08 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610030 - 3125668c-dc81-4c20-bfcb-71a66c9f789b [2024-01-18T20:34:10.168Z] 2024-01-18 20:34:10 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610038] creation - [2024-01-18T20:34:10.168Z] 2024-01-18 20:34:10 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610038] created [2024-01-18T20:34:10.168Z] 2024-01-18 20:34:10 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610038 [2024-01-18T20:34:11.535Z] 2024-01-18 20:34:11 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610032 - c434d6dd-e59a-47b7-b2e6-ff0cafe49b99 [2024-01-18T20:34:12.903Z] 2024-01-18 20:34:12 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610040] creation - [2024-01-18T20:34:12.903Z] 2024-01-18 20:34:12 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610040] created [2024-01-18T20:34:12.903Z] 2024-01-18 20:34:12 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610040 [2024-01-18T20:34:14.795Z] 2024-01-18 20:34:14 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610034 - 98db2477-47c0-4182-b229-a7801d11b9df [2024-01-18T20:34:15.359Z] 2024-01-18 20:34:15 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610042] creation - [2024-01-18T20:34:15.359Z] 2024-01-18 20:34:15 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610042] created [2024-01-18T20:34:15.359Z] 2024-01-18 20:34:15 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610042 [2024-01-18T20:34:16.727Z] I0118 20:34:16.343602 14 request.go:682] Waited for 2.953386241s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-18T20:34:17.655Z] 2024-01-18 20:34:17 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610044] creation - [2024-01-18T20:34:17.655Z] 2024-01-18 20:34:17 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610044] created [2024-01-18T20:34:17.655Z] 2024-01-18 20:34:17 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610044 [2024-01-18T20:34:17.911Z] 2024-01-18 20:34:17 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610036 - 2715aeee-1103-42ed-8c0e-dbbe725053c9 [2024-01-18T20:34:20.445Z] 2024-01-18 20:34:20 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610046] creation - [2024-01-18T20:34:20.445Z] 2024-01-18 20:34:20 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610046] created [2024-01-18T20:34:20.445Z] 2024-01-18 20:34:20 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610046 [2024-01-18T20:34:21.810Z] 2024-01-18 20:34:21 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610038 - 826f6706-3c7e-4be8-958b-a2f5c3cdb339 [2024-01-18T20:34:22.738Z] 2024-01-18 20:34:22 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610048] creation - [2024-01-18T20:34:22.738Z] 2024-01-18 20:34:22 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610048] created [2024-01-18T20:34:22.738Z] 2024-01-18 20:34:22 +0000:[INFO] [{DeleteUsersRole}] Adding role default-roles-master to user testautouser1705610048 [2024-01-18T20:34:25.256Z] INFO[2024-01-18 20:34:24] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T20:34:25.256Z] INFO[2024-01-18 20:34:24] sched stats: tick interval: 18024 < 1.2s (100%) [2024-01-18T20:34:25.256Z] INFO[2024-01-18 20:34:24] sched stats: tick process durations: 18023 < 500ms (100%) [2024-01-18T20:34:25.256Z] 2024-01-18 20:34:24 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610040 - d3ce505b-ed10-4750-82ad-3a8a689612da [2024-01-18T20:34:25.512Z] 2024-01-18 20:34:25 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610050] creation - [2024-01-18T20:34:25.512Z] 2024-01-18 20:34:25 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610050] created [2024-01-18T20:34:25.512Z] 2024-01-18 20:34:25 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610050 [2024-01-18T20:34:26.438Z] I0118 20:34:26.353883 14 request.go:682] Waited for 3.388483184s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/services [2024-01-18T20:34:27.802Z] 2024-01-18 20:34:27 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610052] creation - [2024-01-18T20:34:27.803Z] 2024-01-18 20:34:27 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610052] created [2024-01-18T20:34:27.803Z] 2024-01-18 20:34:27 +0000:[INFO] [{DeleteUsersRole}] Adding role default-roles-master to user testautouser1705610052 [2024-01-18T20:34:28.060Z] 2024-01-18 20:34:27 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610042 - 025770b5-cd44-4ed6-a764-53655a42d7bd [2024-01-18T20:34:30.576Z] 2024-01-18 20:34:30 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610054] creation - [2024-01-18T20:34:30.576Z] 2024-01-18 20:34:30 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610054] created [2024-01-18T20:34:30.576Z] 2024-01-18 20:34:30 +0000:[INFO] [{DeleteUsersRole}] Adding role default-roles-master to user testautouser1705610054 [2024-01-18T20:34:31.137Z] 2024-01-18 20:34:31 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610044 - 7977b2f1-3558-4162-b756-4310234b51aa [2024-01-18T20:34:33.656Z] 2024-01-18 20:34:33 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610056] creation - [2024-01-18T20:34:33.656Z] 2024-01-18 20:34:33 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610056] created [2024-01-18T20:34:33.657Z] 2024-01-18 20:34:33 +0000:[INFO] [{DeleteUsersRole}] Adding role default-roles-master to user testautouser1705610056 [2024-01-18T20:34:34.222Z] 2024-01-18 20:34:33 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610046 - 3161dd97-111e-4e0e-9f3f-58210644d6d1 [2024-01-18T20:34:35.588Z] 2024-01-18 20:34:35 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610058] creation - [2024-01-18T20:34:35.588Z] 2024-01-18 20:34:35 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610058] created [2024-01-18T20:34:35.588Z] 2024-01-18 20:34:35 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610058 [2024-01-18T20:34:36.952Z] I0118 20:34:36.543047 14 request.go:682] Waited for 3.956299939s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-18T20:34:37.516Z] 2024-01-18 20:34:37 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610048 - dde214e6-fd13-48dd-9daf-78e748ebd986 [2024-01-18T20:34:37.776Z] 2024-01-18 20:34:37 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610060] creation - [2024-01-18T20:34:37.776Z] 2024-01-18 20:34:37 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610060] created [2024-01-18T20:34:37.776Z] 2024-01-18 20:34:37 +0000:[INFO] [{DeleteUsersRole}] Adding role default-roles-master to user testautouser1705610060 [2024-01-18T20:34:40.327Z] 2024-01-18 20:34:40 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610062] creation - [2024-01-18T20:34:40.327Z] 2024-01-18 20:34:40 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610062] created [2024-01-18T20:34:40.327Z] 2024-01-18 20:34:40 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610062 [2024-01-18T20:34:41.707Z] 2024-01-18 20:34:41 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610050 - fee7a61a-629a-4226-8cc9-f492a615fcf3 [2024-01-18T20:34:43.072Z] 2024-01-18 20:34:42 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610064] creation - [2024-01-18T20:34:43.072Z] 2024-01-18 20:34:42 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610064] created [2024-01-18T20:34:43.072Z] 2024-01-18 20:34:42 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610064 [2024-01-18T20:34:44.436Z] 2024-01-18 20:34:43 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610052 - dc9577da-177e-4d7d-a282-38db3d1d912e [2024-01-18T20:34:45.801Z] 2024-01-18 20:34:45 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610066] creation - [2024-01-18T20:34:45.801Z] 2024-01-18 20:34:45 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610066] created [2024-01-18T20:34:45.801Z] 2024-01-18 20:34:45 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610066 [2024-01-18T20:34:46.731Z] I0118 20:34:46.543462 14 request.go:682] Waited for 4.558749056s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-18T20:34:46.987Z] 2024-01-18 20:34:46 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610054 - 916ea4a6-ede1-4c37-89fe-3c788488ae3f [2024-01-18T20:34:47.915Z] 2024-01-18 20:34:47 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610068] creation - [2024-01-18T20:34:47.915Z] 2024-01-18 20:34:47 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610068] created [2024-01-18T20:34:47.915Z] 2024-01-18 20:34:47 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610068 [2024-01-18T20:34:51.180Z] 2024-01-18 20:34:50 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610070] creation - [2024-01-18T20:34:51.180Z] 2024-01-18 20:34:50 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610070] created [2024-01-18T20:34:51.180Z] 2024-01-18 20:34:50 +0000:[INFO] [{DeleteUsersRole}] Adding role default-roles-master to user testautouser1705610070 [2024-01-18T20:34:51.180Z] 2024-01-18 20:34:51 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610056 - 49f2c80a-ce57-44ac-a3e3-e7030e5a467e [2024-01-18T20:34:53.694Z] 2024-01-18 20:34:53 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610072] creation - [2024-01-18T20:34:53.694Z] 2024-01-18 20:34:53 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610072] created [2024-01-18T20:34:53.694Z] 2024-01-18 20:34:53 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610072 [2024-01-18T20:34:53.694Z] 2024-01-18 20:34:53 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610058 - 16bde0ab-8f32-4dae-9f70-7c4ea3b5edeb [2024-01-18T20:34:55.581Z] 2024-01-18 20:34:55 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610074] creation - [2024-01-18T20:34:55.582Z] 2024-01-18 20:34:55 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610074] created [2024-01-18T20:34:55.582Z] 2024-01-18 20:34:55 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610074 [2024-01-18T20:34:56.947Z] 2024-01-18 20:34:56 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610060 - c368720b-b33b-409e-b031-b87377ae0121 [2024-01-18T20:34:56.947Z] I0118 20:34:56.742702 14 request.go:682] Waited for 5.163986962s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-18T20:34:58.311Z] 2024-01-18 20:34:58 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610076] creation - [2024-01-18T20:34:58.311Z] 2024-01-18 20:34:58 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610076] created [2024-01-18T20:34:58.311Z] 2024-01-18 20:34:58 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610076 [2024-01-18T20:34:59.675Z] 2024-01-18 20:34:59 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610062 - 1054cc3c-7917-4911-a986-622200496913 [2024-01-18T20:35:00.623Z] 2024-01-18 20:35:00 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610078] creation - [2024-01-18T20:35:00.623Z] 2024-01-18 20:35:00 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610078] created [2024-01-18T20:35:00.623Z] 2024-01-18 20:35:00 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610078 [2024-01-18T20:35:03.138Z] 2024-01-18 20:35:03 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610064 - 0e589b35-6b8e-4ff7-86b4-f7483b6b44bb [2024-01-18T20:35:03.396Z] 2024-01-18 20:35:03 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610080] creation - [2024-01-18T20:35:03.396Z] 2024-01-18 20:35:03 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610080] created [2024-01-18T20:35:03.396Z] 2024-01-18 20:35:03 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610080 [2024-01-18T20:35:05.918Z] 2024-01-18 20:35:05 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610082] creation - [2024-01-18T20:35:05.918Z] 2024-01-18 20:35:05 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610082] created [2024-01-18T20:35:05.918Z] 2024-01-18 20:35:05 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610082 [2024-01-18T20:35:06.481Z] 2024-01-18 20:35:06 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610066 - 83bd9819-043f-4466-8f05-1588d1d7a463 [2024-01-18T20:35:07.042Z] I0118 20:35:06.743041 14 request.go:682] Waited for 5.72674364s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/services [2024-01-18T20:35:08.410Z] 2024-01-18 20:35:08 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610084] creation - [2024-01-18T20:35:08.410Z] 2024-01-18 20:35:08 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610084] created [2024-01-18T20:35:08.410Z] 2024-01-18 20:35:08 +0000:[INFO] [{DeleteUsersRole}] Adding role default-roles-master to user testautouser1705610084 [2024-01-18T20:35:09.773Z] 2024-01-18 20:35:09 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610068 - de9bf072-d0af-427f-9e0c-5c9328a6cbed [2024-01-18T20:35:11.669Z] 2024-01-18 20:35:11 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610086] creation - [2024-01-18T20:35:11.669Z] 2024-01-18 20:35:11 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610086] created [2024-01-18T20:35:11.669Z] 2024-01-18 20:35:11 +0000:[INFO] [{DeleteUsersRole}] Adding role default-roles-master to user testautouser1705610086 [2024-01-18T20:35:13.557Z] 2024-01-18 20:35:13 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610070 - ff66ab06-f279-48ee-a22e-db6a50be452b [2024-01-18T20:35:13.557Z] 2024-01-18 20:35:13 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610088] creation - [2024-01-18T20:35:13.557Z] 2024-01-18 20:35:13 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610088] created [2024-01-18T20:35:13.557Z] 2024-01-18 20:35:13 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610088 [2024-01-18T20:35:16.821Z] 2024-01-18 20:35:16 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610090] creation - [2024-01-18T20:35:16.821Z] 2024-01-18 20:35:16 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610090] created [2024-01-18T20:35:16.821Z] 2024-01-18 20:35:16 +0000:[INFO] [{DeleteUsersRole}] Adding role default-roles-master to user testautouser1705610090 [2024-01-18T20:35:16.821Z] 2024-01-18 20:35:16 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610072 - a643dea8-aed3-4ab6-912b-7f4fa9d485d6 [2024-01-18T20:35:17.077Z] I0118 20:35:16.943218 14 request.go:682] Waited for 6.170852669s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/services [2024-01-18T20:35:18.443Z] 2024-01-18 20:35:18 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610092] creation - [2024-01-18T20:35:18.443Z] 2024-01-18 20:35:18 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610092] created [2024-01-18T20:35:18.443Z] 2024-01-18 20:35:18 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610092 [2024-01-18T20:35:19.378Z] 2024-01-18 20:35:18 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610074 - 877d30b8-8b5a-4b4d-8f7c-9c64ec6cca37 [2024-01-18T20:35:21.269Z] 2024-01-18 20:35:21 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610094] creation - [2024-01-18T20:35:21.269Z] 2024-01-18 20:35:21 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610094] created [2024-01-18T20:35:21.269Z] 2024-01-18 20:35:21 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610094 [2024-01-18T20:35:23.167Z] 2024-01-18 20:35:22 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610076 - fb6341a7-629a-437f-93ac-b3770e24820b [2024-01-18T20:35:23.423Z] 2024-01-18 20:35:23 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610096] creation - [2024-01-18T20:35:23.423Z] 2024-01-18 20:35:23 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610096] created [2024-01-18T20:35:23.423Z] 2024-01-18 20:35:23 +0000:[INFO] [{DeleteUsersRole}] Adding role default-roles-master to user testautouser1705610096 [2024-01-18T20:35:25.316Z] 2024-01-18 20:35:25 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610078 - 28750b2c-002f-4d4e-8dda-62436d29ac85 [2024-01-18T20:35:26.242Z] 2024-01-18 20:35:25 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610098] creation - [2024-01-18T20:35:26.242Z] 2024-01-18 20:35:25 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610098] created [2024-01-18T20:35:26.242Z] 2024-01-18 20:35:25 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610098 [2024-01-18T20:35:27.178Z] I0118 20:35:26.943274 14 request.go:682] Waited for 6.955251373s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-18T20:35:29.068Z] 2024-01-18 20:35:28 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610100] creation - [2024-01-18T20:35:29.068Z] 2024-01-18 20:35:28 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610100] created [2024-01-18T20:35:29.068Z] 2024-01-18 20:35:28 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610100 [2024-01-18T20:35:29.331Z] 2024-01-18 20:35:29 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610080 - f8998bc2-2979-4af0-8ebf-37e9555df970 [2024-01-18T20:35:31.876Z] 2024-01-18 20:35:31 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610102] creation - [2024-01-18T20:35:31.876Z] 2024-01-18 20:35:31 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610102] created [2024-01-18T20:35:31.876Z] 2024-01-18 20:35:31 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610102 [2024-01-18T20:35:32.442Z] 2024-01-18 20:35:32 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610082 - 52acc498-f1df-4642-b478-a947802d85a7 [2024-01-18T20:35:34.331Z] 2024-01-18 20:35:33 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610104] creation - [2024-01-18T20:35:34.331Z] 2024-01-18 20:35:33 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610104] created [2024-01-18T20:35:34.331Z] 2024-01-18 20:35:33 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610104 [2024-01-18T20:35:35.259Z] 2024-01-18 20:35:35 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610084 - 7e1b5ce9-b6e4-4cf6-9e1f-723066b86fe6 [2024-01-18T20:35:36.191Z] 2024-01-18 20:35:36 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610106] creation - [2024-01-18T20:35:36.191Z] 2024-01-18 20:35:36 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610106] created [2024-01-18T20:35:36.191Z] 2024-01-18 20:35:36 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610106 [2024-01-18T20:35:37.583Z] I0118 20:35:37.143727 14 request.go:682] Waited for 7.540437196s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/services [2024-01-18T20:35:38.956Z] 2024-01-18 20:35:38 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610108] creation - [2024-01-18T20:35:38.956Z] 2024-01-18 20:35:38 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610108] created [2024-01-18T20:35:38.956Z] 2024-01-18 20:35:38 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610108 [2024-01-18T20:35:39.517Z] 2024-01-18 20:35:39 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610086 - f654ad17-86b0-44b5-8531-ff4e2b3d0d8e [2024-01-18T20:35:42.053Z] 2024-01-18 20:35:41 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610110] creation - [2024-01-18T20:35:42.053Z] 2024-01-18 20:35:41 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610110] created [2024-01-18T20:35:42.053Z] 2024-01-18 20:35:41 +0000:[INFO] [{DeleteUsersRole}] Adding role default-roles-master to user testautouser1705610110 [2024-01-18T20:35:42.324Z] 2024-01-18 20:35:42 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610088 - 63b2f68b-4b2a-4bf0-b640-d28d0543ed46 [2024-01-18T20:35:44.214Z] 2024-01-18 20:35:43 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610112] creation - [2024-01-18T20:35:44.214Z] 2024-01-18 20:35:43 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610112] created [2024-01-18T20:35:44.214Z] 2024-01-18 20:35:43 +0000:[INFO] [{DeleteUsersRole}] Adding role default-roles-master to user testautouser1705610112 [2024-01-18T20:35:45.580Z] 2024-01-18 20:35:45 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610090 - 75e33520-f78f-4044-b520-6703df801db9 [2024-01-18T20:35:46.510Z] 2024-01-18 20:35:46 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610114] creation - [2024-01-18T20:35:46.510Z] 2024-01-18 20:35:46 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610114] created [2024-01-18T20:35:46.510Z] 2024-01-18 20:35:46 +0000:[INFO] [{DeleteUsersRole}] Adding role default-roles-master to user testautouser1705610114 [2024-01-18T20:35:47.438Z] I0118 20:35:47.342707 14 request.go:682] Waited for 8.126998017s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/services [2024-01-18T20:35:48.369Z] 2024-01-18 20:35:48 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610092 - a44ce439-7030-4050-a318-52347f806d6b [2024-01-18T20:35:49.299Z] 2024-01-18 20:35:49 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610116] creation - [2024-01-18T20:35:49.299Z] 2024-01-18 20:35:49 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610116] created [2024-01-18T20:35:49.299Z] 2024-01-18 20:35:49 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610116 [2024-01-18T20:35:51.819Z] 2024-01-18 20:35:51 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610118] creation - [2024-01-18T20:35:51.819Z] 2024-01-18 20:35:51 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610118] created [2024-01-18T20:35:51.819Z] 2024-01-18 20:35:51 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610118 [2024-01-18T20:35:52.076Z] 2024-01-18 20:35:51 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610094 - b23ab55c-d0c8-4cef-8d06-cb57dbc700ea [2024-01-18T20:35:54.610Z] 2024-01-18 20:35:54 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610120] creation - [2024-01-18T20:35:54.610Z] 2024-01-18 20:35:54 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610120] created [2024-01-18T20:35:54.610Z] 2024-01-18 20:35:54 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610120 [2024-01-18T20:35:54.865Z] 2024-01-18 20:35:54 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610096 - afcf41e9-a71f-47d2-921c-b6938918f311 [2024-01-18T20:35:56.754Z] 2024-01-18 20:35:56 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610122] creation - [2024-01-18T20:35:56.754Z] 2024-01-18 20:35:56 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610122] created [2024-01-18T20:35:56.754Z] 2024-01-18 20:35:56 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610122 [2024-01-18T20:35:57.688Z] I0118 20:35:57.543015 14 request.go:682] Waited for 8.757611659s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/services [2024-01-18T20:35:58.249Z] 2024-01-18 20:35:58 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610098 - 1c5a50cc-50fe-497f-b1df-cbcbf250af9e [2024-01-18T20:35:59.615Z] 2024-01-18 20:35:59 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610124] creation - [2024-01-18T20:35:59.615Z] 2024-01-18 20:35:59 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610124] created [2024-01-18T20:35:59.615Z] 2024-01-18 20:35:59 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610124 [2024-01-18T20:36:01.515Z] 2024-01-18 20:36:01 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610126] creation - [2024-01-18T20:36:01.515Z] 2024-01-18 20:36:01 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610126] created [2024-01-18T20:36:01.515Z] 2024-01-18 20:36:01 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610126 [2024-01-18T20:36:01.515Z] 2024-01-18 20:36:01 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610100 - 86a2a183-4d11-48ed-9a75-7567203642e2 [2024-01-18T20:36:04.791Z] 2024-01-18 20:36:04 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610128] creation - [2024-01-18T20:36:04.791Z] 2024-01-18 20:36:04 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610128] created [2024-01-18T20:36:04.791Z] 2024-01-18 20:36:04 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610128 [2024-01-18T20:36:05.053Z] 2024-01-18 20:36:05 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610102 - ce28004a-64b3-4088-bfd8-42a5e0014c08 [2024-01-18T20:36:06.945Z] 2024-01-18 20:36:06 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610130] creation - [2024-01-18T20:36:06.945Z] 2024-01-18 20:36:06 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610130] created [2024-01-18T20:36:06.945Z] 2024-01-18 20:36:06 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610130 [2024-01-18T20:36:07.876Z] I0118 20:36:07.543555 14 request.go:682] Waited for 9.157382184s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-18T20:36:08.446Z] 2024-01-18 20:36:08 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610104 - 901ad539-24a5-4a25-8daa-2d47e6bd68c3 [2024-01-18T20:36:09.377Z] 2024-01-18 20:36:09 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610132] creation - [2024-01-18T20:36:09.377Z] 2024-01-18 20:36:09 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610132] created [2024-01-18T20:36:09.377Z] 2024-01-18 20:36:09 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610132 [2024-01-18T20:36:10.747Z] 2024-01-18 20:36:10 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610106 - b5943e13-aa12-462c-b394-a1e848dc62ce [2024-01-18T20:36:11.679Z] 2024-01-18 20:36:11 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610134] creation - [2024-01-18T20:36:11.679Z] 2024-01-18 20:36:11 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610134] created [2024-01-18T20:36:11.679Z] 2024-01-18 20:36:11 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610134 [2024-01-18T20:36:14.949Z] 2024-01-18 20:36:14 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610136] creation - [2024-01-18T20:36:14.949Z] 2024-01-18 20:36:14 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610136] created [2024-01-18T20:36:14.949Z] 2024-01-18 20:36:14 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610136 [2024-01-18T20:36:14.949Z] 2024-01-18 20:36:14 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610108 - 7a591a81-146b-4d14-b89e-d3412b6f1370 [2024-01-18T20:36:16.849Z] 2024-01-18 20:36:16 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610138] creation - [2024-01-18T20:36:16.849Z] 2024-01-18 20:36:16 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610138] created [2024-01-18T20:36:16.849Z] 2024-01-18 20:36:16 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610138 [2024-01-18T20:36:17.413Z] 2024-01-18 20:36:17 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610110 - 1a809db7-d012-4542-b732-fc3839876092 [2024-01-18T20:36:17.981Z] I0118 20:36:17.742929 14 request.go:682] Waited for 9.540202968s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/services [2024-01-18T20:36:19.345Z] 2024-01-18 20:36:19 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610140] creation - [2024-01-18T20:36:19.345Z] 2024-01-18 20:36:19 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610140] created [2024-01-18T20:36:19.345Z] 2024-01-18 20:36:19 +0000:[INFO] [{DeleteUsersRole}] Adding role default-roles-master to user testautouser1705610140 [2024-01-18T20:36:20.274Z] 2024-01-18 20:36:20 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610112 - 53b39063-19e1-40a4-9693-8ddfe0db231b [2024-01-18T20:36:21.640Z] 2024-01-18 20:36:21 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610142] creation - [2024-01-18T20:36:21.640Z] 2024-01-18 20:36:21 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610142] created [2024-01-18T20:36:21.640Z] 2024-01-18 20:36:21 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610142 [2024-01-18T20:36:23.005Z] 2024-01-18 20:36:22 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610114 - 940dc0e4-2d22-4490-8925-91cc61ec31e9 [2024-01-18T20:36:23.567Z] 2024-01-18 20:36:23 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610144] creation - [2024-01-18T20:36:23.568Z] 2024-01-18 20:36:23 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610144] created [2024-01-18T20:36:23.568Z] 2024-01-18 20:36:23 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610144 [2024-01-18T20:36:26.086Z] 2024-01-18 20:36:25 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610146] creation - [2024-01-18T20:36:26.086Z] 2024-01-18 20:36:25 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610146] created [2024-01-18T20:36:26.086Z] 2024-01-18 20:36:25 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610146 [2024-01-18T20:36:26.343Z] 2024-01-18 20:36:26 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610116 - 7b0e6957-5a2a-4058-af4e-eea69261a3ea [2024-01-18T20:36:28.233Z] I0118 20:36:27.743119 14 request.go:682] Waited for 9.157029238s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-18T20:36:28.233Z] 2024-01-18 20:36:27 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610148] creation - [2024-01-18T20:36:28.233Z] 2024-01-18 20:36:27 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610148] created [2024-01-18T20:36:28.234Z] 2024-01-18 20:36:27 +0000:[INFO] [{DeleteUsersRole}] Adding role default-roles-master to user testautouser1705610148 [2024-01-18T20:36:28.489Z] 2024-01-18 20:36:28 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610118 - adf61788-400d-4b0d-9d25-0ed58cf00e61 [2024-01-18T20:36:30.376Z] 2024-01-18 20:36:30 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610150] creation - [2024-01-18T20:36:30.376Z] 2024-01-18 20:36:30 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610150] created [2024-01-18T20:36:30.376Z] 2024-01-18 20:36:30 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610150 [2024-01-18T20:36:31.302Z] 2024-01-18 20:36:31 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610120 - b7cd24fd-cdb4-493b-b45f-d8130b181007 [2024-01-18T20:36:32.231Z] 2024-01-18 20:36:32 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610152] creation - [2024-01-18T20:36:32.231Z] 2024-01-18 20:36:32 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610152] created [2024-01-18T20:36:32.231Z] 2024-01-18 20:36:32 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610152 [2024-01-18T20:36:34.119Z] 2024-01-18 20:36:33 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610122 - dbff47c3-0d0e-4011-8620-fc6b5336834a [2024-01-18T20:36:34.119Z] 2024-01-18 20:36:34 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610154] creation - [2024-01-18T20:36:34.119Z] 2024-01-18 20:36:34 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610154] created [2024-01-18T20:36:34.119Z] 2024-01-18 20:36:34 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610154 [2024-01-18T20:36:36.005Z] 2024-01-18 20:36:35 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610156] creation - [2024-01-18T20:36:36.005Z] 2024-01-18 20:36:35 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610156] created [2024-01-18T20:36:36.005Z] 2024-01-18 20:36:35 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-app.admin to user testautouser1705610156 [2024-01-18T20:36:37.865Z] 2024-01-18 20:36:36 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610124 - 30c2883a-9b23-451c-a9fe-4473e94c8371 [2024-01-18T20:36:38.124Z] 2024-01-18 20:36:37 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610158] creation - [2024-01-18T20:36:38.124Z] 2024-01-18 20:36:37 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610158] created [2024-01-18T20:36:38.124Z] 2024-01-18 20:36:37 +0000:[INFO] [{DeleteUsersRole}] Adding role default-roles-master to user testautouser1705610158 [2024-01-18T20:36:38.124Z] I0118 20:36:37.943606 14 request.go:682] Waited for 8.749340057s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-18T20:36:38.688Z] 2024-01-18 20:36:38 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610126 - 26113b48-5de6-4287-b996-d79c8b3d3f2c [2024-01-18T20:36:40.050Z] 2024-01-18 20:36:39 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610160] creation - [2024-01-18T20:36:40.051Z] 2024-01-18 20:36:39 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610160] created [2024-01-18T20:36:40.051Z] 2024-01-18 20:36:39 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610160 [2024-01-18T20:36:40.981Z] 2024-01-18 20:36:40 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610128 - e0c2beaf-bb4b-4438-b618-3e34c39c5340 [2024-01-18T20:36:41.909Z] 2024-01-18 20:36:41 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#728] - Response for user [testautouser1705610162] creation - [2024-01-18T20:36:41.909Z] 2024-01-18 20:36:41 +0000:[INFO] [{DeleteUsersRole}] [backup.AddUser:#733] - User [testautouser1705610162] created [2024-01-18T20:36:41.909Z] 2024-01-18 20:36:41 +0000:[INFO] [{DeleteUsersRole}] Adding role px-backup-infra.admin to user testautouser1705610162 [2024-01-18T20:36:43.272Z] 2024-01-18 20:36:43 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610130 - 917d6368-07a2-4e0b-8a2d-fba8ff42324d [2024-01-18T20:36:45.788Z] 2024-01-18 20:36:45 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610132 - 8efaf076-1d80-4b56-a7be-7e57a5929a98 [2024-01-18T20:36:47.682Z] 2024-01-18 20:36:47 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610134 - 22af8231-a765-4a7c-b55f-a221d2db46a8 [2024-01-18T20:36:48.243Z] I0118 20:36:48.142661 14 request.go:682] Waited for 8.152886719s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-18T20:36:49.614Z] 2024-01-18 20:36:49 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610136 - 7ba4083a-8907-440c-a286-54606b1f2c19 [2024-01-18T20:36:51.501Z] 2024-01-18 20:36:51 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610138 - 55984835-105f-4f03-9f75-ae4343ad20bf [2024-01-18T20:36:54.019Z] 2024-01-18 20:36:53 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610140 - 4d6b2481-e9c8-4fec-9559-da3d377fb702 [2024-01-18T20:36:55.907Z] 2024-01-18 20:36:55 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610142 - cbe78ed9-5980-4870-8b97-2fe3e9a7fde7 [2024-01-18T20:36:57.270Z] 2024-01-18 20:36:56 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610144 - d0c9cadc-16bd-4019-9805-8f84365d646b [2024-01-18T20:36:58.200Z] I0118 20:36:58.142718 14 request.go:682] Waited for 7.52827765s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/services [2024-01-18T20:36:59.125Z] 2024-01-18 20:36:58 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610146 - 1fa99470-8321-4701-bdfe-5ce6c90d70c3 [2024-01-18T20:37:00.488Z] 2024-01-18 20:37:00 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610148 - fd5df611-7f40-4420-b893-841433098e63 [2024-01-18T20:37:03.002Z] 2024-01-18 20:37:02 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610150 - 793d0d8f-e562-42e0-9417-12db14b278e7 [2024-01-18T20:37:03.930Z] 2024-01-18 20:37:03 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610152 - 3f864853-3b52-4c43-bf7e-6a98a067e03f [2024-01-18T20:37:05.826Z] 2024-01-18 20:37:05 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610154 - e69a5d42-cc97-47b3-9d6a-3f1464578131 [2024-01-18T20:37:07.192Z] 2024-01-18 20:37:06 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610156 - 3477536c-ee63-4f69-b374-34452710f43b [2024-01-18T20:37:08.557Z] I0118 20:37:08.143365 14 request.go:682] Waited for 6.754121452s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-18T20:37:08.557Z] 2024-01-18 20:37:08 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610158 - 27657394-487e-4705-a61e-00122cda400b [2024-01-18T20:37:09.927Z] 2024-01-18 20:37:09 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610160 - 11434e64-4fc8-4485-9e1f-adfe328c0df6 [2024-01-18T20:37:11.822Z] 2024-01-18 20:37:11 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610162 - 633ecb26-7233-4da3-9e45-62fac67885d1 [2024-01-18T20:37:18.355Z] I0118 20:37:18.143639 14 request.go:682] Waited for 5.97600211s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/services [2024-01-18T20:37:30.523Z] I0118 20:37:28.343340 14 request.go:682] Waited for 4.777545848s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/services [2024-01-18T20:37:38.606Z] I0118 20:37:38.343636 14 request.go:682] Waited for 2.554715529s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-18T20:37:41.122Z] 2024-01-18 20:37:40 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610012] [2024-01-18T20:37:42.054Z] 2024-01-18 20:37:41 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610012 - 84b1f4be-bffc-4566-8592-4faeb1e00fd9 [2024-01-18T20:37:43.420Z] INFO[2024-01-18 20:37:43] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610012] [2024-01-18T20:37:43.420Z] INFO[2024-01-18 20:37:43] Actual:true, Expected: true [2024-01-18T20:37:43.420Z] 2024-01-18 20:37:43 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [default-roles-master] from the user : [testautouser1705610026] [2024-01-18T20:37:44.360Z] 2024-01-18 20:37:43 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610026 - a0235953-6fbc-4765-bf99-d076797582a4 [2024-01-18T20:37:45.726Z] INFO[2024-01-18 20:37:45] Verifying : Description : Removing role [default-roles-master] from the user [testautouser1705610026] [2024-01-18T20:37:45.726Z] INFO[2024-01-18 20:37:45] Actual:true, Expected: true [2024-01-18T20:37:45.726Z] 2024-01-18 20:37:45 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610036] [2024-01-18T20:37:46.666Z] 2024-01-18 20:37:46 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610036 - 2715aeee-1103-42ed-8c0e-dbbe725053c9 [2024-01-18T20:37:48.558Z] INFO[2024-01-18 20:37:48] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610036] [2024-01-18T20:37:48.558Z] INFO[2024-01-18 20:37:48] Actual:true, Expected: true [2024-01-18T20:37:48.558Z] 2024-01-18 20:37:48 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610050] [2024-01-18T20:37:49.119Z] 2024-01-18 20:37:48 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610050 - fee7a61a-629a-4226-8cc9-f492a615fcf3 [2024-01-18T20:37:50.484Z] INFO[2024-01-18 20:37:50] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610050] [2024-01-18T20:37:50.484Z] INFO[2024-01-18 20:37:50] Actual:true, Expected: true [2024-01-18T20:37:50.484Z] 2024-01-18 20:37:50 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [default-roles-master] from the user : [testautouser1705610140] [2024-01-18T20:37:51.413Z] 2024-01-18 20:37:51 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610140 - 4d6b2481-e9c8-4fec-9559-da3d377fb702 [2024-01-18T20:37:53.311Z] INFO[2024-01-18 20:37:52] Verifying : Description : Removing role [default-roles-master] from the user [testautouser1705610140] [2024-01-18T20:37:53.311Z] INFO[2024-01-18 20:37:52] Actual:true, Expected: true [2024-01-18T20:37:53.311Z] 2024-01-18 20:37:52 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610044] [2024-01-18T20:37:53.873Z] 2024-01-18 20:37:53 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610044 - 7977b2f1-3558-4162-b756-4310234b51aa [2024-01-18T20:37:55.759Z] INFO[2024-01-18 20:37:55] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610044] [2024-01-18T20:37:55.759Z] INFO[2024-01-18 20:37:55] Actual:true, Expected: true [2024-01-18T20:37:55.759Z] 2024-01-18 20:37:55 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610058] [2024-01-18T20:37:56.320Z] 2024-01-18 20:37:55 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610058 - 16bde0ab-8f32-4dae-9f70-7c4ea3b5edeb [2024-01-18T20:37:57.687Z] INFO[2024-01-18 20:37:57] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610058] [2024-01-18T20:37:57.687Z] INFO[2024-01-18 20:37:57] Actual:true, Expected: true [2024-01-18T20:37:57.687Z] 2024-01-18 20:37:57 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610066] [2024-01-18T20:37:58.620Z] 2024-01-18 20:37:58 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610066 - 83bd9819-043f-4466-8f05-1588d1d7a463 [2024-01-18T20:38:00.504Z] INFO[2024-01-18 20:38:00] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610066] [2024-01-18T20:38:00.504Z] INFO[2024-01-18 20:38:00] Actual:true, Expected: true [2024-01-18T20:38:00.504Z] 2024-01-18 20:38:00 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [default-roles-master] from the user : [testautouser1705610084] [2024-01-18T20:38:01.066Z] 2024-01-18 20:38:00 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610084 - 7e1b5ce9-b6e4-4cf6-9e1f-723066b86fe6 [2024-01-18T20:38:02.467Z] INFO[2024-01-18 20:38:02] Verifying : Description : Removing role [default-roles-master] from the user [testautouser1705610084] [2024-01-18T20:38:02.467Z] INFO[2024-01-18 20:38:02] Actual:true, Expected: true [2024-01-18T20:38:02.467Z] 2024-01-18 20:38:02 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610118] [2024-01-18T20:38:03.397Z] 2024-01-18 20:38:03 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610118 - adf61788-400d-4b0d-9d25-0ed58cf00e61 [2024-01-18T20:38:05.310Z] INFO[2024-01-18 20:38:04] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610118] [2024-01-18T20:38:05.310Z] INFO[2024-01-18 20:38:04] Actual:true, Expected: true [2024-01-18T20:38:05.310Z] 2024-01-18 20:38:04 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [default-roles-master] from the user : [testautouser1705610024] [2024-01-18T20:38:05.873Z] 2024-01-18 20:38:05 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610024 - 878b64e4-b6cd-464c-888e-e6ffc9769335 [2024-01-18T20:38:07.765Z] INFO[2024-01-18 20:38:07] Verifying : Description : Removing role [default-roles-master] from the user [testautouser1705610024] [2024-01-18T20:38:07.765Z] INFO[2024-01-18 20:38:07] Actual:true, Expected: true [2024-01-18T20:38:07.765Z] 2024-01-18 20:38:07 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610072] [2024-01-18T20:38:08.021Z] 2024-01-18 20:38:07 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610072 - a643dea8-aed3-4ab6-912b-7f4fa9d485d6 [2024-01-18T20:38:09.908Z] INFO[2024-01-18 20:38:09] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610072] [2024-01-18T20:38:09.908Z] INFO[2024-01-18 20:38:09] Actual:true, Expected: true [2024-01-18T20:38:09.909Z] 2024-01-18 20:38:09 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610078] [2024-01-18T20:38:10.470Z] 2024-01-18 20:38:10 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610078 - 28750b2c-002f-4d4e-8dda-62436d29ac85 [2024-01-18T20:38:12.359Z] INFO[2024-01-18 20:38:12] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610078] [2024-01-18T20:38:12.359Z] INFO[2024-01-18 20:38:12] Actual:true, Expected: true [2024-01-18T20:38:12.359Z] 2024-01-18 20:38:12 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610092] [2024-01-18T20:38:12.925Z] 2024-01-18 20:38:12 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610092 - a44ce439-7030-4050-a318-52347f806d6b [2024-01-18T20:38:14.816Z] INFO[2024-01-18 20:38:14] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610092] [2024-01-18T20:38:14.816Z] INFO[2024-01-18 20:38:14] Actual:true, Expected: true [2024-01-18T20:38:14.816Z] 2024-01-18 20:38:14 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610102] [2024-01-18T20:38:15.381Z] 2024-01-18 20:38:15 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610102 - ce28004a-64b3-4088-bfd8-42a5e0014c08 [2024-01-18T20:38:17.273Z] INFO[2024-01-18 20:38:16] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610102] [2024-01-18T20:38:17.273Z] INFO[2024-01-18 20:38:16] Actual:true, Expected: true [2024-01-18T20:38:17.273Z] 2024-01-18 20:38:16 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610132] [2024-01-18T20:38:17.834Z] 2024-01-18 20:38:17 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610132 - 8efaf076-1d80-4b56-a7be-7e57a5929a98 [2024-01-18T20:38:19.727Z] INFO[2024-01-18 20:38:19] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610132] [2024-01-18T20:38:19.727Z] INFO[2024-01-18 20:38:19] Actual:true, Expected: true [2024-01-18T20:38:19.727Z] 2024-01-18 20:38:19 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610160] [2024-01-18T20:38:20.291Z] 2024-01-18 20:38:19 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610160 - 11434e64-4fc8-4485-9e1f-adfe328c0df6 [2024-01-18T20:38:22.185Z] INFO[2024-01-18 20:38:21] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610160] [2024-01-18T20:38:22.185Z] INFO[2024-01-18 20:38:21] Actual:true, Expected: true [2024-01-18T20:38:22.185Z] 2024-01-18 20:38:21 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610032] [2024-01-18T20:38:22.441Z] 2024-01-18 20:38:22 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610032 - c434d6dd-e59a-47b7-b2e6-ff0cafe49b99 [2024-01-18T20:38:24.331Z] INFO[2024-01-18 20:38:24] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610032] [2024-01-18T20:38:24.331Z] INFO[2024-01-18 20:38:24] Actual:true, Expected: true [2024-01-18T20:38:24.331Z] 2024-01-18 20:38:24 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [default-roles-master] from the user : [testautouser1705610054] [2024-01-18T20:38:24.892Z] 2024-01-18 20:38:24 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610054 - 916ea4a6-ede1-4c37-89fe-3c788488ae3f [2024-01-18T20:38:26.780Z] INFO[2024-01-18 20:38:26] Verifying : Description : Removing role [default-roles-master] from the user [testautouser1705610054] [2024-01-18T20:38:26.780Z] INFO[2024-01-18 20:38:26] Actual:true, Expected: true [2024-01-18T20:38:26.780Z] 2024-01-18 20:38:26 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610108] [2024-01-18T20:38:27.340Z] 2024-01-18 20:38:27 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610108 - 7a591a81-146b-4d14-b89e-d3412b6f1370 [2024-01-18T20:38:29.226Z] INFO[2024-01-18 20:38:28] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610108] [2024-01-18T20:38:29.226Z] INFO[2024-01-18 20:38:28] Actual:true, Expected: true [2024-01-18T20:38:29.226Z] 2024-01-18 20:38:28 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [default-roles-master] from the user : [testautouser1705610114] [2024-01-18T20:38:29.786Z] 2024-01-18 20:38:29 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610114 - 940dc0e4-2d22-4490-8925-91cc61ec31e9 [2024-01-18T20:38:31.672Z] INFO[2024-01-18 20:38:31] Verifying : Description : Removing role [default-roles-master] from the user [testautouser1705610114] [2024-01-18T20:38:31.672Z] INFO[2024-01-18 20:38:31] Actual:true, Expected: true [2024-01-18T20:38:31.672Z] 2024-01-18 20:38:31 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610124] [2024-01-18T20:38:32.233Z] 2024-01-18 20:38:32 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610124 - 30c2883a-9b23-451c-a9fe-4473e94c8371 [2024-01-18T20:38:34.120Z] INFO[2024-01-18 20:38:33] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610124] [2024-01-18T20:38:34.120Z] INFO[2024-01-18 20:38:33] Actual:true, Expected: true [2024-01-18T20:38:34.120Z] 2024-01-18 20:38:33 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610156] [2024-01-18T20:38:34.681Z] 2024-01-18 20:38:34 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610156 - 3477536c-ee63-4f69-b374-34452710f43b [2024-01-18T20:38:36.568Z] INFO[2024-01-18 20:38:36] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610156] [2024-01-18T20:38:36.568Z] INFO[2024-01-18 20:38:36] Actual:true, Expected: true [2024-01-18T20:38:36.568Z] 2024-01-18 20:38:36 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610136] [2024-01-18T20:38:37.130Z] 2024-01-18 20:38:36 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610136 - 7ba4083a-8907-440c-a286-54606b1f2c19 [2024-01-18T20:38:38.493Z] INFO[2024-01-18 20:38:38] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610136] [2024-01-18T20:38:38.493Z] INFO[2024-01-18 20:38:38] Actual:true, Expected: true [2024-01-18T20:38:38.493Z] 2024-01-18 20:38:38 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610004] [2024-01-18T20:38:39.450Z] 2024-01-18 20:38:39 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610004 - 5c93ae4c-682e-4b77-92e8-36dcf764a6ab [2024-01-18T20:38:41.340Z] INFO[2024-01-18 20:38:40] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610004] [2024-01-18T20:38:41.340Z] INFO[2024-01-18 20:38:40] Actual:true, Expected: true [2024-01-18T20:38:41.340Z] 2024-01-18 20:38:40 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610022] [2024-01-18T20:38:41.899Z] 2024-01-18 20:38:41 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610022 - c3b79415-4826-451a-be4d-707b5011e3f7 [2024-01-18T20:38:43.787Z] INFO[2024-01-18 20:38:43] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610022] [2024-01-18T20:38:43.787Z] INFO[2024-01-18 20:38:43] Actual:true, Expected: true [2024-01-18T20:38:43.787Z] 2024-01-18 20:38:43 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610064] [2024-01-18T20:38:44.044Z] 2024-01-18 20:38:44 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610064 - 0e589b35-6b8e-4ff7-86b4-f7483b6b44bb [2024-01-18T20:38:45.935Z] INFO[2024-01-18 20:38:45] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610064] [2024-01-18T20:38:45.935Z] INFO[2024-01-18 20:38:45] Actual:true, Expected: true [2024-01-18T20:38:45.935Z] 2024-01-18 20:38:45 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610076] [2024-01-18T20:38:46.501Z] 2024-01-18 20:38:46 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610076 - fb6341a7-629a-437f-93ac-b3770e24820b [2024-01-18T20:38:48.391Z] INFO[2024-01-18 20:38:48] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610076] [2024-01-18T20:38:48.391Z] INFO[2024-01-18 20:38:48] Actual:true, Expected: true [2024-01-18T20:38:48.391Z] 2024-01-18 20:38:48 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610104] [2024-01-18T20:38:48.953Z] 2024-01-18 20:38:48 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610104 - 901ad539-24a5-4a25-8daa-2d47e6bd68c3 [2024-01-18T20:38:50.844Z] INFO[2024-01-18 20:38:50] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610104] [2024-01-18T20:38:50.844Z] INFO[2024-01-18 20:38:50] Actual:true, Expected: true [2024-01-18T20:38:50.844Z] 2024-01-18 20:38:50 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610130] [2024-01-18T20:38:51.405Z] 2024-01-18 20:38:51 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610130 - 917d6368-07a2-4e0b-8a2d-fba8ff42324d [2024-01-18T20:38:53.291Z] INFO[2024-01-18 20:38:52] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610130] [2024-01-18T20:38:53.291Z] INFO[2024-01-18 20:38:52] Actual:true, Expected: true [2024-01-18T20:38:53.291Z] 2024-01-18 20:38:52 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [default-roles-master] from the user : [testautouser1705610158] [2024-01-18T20:38:53.852Z] 2024-01-18 20:38:53 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610158 - 27657394-487e-4705-a61e-00122cda400b [2024-01-18T20:38:55.739Z] INFO[2024-01-18 20:38:55] Verifying : Description : Removing role [default-roles-master] from the user [testautouser1705610158] [2024-01-18T20:38:55.739Z] INFO[2024-01-18 20:38:55] Actual:true, Expected: true [2024-01-18T20:38:55.739Z] 2024-01-18 20:38:55 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [default-roles-master] from the user : [testautouser1705610008] [2024-01-18T20:38:56.299Z] 2024-01-18 20:38:56 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610008 - fd9cfa2f-6ba4-40a1-a0f1-5c111aa0a5e8 [2024-01-18T20:38:57.664Z] INFO[2024-01-18 20:38:57] Verifying : Description : Removing role [default-roles-master] from the user [testautouser1705610008] [2024-01-18T20:38:57.664Z] INFO[2024-01-18 20:38:57] Actual:true, Expected: true [2024-01-18T20:38:57.664Z] 2024-01-18 20:38:57 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [default-roles-master] from the user : [testautouser1705610020] [2024-01-18T20:38:58.593Z] 2024-01-18 20:38:58 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610020 - 3d974540-f2fb-4df5-becb-1b26936765f6 [2024-01-18T20:39:00.485Z] INFO[2024-01-18 20:39:00] Verifying : Description : Removing role [default-roles-master] from the user [testautouser1705610020] [2024-01-18T20:39:00.485Z] INFO[2024-01-18 20:39:00] Actual:true, Expected: true [2024-01-18T20:39:00.485Z] 2024-01-18 20:39:00 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610038] [2024-01-18T20:39:01.046Z] 2024-01-18 20:39:00 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610038 - 826f6706-3c7e-4be8-958b-a2f5c3cdb339 [2024-01-18T20:39:02.937Z] INFO[2024-01-18 20:39:02] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610038] [2024-01-18T20:39:02.937Z] INFO[2024-01-18 20:39:02] Actual:true, Expected: true [2024-01-18T20:39:02.937Z] 2024-01-18 20:39:02 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610088] [2024-01-18T20:39:03.497Z] 2024-01-18 20:39:03 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610088 - 63b2f68b-4b2a-4bf0-b640-d28d0543ed46 [2024-01-18T20:39:04.862Z] INFO[2024-01-18 20:39:04] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610088] [2024-01-18T20:39:04.862Z] INFO[2024-01-18 20:39:04] Actual:true, Expected: true [2024-01-18T20:39:04.862Z] 2024-01-18 20:39:04 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [default-roles-master] from the user : [testautouser1705610090] [2024-01-18T20:39:05.792Z] 2024-01-18 20:39:05 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610090 - 75e33520-f78f-4044-b520-6703df801db9 [2024-01-18T20:39:07.713Z] INFO[2024-01-18 20:39:07] Verifying : Description : Removing role [default-roles-master] from the user [testautouser1705610090] [2024-01-18T20:39:07.713Z] INFO[2024-01-18 20:39:07] Actual:true, Expected: true [2024-01-18T20:39:07.713Z] 2024-01-18 20:39:07 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610094] [2024-01-18T20:39:08.274Z] 2024-01-18 20:39:08 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610094 - b23ab55c-d0c8-4cef-8d06-cb57dbc700ea [2024-01-18T20:39:10.165Z] INFO[2024-01-18 20:39:09] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610094] [2024-01-18T20:39:10.165Z] INFO[2024-01-18 20:39:09] Actual:true, Expected: true [2024-01-18T20:39:10.165Z] 2024-01-18 20:39:09 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [default-roles-master] from the user : [testautouser1705610030] [2024-01-18T20:39:10.729Z] 2024-01-18 20:39:10 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610030 - 3125668c-dc81-4c20-bfcb-71a66c9f789b [2024-01-18T20:39:12.093Z] INFO[2024-01-18 20:39:12] Verifying : Description : Removing role [default-roles-master] from the user [testautouser1705610030] [2024-01-18T20:39:12.093Z] INFO[2024-01-18 20:39:12] Actual:true, Expected: true [2024-01-18T20:39:12.093Z] 2024-01-18 20:39:12 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610062] [2024-01-18T20:39:13.023Z] 2024-01-18 20:39:12 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610062 - 1054cc3c-7917-4911-a986-622200496913 [2024-01-18T20:39:14.912Z] INFO[2024-01-18 20:39:14] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610062] [2024-01-18T20:39:14.912Z] INFO[2024-01-18 20:39:14] Actual:true, Expected: true [2024-01-18T20:39:14.912Z] 2024-01-18 20:39:14 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610068] [2024-01-18T20:39:15.472Z] 2024-01-18 20:39:15 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610068 - de9bf072-d0af-427f-9e0c-5c9328a6cbed [2024-01-18T20:39:17.363Z] INFO[2024-01-18 20:39:16] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610068] [2024-01-18T20:39:17.363Z] INFO[2024-01-18 20:39:16] Actual:true, Expected: true [2024-01-18T20:39:17.363Z] 2024-01-18 20:39:16 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [default-roles-master] from the user : [testautouser1705610070] [2024-01-18T20:39:17.923Z] 2024-01-18 20:39:17 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610070 - ff66ab06-f279-48ee-a22e-db6a50be452b [2024-01-18T20:39:19.288Z] INFO[2024-01-18 20:39:19] Verifying : Description : Removing role [default-roles-master] from the user [testautouser1705610070] [2024-01-18T20:39:19.288Z] INFO[2024-01-18 20:39:19] Actual:true, Expected: true [2024-01-18T20:39:19.288Z] 2024-01-18 20:39:19 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [default-roles-master] from the user : [testautouser1705610096] [2024-01-18T20:39:20.213Z] 2024-01-18 20:39:20 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610096 - afcf41e9-a71f-47d2-921c-b6938918f311 [2024-01-18T20:39:22.099Z] INFO[2024-01-18 20:39:21] Verifying : Description : Removing role [default-roles-master] from the user [testautouser1705610096] [2024-01-18T20:39:22.099Z] INFO[2024-01-18 20:39:21] Actual:true, Expected: true [2024-01-18T20:39:22.099Z] 2024-01-18 20:39:21 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610154] [2024-01-18T20:39:22.660Z] 2024-01-18 20:39:22 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610154 - e69a5d42-cc97-47b3-9d6a-3f1464578131 [2024-01-18T20:39:24.550Z] INFO[2024-01-18 20:39:24] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610154] [2024-01-18T20:39:24.550Z] INFO[2024-01-18 20:39:24] Actual:true, Expected: true [2024-01-18T20:39:24.550Z] 2024-01-18 20:39:24 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610134] [2024-01-18T20:39:25.111Z] 2024-01-18 20:39:24 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610134 - 22af8231-a765-4a7c-b55f-a221d2db46a8 [2024-01-18T20:39:26.480Z] INFO[2024-01-18 20:39:26] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610134] [2024-01-18T20:39:26.480Z] INFO[2024-01-18 20:39:26] Actual:true, Expected: true [2024-01-18T20:39:26.480Z] 2024-01-18 20:39:26 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610138] [2024-01-18T20:39:27.409Z] 2024-01-18 20:39:27 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610138 - 55984835-105f-4f03-9f75-ae4343ad20bf [2024-01-18T20:39:29.297Z] INFO[2024-01-18 20:39:28] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610138] [2024-01-18T20:39:29.297Z] INFO[2024-01-18 20:39:28] Actual:true, Expected: true [2024-01-18T20:39:29.297Z] 2024-01-18 20:39:28 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610150] [2024-01-18T20:39:29.858Z] 2024-01-18 20:39:29 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610150 - 793d0d8f-e562-42e0-9417-12db14b278e7 [2024-01-18T20:39:31.745Z] INFO[2024-01-18 20:39:31] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610150] [2024-01-18T20:39:31.745Z] INFO[2024-01-18 20:39:31] Actual:true, Expected: true [2024-01-18T20:39:31.746Z] 2024-01-18 20:39:31 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610162] [2024-01-18T20:39:32.308Z] 2024-01-18 20:39:31 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610162 - 633ecb26-7233-4da3-9e45-62fac67885d1 [2024-01-18T20:39:33.671Z] INFO[2024-01-18 20:39:33] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610162] [2024-01-18T20:39:33.671Z] INFO[2024-01-18 20:39:33] Actual:true, Expected: true [2024-01-18T20:39:33.671Z] 2024-01-18 20:39:33 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610010] [2024-01-18T20:39:34.597Z] 2024-01-18 20:39:34 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610010 - c78a9cc3-7668-40be-9f4a-71cf45ade350 [2024-01-18T20:39:36.486Z] INFO[2024-01-18 20:39:36] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610010] [2024-01-18T20:39:36.486Z] INFO[2024-01-18 20:39:36] Actual:true, Expected: true [2024-01-18T20:39:36.486Z] 2024-01-18 20:39:36 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610016] [2024-01-18T20:39:37.047Z] 2024-01-18 20:39:36 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610016 - fbd35ec6-1a78-49ff-b7b0-74d1b338037c [2024-01-18T20:39:38.936Z] INFO[2024-01-18 20:39:38] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610016] [2024-01-18T20:39:38.936Z] INFO[2024-01-18 20:39:38] Actual:true, Expected: true [2024-01-18T20:39:38.936Z] 2024-01-18 20:39:38 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [default-roles-master] from the user : [testautouser1705610112] [2024-01-18T20:39:39.497Z] 2024-01-18 20:39:39 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610112 - 53b39063-19e1-40a4-9693-8ddfe0db231b [2024-01-18T20:39:40.862Z] INFO[2024-01-18 20:39:40] Verifying : Description : Removing role [default-roles-master] from the user [testautouser1705610112] [2024-01-18T20:39:40.862Z] INFO[2024-01-18 20:39:40] Actual:true, Expected: true [2024-01-18T20:39:40.862Z] 2024-01-18 20:39:40 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610122] [2024-01-18T20:39:41.788Z] 2024-01-18 20:39:41 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610122 - dbff47c3-0d0e-4011-8620-fc6b5336834a [2024-01-18T20:39:43.687Z] INFO[2024-01-18 20:39:43] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610122] [2024-01-18T20:39:43.687Z] INFO[2024-01-18 20:39:43] Actual:true, Expected: true [2024-01-18T20:39:43.687Z] 2024-01-18 20:39:43 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610152] [2024-01-18T20:39:44.248Z] 2024-01-18 20:39:43 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610152 - 3f864853-3b52-4c43-bf7e-6a98a067e03f [2024-01-18T20:39:46.142Z] INFO[2024-01-18 20:39:45] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610152] [2024-01-18T20:39:46.142Z] INFO[2024-01-18 20:39:45] Actual:true, Expected: true [2024-01-18T20:39:46.142Z] 2024-01-18 20:39:45 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610126] [2024-01-18T20:39:46.706Z] 2024-01-18 20:39:46 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610126 - 26113b48-5de6-4287-b996-d79c8b3d3f2c [2024-01-18T20:39:48.074Z] INFO[2024-01-18 20:39:48] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610126] [2024-01-18T20:39:48.074Z] INFO[2024-01-18 20:39:48] Actual:true, Expected: true [2024-01-18T20:39:48.074Z] 2024-01-18 20:39:48 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610144] [2024-01-18T20:39:48.999Z] 2024-01-18 20:39:48 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610144 - d0c9cadc-16bd-4019-9805-8f84365d646b [2024-01-18T20:39:50.890Z] INFO[2024-01-18 20:39:50] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610144] [2024-01-18T20:39:50.890Z] INFO[2024-01-18 20:39:50] Actual:true, Expected: true [2024-01-18T20:39:50.890Z] 2024-01-18 20:39:50 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [default-roles-master] from the user : [testautouser1705610014] [2024-01-18T20:39:51.451Z] 2024-01-18 20:39:51 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610014 - 769b0e6a-d44f-4666-a9dd-c5820ce32ff1 [2024-01-18T20:39:53.339Z] INFO[2024-01-18 20:39:52] Verifying : Description : Removing role [default-roles-master] from the user [testautouser1705610014] [2024-01-18T20:39:53.339Z] INFO[2024-01-18 20:39:52] Actual:true, Expected: true [2024-01-18T20:39:53.339Z] 2024-01-18 20:39:52 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610040] [2024-01-18T20:39:53.900Z] 2024-01-18 20:39:53 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610040 - d3ce505b-ed10-4750-82ad-3a8a689612da [2024-01-18T20:39:55.266Z] INFO[2024-01-18 20:39:55] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610040] [2024-01-18T20:39:55.266Z] INFO[2024-01-18 20:39:55] Actual:true, Expected: true [2024-01-18T20:39:55.266Z] 2024-01-18 20:39:55 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610046] [2024-01-18T20:39:56.192Z] 2024-01-18 20:39:56 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610046 - 3161dd97-111e-4e0e-9f3f-58210644d6d1 [2024-01-18T20:39:58.150Z] INFO[2024-01-18 20:39:57] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610046] [2024-01-18T20:39:58.150Z] INFO[2024-01-18 20:39:57] Actual:true, Expected: true [2024-01-18T20:39:58.150Z] 2024-01-18 20:39:57 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [default-roles-master] from the user : [testautouser1705610048] [2024-01-18T20:39:58.711Z] 2024-01-18 20:39:58 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610048 - dde214e6-fd13-48dd-9daf-78e748ebd986 [2024-01-18T20:40:00.079Z] INFO[2024-01-18 20:40:00] Verifying : Description : Removing role [default-roles-master] from the user [testautouser1705610048] [2024-01-18T20:40:00.079Z] INFO[2024-01-18 20:40:00] Actual:true, Expected: true [2024-01-18T20:40:00.079Z] 2024-01-18 20:40:00 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [default-roles-master] from the user : [testautouser1705610060] [2024-01-18T20:40:01.008Z] 2024-01-18 20:40:00 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610060 - c368720b-b33b-409e-b031-b87377ae0121 [2024-01-18T20:40:02.896Z] INFO[2024-01-18 20:40:02] Verifying : Description : Removing role [default-roles-master] from the user [testautouser1705610060] [2024-01-18T20:40:02.896Z] INFO[2024-01-18 20:40:02] Actual:true, Expected: true [2024-01-18T20:40:02.897Z] 2024-01-18 20:40:02 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610074] [2024-01-18T20:40:03.460Z] 2024-01-18 20:40:03 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610074 - 877d30b8-8b5a-4b4d-8f7c-9c64ec6cca37 [2024-01-18T20:40:05.351Z] INFO[2024-01-18 20:40:04] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610074] [2024-01-18T20:40:05.351Z] INFO[2024-01-18 20:40:04] Actual:true, Expected: true [2024-01-18T20:40:05.351Z] 2024-01-18 20:40:04 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610080] [2024-01-18T20:40:05.914Z] 2024-01-18 20:40:05 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610080 - f8998bc2-2979-4af0-8ebf-37e9555df970 [2024-01-18T20:40:07.280Z] INFO[2024-01-18 20:40:07] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610080] [2024-01-18T20:40:07.280Z] INFO[2024-01-18 20:40:07] Actual:true, Expected: true [2024-01-18T20:40:07.280Z] 2024-01-18 20:40:07 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [default-roles-master] from the user : [testautouser1705610148] [2024-01-18T20:40:08.207Z] 2024-01-18 20:40:07 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610148 - fd5df611-7f40-4420-b893-841433098e63 [2024-01-18T20:40:10.095Z] INFO[2024-01-18 20:40:09] Verifying : Description : Removing role [default-roles-master] from the user [testautouser1705610148] [2024-01-18T20:40:10.095Z] INFO[2024-01-18 20:40:09] Actual:true, Expected: true [2024-01-18T20:40:10.095Z] 2024-01-18 20:40:09 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [default-roles-master] from the user : [testautouser1705610086] [2024-01-18T20:40:10.656Z] 2024-01-18 20:40:10 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610086 - f654ad17-86b0-44b5-8531-ff4e2b3d0d8e [2024-01-18T20:40:12.546Z] INFO[2024-01-18 20:40:12] Verifying : Description : Removing role [default-roles-master] from the user [testautouser1705610086] [2024-01-18T20:40:12.546Z] INFO[2024-01-18 20:40:12] Actual:true, Expected: true [2024-01-18T20:40:12.546Z] 2024-01-18 20:40:12 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610098] [2024-01-18T20:40:13.107Z] 2024-01-18 20:40:12 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610098 - 1c5a50cc-50fe-497f-b1df-cbcbf250af9e [2024-01-18T20:40:14.473Z] INFO[2024-01-18 20:40:14] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610098] [2024-01-18T20:40:14.473Z] INFO[2024-01-18 20:40:14] Actual:true, Expected: true [2024-01-18T20:40:14.473Z] 2024-01-18 20:40:14 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610100] [2024-01-18T20:40:15.403Z] 2024-01-18 20:40:15 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610100 - 86a2a183-4d11-48ed-9a75-7567203642e2 [2024-01-18T20:40:17.291Z] INFO[2024-01-18 20:40:16] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610100] [2024-01-18T20:40:17.291Z] INFO[2024-01-18 20:40:16] Actual:true, Expected: true [2024-01-18T20:40:17.291Z] 2024-01-18 20:40:16 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610106] [2024-01-18T20:40:17.852Z] 2024-01-18 20:40:17 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610106 - b5943e13-aa12-462c-b394-a1e848dc62ce [2024-01-18T20:40:19.741Z] INFO[2024-01-18 20:40:19] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610106] [2024-01-18T20:40:19.741Z] INFO[2024-01-18 20:40:19] Actual:true, Expected: true [2024-01-18T20:40:19.741Z] 2024-01-18 20:40:19 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610142] [2024-01-18T20:40:20.307Z] 2024-01-18 20:40:20 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610142 - cbe78ed9-5980-4870-8b97-2fe3e9a7fde7 [2024-01-18T20:40:21.685Z] INFO[2024-01-18 20:40:21] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610142] [2024-01-18T20:40:21.685Z] INFO[2024-01-18 20:40:21] Actual:true, Expected: true [2024-01-18T20:40:21.685Z] 2024-01-18 20:40:21 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610042] [2024-01-18T20:40:22.619Z] 2024-01-18 20:40:22 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610042 - 025770b5-cd44-4ed6-a764-53655a42d7bd [2024-01-18T20:40:24.520Z] INFO[2024-01-18 20:40:24] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610042] [2024-01-18T20:40:24.520Z] INFO[2024-01-18 20:40:24] Actual:true, Expected: true [2024-01-18T20:40:24.520Z] 2024-01-18 20:40:24 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [default-roles-master] from the user : [testautouser1705610056] [2024-01-18T20:40:25.082Z] 2024-01-18 20:40:24 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610056 - 49f2c80a-ce57-44ac-a3e3-e7030e5a467e [2024-01-18T20:40:26.449Z] INFO[2024-01-18 20:40:26] Verifying : Description : Removing role [default-roles-master] from the user [testautouser1705610056] [2024-01-18T20:40:26.449Z] INFO[2024-01-18 20:40:26] Actual:true, Expected: true [2024-01-18T20:40:26.449Z] 2024-01-18 20:40:26 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610146] [2024-01-18T20:40:27.382Z] 2024-01-18 20:40:27 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610146 - 1fa99470-8321-4701-bdfe-5ce6c90d70c3 [2024-01-18T20:40:29.327Z] INFO[2024-01-18 20:40:28] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610146] [2024-01-18T20:40:29.327Z] INFO[2024-01-18 20:40:28] Actual:true, Expected: true [2024-01-18T20:40:29.327Z] 2024-01-18 20:40:28 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610018] [2024-01-18T20:40:29.889Z] 2024-01-18 20:40:29 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610018 - 48d8a9a5-648d-46d3-b1a6-3e27c16d3a2e [2024-01-18T20:40:31.257Z] INFO[2024-01-18 20:40:31] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610018] [2024-01-18T20:40:31.257Z] INFO[2024-01-18 20:40:31] Actual:true, Expected: true [2024-01-18T20:40:31.258Z] 2024-01-18 20:40:31 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610082] [2024-01-18T20:40:32.184Z] 2024-01-18 20:40:32 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610082 - 52acc498-f1df-4642-b478-a947802d85a7 [2024-01-18T20:40:34.071Z] INFO[2024-01-18 20:40:33] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610082] [2024-01-18T20:40:34.071Z] INFO[2024-01-18 20:40:33] Actual:true, Expected: true [2024-01-18T20:40:34.071Z] 2024-01-18 20:40:33 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610116] [2024-01-18T20:40:34.633Z] 2024-01-18 20:40:34 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610116 - 7b0e6957-5a2a-4058-af4e-eea69261a3ea [2024-01-18T20:40:36.521Z] INFO[2024-01-18 20:40:36] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610116] [2024-01-18T20:40:36.521Z] INFO[2024-01-18 20:40:36] Actual:true, Expected: true [2024-01-18T20:40:36.522Z] 2024-01-18 20:40:36 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610120] [2024-01-18T20:40:37.083Z] 2024-01-18 20:40:36 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610120 - b7cd24fd-cdb4-493b-b45f-d8130b181007 [2024-01-18T20:40:38.972Z] INFO[2024-01-18 20:40:38] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610120] [2024-01-18T20:40:38.972Z] INFO[2024-01-18 20:40:38] Actual:true, Expected: true [2024-01-18T20:40:38.972Z] 2024-01-18 20:40:38 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-infra.admin] from the user : [testautouser1705610006] [2024-01-18T20:40:39.227Z] 2024-01-18 20:40:39 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610006 - e67bd13c-0032-4115-8a8e-fb7a4bb412ac [2024-01-18T20:40:41.116Z] INFO[2024-01-18 20:40:40] Verifying : Description : Removing role [px-backup-infra.admin] from the user [testautouser1705610006] [2024-01-18T20:40:41.116Z] INFO[2024-01-18 20:40:40] Actual:true, Expected: true [2024-01-18T20:40:41.116Z] 2024-01-18 20:40:40 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610028] [2024-01-18T20:40:41.677Z] 2024-01-18 20:40:41 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610028 - e08f60f9-2318-418d-a57f-bbf03280c55d [2024-01-18T20:40:43.566Z] INFO[2024-01-18 20:40:43] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610028] [2024-01-18T20:40:43.566Z] INFO[2024-01-18 20:40:43] Actual:true, Expected: true [2024-01-18T20:40:43.566Z] 2024-01-18 20:40:43 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610034] [2024-01-18T20:40:44.129Z] 2024-01-18 20:40:43 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610034 - 98db2477-47c0-4182-b229-a7801d11b9df [2024-01-18T20:40:46.019Z] INFO[2024-01-18 20:40:45] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610034] [2024-01-18T20:40:46.019Z] INFO[2024-01-18 20:40:45] Actual:true, Expected: true [2024-01-18T20:40:46.019Z] 2024-01-18 20:40:45 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [default-roles-master] from the user : [testautouser1705610052] [2024-01-18T20:40:46.579Z] 2024-01-18 20:40:46 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610052 - dc9577da-177e-4d7d-a282-38db3d1d912e [2024-01-18T20:40:48.465Z] INFO[2024-01-18 20:40:48] Verifying : Description : Removing role [default-roles-master] from the user [testautouser1705610052] [2024-01-18T20:40:48.465Z] INFO[2024-01-18 20:40:48] Actual:true, Expected: true [2024-01-18T20:40:48.465Z] 2024-01-18 20:40:48 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [default-roles-master] from the user : [testautouser1705610110] [2024-01-18T20:40:49.027Z] 2024-01-18 20:40:48 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610110 - 1a809db7-d012-4542-b732-fc3839876092 [2024-01-18T20:40:50.918Z] INFO[2024-01-18 20:40:50] Verifying : Description : Removing role [default-roles-master] from the user [testautouser1705610110] [2024-01-18T20:40:50.918Z] INFO[2024-01-18 20:40:50] Actual:true, Expected: true [2024-01-18T20:40:50.918Z] 2024-01-18 20:40:50 +0000:[INFO] [{DeleteUsersRole}] [backup.glob..func95.2.2:#3681] - Deleting [px-backup-app.admin] from the user : [testautouser1705610128] [2024-01-18T20:40:51.482Z] 2024-01-18 20:40:51 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610128 - e0c2beaf-bb4b-4438-b618-3e34c39c5340 [2024-01-18T20:40:52.848Z] INFO[2024-01-18 20:40:52] Verifying : Description : Removing role [px-backup-app.admin] from the user [testautouser1705610128] [2024-01-18T20:40:52.848Z] INFO[2024-01-18 20:40:52] Actual:true, Expected: true [2024-01-18T20:40:54.742Z] 2024-01-18 20:40:54 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610004 - 5c93ae4c-682e-4b77-92e8-36dcf764a6ab [2024-01-18T20:40:54.742Z] INFO[2024-01-18 20:40:54] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610004] [2024-01-18T20:40:54.742Z] INFO[2024-01-18 20:40:54] Actual:false, Expected: false [2024-01-18T20:40:56.113Z] 2024-01-18 20:40:55 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610022 - c3b79415-4826-451a-be4d-707b5011e3f7 [2024-01-18T20:40:56.113Z] INFO[2024-01-18 20:40:56] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610022] [2024-01-18T20:40:56.113Z] INFO[2024-01-18 20:40:56] Actual:false, Expected: false [2024-01-18T20:40:57.999Z] 2024-01-18 20:40:57 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610064 - 0e589b35-6b8e-4ff7-86b4-f7483b6b44bb [2024-01-18T20:40:57.999Z] INFO[2024-01-18 20:40:57] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610064] [2024-01-18T20:40:58.000Z] INFO[2024-01-18 20:40:57] Actual:false, Expected: false [2024-01-18T20:40:59.363Z] 2024-01-18 20:40:59 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610076 - fb6341a7-629a-437f-93ac-b3770e24820b [2024-01-18T20:40:59.363Z] INFO[2024-01-18 20:40:59] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610076] [2024-01-18T20:40:59.363Z] INFO[2024-01-18 20:40:59] Actual:false, Expected: false [2024-01-18T20:41:01.251Z] 2024-01-18 20:41:00 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610104 - 901ad539-24a5-4a25-8daa-2d47e6bd68c3 [2024-01-18T20:41:01.251Z] INFO[2024-01-18 20:41:00] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610104] [2024-01-18T20:41:01.251Z] INFO[2024-01-18 20:41:00] Actual:false, Expected: false [2024-01-18T20:41:02.618Z] 2024-01-18 20:41:02 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610130 - 917d6368-07a2-4e0b-8a2d-fba8ff42324d [2024-01-18T20:41:02.618Z] INFO[2024-01-18 20:41:02] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610130] [2024-01-18T20:41:02.618Z] INFO[2024-01-18 20:41:02] Actual:false, Expected: false [2024-01-18T20:41:04.510Z] 2024-01-18 20:41:03 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610136 - 7ba4083a-8907-440c-a286-54606b1f2c19 [2024-01-18T20:41:04.510Z] INFO[2024-01-18 20:41:04] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610136] [2024-01-18T20:41:04.510Z] INFO[2024-01-18 20:41:04] Actual:false, Expected: false [2024-01-18T20:41:05.876Z] 2024-01-18 20:41:05 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610008 - fd9cfa2f-6ba4-40a1-a0f1-5c111aa0a5e8 [2024-01-18T20:41:05.876Z] INFO[2024-01-18 20:41:05] Verifying : Description : validation of deleted role [default-roles-master] from user [testautouser1705610008] [2024-01-18T20:41:05.876Z] INFO[2024-01-18 20:41:05] Actual:false, Expected: false [2024-01-18T20:41:07.242Z] 2024-01-18 20:41:07 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610020 - 3d974540-f2fb-4df5-becb-1b26936765f6 [2024-01-18T20:41:07.242Z] INFO[2024-01-18 20:41:07] Verifying : Description : validation of deleted role [default-roles-master] from user [testautouser1705610020] [2024-01-18T20:41:07.242Z] INFO[2024-01-18 20:41:07] Actual:false, Expected: false [2024-01-18T20:41:09.132Z] 2024-01-18 20:41:08 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610038 - 826f6706-3c7e-4be8-958b-a2f5c3cdb339 [2024-01-18T20:41:09.132Z] INFO[2024-01-18 20:41:08] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610038] [2024-01-18T20:41:09.132Z] INFO[2024-01-18 20:41:08] Actual:false, Expected: false [2024-01-18T20:41:10.499Z] 2024-01-18 20:41:10 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610088 - 63b2f68b-4b2a-4bf0-b640-d28d0543ed46 [2024-01-18T20:41:10.499Z] INFO[2024-01-18 20:41:10] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610088] [2024-01-18T20:41:10.499Z] INFO[2024-01-18 20:41:10] Actual:false, Expected: false [2024-01-18T20:41:12.388Z] 2024-01-18 20:41:11 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610090 - 75e33520-f78f-4044-b520-6703df801db9 [2024-01-18T20:41:12.388Z] INFO[2024-01-18 20:41:11] Verifying : Description : validation of deleted role [default-roles-master] from user [testautouser1705610090] [2024-01-18T20:41:12.388Z] INFO[2024-01-18 20:41:11] Actual:false, Expected: false [2024-01-18T20:41:13.751Z] 2024-01-18 20:41:13 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610094 - b23ab55c-d0c8-4cef-8d06-cb57dbc700ea [2024-01-18T20:41:13.752Z] INFO[2024-01-18 20:41:13] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610094] [2024-01-18T20:41:13.752Z] INFO[2024-01-18 20:41:13] Actual:false, Expected: false [2024-01-18T20:41:15.640Z] 2024-01-18 20:41:15 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610158 - 27657394-487e-4705-a61e-00122cda400b [2024-01-18T20:41:15.640Z] INFO[2024-01-18 20:41:15] Verifying : Description : validation of deleted role [default-roles-master] from user [testautouser1705610158] [2024-01-18T20:41:15.640Z] INFO[2024-01-18 20:41:15] Actual:false, Expected: false [2024-01-18T20:41:17.007Z] 2024-01-18 20:41:16 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610030 - 3125668c-dc81-4c20-bfcb-71a66c9f789b [2024-01-18T20:41:17.007Z] INFO[2024-01-18 20:41:16] Verifying : Description : validation of deleted role [default-roles-master] from user [testautouser1705610030] [2024-01-18T20:41:17.007Z] INFO[2024-01-18 20:41:16] Actual:false, Expected: false [2024-01-18T20:41:18.898Z] 2024-01-18 20:41:18 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610062 - 1054cc3c-7917-4911-a986-622200496913 [2024-01-18T20:41:18.898Z] INFO[2024-01-18 20:41:18] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610062] [2024-01-18T20:41:18.898Z] INFO[2024-01-18 20:41:18] Actual:false, Expected: false [2024-01-18T20:41:20.267Z] 2024-01-18 20:41:19 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610068 - de9bf072-d0af-427f-9e0c-5c9328a6cbed [2024-01-18T20:41:20.267Z] INFO[2024-01-18 20:41:20] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610068] [2024-01-18T20:41:20.267Z] INFO[2024-01-18 20:41:20] Actual:false, Expected: false [2024-01-18T20:41:21.635Z] 2024-01-18 20:41:21 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610070 - ff66ab06-f279-48ee-a22e-db6a50be452b [2024-01-18T20:41:21.635Z] INFO[2024-01-18 20:41:21] Verifying : Description : validation of deleted role [default-roles-master] from user [testautouser1705610070] [2024-01-18T20:41:21.636Z] INFO[2024-01-18 20:41:21] Actual:false, Expected: false [2024-01-18T20:41:23.527Z] 2024-01-18 20:41:23 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610096 - afcf41e9-a71f-47d2-921c-b6938918f311 [2024-01-18T20:41:23.527Z] INFO[2024-01-18 20:41:23] Verifying : Description : validation of deleted role [default-roles-master] from user [testautouser1705610096] [2024-01-18T20:41:23.527Z] INFO[2024-01-18 20:41:23] Actual:false, Expected: false [2024-01-18T20:41:24.892Z] 2024-01-18 20:41:24 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610154 - e69a5d42-cc97-47b3-9d6a-3f1464578131 [2024-01-18T20:41:24.892Z] INFO[2024-01-18 20:41:24] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610154] [2024-01-18T20:41:24.892Z] INFO[2024-01-18 20:41:24] Actual:false, Expected: false [2024-01-18T20:41:26.782Z] 2024-01-18 20:41:26 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610134 - 22af8231-a765-4a7c-b55f-a221d2db46a8 [2024-01-18T20:41:26.782Z] INFO[2024-01-18 20:41:26] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610134] [2024-01-18T20:41:26.782Z] INFO[2024-01-18 20:41:26] Actual:false, Expected: false [2024-01-18T20:41:28.147Z] 2024-01-18 20:41:27 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610138 - 55984835-105f-4f03-9f75-ae4343ad20bf [2024-01-18T20:41:28.147Z] INFO[2024-01-18 20:41:27] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610138] [2024-01-18T20:41:28.147Z] INFO[2024-01-18 20:41:27] Actual:false, Expected: false [2024-01-18T20:41:30.044Z] 2024-01-18 20:41:29 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610150 - 793d0d8f-e562-42e0-9417-12db14b278e7 [2024-01-18T20:41:30.044Z] INFO[2024-01-18 20:41:29] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610150] [2024-01-18T20:41:30.044Z] INFO[2024-01-18 20:41:29] Actual:false, Expected: false [2024-01-18T20:41:31.410Z] 2024-01-18 20:41:31 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610162 - 633ecb26-7233-4da3-9e45-62fac67885d1 [2024-01-18T20:41:31.410Z] INFO[2024-01-18 20:41:31] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610162] [2024-01-18T20:41:31.410Z] INFO[2024-01-18 20:41:31] Actual:false, Expected: false [2024-01-18T20:41:33.298Z] 2024-01-18 20:41:32 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610010 - c78a9cc3-7668-40be-9f4a-71cf45ade350 [2024-01-18T20:41:33.298Z] INFO[2024-01-18 20:41:32] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610010] [2024-01-18T20:41:33.298Z] INFO[2024-01-18 20:41:32] Actual:false, Expected: false [2024-01-18T20:41:34.662Z] 2024-01-18 20:41:34 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610016 - fbd35ec6-1a78-49ff-b7b0-74d1b338037c [2024-01-18T20:41:34.662Z] INFO[2024-01-18 20:41:34] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610016] [2024-01-18T20:41:34.662Z] INFO[2024-01-18 20:41:34] Actual:false, Expected: false [2024-01-18T20:41:36.027Z] 2024-01-18 20:41:35 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610112 - 53b39063-19e1-40a4-9693-8ddfe0db231b [2024-01-18T20:41:36.283Z] INFO[2024-01-18 20:41:36] Verifying : Description : validation of deleted role [default-roles-master] from user [testautouser1705610112] [2024-01-18T20:41:36.283Z] INFO[2024-01-18 20:41:36] Actual:false, Expected: false [2024-01-18T20:41:37.651Z] 2024-01-18 20:41:37 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610122 - dbff47c3-0d0e-4011-8620-fc6b5336834a [2024-01-18T20:41:37.652Z] INFO[2024-01-18 20:41:37] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610122] [2024-01-18T20:41:37.652Z] INFO[2024-01-18 20:41:37] Actual:false, Expected: false [2024-01-18T20:41:39.556Z] 2024-01-18 20:41:39 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610152 - 3f864853-3b52-4c43-bf7e-6a98a067e03f [2024-01-18T20:41:39.556Z] INFO[2024-01-18 20:41:39] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610152] [2024-01-18T20:41:39.556Z] INFO[2024-01-18 20:41:39] Actual:false, Expected: false [2024-01-18T20:41:40.923Z] 2024-01-18 20:41:40 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610014 - 769b0e6a-d44f-4666-a9dd-c5820ce32ff1 [2024-01-18T20:41:40.923Z] INFO[2024-01-18 20:41:40] Verifying : Description : validation of deleted role [default-roles-master] from user [testautouser1705610014] [2024-01-18T20:41:40.923Z] INFO[2024-01-18 20:41:40] Actual:false, Expected: false [2024-01-18T20:41:42.857Z] 2024-01-18 20:41:42 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610040 - d3ce505b-ed10-4750-82ad-3a8a689612da [2024-01-18T20:41:42.857Z] INFO[2024-01-18 20:41:42] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610040] [2024-01-18T20:41:42.857Z] INFO[2024-01-18 20:41:42] Actual:false, Expected: false [2024-01-18T20:41:44.223Z] 2024-01-18 20:41:43 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610046 - 3161dd97-111e-4e0e-9f3f-58210644d6d1 [2024-01-18T20:41:44.223Z] INFO[2024-01-18 20:41:44] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610046] [2024-01-18T20:41:44.223Z] INFO[2024-01-18 20:41:44] Actual:false, Expected: false [2024-01-18T20:41:46.114Z] 2024-01-18 20:41:45 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610048 - dde214e6-fd13-48dd-9daf-78e748ebd986 [2024-01-18T20:41:46.114Z] INFO[2024-01-18 20:41:45] Verifying : Description : validation of deleted role [default-roles-master] from user [testautouser1705610048] [2024-01-18T20:41:46.114Z] INFO[2024-01-18 20:41:45] Actual:false, Expected: false [2024-01-18T20:41:47.484Z] 2024-01-18 20:41:47 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610060 - c368720b-b33b-409e-b031-b87377ae0121 [2024-01-18T20:41:47.484Z] INFO[2024-01-18 20:41:47] Verifying : Description : validation of deleted role [default-roles-master] from user [testautouser1705610060] [2024-01-18T20:41:47.484Z] INFO[2024-01-18 20:41:47] Actual:false, Expected: false [2024-01-18T20:41:48.856Z] 2024-01-18 20:41:48 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610074 - 877d30b8-8b5a-4b4d-8f7c-9c64ec6cca37 [2024-01-18T20:41:48.856Z] INFO[2024-01-18 20:41:48] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610074] [2024-01-18T20:41:48.856Z] INFO[2024-01-18 20:41:48] Actual:false, Expected: false [2024-01-18T20:41:50.753Z] 2024-01-18 20:41:50 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610126 - 26113b48-5de6-4287-b996-d79c8b3d3f2c [2024-01-18T20:41:50.753Z] INFO[2024-01-18 20:41:50] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610126] [2024-01-18T20:41:50.753Z] INFO[2024-01-18 20:41:50] Actual:false, Expected: false [2024-01-18T20:41:52.121Z] 2024-01-18 20:41:51 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610144 - d0c9cadc-16bd-4019-9805-8f84365d646b [2024-01-18T20:41:52.121Z] INFO[2024-01-18 20:41:52] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610144] [2024-01-18T20:41:52.121Z] INFO[2024-01-18 20:41:52] Actual:false, Expected: false [2024-01-18T20:41:54.012Z] 2024-01-18 20:41:53 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610080 - f8998bc2-2979-4af0-8ebf-37e9555df970 [2024-01-18T20:41:54.012Z] INFO[2024-01-18 20:41:53] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610080] [2024-01-18T20:41:54.012Z] INFO[2024-01-18 20:41:53] Actual:false, Expected: false [2024-01-18T20:41:55.378Z] 2024-01-18 20:41:55 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610148 - fd5df611-7f40-4420-b893-841433098e63 [2024-01-18T20:41:55.378Z] INFO[2024-01-18 20:41:55] Verifying : Description : validation of deleted role [default-roles-master] from user [testautouser1705610148] [2024-01-18T20:41:55.378Z] INFO[2024-01-18 20:41:55] Actual:false, Expected: false [2024-01-18T20:41:57.266Z] 2024-01-18 20:41:56 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610086 - f654ad17-86b0-44b5-8531-ff4e2b3d0d8e [2024-01-18T20:41:57.266Z] INFO[2024-01-18 20:41:56] Verifying : Description : validation of deleted role [default-roles-master] from user [testautouser1705610086] [2024-01-18T20:41:57.266Z] INFO[2024-01-18 20:41:56] Actual:false, Expected: false [2024-01-18T20:41:58.630Z] 2024-01-18 20:41:58 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610098 - 1c5a50cc-50fe-497f-b1df-cbcbf250af9e [2024-01-18T20:41:58.630Z] INFO[2024-01-18 20:41:58] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610098] [2024-01-18T20:41:58.630Z] INFO[2024-01-18 20:41:58] Actual:false, Expected: false [2024-01-18T20:42:00.517Z] 2024-01-18 20:41:59 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610100 - 86a2a183-4d11-48ed-9a75-7567203642e2 [2024-01-18T20:42:00.517Z] INFO[2024-01-18 20:42:00] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610100] [2024-01-18T20:42:00.517Z] INFO[2024-01-18 20:42:00] Actual:false, Expected: false [2024-01-18T20:42:01.886Z] 2024-01-18 20:42:01 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610106 - b5943e13-aa12-462c-b394-a1e848dc62ce [2024-01-18T20:42:01.887Z] INFO[2024-01-18 20:42:01] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610106] [2024-01-18T20:42:01.887Z] INFO[2024-01-18 20:42:01] Actual:false, Expected: false [2024-01-18T20:42:03.252Z] 2024-01-18 20:42:03 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610142 - cbe78ed9-5980-4870-8b97-2fe3e9a7fde7 [2024-01-18T20:42:03.252Z] INFO[2024-01-18 20:42:03] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610142] [2024-01-18T20:42:03.252Z] INFO[2024-01-18 20:42:03] Actual:false, Expected: false [2024-01-18T20:42:05.141Z] 2024-01-18 20:42:04 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610042 - 025770b5-cd44-4ed6-a764-53655a42d7bd [2024-01-18T20:42:05.141Z] INFO[2024-01-18 20:42:04] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610042] [2024-01-18T20:42:05.141Z] INFO[2024-01-18 20:42:04] Actual:false, Expected: false [2024-01-18T20:42:06.506Z] 2024-01-18 20:42:06 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610056 - 49f2c80a-ce57-44ac-a3e3-e7030e5a467e [2024-01-18T20:42:06.506Z] INFO[2024-01-18 20:42:06] Verifying : Description : validation of deleted role [default-roles-master] from user [testautouser1705610056] [2024-01-18T20:42:06.506Z] INFO[2024-01-18 20:42:06] Actual:false, Expected: false [2024-01-18T20:42:08.394Z] 2024-01-18 20:42:07 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610146 - 1fa99470-8321-4701-bdfe-5ce6c90d70c3 [2024-01-18T20:42:08.394Z] INFO[2024-01-18 20:42:08] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610146] [2024-01-18T20:42:08.395Z] INFO[2024-01-18 20:42:08] Actual:false, Expected: false [2024-01-18T20:42:09.761Z] 2024-01-18 20:42:09 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610018 - 48d8a9a5-648d-46d3-b1a6-3e27c16d3a2e [2024-01-18T20:42:09.761Z] INFO[2024-01-18 20:42:09] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610018] [2024-01-18T20:42:09.761Z] INFO[2024-01-18 20:42:09] Actual:false, Expected: false [2024-01-18T20:42:11.666Z] 2024-01-18 20:42:11 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610082 - 52acc498-f1df-4642-b478-a947802d85a7 [2024-01-18T20:42:11.666Z] INFO[2024-01-18 20:42:11] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610082] [2024-01-18T20:42:11.666Z] INFO[2024-01-18 20:42:11] Actual:false, Expected: false [2024-01-18T20:42:13.032Z] 2024-01-18 20:42:12 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610116 - 7b0e6957-5a2a-4058-af4e-eea69261a3ea [2024-01-18T20:42:13.032Z] INFO[2024-01-18 20:42:12] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610116] [2024-01-18T20:42:13.032Z] INFO[2024-01-18 20:42:12] Actual:false, Expected: false [2024-01-18T20:42:14.923Z] 2024-01-18 20:42:14 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610120 - b7cd24fd-cdb4-493b-b45f-d8130b181007 [2024-01-18T20:42:14.923Z] INFO[2024-01-18 20:42:14] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610120] [2024-01-18T20:42:14.923Z] INFO[2024-01-18 20:42:14] Actual:false, Expected: false [2024-01-18T20:42:16.290Z] 2024-01-18 20:42:16 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610006 - e67bd13c-0032-4115-8a8e-fb7a4bb412ac [2024-01-18T20:42:16.290Z] INFO[2024-01-18 20:42:16] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610006] [2024-01-18T20:42:16.290Z] INFO[2024-01-18 20:42:16] Actual:false, Expected: false [2024-01-18T20:42:17.659Z] 2024-01-18 20:42:17 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610028 - e08f60f9-2318-418d-a57f-bbf03280c55d [2024-01-18T20:42:17.659Z] INFO[2024-01-18 20:42:17] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610028] [2024-01-18T20:42:17.659Z] INFO[2024-01-18 20:42:17] Actual:false, Expected: false [2024-01-18T20:42:19.549Z] 2024-01-18 20:42:19 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610034 - 98db2477-47c0-4182-b229-a7801d11b9df [2024-01-18T20:42:19.550Z] INFO[2024-01-18 20:42:19] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610034] [2024-01-18T20:42:19.550Z] INFO[2024-01-18 20:42:19] Actual:false, Expected: false [2024-01-18T20:42:20.913Z] 2024-01-18 20:42:20 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610052 - dc9577da-177e-4d7d-a282-38db3d1d912e [2024-01-18T20:42:20.913Z] INFO[2024-01-18 20:42:20] Verifying : Description : validation of deleted role [default-roles-master] from user [testautouser1705610052] [2024-01-18T20:42:20.913Z] INFO[2024-01-18 20:42:20] Actual:false, Expected: false [2024-01-18T20:42:22.804Z] 2024-01-18 20:42:22 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610110 - 1a809db7-d012-4542-b732-fc3839876092 [2024-01-18T20:42:22.804Z] INFO[2024-01-18 20:42:22] Verifying : Description : validation of deleted role [default-roles-master] from user [testautouser1705610110] [2024-01-18T20:42:22.804Z] INFO[2024-01-18 20:42:22] Actual:false, Expected: false [2024-01-18T20:42:24.169Z] 2024-01-18 20:42:23 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610128 - e0c2beaf-bb4b-4438-b618-3e34c39c5340 [2024-01-18T20:42:24.169Z] INFO[2024-01-18 20:42:24] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610128] [2024-01-18T20:42:24.169Z] INFO[2024-01-18 20:42:24] Actual:false, Expected: false [2024-01-18T20:42:26.062Z] 2024-01-18 20:42:25 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610012 - 84b1f4be-bffc-4566-8592-4faeb1e00fd9 [2024-01-18T20:42:26.062Z] INFO[2024-01-18 20:42:25] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610012] [2024-01-18T20:42:26.062Z] INFO[2024-01-18 20:42:25] Actual:false, Expected: false [2024-01-18T20:42:27.428Z] 2024-01-18 20:42:27 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610026 - a0235953-6fbc-4765-bf99-d076797582a4 [2024-01-18T20:42:27.428Z] INFO[2024-01-18 20:42:27] Verifying : Description : validation of deleted role [default-roles-master] from user [testautouser1705610026] [2024-01-18T20:42:27.428Z] INFO[2024-01-18 20:42:27] Actual:false, Expected: false [2024-01-18T20:42:29.316Z] 2024-01-18 20:42:28 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610036 - 2715aeee-1103-42ed-8c0e-dbbe725053c9 [2024-01-18T20:42:29.316Z] INFO[2024-01-18 20:42:28] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610036] [2024-01-18T20:42:29.316Z] INFO[2024-01-18 20:42:28] Actual:false, Expected: false [2024-01-18T20:42:30.684Z] 2024-01-18 20:42:30 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610050 - fee7a61a-629a-4226-8cc9-f492a615fcf3 [2024-01-18T20:42:30.684Z] INFO[2024-01-18 20:42:30] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610050] [2024-01-18T20:42:30.684Z] INFO[2024-01-18 20:42:30] Actual:false, Expected: false [2024-01-18T20:42:32.050Z] 2024-01-18 20:42:31 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610140 - 4d6b2481-e9c8-4fec-9559-da3d377fb702 [2024-01-18T20:42:32.050Z] INFO[2024-01-18 20:42:32] Verifying : Description : validation of deleted role [default-roles-master] from user [testautouser1705610140] [2024-01-18T20:42:32.050Z] INFO[2024-01-18 20:42:32] Actual:false, Expected: false [2024-01-18T20:42:33.941Z] 2024-01-18 20:42:33 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610044 - 7977b2f1-3558-4162-b756-4310234b51aa [2024-01-18T20:42:33.941Z] INFO[2024-01-18 20:42:33] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610044] [2024-01-18T20:42:33.941Z] INFO[2024-01-18 20:42:33] Actual:false, Expected: false [2024-01-18T20:42:35.307Z] 2024-01-18 20:42:35 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610058 - 16bde0ab-8f32-4dae-9f70-7c4ea3b5edeb [2024-01-18T20:42:35.308Z] INFO[2024-01-18 20:42:35] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610058] [2024-01-18T20:42:35.308Z] INFO[2024-01-18 20:42:35] Actual:false, Expected: false [2024-01-18T20:42:37.197Z] 2024-01-18 20:42:36 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610066 - 83bd9819-043f-4466-8f05-1588d1d7a463 [2024-01-18T20:42:37.197Z] INFO[2024-01-18 20:42:36] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610066] [2024-01-18T20:42:37.197Z] INFO[2024-01-18 20:42:36] Actual:false, Expected: false [2024-01-18T20:42:38.562Z] 2024-01-18 20:42:38 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610084 - 7e1b5ce9-b6e4-4cf6-9e1f-723066b86fe6 [2024-01-18T20:42:38.562Z] INFO[2024-01-18 20:42:38] Verifying : Description : validation of deleted role [default-roles-master] from user [testautouser1705610084] [2024-01-18T20:42:38.562Z] INFO[2024-01-18 20:42:38] Actual:false, Expected: false [2024-01-18T20:42:40.451Z] 2024-01-18 20:42:39 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610118 - adf61788-400d-4b0d-9d25-0ed58cf00e61 [2024-01-18T20:42:40.451Z] INFO[2024-01-18 20:42:40] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610118] [2024-01-18T20:42:40.451Z] INFO[2024-01-18 20:42:40] Actual:false, Expected: false [2024-01-18T20:42:41.837Z] 2024-01-18 20:42:41 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610024 - 878b64e4-b6cd-464c-888e-e6ffc9769335 [2024-01-18T20:42:41.837Z] INFO[2024-01-18 20:42:41] Verifying : Description : validation of deleted role [default-roles-master] from user [testautouser1705610024] [2024-01-18T20:42:41.837Z] INFO[2024-01-18 20:42:41] Actual:false, Expected: false [2024-01-18T20:42:43.728Z] 2024-01-18 20:42:43 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610072 - a643dea8-aed3-4ab6-912b-7f4fa9d485d6 [2024-01-18T20:42:43.728Z] INFO[2024-01-18 20:42:43] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610072] [2024-01-18T20:42:43.728Z] INFO[2024-01-18 20:42:43] Actual:false, Expected: false [2024-01-18T20:42:45.095Z] 2024-01-18 20:42:44 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610078 - 28750b2c-002f-4d4e-8dda-62436d29ac85 [2024-01-18T20:42:45.095Z] INFO[2024-01-18 20:42:44] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610078] [2024-01-18T20:42:45.095Z] INFO[2024-01-18 20:42:44] Actual:false, Expected: false [2024-01-18T20:42:46.459Z] 2024-01-18 20:42:46 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610092 - a44ce439-7030-4050-a318-52347f806d6b [2024-01-18T20:42:46.459Z] INFO[2024-01-18 20:42:46] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610092] [2024-01-18T20:42:46.459Z] INFO[2024-01-18 20:42:46] Actual:false, Expected: false [2024-01-18T20:42:48.348Z] 2024-01-18 20:42:47 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610102 - ce28004a-64b3-4088-bfd8-42a5e0014c08 [2024-01-18T20:42:48.349Z] INFO[2024-01-18 20:42:48] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610102] [2024-01-18T20:42:48.349Z] INFO[2024-01-18 20:42:48] Actual:false, Expected: false [2024-01-18T20:42:49.717Z] 2024-01-18 20:42:49 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610132 - 8efaf076-1d80-4b56-a7be-7e57a5929a98 [2024-01-18T20:42:49.717Z] INFO[2024-01-18 20:42:49] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610132] [2024-01-18T20:42:49.717Z] INFO[2024-01-18 20:42:49] Actual:false, Expected: false [2024-01-18T20:42:51.618Z] 2024-01-18 20:42:51 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610032 - c434d6dd-e59a-47b7-b2e6-ff0cafe49b99 [2024-01-18T20:42:51.618Z] INFO[2024-01-18 20:42:51] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610032] [2024-01-18T20:42:51.618Z] INFO[2024-01-18 20:42:51] Actual:false, Expected: false [2024-01-18T20:42:53.047Z] 2024-01-18 20:42:52 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610054 - 916ea4a6-ede1-4c37-89fe-3c788488ae3f [2024-01-18T20:42:53.047Z] INFO[2024-01-18 20:42:52] Verifying : Description : validation of deleted role [default-roles-master] from user [testautouser1705610054] [2024-01-18T20:42:53.047Z] INFO[2024-01-18 20:42:52] Actual:false, Expected: false [2024-01-18T20:42:54.939Z] 2024-01-18 20:42:54 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610108 - 7a591a81-146b-4d14-b89e-d3412b6f1370 [2024-01-18T20:42:54.939Z] INFO[2024-01-18 20:42:54] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610108] [2024-01-18T20:42:54.939Z] INFO[2024-01-18 20:42:54] Actual:false, Expected: false [2024-01-18T20:42:56.308Z] 2024-01-18 20:42:55 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610114 - 940dc0e4-2d22-4490-8925-91cc61ec31e9 [2024-01-18T20:42:56.308Z] INFO[2024-01-18 20:42:55] Verifying : Description : validation of deleted role [default-roles-master] from user [testautouser1705610114] [2024-01-18T20:42:56.308Z] INFO[2024-01-18 20:42:55] Actual:false, Expected: false [2024-01-18T20:42:57.676Z] 2024-01-18 20:42:57 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610124 - 30c2883a-9b23-451c-a9fe-4473e94c8371 [2024-01-18T20:42:57.676Z] INFO[2024-01-18 20:42:57] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610124] [2024-01-18T20:42:57.676Z] INFO[2024-01-18 20:42:57] Actual:false, Expected: false [2024-01-18T20:42:59.563Z] 2024-01-18 20:42:59 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610156 - 3477536c-ee63-4f69-b374-34452710f43b [2024-01-18T20:42:59.563Z] INFO[2024-01-18 20:42:59] Verifying : Description : validation of deleted role [px-backup-app.admin] from user [testautouser1705610156] [2024-01-18T20:42:59.563Z] INFO[2024-01-18 20:42:59] Actual:false, Expected: false [2024-01-18T20:43:00.927Z] 2024-01-18 20:43:00 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610160 - 11434e64-4fc8-4485-9e1f-adfe328c0df6 [2024-01-18T20:43:00.927Z] INFO[2024-01-18 20:43:00] Verifying : Description : validation of deleted role [px-backup-infra.admin] from user [testautouser1705610160] [2024-01-18T20:43:00.927Z] INFO[2024-01-18 20:43:00] Actual:false, Expected: false [2024-01-18T20:43:02.292Z] 2024-01-18 20:43:02 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610080 - f8998bc2-2979-4af0-8ebf-37e9555df970 [2024-01-18T20:43:02.548Z] 2024-01-18 20:43:02 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610080] deletion - [2024-01-18T20:43:02.548Z] 2024-01-18 20:43:02 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610080 [2024-01-18T20:43:02.548Z] INFO[2024-01-18 20:43:02] Verifying : Description : Deleting the user [testautouser1705610080] [2024-01-18T20:43:02.548Z] INFO[2024-01-18 20:43:02] Actual:true, Expected: true [2024-01-18T20:43:03.920Z] 2024-01-18 20:43:03 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610148 - fd5df611-7f40-4420-b893-841433098e63 [2024-01-18T20:43:04.176Z] 2024-01-18 20:43:04 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610148] deletion - [2024-01-18T20:43:04.176Z] 2024-01-18 20:43:04 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610148 [2024-01-18T20:43:04.176Z] INFO[2024-01-18 20:43:04] Verifying : Description : Deleting the user [testautouser1705610148] [2024-01-18T20:43:04.176Z] INFO[2024-01-18 20:43:04] Actual:true, Expected: true [2024-01-18T20:43:05.541Z] 2024-01-18 20:43:05 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610086 - f654ad17-86b0-44b5-8531-ff4e2b3d0d8e [2024-01-18T20:43:05.797Z] 2024-01-18 20:43:05 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610086] deletion - [2024-01-18T20:43:05.797Z] 2024-01-18 20:43:05 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610086 [2024-01-18T20:43:05.797Z] INFO[2024-01-18 20:43:05] Verifying : Description : Deleting the user [testautouser1705610086] [2024-01-18T20:43:05.797Z] INFO[2024-01-18 20:43:05] Actual:true, Expected: true [2024-01-18T20:43:07.165Z] 2024-01-18 20:43:06 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610098 - 1c5a50cc-50fe-497f-b1df-cbcbf250af9e [2024-01-18T20:43:07.421Z] 2024-01-18 20:43:07 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610098] deletion - [2024-01-18T20:43:07.421Z] 2024-01-18 20:43:07 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610098 [2024-01-18T20:43:07.421Z] INFO[2024-01-18 20:43:07] Verifying : Description : Deleting the user [testautouser1705610098] [2024-01-18T20:43:07.421Z] INFO[2024-01-18 20:43:07] Actual:true, Expected: true [2024-01-18T20:43:08.786Z] 2024-01-18 20:43:08 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610100 - 86a2a183-4d11-48ed-9a75-7567203642e2 [2024-01-18T20:43:09.043Z] 2024-01-18 20:43:08 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610100] deletion - [2024-01-18T20:43:09.043Z] 2024-01-18 20:43:08 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610100 [2024-01-18T20:43:09.043Z] INFO[2024-01-18 20:43:08] Verifying : Description : Deleting the user [testautouser1705610100] [2024-01-18T20:43:09.043Z] INFO[2024-01-18 20:43:08] Actual:true, Expected: true [2024-01-18T20:43:10.407Z] 2024-01-18 20:43:09 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610106 - b5943e13-aa12-462c-b394-a1e848dc62ce [2024-01-18T20:43:10.663Z] 2024-01-18 20:43:10 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610106] deletion - [2024-01-18T20:43:10.663Z] 2024-01-18 20:43:10 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610106 [2024-01-18T20:43:10.663Z] INFO[2024-01-18 20:43:10] Verifying : Description : Deleting the user [testautouser1705610106] [2024-01-18T20:43:10.663Z] INFO[2024-01-18 20:43:10] Actual:true, Expected: true [2024-01-18T20:43:12.029Z] 2024-01-18 20:43:11 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610142 - cbe78ed9-5980-4870-8b97-2fe3e9a7fde7 [2024-01-18T20:43:12.284Z] 2024-01-18 20:43:12 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610142] deletion - [2024-01-18T20:43:12.284Z] 2024-01-18 20:43:12 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610142 [2024-01-18T20:43:12.284Z] INFO[2024-01-18 20:43:12] Verifying : Description : Deleting the user [testautouser1705610142] [2024-01-18T20:43:12.284Z] INFO[2024-01-18 20:43:12] Actual:true, Expected: true [2024-01-18T20:43:13.649Z] 2024-01-18 20:43:13 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610042 - 025770b5-cd44-4ed6-a764-53655a42d7bd [2024-01-18T20:43:13.912Z] 2024-01-18 20:43:13 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610042] deletion - [2024-01-18T20:43:13.912Z] 2024-01-18 20:43:13 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610042 [2024-01-18T20:43:13.912Z] INFO[2024-01-18 20:43:13] Verifying : Description : Deleting the user [testautouser1705610042] [2024-01-18T20:43:13.912Z] INFO[2024-01-18 20:43:13] Actual:true, Expected: true [2024-01-18T20:43:14.844Z] 2024-01-18 20:43:14 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610056 - 49f2c80a-ce57-44ac-a3e3-e7030e5a467e [2024-01-18T20:43:15.406Z] 2024-01-18 20:43:15 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610056] deletion - [2024-01-18T20:43:15.406Z] 2024-01-18 20:43:15 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610056 [2024-01-18T20:43:15.406Z] INFO[2024-01-18 20:43:15] Verifying : Description : Deleting the user [testautouser1705610056] [2024-01-18T20:43:15.406Z] INFO[2024-01-18 20:43:15] Actual:true, Expected: true [2024-01-18T20:43:16.770Z] 2024-01-18 20:43:16 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610146 - 1fa99470-8321-4701-bdfe-5ce6c90d70c3 [2024-01-18T20:43:17.026Z] 2024-01-18 20:43:16 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610146] deletion - [2024-01-18T20:43:17.026Z] 2024-01-18 20:43:16 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610146 [2024-01-18T20:43:17.026Z] INFO[2024-01-18 20:43:16] Verifying : Description : Deleting the user [testautouser1705610146] [2024-01-18T20:43:17.026Z] INFO[2024-01-18 20:43:16] Actual:true, Expected: true [2024-01-18T20:43:18.394Z] 2024-01-18 20:43:17 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610018 - 48d8a9a5-648d-46d3-b1a6-3e27c16d3a2e [2024-01-18T20:43:18.649Z] 2024-01-18 20:43:18 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610018] deletion - [2024-01-18T20:43:18.649Z] 2024-01-18 20:43:18 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610018 [2024-01-18T20:43:18.649Z] INFO[2024-01-18 20:43:18] Verifying : Description : Deleting the user [testautouser1705610018] [2024-01-18T20:43:18.650Z] INFO[2024-01-18 20:43:18] Actual:true, Expected: true [2024-01-18T20:43:20.013Z] 2024-01-18 20:43:19 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610082 - 52acc498-f1df-4642-b478-a947802d85a7 [2024-01-18T20:43:20.269Z] 2024-01-18 20:43:20 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610082] deletion - [2024-01-18T20:43:20.269Z] 2024-01-18 20:43:20 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610082 [2024-01-18T20:43:20.269Z] INFO[2024-01-18 20:43:20] Verifying : Description : Deleting the user [testautouser1705610082] [2024-01-18T20:43:20.269Z] INFO[2024-01-18 20:43:20] Actual:true, Expected: true [2024-01-18T20:43:21.632Z] 2024-01-18 20:43:21 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610116 - 7b0e6957-5a2a-4058-af4e-eea69261a3ea [2024-01-18T20:43:21.889Z] 2024-01-18 20:43:21 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610116] deletion - [2024-01-18T20:43:21.889Z] 2024-01-18 20:43:21 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610116 [2024-01-18T20:43:21.889Z] INFO[2024-01-18 20:43:21] Verifying : Description : Deleting the user [testautouser1705610116] [2024-01-18T20:43:21.889Z] INFO[2024-01-18 20:43:21] Actual:true, Expected: true [2024-01-18T20:43:22.818Z] 2024-01-18 20:43:22 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610120 - b7cd24fd-cdb4-493b-b45f-d8130b181007 [2024-01-18T20:43:23.379Z] 2024-01-18 20:43:23 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610120] deletion - [2024-01-18T20:43:23.379Z] 2024-01-18 20:43:23 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610120 [2024-01-18T20:43:23.379Z] INFO[2024-01-18 20:43:23] Verifying : Description : Deleting the user [testautouser1705610120] [2024-01-18T20:43:23.379Z] INFO[2024-01-18 20:43:23] Actual:true, Expected: true [2024-01-18T20:43:24.746Z] 2024-01-18 20:43:24 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610006 - e67bd13c-0032-4115-8a8e-fb7a4bb412ac [2024-01-18T20:43:25.003Z] 2024-01-18 20:43:24 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610006] deletion - [2024-01-18T20:43:25.003Z] 2024-01-18 20:43:24 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610006 [2024-01-18T20:43:25.003Z] INFO[2024-01-18 20:43:24] Verifying : Description : Deleting the user [testautouser1705610006] [2024-01-18T20:43:25.003Z] INFO[2024-01-18 20:43:24] Actual:true, Expected: true [2024-01-18T20:43:26.367Z] 2024-01-18 20:43:26 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610028 - e08f60f9-2318-418d-a57f-bbf03280c55d [2024-01-18T20:43:26.625Z] 2024-01-18 20:43:26 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610028] deletion - [2024-01-18T20:43:26.625Z] 2024-01-18 20:43:26 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610028 [2024-01-18T20:43:26.625Z] INFO[2024-01-18 20:43:26] Verifying : Description : Deleting the user [testautouser1705610028] [2024-01-18T20:43:26.625Z] INFO[2024-01-18 20:43:26] Actual:true, Expected: true [2024-01-18T20:43:27.987Z] 2024-01-18 20:43:27 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610034 - 98db2477-47c0-4182-b229-a7801d11b9df [2024-01-18T20:43:28.243Z] 2024-01-18 20:43:28 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610034] deletion - [2024-01-18T20:43:28.243Z] 2024-01-18 20:43:28 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610034 [2024-01-18T20:43:28.243Z] INFO[2024-01-18 20:43:28] Verifying : Description : Deleting the user [testautouser1705610034] [2024-01-18T20:43:28.243Z] INFO[2024-01-18 20:43:28] Actual:true, Expected: true [2024-01-18T20:43:29.609Z] 2024-01-18 20:43:29 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610052 - dc9577da-177e-4d7d-a282-38db3d1d912e [2024-01-18T20:43:29.864Z] 2024-01-18 20:43:29 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610052] deletion - [2024-01-18T20:43:29.864Z] 2024-01-18 20:43:29 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610052 [2024-01-18T20:43:29.864Z] INFO[2024-01-18 20:43:29] Verifying : Description : Deleting the user [testautouser1705610052] [2024-01-18T20:43:29.864Z] INFO[2024-01-18 20:43:29] Actual:true, Expected: true [2024-01-18T20:43:31.244Z] 2024-01-18 20:43:30 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610110 - 1a809db7-d012-4542-b732-fc3839876092 [2024-01-18T20:43:31.499Z] 2024-01-18 20:43:31 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610110] deletion - [2024-01-18T20:43:31.500Z] 2024-01-18 20:43:31 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610110 [2024-01-18T20:43:31.500Z] INFO[2024-01-18 20:43:31] Verifying : Description : Deleting the user [testautouser1705610110] [2024-01-18T20:43:31.500Z] INFO[2024-01-18 20:43:31] Actual:true, Expected: true [2024-01-18T20:43:32.430Z] 2024-01-18 20:43:32 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610128 - e0c2beaf-bb4b-4438-b618-3e34c39c5340 [2024-01-18T20:43:32.997Z] 2024-01-18 20:43:32 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610128] deletion - [2024-01-18T20:43:32.997Z] 2024-01-18 20:43:32 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610128 [2024-01-18T20:43:32.997Z] INFO[2024-01-18 20:43:32] Verifying : Description : Deleting the user [testautouser1705610128] [2024-01-18T20:43:32.997Z] INFO[2024-01-18 20:43:32] Actual:true, Expected: true [2024-01-18T20:43:34.362Z] 2024-01-18 20:43:33 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610012 - 84b1f4be-bffc-4566-8592-4faeb1e00fd9 [2024-01-18T20:43:34.617Z] 2024-01-18 20:43:34 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610012] deletion - [2024-01-18T20:43:34.617Z] 2024-01-18 20:43:34 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610012 [2024-01-18T20:43:34.617Z] INFO[2024-01-18 20:43:34] Verifying : Description : Deleting the user [testautouser1705610012] [2024-01-18T20:43:34.617Z] INFO[2024-01-18 20:43:34] Actual:true, Expected: true [2024-01-18T20:43:35.981Z] 2024-01-18 20:43:35 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610026 - a0235953-6fbc-4765-bf99-d076797582a4 [2024-01-18T20:43:36.239Z] 2024-01-18 20:43:36 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610026] deletion - [2024-01-18T20:43:36.239Z] 2024-01-18 20:43:36 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610026 [2024-01-18T20:43:36.239Z] INFO[2024-01-18 20:43:36] Verifying : Description : Deleting the user [testautouser1705610026] [2024-01-18T20:43:36.239Z] INFO[2024-01-18 20:43:36] Actual:true, Expected: true [2024-01-18T20:43:37.606Z] 2024-01-18 20:43:37 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610036 - 2715aeee-1103-42ed-8c0e-dbbe725053c9 [2024-01-18T20:43:37.862Z] 2024-01-18 20:43:37 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610036] deletion - [2024-01-18T20:43:37.862Z] 2024-01-18 20:43:37 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610036 [2024-01-18T20:43:37.862Z] INFO[2024-01-18 20:43:37] Verifying : Description : Deleting the user [testautouser1705610036] [2024-01-18T20:43:37.862Z] INFO[2024-01-18 20:43:37] Actual:true, Expected: true [2024-01-18T20:43:39.225Z] 2024-01-18 20:43:38 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610050 - fee7a61a-629a-4226-8cc9-f492a615fcf3 [2024-01-18T20:43:39.483Z] 2024-01-18 20:43:39 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610050] deletion - [2024-01-18T20:43:39.483Z] 2024-01-18 20:43:39 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610050 [2024-01-18T20:43:39.483Z] INFO[2024-01-18 20:43:39] Verifying : Description : Deleting the user [testautouser1705610050] [2024-01-18T20:43:39.483Z] INFO[2024-01-18 20:43:39] Actual:true, Expected: true [2024-01-18T20:43:40.409Z] 2024-01-18 20:43:40 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610140 - 4d6b2481-e9c8-4fec-9559-da3d377fb702 [2024-01-18T20:43:40.970Z] 2024-01-18 20:43:40 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610140] deletion - [2024-01-18T20:43:40.970Z] 2024-01-18 20:43:40 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610140 [2024-01-18T20:43:40.970Z] INFO[2024-01-18 20:43:40] Verifying : Description : Deleting the user [testautouser1705610140] [2024-01-18T20:43:40.970Z] INFO[2024-01-18 20:43:40] Actual:true, Expected: true [2024-01-18T20:43:42.334Z] 2024-01-18 20:43:41 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610044 - 7977b2f1-3558-4162-b756-4310234b51aa [2024-01-18T20:43:42.589Z] 2024-01-18 20:43:42 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610044] deletion - [2024-01-18T20:43:42.589Z] 2024-01-18 20:43:42 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610044 [2024-01-18T20:43:42.589Z] INFO[2024-01-18 20:43:42] Verifying : Description : Deleting the user [testautouser1705610044] [2024-01-18T20:43:42.589Z] INFO[2024-01-18 20:43:42] Actual:true, Expected: true [2024-01-18T20:43:43.955Z] 2024-01-18 20:43:43 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610058 - 16bde0ab-8f32-4dae-9f70-7c4ea3b5edeb [2024-01-18T20:43:44.213Z] 2024-01-18 20:43:44 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610058] deletion - [2024-01-18T20:43:44.213Z] 2024-01-18 20:43:44 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610058 [2024-01-18T20:43:44.213Z] INFO[2024-01-18 20:43:44] Verifying : Description : Deleting the user [testautouser1705610058] [2024-01-18T20:43:44.213Z] INFO[2024-01-18 20:43:44] Actual:true, Expected: true [2024-01-18T20:43:45.580Z] 2024-01-18 20:43:45 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610066 - 83bd9819-043f-4466-8f05-1588d1d7a463 [2024-01-18T20:43:45.836Z] 2024-01-18 20:43:45 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610066] deletion - [2024-01-18T20:43:45.836Z] 2024-01-18 20:43:45 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610066 [2024-01-18T20:43:45.836Z] INFO[2024-01-18 20:43:45] Verifying : Description : Deleting the user [testautouser1705610066] [2024-01-18T20:43:45.836Z] INFO[2024-01-18 20:43:45] Actual:true, Expected: true [2024-01-18T20:43:47.204Z] 2024-01-18 20:43:46 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610084 - 7e1b5ce9-b6e4-4cf6-9e1f-723066b86fe6 [2024-01-18T20:43:47.461Z] 2024-01-18 20:43:47 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610084] deletion - [2024-01-18T20:43:47.461Z] 2024-01-18 20:43:47 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610084 [2024-01-18T20:43:47.461Z] INFO[2024-01-18 20:43:47] Verifying : Description : Deleting the user [testautouser1705610084] [2024-01-18T20:43:47.461Z] INFO[2024-01-18 20:43:47] Actual:true, Expected: true [2024-01-18T20:43:48.823Z] 2024-01-18 20:43:48 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610118 - adf61788-400d-4b0d-9d25-0ed58cf00e61 [2024-01-18T20:43:49.079Z] 2024-01-18 20:43:48 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610118] deletion - [2024-01-18T20:43:49.079Z] 2024-01-18 20:43:48 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610118 [2024-01-18T20:43:49.079Z] INFO[2024-01-18 20:43:48] Verifying : Description : Deleting the user [testautouser1705610118] [2024-01-18T20:43:49.080Z] INFO[2024-01-18 20:43:48] Actual:true, Expected: true [2024-01-18T20:43:50.013Z] 2024-01-18 20:43:49 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610024 - 878b64e4-b6cd-464c-888e-e6ffc9769335 [2024-01-18T20:43:50.574Z] 2024-01-18 20:43:50 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610024] deletion - [2024-01-18T20:43:50.574Z] 2024-01-18 20:43:50 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610024 [2024-01-18T20:43:50.574Z] INFO[2024-01-18 20:43:50] Verifying : Description : Deleting the user [testautouser1705610024] [2024-01-18T20:43:50.575Z] INFO[2024-01-18 20:43:50] Actual:true, Expected: true [2024-01-18T20:43:51.939Z] 2024-01-18 20:43:51 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610072 - a643dea8-aed3-4ab6-912b-7f4fa9d485d6 [2024-01-18T20:43:52.199Z] 2024-01-18 20:43:52 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610072] deletion - [2024-01-18T20:43:52.199Z] 2024-01-18 20:43:52 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610072 [2024-01-18T20:43:52.199Z] INFO[2024-01-18 20:43:52] Verifying : Description : Deleting the user [testautouser1705610072] [2024-01-18T20:43:52.199Z] INFO[2024-01-18 20:43:52] Actual:true, Expected: true [2024-01-18T20:43:53.563Z] 2024-01-18 20:43:53 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610078 - 28750b2c-002f-4d4e-8dda-62436d29ac85 [2024-01-18T20:43:53.820Z] 2024-01-18 20:43:53 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610078] deletion - [2024-01-18T20:43:53.820Z] 2024-01-18 20:43:53 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610078 [2024-01-18T20:43:53.821Z] INFO[2024-01-18 20:43:53] Verifying : Description : Deleting the user [testautouser1705610078] [2024-01-18T20:43:53.821Z] INFO[2024-01-18 20:43:53] Actual:true, Expected: true [2024-01-18T20:43:55.211Z] 2024-01-18 20:43:54 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610092 - a44ce439-7030-4050-a318-52347f806d6b [2024-01-18T20:43:55.467Z] 2024-01-18 20:43:55 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610092] deletion - [2024-01-18T20:43:55.467Z] 2024-01-18 20:43:55 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610092 [2024-01-18T20:43:55.467Z] INFO[2024-01-18 20:43:55] Verifying : Description : Deleting the user [testautouser1705610092] [2024-01-18T20:43:55.467Z] INFO[2024-01-18 20:43:55] Actual:true, Expected: true [2024-01-18T20:43:56.830Z] 2024-01-18 20:43:56 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610102 - ce28004a-64b3-4088-bfd8-42a5e0014c08 [2024-01-18T20:43:57.086Z] 2024-01-18 20:43:56 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610102] deletion - [2024-01-18T20:43:57.086Z] 2024-01-18 20:43:56 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610102 [2024-01-18T20:43:57.086Z] INFO[2024-01-18 20:43:56] Verifying : Description : Deleting the user [testautouser1705610102] [2024-01-18T20:43:57.086Z] INFO[2024-01-18 20:43:56] Actual:true, Expected: true [2024-01-18T20:43:58.018Z] 2024-01-18 20:43:57 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610132 - 8efaf076-1d80-4b56-a7be-7e57a5929a98 [2024-01-18T20:43:58.585Z] 2024-01-18 20:43:58 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610132] deletion - [2024-01-18T20:43:58.585Z] 2024-01-18 20:43:58 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610132 [2024-01-18T20:43:58.585Z] INFO[2024-01-18 20:43:58] Verifying : Description : Deleting the user [testautouser1705610132] [2024-01-18T20:43:58.585Z] INFO[2024-01-18 20:43:58] Actual:true, Expected: true [2024-01-18T20:43:59.954Z] 2024-01-18 20:43:59 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610032 - c434d6dd-e59a-47b7-b2e6-ff0cafe49b99 [2024-01-18T20:44:00.209Z] 2024-01-18 20:44:00 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610032] deletion - [2024-01-18T20:44:00.210Z] 2024-01-18 20:44:00 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610032 [2024-01-18T20:44:00.210Z] INFO[2024-01-18 20:44:00] Verifying : Description : Deleting the user [testautouser1705610032] [2024-01-18T20:44:00.210Z] INFO[2024-01-18 20:44:00] Actual:true, Expected: true [2024-01-18T20:44:01.576Z] 2024-01-18 20:44:01 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610054 - 916ea4a6-ede1-4c37-89fe-3c788488ae3f [2024-01-18T20:44:01.832Z] 2024-01-18 20:44:01 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610054] deletion - [2024-01-18T20:44:01.832Z] 2024-01-18 20:44:01 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610054 [2024-01-18T20:44:01.833Z] INFO[2024-01-18 20:44:01] Verifying : Description : Deleting the user [testautouser1705610054] [2024-01-18T20:44:01.833Z] INFO[2024-01-18 20:44:01] Actual:true, Expected: true [2024-01-18T20:44:03.207Z] 2024-01-18 20:44:02 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610108 - 7a591a81-146b-4d14-b89e-d3412b6f1370 [2024-01-18T20:44:03.463Z] 2024-01-18 20:44:03 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610108] deletion - [2024-01-18T20:44:03.463Z] 2024-01-18 20:44:03 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610108 [2024-01-18T20:44:03.463Z] INFO[2024-01-18 20:44:03] Verifying : Description : Deleting the user [testautouser1705610108] [2024-01-18T20:44:03.463Z] INFO[2024-01-18 20:44:03] Actual:true, Expected: true [2024-01-18T20:44:04.835Z] 2024-01-18 20:44:04 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610114 - 940dc0e4-2d22-4490-8925-91cc61ec31e9 [2024-01-18T20:44:05.092Z] 2024-01-18 20:44:04 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610114] deletion - [2024-01-18T20:44:05.092Z] 2024-01-18 20:44:04 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610114 [2024-01-18T20:44:05.092Z] INFO[2024-01-18 20:44:04] Verifying : Description : Deleting the user [testautouser1705610114] [2024-01-18T20:44:05.092Z] INFO[2024-01-18 20:44:04] Actual:true, Expected: true [2024-01-18T20:44:06.454Z] 2024-01-18 20:44:06 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610124 - 30c2883a-9b23-451c-a9fe-4473e94c8371 [2024-01-18T20:44:06.710Z] 2024-01-18 20:44:06 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610124] deletion - [2024-01-18T20:44:06.710Z] 2024-01-18 20:44:06 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610124 [2024-01-18T20:44:06.710Z] INFO[2024-01-18 20:44:06] Verifying : Description : Deleting the user [testautouser1705610124] [2024-01-18T20:44:06.710Z] INFO[2024-01-18 20:44:06] Actual:true, Expected: true [2024-01-18T20:44:07.635Z] 2024-01-18 20:44:07 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610156 - 3477536c-ee63-4f69-b374-34452710f43b [2024-01-18T20:44:08.197Z] 2024-01-18 20:44:08 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610156] deletion - [2024-01-18T20:44:08.197Z] 2024-01-18 20:44:08 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610156 [2024-01-18T20:44:08.197Z] INFO[2024-01-18 20:44:08] Verifying : Description : Deleting the user [testautouser1705610156] [2024-01-18T20:44:08.197Z] INFO[2024-01-18 20:44:08] Actual:true, Expected: true [2024-01-18T20:44:09.583Z] 2024-01-18 20:44:09 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610160 - 11434e64-4fc8-4485-9e1f-adfe328c0df6 [2024-01-18T20:44:09.838Z] 2024-01-18 20:44:09 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610160] deletion - [2024-01-18T20:44:09.838Z] 2024-01-18 20:44:09 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610160 [2024-01-18T20:44:09.838Z] INFO[2024-01-18 20:44:09] Verifying : Description : Deleting the user [testautouser1705610160] [2024-01-18T20:44:09.838Z] INFO[2024-01-18 20:44:09] Actual:true, Expected: true [2024-01-18T20:44:11.202Z] 2024-01-18 20:44:10 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610004 - 5c93ae4c-682e-4b77-92e8-36dcf764a6ab [2024-01-18T20:44:11.458Z] 2024-01-18 20:44:11 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610004] deletion - [2024-01-18T20:44:11.458Z] 2024-01-18 20:44:11 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610004 [2024-01-18T20:44:11.458Z] INFO[2024-01-18 20:44:11] Verifying : Description : Deleting the user [testautouser1705610004] [2024-01-18T20:44:11.458Z] INFO[2024-01-18 20:44:11] Actual:true, Expected: true [2024-01-18T20:44:12.824Z] 2024-01-18 20:44:12 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610022 - c3b79415-4826-451a-be4d-707b5011e3f7 [2024-01-18T20:44:13.080Z] 2024-01-18 20:44:12 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610022] deletion - [2024-01-18T20:44:13.080Z] 2024-01-18 20:44:12 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610022 [2024-01-18T20:44:13.080Z] INFO[2024-01-18 20:44:12] Verifying : Description : Deleting the user [testautouser1705610022] [2024-01-18T20:44:13.080Z] INFO[2024-01-18 20:44:12] Actual:true, Expected: true [2024-01-18T20:44:14.009Z] 2024-01-18 20:44:13 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610064 - 0e589b35-6b8e-4ff7-86b4-f7483b6b44bb [2024-01-18T20:44:14.569Z] 2024-01-18 20:44:14 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610064] deletion - [2024-01-18T20:44:14.569Z] 2024-01-18 20:44:14 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610064 [2024-01-18T20:44:14.569Z] INFO[2024-01-18 20:44:14] Verifying : Description : Deleting the user [testautouser1705610064] [2024-01-18T20:44:14.569Z] INFO[2024-01-18 20:44:14] Actual:true, Expected: true [2024-01-18T20:44:15.933Z] 2024-01-18 20:44:15 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610076 - fb6341a7-629a-437f-93ac-b3770e24820b [2024-01-18T20:44:16.190Z] 2024-01-18 20:44:16 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610076] deletion - [2024-01-18T20:44:16.190Z] 2024-01-18 20:44:16 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610076 [2024-01-18T20:44:16.190Z] INFO[2024-01-18 20:44:16] Verifying : Description : Deleting the user [testautouser1705610076] [2024-01-18T20:44:16.190Z] INFO[2024-01-18 20:44:16] Actual:true, Expected: true [2024-01-18T20:44:17.559Z] 2024-01-18 20:44:17 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610104 - 901ad539-24a5-4a25-8daa-2d47e6bd68c3 [2024-01-18T20:44:17.815Z] 2024-01-18 20:44:17 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610104] deletion - [2024-01-18T20:44:17.815Z] 2024-01-18 20:44:17 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610104 [2024-01-18T20:44:17.815Z] INFO[2024-01-18 20:44:17] Verifying : Description : Deleting the user [testautouser1705610104] [2024-01-18T20:44:17.815Z] INFO[2024-01-18 20:44:17] Actual:true, Expected: true [2024-01-18T20:44:19.178Z] 2024-01-18 20:44:18 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610130 - 917d6368-07a2-4e0b-8a2d-fba8ff42324d [2024-01-18T20:44:19.434Z] 2024-01-18 20:44:19 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610130] deletion - [2024-01-18T20:44:19.434Z] 2024-01-18 20:44:19 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610130 [2024-01-18T20:44:19.434Z] INFO[2024-01-18 20:44:19] Verifying : Description : Deleting the user [testautouser1705610130] [2024-01-18T20:44:19.434Z] INFO[2024-01-18 20:44:19] Actual:true, Expected: true [2024-01-18T20:44:20.795Z] 2024-01-18 20:44:20 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610136 - 7ba4083a-8907-440c-a286-54606b1f2c19 [2024-01-18T20:44:21.050Z] 2024-01-18 20:44:20 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610136] deletion - [2024-01-18T20:44:21.050Z] 2024-01-18 20:44:20 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610136 [2024-01-18T20:44:21.050Z] INFO[2024-01-18 20:44:20] Verifying : Description : Deleting the user [testautouser1705610136] [2024-01-18T20:44:21.050Z] INFO[2024-01-18 20:44:20] Actual:true, Expected: true [2024-01-18T20:44:22.415Z] 2024-01-18 20:44:21 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610008 - fd9cfa2f-6ba4-40a1-a0f1-5c111aa0a5e8 [2024-01-18T20:44:22.671Z] 2024-01-18 20:44:22 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610008] deletion - [2024-01-18T20:44:22.671Z] 2024-01-18 20:44:22 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610008 [2024-01-18T20:44:22.671Z] INFO[2024-01-18 20:44:22] Verifying : Description : Deleting the user [testautouser1705610008] [2024-01-18T20:44:22.671Z] INFO[2024-01-18 20:44:22] Actual:true, Expected: true [2024-01-18T20:44:24.034Z] 2024-01-18 20:44:23 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610020 - 3d974540-f2fb-4df5-becb-1b26936765f6 [2024-01-18T20:44:24.290Z] 2024-01-18 20:44:24 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610020] deletion - [2024-01-18T20:44:24.290Z] 2024-01-18 20:44:24 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610020 [2024-01-18T20:44:24.290Z] INFO[2024-01-18 20:44:24] Verifying : Description : Deleting the user [testautouser1705610020] [2024-01-18T20:44:24.290Z] INFO[2024-01-18 20:44:24] Actual:true, Expected: true [2024-01-18T20:44:25.653Z] 2024-01-18 20:44:25 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610038 - 826f6706-3c7e-4be8-958b-a2f5c3cdb339 [2024-01-18T20:44:25.653Z] INFO[2024-01-18 20:44:25] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T20:44:25.653Z] INFO[2024-01-18 20:44:25] sched stats: tick interval: 18625 < 1.2s (100%) [2024-01-18T20:44:25.653Z] INFO[2024-01-18 20:44:25] sched stats: tick process durations: 18624 < 500ms (100%) [2024-01-18T20:44:25.908Z] 2024-01-18 20:44:25 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610038] deletion - [2024-01-18T20:44:25.908Z] 2024-01-18 20:44:25 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610038 [2024-01-18T20:44:25.908Z] INFO[2024-01-18 20:44:25] Verifying : Description : Deleting the user [testautouser1705610038] [2024-01-18T20:44:25.908Z] INFO[2024-01-18 20:44:25] Actual:true, Expected: true [2024-01-18T20:44:26.836Z] 2024-01-18 20:44:26 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610088 - 63b2f68b-4b2a-4bf0-b640-d28d0543ed46 [2024-01-18T20:44:27.398Z] 2024-01-18 20:44:27 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610088] deletion - [2024-01-18T20:44:27.398Z] 2024-01-18 20:44:27 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610088 [2024-01-18T20:44:27.398Z] INFO[2024-01-18 20:44:27] Verifying : Description : Deleting the user [testautouser1705610088] [2024-01-18T20:44:27.398Z] INFO[2024-01-18 20:44:27] Actual:true, Expected: true [2024-01-18T20:44:28.765Z] 2024-01-18 20:44:28 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610090 - 75e33520-f78f-4044-b520-6703df801db9 [2024-01-18T20:44:29.020Z] 2024-01-18 20:44:28 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610090] deletion - [2024-01-18T20:44:29.020Z] 2024-01-18 20:44:28 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610090 [2024-01-18T20:44:29.020Z] INFO[2024-01-18 20:44:28] Verifying : Description : Deleting the user [testautouser1705610090] [2024-01-18T20:44:29.020Z] INFO[2024-01-18 20:44:28] Actual:true, Expected: true [2024-01-18T20:44:30.388Z] 2024-01-18 20:44:29 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610094 - b23ab55c-d0c8-4cef-8d06-cb57dbc700ea [2024-01-18T20:44:30.646Z] 2024-01-18 20:44:30 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610094] deletion - [2024-01-18T20:44:30.646Z] 2024-01-18 20:44:30 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610094 [2024-01-18T20:44:30.646Z] INFO[2024-01-18 20:44:30] Verifying : Description : Deleting the user [testautouser1705610094] [2024-01-18T20:44:30.646Z] INFO[2024-01-18 20:44:30] Actual:true, Expected: true [2024-01-18T20:44:31.646Z] 2024-01-18 20:44:31 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610158 - 27657394-487e-4705-a61e-00122cda400b [2024-01-18T20:44:32.209Z] 2024-01-18 20:44:32 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610158] deletion - [2024-01-18T20:44:32.209Z] 2024-01-18 20:44:32 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610158 [2024-01-18T20:44:32.209Z] INFO[2024-01-18 20:44:32] Verifying : Description : Deleting the user [testautouser1705610158] [2024-01-18T20:44:32.209Z] INFO[2024-01-18 20:44:32] Actual:true, Expected: true [2024-01-18T20:44:33.573Z] 2024-01-18 20:44:33 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610030 - 3125668c-dc81-4c20-bfcb-71a66c9f789b [2024-01-18T20:44:33.829Z] 2024-01-18 20:44:33 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610030] deletion - [2024-01-18T20:44:33.829Z] 2024-01-18 20:44:33 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610030 [2024-01-18T20:44:33.829Z] INFO[2024-01-18 20:44:33] Verifying : Description : Deleting the user [testautouser1705610030] [2024-01-18T20:44:33.829Z] INFO[2024-01-18 20:44:33] Actual:true, Expected: true [2024-01-18T20:44:35.198Z] 2024-01-18 20:44:34 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610062 - 1054cc3c-7917-4911-a986-622200496913 [2024-01-18T20:44:35.459Z] 2024-01-18 20:44:35 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610062] deletion - [2024-01-18T20:44:35.459Z] 2024-01-18 20:44:35 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610062 [2024-01-18T20:44:35.459Z] INFO[2024-01-18 20:44:35] Verifying : Description : Deleting the user [testautouser1705610062] [2024-01-18T20:44:35.459Z] INFO[2024-01-18 20:44:35] Actual:true, Expected: true [2024-01-18T20:44:36.832Z] 2024-01-18 20:44:36 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610068 - de9bf072-d0af-427f-9e0c-5c9328a6cbed [2024-01-18T20:44:37.094Z] 2024-01-18 20:44:36 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610068] deletion - [2024-01-18T20:44:37.094Z] 2024-01-18 20:44:36 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610068 [2024-01-18T20:44:37.094Z] INFO[2024-01-18 20:44:36] Verifying : Description : Deleting the user [testautouser1705610068] [2024-01-18T20:44:37.094Z] INFO[2024-01-18 20:44:36] Actual:true, Expected: true [2024-01-18T20:44:38.029Z] 2024-01-18 20:44:37 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610070 - ff66ab06-f279-48ee-a22e-db6a50be452b [2024-01-18T20:44:38.597Z] 2024-01-18 20:44:38 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610070] deletion - [2024-01-18T20:44:38.597Z] 2024-01-18 20:44:38 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610070 [2024-01-18T20:44:38.597Z] INFO[2024-01-18 20:44:38] Verifying : Description : Deleting the user [testautouser1705610070] [2024-01-18T20:44:38.597Z] INFO[2024-01-18 20:44:38] Actual:true, Expected: true [2024-01-18T20:44:39.965Z] 2024-01-18 20:44:39 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610096 - afcf41e9-a71f-47d2-921c-b6938918f311 [2024-01-18T20:44:40.220Z] 2024-01-18 20:44:40 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610096] deletion - [2024-01-18T20:44:40.220Z] 2024-01-18 20:44:40 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610096 [2024-01-18T20:44:40.220Z] INFO[2024-01-18 20:44:40] Verifying : Description : Deleting the user [testautouser1705610096] [2024-01-18T20:44:40.220Z] INFO[2024-01-18 20:44:40] Actual:true, Expected: true [2024-01-18T20:44:41.584Z] 2024-01-18 20:44:41 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610154 - e69a5d42-cc97-47b3-9d6a-3f1464578131 [2024-01-18T20:44:41.842Z] 2024-01-18 20:44:41 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610154] deletion - [2024-01-18T20:44:41.842Z] 2024-01-18 20:44:41 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610154 [2024-01-18T20:44:41.842Z] INFO[2024-01-18 20:44:41] Verifying : Description : Deleting the user [testautouser1705610154] [2024-01-18T20:44:41.842Z] INFO[2024-01-18 20:44:41] Actual:true, Expected: true [2024-01-18T20:44:43.204Z] 2024-01-18 20:44:42 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610134 - 22af8231-a765-4a7c-b55f-a221d2db46a8 [2024-01-18T20:44:43.460Z] 2024-01-18 20:44:43 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610134] deletion - [2024-01-18T20:44:43.460Z] 2024-01-18 20:44:43 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610134 [2024-01-18T20:44:43.460Z] INFO[2024-01-18 20:44:43] Verifying : Description : Deleting the user [testautouser1705610134] [2024-01-18T20:44:43.460Z] INFO[2024-01-18 20:44:43] Actual:true, Expected: true [2024-01-18T20:44:44.825Z] 2024-01-18 20:44:44 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610138 - 55984835-105f-4f03-9f75-ae4343ad20bf [2024-01-18T20:44:45.081Z] 2024-01-18 20:44:44 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610138] deletion - [2024-01-18T20:44:45.081Z] 2024-01-18 20:44:44 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610138 [2024-01-18T20:44:45.081Z] INFO[2024-01-18 20:44:44] Verifying : Description : Deleting the user [testautouser1705610138] [2024-01-18T20:44:45.081Z] INFO[2024-01-18 20:44:44] Actual:true, Expected: true [2024-01-18T20:44:46.010Z] 2024-01-18 20:44:45 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610150 - 793d0d8f-e562-42e0-9417-12db14b278e7 [2024-01-18T20:44:46.571Z] 2024-01-18 20:44:46 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610150] deletion - [2024-01-18T20:44:46.571Z] 2024-01-18 20:44:46 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610150 [2024-01-18T20:44:46.571Z] INFO[2024-01-18 20:44:46] Verifying : Description : Deleting the user [testautouser1705610150] [2024-01-18T20:44:46.572Z] INFO[2024-01-18 20:44:46] Actual:true, Expected: true [2024-01-18T20:44:47.938Z] 2024-01-18 20:44:47 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610162 - 633ecb26-7233-4da3-9e45-62fac67885d1 [2024-01-18T20:44:48.195Z] 2024-01-18 20:44:48 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610162] deletion - [2024-01-18T20:44:48.195Z] 2024-01-18 20:44:48 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610162 [2024-01-18T20:44:48.195Z] INFO[2024-01-18 20:44:48] Verifying : Description : Deleting the user [testautouser1705610162] [2024-01-18T20:44:48.195Z] INFO[2024-01-18 20:44:48] Actual:true, Expected: true [2024-01-18T20:44:49.563Z] 2024-01-18 20:44:49 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610010 - c78a9cc3-7668-40be-9f4a-71cf45ade350 [2024-01-18T20:44:49.819Z] 2024-01-18 20:44:49 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610010] deletion - [2024-01-18T20:44:49.819Z] 2024-01-18 20:44:49 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610010 [2024-01-18T20:44:49.819Z] INFO[2024-01-18 20:44:49] Verifying : Description : Deleting the user [testautouser1705610010] [2024-01-18T20:44:49.819Z] INFO[2024-01-18 20:44:49] Actual:true, Expected: true [2024-01-18T20:44:51.190Z] 2024-01-18 20:44:50 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610016 - fbd35ec6-1a78-49ff-b7b0-74d1b338037c [2024-01-18T20:44:51.449Z] 2024-01-18 20:44:51 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610016] deletion - [2024-01-18T20:44:51.449Z] 2024-01-18 20:44:51 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610016 [2024-01-18T20:44:51.449Z] INFO[2024-01-18 20:44:51] Verifying : Description : Deleting the user [testautouser1705610016] [2024-01-18T20:44:51.449Z] INFO[2024-01-18 20:44:51] Actual:true, Expected: true [2024-01-18T20:44:52.812Z] 2024-01-18 20:44:52 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610112 - 53b39063-19e1-40a4-9693-8ddfe0db231b [2024-01-18T20:44:53.069Z] 2024-01-18 20:44:52 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610112] deletion - [2024-01-18T20:44:53.069Z] 2024-01-18 20:44:52 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610112 [2024-01-18T20:44:53.069Z] INFO[2024-01-18 20:44:52] Verifying : Description : Deleting the user [testautouser1705610112] [2024-01-18T20:44:53.069Z] INFO[2024-01-18 20:44:52] Actual:true, Expected: true [2024-01-18T20:44:54.003Z] 2024-01-18 20:44:53 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610122 - dbff47c3-0d0e-4011-8620-fc6b5336834a [2024-01-18T20:44:54.565Z] 2024-01-18 20:44:54 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610122] deletion - [2024-01-18T20:44:54.565Z] 2024-01-18 20:44:54 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610122 [2024-01-18T20:44:54.565Z] INFO[2024-01-18 20:44:54] Verifying : Description : Deleting the user [testautouser1705610122] [2024-01-18T20:44:54.565Z] INFO[2024-01-18 20:44:54] Actual:true, Expected: true [2024-01-18T20:44:55.931Z] 2024-01-18 20:44:55 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610152 - 3f864853-3b52-4c43-bf7e-6a98a067e03f [2024-01-18T20:44:56.188Z] 2024-01-18 20:44:56 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610152] deletion - [2024-01-18T20:44:56.188Z] 2024-01-18 20:44:56 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610152 [2024-01-18T20:44:56.188Z] INFO[2024-01-18 20:44:56] Verifying : Description : Deleting the user [testautouser1705610152] [2024-01-18T20:44:56.188Z] INFO[2024-01-18 20:44:56] Actual:true, Expected: true [2024-01-18T20:44:57.553Z] 2024-01-18 20:44:57 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610014 - 769b0e6a-d44f-4666-a9dd-c5820ce32ff1 [2024-01-18T20:44:57.809Z] 2024-01-18 20:44:57 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610014] deletion - [2024-01-18T20:44:57.809Z] 2024-01-18 20:44:57 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610014 [2024-01-18T20:44:57.809Z] INFO[2024-01-18 20:44:57] Verifying : Description : Deleting the user [testautouser1705610014] [2024-01-18T20:44:57.809Z] INFO[2024-01-18 20:44:57] Actual:true, Expected: true [2024-01-18T20:44:59.172Z] 2024-01-18 20:44:58 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610040 - d3ce505b-ed10-4750-82ad-3a8a689612da [2024-01-18T20:44:59.427Z] 2024-01-18 20:44:59 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610040] deletion - [2024-01-18T20:44:59.428Z] 2024-01-18 20:44:59 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610040 [2024-01-18T20:44:59.428Z] INFO[2024-01-18 20:44:59] Verifying : Description : Deleting the user [testautouser1705610040] [2024-01-18T20:44:59.428Z] INFO[2024-01-18 20:44:59] Actual:true, Expected: true [2024-01-18T20:45:00.817Z] 2024-01-18 20:45:00 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610046 - 3161dd97-111e-4e0e-9f3f-58210644d6d1 [2024-01-18T20:45:01.076Z] 2024-01-18 20:45:00 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610046] deletion - [2024-01-18T20:45:01.076Z] 2024-01-18 20:45:00 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610046 [2024-01-18T20:45:01.076Z] INFO[2024-01-18 20:45:00] Verifying : Description : Deleting the user [testautouser1705610046] [2024-01-18T20:45:01.076Z] INFO[2024-01-18 20:45:00] Actual:true, Expected: true [2024-01-18T20:45:02.443Z] 2024-01-18 20:45:01 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610048 - dde214e6-fd13-48dd-9daf-78e748ebd986 [2024-01-18T20:45:02.700Z] 2024-01-18 20:45:02 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610048] deletion - [2024-01-18T20:45:02.700Z] 2024-01-18 20:45:02 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610048 [2024-01-18T20:45:02.700Z] INFO[2024-01-18 20:45:02] Verifying : Description : Deleting the user [testautouser1705610048] [2024-01-18T20:45:02.700Z] INFO[2024-01-18 20:45:02] Actual:true, Expected: true [2024-01-18T20:45:03.637Z] 2024-01-18 20:45:03 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610060 - c368720b-b33b-409e-b031-b87377ae0121 [2024-01-18T20:45:04.199Z] 2024-01-18 20:45:04 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610060] deletion - [2024-01-18T20:45:04.199Z] 2024-01-18 20:45:04 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610060 [2024-01-18T20:45:04.199Z] INFO[2024-01-18 20:45:04] Verifying : Description : Deleting the user [testautouser1705610060] [2024-01-18T20:45:04.199Z] INFO[2024-01-18 20:45:04] Actual:true, Expected: true [2024-01-18T20:45:05.565Z] 2024-01-18 20:45:05 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610074 - 877d30b8-8b5a-4b4d-8f7c-9c64ec6cca37 [2024-01-18T20:45:05.821Z] 2024-01-18 20:45:05 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610074] deletion - [2024-01-18T20:45:05.821Z] 2024-01-18 20:45:05 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610074 [2024-01-18T20:45:05.821Z] INFO[2024-01-18 20:45:05] Verifying : Description : Deleting the user [testautouser1705610074] [2024-01-18T20:45:05.821Z] INFO[2024-01-18 20:45:05] Actual:true, Expected: true [2024-01-18T20:45:07.185Z] 2024-01-18 20:45:06 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610126 - 26113b48-5de6-4287-b996-d79c8b3d3f2c [2024-01-18T20:45:07.441Z] 2024-01-18 20:45:07 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610126] deletion - [2024-01-18T20:45:07.441Z] 2024-01-18 20:45:07 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610126 [2024-01-18T20:45:07.441Z] INFO[2024-01-18 20:45:07] Verifying : Description : Deleting the user [testautouser1705610126] [2024-01-18T20:45:07.441Z] INFO[2024-01-18 20:45:07] Actual:true, Expected: true [2024-01-18T20:45:08.805Z] 2024-01-18 20:45:08 +0000:[INFO] [{DeleteUsersRole}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705610144 - d0c9cadc-16bd-4019-9805-8f84365d646b [2024-01-18T20:45:09.061Z] 2024-01-18 20:45:08 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#755] - Response for user [testautouser1705610144] deletion - [2024-01-18T20:45:09.061Z] 2024-01-18 20:45:08 +0000:[INFO] [{DeleteUsersRole}] [backup.DeleteUser:#759] - Deleted User - testautouser1705610144 [2024-01-18T20:45:09.061Z] INFO[2024-01-18 20:45:08] Verifying : Description : Deleting the user [testautouser1705610144] [2024-01-18T20:45:09.061Z] INFO[2024-01-18 20:45:08] Actual:true, Expected: true [2024-01-18T20:45:09.621Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610114] [2024-01-18T20:45:09.621Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.621Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610124] [2024-01-18T20:45:09.621Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.621Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610156] [2024-01-18T20:45:09.621Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.621Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610160] [2024-01-18T20:45:09.621Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.621Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610032] [2024-01-18T20:45:09.621Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.621Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610054] [2024-01-18T20:45:09.621Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.621Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610108] [2024-01-18T20:45:09.621Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.621Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610076] [2024-01-18T20:45:09.621Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.621Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610104] [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610130] [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610136] [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610004] [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610022] [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610064] [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610088] [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610090] [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610094] [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610158] [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610008] [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610020] [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610038] [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610070] [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610096] [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610154] [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610030] [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.622Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610062] [2024-01-18T20:45:09.623Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.623Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610068] [2024-01-18T20:45:09.623Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.623Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610162] [2024-01-18T20:45:09.623Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.623Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610134] [2024-01-18T20:45:09.623Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.623Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610138] [2024-01-18T20:45:09.623Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.623Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610150] [2024-01-18T20:45:09.623Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.623Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610122] [2024-01-18T20:45:09.623Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.623Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610152] [2024-01-18T20:45:09.623Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.623Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610010] [2024-01-18T20:45:09.623Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.623Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610016] [2024-01-18T20:45:09.623Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.623Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610112] [2024-01-18T20:45:09.623Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.623Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610048] [2024-01-18T20:45:09.623Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.624Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610060] [2024-01-18T20:45:09.624Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.624Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610074] [2024-01-18T20:45:09.624Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.624Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610126] [2024-01-18T20:45:09.624Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.624Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610144] [2024-01-18T20:45:09.624Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.624Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610014] [2024-01-18T20:45:09.624Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.624Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610040] [2024-01-18T20:45:09.880Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.880Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610046] [2024-01-18T20:45:09.880Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.880Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610080] [2024-01-18T20:45:09.880Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.880Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610148] [2024-01-18T20:45:09.880Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.880Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610106] [2024-01-18T20:45:09.880Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.880Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610142] [2024-01-18T20:45:09.880Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.880Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610086] [2024-01-18T20:45:09.880Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.880Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610098] [2024-01-18T20:45:09.880Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.880Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610100] [2024-01-18T20:45:09.880Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.880Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610042] [2024-01-18T20:45:09.880Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610056] [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610146] [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610120] [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610018] [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610082] [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610116] [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610052] [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610110] [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610128] [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610006] [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610028] [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610034] [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610050] [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610140] [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610012] [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610026] [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610036] [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610084] [2024-01-18T20:45:09.881Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.882Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610118] [2024-01-18T20:45:09.882Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.882Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610044] [2024-01-18T20:45:09.882Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.882Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610058] [2024-01-18T20:45:09.882Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.882Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610066] [2024-01-18T20:45:09.882Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.882Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610092] [2024-01-18T20:45:09.882Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.882Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610102] [2024-01-18T20:45:09.882Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.882Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610132] [2024-01-18T20:45:09.882Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.882Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610024] [2024-01-18T20:45:09.882Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.882Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610072] [2024-01-18T20:45:09.882Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.882Z] INFO[2024-01-18 20:45:09] Verifying : Description : validation of deleted user [testautouser1705610078] [2024-01-18T20:45:09.882Z] INFO[2024-01-18 20:45:09] Actual:false, Expected: false [2024-01-18T20:45:09.882Z] INFO[2024-01-18 20:45:09] --------Test End------ [2024-01-18T20:45:09.882Z] INFO[2024-01-18 20:45:09] #Test: [2024-01-18T20:45:09.882Z] INFO[2024-01-18 20:45:09] #Description: [2024-01-18T20:45:09.882Z] INFO[2024-01-18 20:45:09] ------------------------ [2024-01-18T20:45:09.882Z] 2024-01-18 20:45:09 +0000:[INFO] [{DeleteUsersRole}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T20:45:09.882Z] 2024-01-18 20:45:09 +0000:[INFO] [{DeleteUsersRole}] Switching context: The context is already [/tmp/destination-config] [2024-01-18T20:45:10.137Z] 2024-01-18 20:45:10 +0000:[INFO] [{DeleteUsersRole}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {DeleteUsersRole} Delete user and roles [2024-01-18T20:45:10.137Z] 2024-01-18 20:45:10 +0000:[INFO] [{DeleteUsersRole}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-18T20:45:10.137Z] 2024/01/18 20:45:10 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:45:15.383Z] 2024/01/18 20:45:15 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:45:20.649Z] 2024/01/18 20:45:20 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:45:25.896Z] 2024/01/18 20:45:25 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:45:30.067Z] 2024/01/18 20:45:30 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:45:35.311Z] 2024/01/18 20:45:35 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:45:40.566Z] 2024/01/18 20:45:40 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:45:46.292Z] 2024/01/18 20:45:45 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:45:50.458Z] 2024/01/18 20:45:50 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.0.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.7 10.240.0.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-18T20:45:50.458Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-18T20:45:50.458Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-18T20:45:50.458Z] {dial tcp 10.240.0.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-18T20:45:55.700Z] 2024-01-18 20:45:55 +0000:[WARNING] [{DeleteUsersRole}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.7 10.240.0.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-18T20:45:55.700Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-18T20:45:55.700Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-18T20:45:55.700Z] {dial tcp 10.240.0.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T20:45:55.700Z] 2024-01-18 20:45:55 +0000:[ERROR] [{DeleteUsersRole}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.7 10.240.0.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-18T20:45:55.700Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-18T20:45:55.700Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-18T20:45:55.700Z] {dial tcp 10.240.0.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T20:45:55.700Z] 2024-01-18 20:45:55 +0000:[INFO] [{DeleteUsersRole}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T20:45:55.700Z] 2024-01-18 20:45:55 +0000:[INFO] [{DeleteUsersRole}] Switching context to [/tmp/source-config] [2024-01-18T20:45:55.700Z] 2024-01-18 20:45:55 +0000:[INFO] [{DeleteUsersRole}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T20:45:55.701Z] 2024-01-18 20:45:55 +0000:[INFO] [{DeleteUsersRole}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T20:45:55.701Z] 2024-01-18 20:45:55 +0000:[INFO] [{DeleteUsersRole}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T20:45:55.701Z] 2024-01-18 20:45:55 +0000:[INFO] [{DeleteUsersRole}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T20:45:55.701Z] 2024-01-18 20:45:55 +0000:[INFO] [{DeleteUsersRole}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T20:45:55.701Z] 2024-01-18 20:45:55 +0000:[INFO] [{DeleteUsersRole}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T20:45:55.956Z] 2024-01-18 20:45:55 +0000:[INFO] [{DeleteUsersRole}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T20:45:56.517Z] 2024-01-18 20:45:56 +0000:[INFO] [{DeleteUsersRole}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T20:45:56.773Z] 2024-01-18 20:45:56 +0000:[INFO] [{DeleteUsersRole}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T20:45:57.028Z] 2024/01/18 20:45:56 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:46:06.964Z] 2024/01/18 20:46:06 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:46:19.125Z] 2024/01/18 20:46:16 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:46:27.206Z] 2024/01/18 20:46:26 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:46:37.194Z] 2024/01/18 20:46:36 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:46:47.141Z] 2024/01/18 20:46:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:46:57.086Z] 2024/01/18 20:46:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:47:09.250Z] 2024/01/18 20:47:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:47:17.343Z] 2024/01/18 20:47:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:47:27.280Z] 2024/01/18 20:47:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:47:37.218Z] 2024/01/18 20:47:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:47:49.417Z] 2024/01/18 20:47:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:47:57.501Z] 2024/01/18 20:47:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:48:07.447Z] 2024/01/18 20:48:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:48:17.392Z] 2024/01/18 20:48:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:48:27.338Z] 2024/01/18 20:48:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:48:39.499Z] 2024/01/18 20:48:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:48:47.638Z] 2024/01/18 20:48:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:48:57.573Z] 2024/01/18 20:48:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:49:07.532Z] 2024/01/18 20:49:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:49:17.474Z] 2024/01/18 20:49:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:49:29.647Z] 2024/01/18 20:49:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:49:37.741Z] 2024/01/18 20:49:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:49:47.685Z] 2024/01/18 20:49:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:49:57.626Z] 2024/01/18 20:49:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:50:09.823Z] 2024/01/18 20:50:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:50:17.903Z] 2024/01/18 20:50:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:50:27.844Z] 2024/01/18 20:50:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:50:37.783Z] 2024/01/18 20:50:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:50:47.758Z] 2024/01/18 20:50:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:50:59.937Z] 2024-01-18 20:50:57 +0000:[ERROR] [{DeleteUsersRole}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.937Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.937Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.937Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.937Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.937Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.937Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.937Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.937Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.937Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.938Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.938Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.938Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.938Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.938Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.938Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.938Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.938Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.938Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.938Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.938Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.938Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.938Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.938Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.938Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.938Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.938Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.938Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.938Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.938Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-18T20:50:59.938Z] [2024-01-18T20:50:59.938Z] ------------------------------ [2024-01-18T20:50:59.938Z] • Failure [1052.962 seconds] [2024-01-18T20:50:59.938Z] {DeleteUsersRole} [2024-01-18T20:50:59.938Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:3644 [2024-01-18T20:50:59.939Z] Delete user and roles [] [2024-01-18T20:50:59.939Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:3654 [2024-01-18T20:50:59.939Z] [2024-01-18T20:50:59.939Z] Unexpected error: [2024-01-18T20:50:59.939Z] <*errors.errorString | 0xc0016cbb40>: { [2024-01-18T20:50:59.940Z] s: "failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{No... [2024-01-18T20:50:59.940Z] [2024-01-18T20:50:59.940Z] Gomega truncated this representation as it exceeds 'format.MaxLength'. [2024-01-18T20:50:59.940Z] Consider having the object provide a custom 'GomegaStringer' representation [2024-01-18T20:50:59.940Z] or adjust the parameters in Gomega's 'format' package. [2024-01-18T20:50:59.940Z] [2024-01-18T20:50:59.940Z] Learn more here: https://onsi.github.io/gomega/#adjusting-output [2024-01-18T20:50:59.940Z] [2024-01-18T20:50:59.940Z] failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.940Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.940Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.940Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.940Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.940Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.940Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.940Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.940Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.940Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.940Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.940Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.940Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.940Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.940Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.940Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.940Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.941Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.941Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.941Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.941Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.941Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.941Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.941Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.941Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.941Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.941Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.941Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.941Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T20:50:59.941Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-18T20:50:59.941Z] occurred [2024-01-18T20:50:59.941Z] [2024-01-18T20:50:59.941Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-18T20:50:59.941Z] [2024-01-18T20:50:59.941Z] Full Stack Trace [2024-01-18T20:50:59.941Z] github.com/portworx/torpedo/pkg/log.FailOnError({0x88a5060, 0xc0016cbb40}, {0x7f74bfd?, 0x6cc0320?}, {0x0?, 0x6cc0320?, 0xc0015ae1c0?}) [2024-01-18T20:50:59.941Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 +0x290 [2024-01-18T20:50:59.941Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest.func1() [2024-01-18T20:50:59.941Z] /go/src/github.com/portworx/torpedo/tests/common.go:7283 +0x32 [2024-01-18T20:50:59.941Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest({0xb924cc0, 0x0, 0x0}) [2024-01-18T20:50:59.941Z] /go/src/github.com/portworx/torpedo/tests/common.go:7299 +0x778 [2024-01-18T20:50:59.941Z] github.com/portworx/torpedo/tests/backup.glob..func95.3() [2024-01-18T20:50:59.941Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:3723 +0x76 [2024-01-18T20:50:59.941Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-18T20:50:59.941Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-18T20:50:59.941Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-18T20:50:59.941Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-18T20:50:59.941Z] created by testing.(*T).Run [2024-01-18T20:50:59.941Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-18T20:50:59.941Z] ------------------------------ [2024-01-18T20:50:59.941Z] STEP: Create Users add roles [2024-01-18T20:50:59.941Z] STEP: Delete roles from the users [2024-01-18T20:50:59.942Z] STEP: Validate if the roles are deleted from the users [2024-01-18T20:50:59.942Z] STEP: Delete users [2024-01-18T20:50:59.942Z] STEP: Validate if all the created users are deleted [2024-01-18T20:50:59.942Z] S2024-01-18 20:50:57 +0000:[INFO] [{CancelAllRunningBackupJobs}] Log Dir: /testresults//CancelAllRunningBackupJobs.log [2024-01-18T20:50:59.942Z] INFO[2024-01-18 20:50:57] --------Test Start------ [2024-01-18T20:50:59.942Z] INFO[2024-01-18 20:50:57] #Test: CancelAllRunningBackupJobs [2024-01-18T20:50:59.942Z] INFO[2024-01-18 20:50:57] #Description: Cancel all the running backup jobs while backups are in progress [2024-01-18T20:50:59.942Z] INFO[2024-01-18 20:50:57] ------------------------ [2024-01-18T20:50:59.942Z] 2024-01-18 20:50:57 +0000:[INFO] [{CancelAllRunningBackupJobs}] Deploying applications required for the testcase [2024-01-18T20:50:59.942Z] 2024-01-18 20:50:57 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T20:50:59.942Z] 2024-01-18 20:50:57 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T20:50:59.942Z] 2024-01-18 20:50:57 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T20:50:59.942Z] 2024-01-18 20:50:57 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T20:50:59.942Z] 2024-01-18 20:50:57 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T20:50:59.942Z] 2024-01-18 20:50:57 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T20:50:59.943Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T20:50:59.943Z] 2024-01-18 20:50:57 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T20:50:59.943Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T20:50:59.943Z] 2024-01-18 20:50:57 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T20:50:59.943Z] 2024-01-18 20:50:57 +0000:[INFO] [{CancelAllRunningBackupJobs}] Validating the deployed applications [2024-01-18T20:50:59.943Z] 2024-01-18 20:50:57 +0000:[INFO] [{CancelAllRunningBackupJobs}] Validate applications [2024-01-18T20:50:59.943Z] 2024-01-18 20:50:57 +0000:[INFO] [{CancelAllRunningBackupJobs}] Validating mysql-ibm app [2024-01-18T20:50:59.943Z] 2024-01-18 20:50:57 +0000:[INFO] [{CancelAllRunningBackupJobs}] Validating mysql-ibm app's volumes [2024-01-18T20:50:59.943Z] 2024-01-18 20:50:57 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:50:59.943Z] 2024/01/18 20:50:58 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T20:51:09.884Z] 2024/01/18 20:51:08 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T20:51:19.832Z] 2024-01-18 20:51:18 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-58045-01-18-20h50m57s [2024-01-18T20:51:19.832Z] 2024-01-18 20:51:18 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T20:51:19.832Z] 2024-01-18 20:51:18 +0000:[WARNING] [{CancelAllRunningBackupJobs}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T20:51:19.832Z] 2024-01-18 20:51:18 +0000:[INFO] [{CancelAllRunningBackupJobs}] wait for mysql-ibm app to start running [2024-01-18T20:51:19.832Z] 2024-01-18 20:51:18 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T20:51:19.832Z] 2024/01/18 20:51:18 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:51:19.832Z] pod name:mysql-7d766cd65c-r6dq5 namespace:mysql-ibm-pxb-0-58045-01-18-20h50m57s running:false ready:false node:10.240.64.27 [2024-01-18T20:51:19.832Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:51:29.786Z] 2024/01/18 20:51:28 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:51:29.786Z] pod name:mysql-7d766cd65c-r6dq5 namespace:mysql-ibm-pxb-0-58045-01-18-20h50m57s running:true ready:false node:10.240.64.27 [2024-01-18T20:51:29.786Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:51:39.762Z] 2024/01/18 20:51:38 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:51:39.762Z] pod name:mysql-7d766cd65c-r6dq5 namespace:mysql-ibm-pxb-0-58045-01-18-20h50m57s running:true ready:false node:10.240.64.27 [2024-01-18T20:51:39.762Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:51:49.714Z] 2024/01/18 20:51:48 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:51:49.714Z] pod name:mysql-7d766cd65c-r6dq5 namespace:mysql-ibm-pxb-0-58045-01-18-20h50m57s running:true ready:false node:10.240.64.27 [2024-01-18T20:51:49.714Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:51:59.702Z] 2024/01/18 20:51:58 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:51:59.702Z] pod name:mysql-7d766cd65c-r6dq5 namespace:mysql-ibm-pxb-0-58045-01-18-20h50m57s running:true ready:false node:10.240.64.27 [2024-01-18T20:51:59.702Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:52:09.645Z] 2024/01/18 20:52:08 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:52:09.645Z] pod name:mysql-7d766cd65c-r6dq5 namespace:mysql-ibm-pxb-0-58045-01-18-20h50m57s running:true ready:false node:10.240.64.27 [2024-01-18T20:52:09.645Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:52:19.628Z] 2024/01/18 20:52:18 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:52:19.628Z] pod name:mysql-7d766cd65c-r6dq5 namespace:mysql-ibm-pxb-0-58045-01-18-20h50m57s running:true ready:false node:10.240.64.27 [2024-01-18T20:52:19.628Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:52:29.577Z] 2024/01/18 20:52:28 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T20:52:29.577Z] pod name:mysql-7d766cd65c-r6dq5 namespace:mysql-ibm-pxb-0-58045-01-18-20h50m57s running:true ready:false node:10.240.64.27 [2024-01-18T20:52:29.577Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T20:52:39.523Z] 2024-01-18 20:52:38 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T20:52:39.523Z] 2024-01-18 20:52:39 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T20:52:39.523Z] 2024-01-18 20:52:39 +0000:[INFO] [{CancelAllRunningBackupJobs}] validate if mysql-ibm app's volumes are setup [2024-01-18T20:52:39.523Z] 2024-01-18 20:52:39 +0000:[WARNING] [{CancelAllRunningBackupJobs}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T20:52:39.523Z] 2024-01-18 20:52:39 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:52:39.523Z] 2024-01-18 20:52:39 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T20:52:39.523Z] 2024-01-18 20:52:39 +0000:[WARNING] [{CancelAllRunningBackupJobs}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T20:52:39.523Z] 2024-01-18 20:52:39 +0000:[INFO] [{CancelAllRunningBackupJobs}] Adding cloud credential and backup location [2024-01-18T20:52:39.779Z] 2024-01-18 20:52:39 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [cloudcred-aws-1705611159] for org [default] with [aws] as provider [2024-01-18T20:52:39.779Z] 2024-01-18 20:52:39 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-18T20:52:39.779Z] 2024-01-18 20:52:39 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cloudcred-aws-1705611159] is present [2024-01-18T20:52:39.779Z] INFO[2024-01-18 20:52:39] Verifying : Description : Verifying creation of cloud credential named [cloudcred-aws-1705611159] for org [default] with [aws] as provider [2024-01-18T20:52:39.780Z] INFO[2024-01-18 20:52:39] Actual:true, Expected: true [2024-01-18T20:52:43.943Z] INFO[2024-01-18 20:52:43] Verifying : Description : Creating backup location aws-global-aws-px-backup-system-test-4cac76dc-1705611159-bl [2024-01-18T20:52:43.943Z] INFO[2024-01-18 20:52:43] Actual:true, Expected: true [2024-01-18T20:52:43.943Z] 2024-01-18 20:52:43 +0000:[INFO] [{CancelAllRunningBackupJobs}] Registering source and destination clusters for backup [2024-01-18T20:52:44.201Z] INFO[2024-01-18 20:52:44] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T20:52:44.201Z] INFO[2024-01-18 20:52:44] Actual:true, Expected: true [2024-01-18T20:52:44.201Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T20:52:44.201Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T20:52:44.201Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T20:52:44.201Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T20:52:44.201Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] Create cluster [source-cluster] in org [default] [2024-01-18T20:52:44.201Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T20:52:44.201Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T20:52:44.201Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] Create cluster [destination-cluster] in org [default] [2024-01-18T20:52:44.201Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T20:52:44.201Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T20:52:44.201Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-ckjxv] for IBM [2024-01-18T20:52:44.201Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-ckjxv] for org [default] with [ibm] as provider [2024-01-18T20:52:44.201Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T20:52:44.201Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T20:52:44.459Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-ckjxv] is present [2024-01-18T20:52:44.459Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] Create cluster [source-cluster] in org [default] [2024-01-18T20:52:44.459Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-qprlh] for IBM [2024-01-18T20:52:44.459Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-qprlh] for org [default] with [ibm] as provider [2024-01-18T20:52:44.459Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T20:52:44.459Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T20:52:44.459Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-qprlh] is present [2024-01-18T20:52:44.459Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] Create cluster [destination-cluster] in org [default] [2024-01-18T20:52:44.459Z] INFO[2024-01-18 20:52:44] Verifying : Description : Creating source cluster source-cluster and destination cluster destination-cluster [2024-01-18T20:52:44.459Z] INFO[2024-01-18 20:52:44] Actual:true, Expected: true [2024-01-18T20:52:44.716Z] INFO[2024-01-18 20:52:44] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-18T20:52:44.716Z] INFO[2024-01-18 20:52:44] Actual:Online, Expected: Online [2024-01-18T20:52:44.716Z] INFO[2024-01-18 20:52:44] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-18T20:52:44.716Z] INFO[2024-01-18 20:52:44] Actual:true, Expected: true [2024-01-18T20:52:44.717Z] INFO[2024-01-18 20:52:44] Verifying : Description : Verifying if [destination-cluster] cluster is online [2024-01-18T20:52:44.717Z] INFO[2024-01-18 20:52:44] Actual:Online, Expected: Online [2024-01-18T20:52:44.717Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] Taking backup of applications [2024-01-18T20:52:44.973Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T20:52:44.973Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T20:52:44.973Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T20:52:44.973Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T20:52:44.973Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] Volume snapshot classes found in the cluster - [2024-01-18T20:52:44.973Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] ibmc-vpcblock-snapshot [2024-01-18T20:52:44.973Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T20:52:44.973Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] Volume snapshot classes found in the cluster - [2024-01-18T20:52:44.973Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] ibmc-vpcblock-snapshot [2024-01-18T20:52:44.973Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T20:52:44.973Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] Volume snapshot classes found in the cluster - [2024-01-18T20:52:44.973Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] ibmc-vpcblock-snapshot [2024-01-18T20:52:44.973Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T20:52:44.973Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] Volume snapshot classes found in the cluster - [2024-01-18T20:52:44.973Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] ibmc-vpcblock-snapshot [2024-01-18T20:52:44.973Z] 2024-01-18 20:52:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T20:52:45.534Z] INFO[2024-01-18 20:52:45] Verifying : Description : Taking backup tp-backup-mysql-ibm-pxb-0-58045-01-18-20h50m57s-3-1705611164 of application- mysql-ibm-pxb-0-58045-01-18-20h50m57s [2024-01-18T20:52:45.534Z] INFO[2024-01-18 20:52:45] Actual:true, Expected: true [2024-01-18T20:52:46.096Z] INFO[2024-01-18 20:52:45] Verifying : Description : Taking backup tp-backup-mysql-ibm-pxb-0-58045-01-18-20h50m57s-2-1705611164 of application- mysql-ibm-pxb-0-58045-01-18-20h50m57s [2024-01-18T20:52:46.096Z] INFO[2024-01-18 20:52:45] Actual:true, Expected: true [2024-01-18T20:52:46.657Z] INFO[2024-01-18 20:52:46] Verifying : Description : Taking backup tp-backup-mysql-ibm-pxb-0-58045-01-18-20h50m57s-0-1705611164 of application- mysql-ibm-pxb-0-58045-01-18-20h50m57s [2024-01-18T20:52:46.657Z] INFO[2024-01-18 20:52:46] Actual:true, Expected: true [2024-01-18T20:52:46.657Z] INFO[2024-01-18 20:52:46] Verifying : Description : Taking backup tp-backup-mysql-ibm-pxb-0-58045-01-18-20h50m57s-1-1705611164 of application- mysql-ibm-pxb-0-58045-01-18-20h50m57s [2024-01-18T20:52:46.657Z] INFO[2024-01-18 20:52:46] Actual:true, Expected: true [2024-01-18T20:52:46.657Z] 2024-01-18 20:52:46 +0000:[INFO] [{CancelAllRunningBackupJobs}] [backup.glob..func50.2.4:#708] - The list of backups taken are: [tp-backup-mysql-ibm-pxb-0-58045-01-18-20h50m57s-0-1705611164 tp-backup-mysql-ibm-pxb-0-58045-01-18-20h50m57s-1-1705611164 tp-backup-mysql-ibm-pxb-0-58045-01-18-20h50m57s-2-1705611164 tp-backup-mysql-ibm-pxb-0-58045-01-18-20h50m57s-3-1705611164] [2024-01-18T20:52:46.657Z] 2024-01-18 20:52:46 +0000:[INFO] [{CancelAllRunningBackupJobs}] Cancelling the ongoing backups [2024-01-18T20:52:47.219Z] 2024/01/18 20:52:46 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58045-01-18-20h50m57s-2-1705611164] expected was [InProgress] but got [Pending]}, Next try in [5s], timeout [20m0s] [2024-01-18T20:52:47.219Z] 2024/01/18 20:52:46 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58045-01-18-20h50m57s-1-1705611164] expected was [InProgress] but got [Pending]}, Next try in [5s], timeout [20m0s] [2024-01-18T20:52:47.219Z] INFO[2024-01-18 20:52:46] Verifying : Description : Verfiying backup tp-backup-mysql-ibm-pxb-0-58045-01-18-20h50m57s-3-1705611164 is in progress [2024-01-18T20:52:47.219Z] INFO[2024-01-18 20:52:46] Actual:true, Expected: true [2024-01-18T20:52:47.219Z] 2024/01/18 20:52:46 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58045-01-18-20h50m57s-0-1705611164] expected was [InProgress] but got [Pending]}, Next try in [5s], timeout [20m0s] [2024-01-18T20:52:47.475Z] INFO[2024-01-18 20:52:47] Verifying : Description : Deleting backup tp-backup-mysql-ibm-pxb-0-58045-01-18-20h50m57s-3-1705611164 while backup is in progress [2024-01-18T20:52:47.475Z] INFO[2024-01-18 20:52:47] Actual:true, Expected: true [2024-01-18T20:52:52.743Z] INFO[2024-01-18 20:52:51] Verifying : Description : Verfiying backup tp-backup-mysql-ibm-pxb-0-58045-01-18-20h50m57s-2-1705611164 is in progress [2024-01-18T20:52:52.743Z] INFO[2024-01-18 20:52:51] Actual:true, Expected: true [2024-01-18T20:52:52.743Z] INFO[2024-01-18 20:52:51] Verifying : Description : Verfiying backup tp-backup-mysql-ibm-pxb-0-58045-01-18-20h50m57s-1-1705611164 is in progress [2024-01-18T20:52:52.743Z] INFO[2024-01-18 20:52:51] Actual:true, Expected: true [2024-01-18T20:52:52.743Z] INFO[2024-01-18 20:52:52] Verifying : Description : Verfiying backup tp-backup-mysql-ibm-pxb-0-58045-01-18-20h50m57s-0-1705611164 is in progress [2024-01-18T20:52:52.743Z] INFO[2024-01-18 20:52:52] Actual:true, Expected: true [2024-01-18T20:52:52.743Z] INFO[2024-01-18 20:52:52] Verifying : Description : Deleting backup tp-backup-mysql-ibm-pxb-0-58045-01-18-20h50m57s-0-1705611164 while backup is in progress [2024-01-18T20:52:52.743Z] INFO[2024-01-18 20:52:52] Actual:true, Expected: true [2024-01-18T20:52:52.743Z] INFO[2024-01-18 20:52:52] Verifying : Description : Deleting backup tp-backup-mysql-ibm-pxb-0-58045-01-18-20h50m57s-2-1705611164 while backup is in progress [2024-01-18T20:52:52.743Z] INFO[2024-01-18 20:52:52] Actual:true, Expected: true [2024-01-18T20:52:52.743Z] INFO[2024-01-18 20:52:52] Verifying : Description : Deleting backup tp-backup-mysql-ibm-pxb-0-58045-01-18-20h50m57s-1-1705611164 while backup is in progress [2024-01-18T20:52:52.743Z] INFO[2024-01-18 20:52:52] Actual:true, Expected: true [2024-01-18T20:52:52.743Z] 2024-01-18 20:52:52 +0000:[INFO] [{CancelAllRunningBackupJobs}] Sleeping for 60 seconds for the backup cancellation to take place [2024-01-18T20:54:00.373Z] 2024-01-18 20:53:52 +0000:[INFO] [{CancelAllRunningBackupJobs}] Verifying if all the backup creation is cancelled [2024-01-18T20:54:00.374Z] 2024-01-18 20:53:52 +0000:[INFO] [{CancelAllRunningBackupJobs}] [backup.glob..func50.2.6:#758] - The list of backups after backup cancellation is [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-203954 tp-backup-1705608575 tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-200712 tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-195204 tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-193657 tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-192150 tp-backup-mysql-ibm-pxb-1-58051-01-18-12h40m37s-mkmce tp-backup-mysql-ibm-pxb-3-58051-01-18-12h40m37s-otfyc tp-backup-mysql-ibm-pxb-2-58051-01-18-12h40m37s-hhxop tp-backup-mysql-ibm-pxb-4-58051-01-18-12h40m37s-ltnvx tp-backup-mysql-ibm-pxb-0-58051-01-18-12h40m37s-gllfn tp-backup-all-1705577915] [2024-01-18T20:54:00.374Z] INFO[2024-01-18 20:53:53] Verifying : Description : Verifying backup jobs cancellation while backups are in progress [2024-01-18T20:54:00.374Z] INFO[2024-01-18 20:53:53] Actual:true, Expected: true [2024-01-18T20:54:00.374Z] 2024-01-18 20:53:53 +0000:[INFO] [{CancelAllRunningBackupJobs}] [backup.glob..func50.2.6:#780] - All the backups created by this testcase is deleted after backup cancellation [2024-01-18T20:54:00.374Z] 2024-01-18 20:53:54 +0000:[INFO] [{CancelAllRunningBackupJobs}] [backup.glob..func50.3:#788] - Deleting the deployed applications [2024-01-18T20:54:00.374Z] 2024-01-18 20:53:54 +0000:[INFO] [{CancelAllRunningBackupJobs}] Validating apps [2024-01-18T20:54:00.374Z] 2024-01-18 20:53:54 +0000:[INFO] [{CancelAllRunningBackupJobs}] Validating mysql-ibm app [2024-01-18T20:54:00.374Z] 2024-01-18 20:53:54 +0000:[INFO] [{CancelAllRunningBackupJobs}] Validating mysql-ibm app's volumes [2024-01-18T20:54:00.374Z] 2024-01-18 20:53:54 +0000:[WARNING] [{CancelAllRunningBackupJobs}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T20:54:00.374Z] 2024-01-18 20:53:54 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:54:00.374Z] 2024-01-18 20:53:55 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-58045-01-18-20h50m57s [2024-01-18T20:54:00.374Z] 2024-01-18 20:53:55 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T20:54:00.374Z] 2024-01-18 20:53:55 +0000:[WARNING] [{CancelAllRunningBackupJobs}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T20:54:00.374Z] 2024-01-18 20:53:55 +0000:[INFO] [{CancelAllRunningBackupJobs}] wait for mysql-ibm app to start running [2024-01-18T20:54:00.374Z] 2024-01-18 20:53:56 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T20:54:00.374Z] 2024-01-18 20:53:56 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T20:54:00.374Z] 2024-01-18 20:53:56 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T20:54:00.374Z] 2024-01-18 20:53:56 +0000:[INFO] [{CancelAllRunningBackupJobs}] validate if mysql-ibm app's volumes are setup [2024-01-18T20:54:00.375Z] 2024-01-18 20:53:56 +0000:[WARNING] [{CancelAllRunningBackupJobs}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T20:54:00.375Z] 2024-01-18 20:53:56 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T20:54:00.375Z] 2024-01-18 20:53:56 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T20:54:00.375Z] 2024-01-18 20:53:56 +0000:[WARNING] [{CancelAllRunningBackupJobs}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T20:54:00.375Z] 2024-01-18 20:53:56 +0000:[INFO] [{CancelAllRunningBackupJobs}] Destroying apps [2024-01-18T20:54:00.375Z] 2024-01-18 20:53:56 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T20:54:00.375Z] 2024-01-18 20:53:57 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T20:54:00.375Z] 2024-01-18 20:53:57 +0000:[INFO] [{CancelAllRunningBackupJobs}] start destroying mysql-ibm app [2024-01-18T20:54:00.375Z] 2024-01-18 20:53:57 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T20:54:00.375Z] 2024-01-18 20:53:57 +0000:[INFO] [{CancelAllRunningBackupJobs}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T20:54:00.375Z] 2024-01-18 20:53:57 +0000:[WARNING] [{CancelAllRunningBackupJobs}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T20:54:00.375Z] INFO[2024-01-18 20:53:57] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T20:54:00.375Z] INFO[2024-01-18 20:53:57] Actual:true, Expected: true [2024-01-18T20:54:00.375Z] 2024-01-18 20:53:57 +0000:[INFO] [{CancelAllRunningBackupJobs}] Cleaning backup locations in map [map[ead9981f-bf9f-4b5c-94ac-90701d81964f:aws-global-aws-px-backup-system-test-4cac76dc-1705611159-bl]], cloud credential [cloudcred-aws-1705611159], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T20:54:00.375Z] INFO[2024-01-18 20:53:57] Verifying : Description : Verifying deletion of backup location [aws-global-aws-px-backup-system-test-4cac76dc-1705611159-bl] [2024-01-18T20:54:00.375Z] INFO[2024-01-18 20:53:57] Actual:true, Expected: true [2024-01-18T20:54:00.375Z] 2024-01-18 20:53:57 +0000:[INFO] [{CancelAllRunningBackupJobs}] [backup.IsBackupLocationPresent:#2442] - Backup location [aws-global-aws-px-backup-system-test-4cac76dc-1705611159-bl] is present [2024-01-18T20:54:00.375Z] 2024/01/18 20:53:57 DoRetryWithTimeout - Error: {backup location aws-global-aws-px-backup-system-test-4cac76dc-1705611159-bl is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T20:54:26.873Z] INFO[2024-01-18 20:54:25] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T20:54:26.873Z] INFO[2024-01-18 20:54:25] sched stats: tick interval: 19225 < 1.2s (100%) [2024-01-18T20:54:26.873Z] INFO[2024-01-18 20:54:25] sched stats: tick process durations: 19224 < 500ms (100%) [2024-01-18T20:54:27.802Z] 2024-01-18 20:54:27 +0000:[INFO] [{CancelAllRunningBackupJobs}] [backup.IsBackupLocationPresent:#2442] - Backup location [aws-global-aws-px-backup-system-test-4cac76dc-1705611159-bl] is present [2024-01-18T20:54:27.802Z] 2024/01/18 20:54:27 DoRetryWithTimeout - Error: {backup location aws-global-aws-px-backup-system-test-4cac76dc-1705611159-bl is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T20:54:59.830Z] 2024-01-18 20:54:57 +0000:[INFO] [{CancelAllRunningBackupJobs}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 aws-global-aws-px-backup-system-test-4cac76dc-bl] [2024-01-18T20:54:59.830Z] INFO[2024-01-18 20:54:57] Verifying : Description : Verifying backup location deletion status aws-global-aws-px-backup-system-test-4cac76dc-1705611159-bl [2024-01-18T20:54:59.830Z] INFO[2024-01-18 20:54:57] Actual:true, Expected: true [2024-01-18T20:54:59.830Z] 2024-01-18 20:54:57 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cloudcred-aws-1705611159] is present [2024-01-18T20:54:59.830Z] INFO[2024-01-18 20:54:57] Verifying : Description : Verifying if cloud cred [cloudcred-aws-1705611159] is present [2024-01-18T20:54:59.830Z] INFO[2024-01-18 20:54:57] Actual:true, Expected: true [2024-01-18T20:54:59.830Z] INFO[2024-01-18 20:54:57] Verifying : Description : Verifying deletion of cloud cred [cloudcred-aws-1705611159] [2024-01-18T20:54:59.830Z] INFO[2024-01-18 20:54:57] Actual:true, Expected: true [2024-01-18T20:54:59.831Z] INFO[2024-01-18 20:54:58] Verifying : Description : Deleting cloud cred cloudcred-aws-1705611159 [2024-01-18T20:54:59.831Z] INFO[2024-01-18 20:54:58] Actual:true, Expected: true [2024-01-18T20:54:59.831Z] INFO[2024-01-18 20:54:58] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T20:54:59.831Z] INFO[2024-01-18 20:54:58] Actual:true, Expected: true [2024-01-18T20:54:59.831Z] INFO[2024-01-18 20:54:58] Verifying : Description : Deleting cluster source-cluster [2024-01-18T20:54:59.831Z] INFO[2024-01-18 20:54:58] Actual:true, Expected: true [2024-01-18T20:54:59.831Z] INFO[2024-01-18 20:54:58] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-seowm] [2024-01-18T20:54:59.831Z] INFO[2024-01-18 20:54:58] Actual:true, Expected: true [2024-01-18T20:54:59.831Z] INFO[2024-01-18 20:54:58] Verifying : Description : Deleting cluster destination-cluster [2024-01-18T20:54:59.831Z] ERRO[2024-01-18 20:54:58] Actual:rpc error: code = FailedPrecondition desc = cluster [destination-cluster] in use by backupschedule [tp-backup-schedule-1705605709], cannot be deleted, Expected:  [2024-01-18T20:54:59.831Z] INFO[2024-01-18 20:54:59] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-zbbvt] [2024-01-18T20:54:59.831Z] ERRO[2024-01-18 20:54:59] Actual:rpc error: code = Unknown desc = cluster [destination-cluster] has reference to cloud credential [ibm-destination-config-cloud-cred-zbbvt:a271ea5b-dc9d-40f1-a82a-651539c7e486], Expected:  [2024-01-18T20:54:59.831Z] INFO[2024-01-18 20:54:59] --------Test End------ [2024-01-18T20:54:59.831Z] INFO[2024-01-18 20:54:59] #Test: [2024-01-18T20:54:59.831Z] INFO[2024-01-18 20:54:59] #Description: [2024-01-18T20:54:59.831Z] INFO[2024-01-18 20:54:59] ------------------------ [2024-01-18T20:54:59.831Z] 2024-01-18 20:54:59 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-58045-01-18-20h50m57s] [2024-01-18T20:54:59.831Z] 2024-01-18 20:54:59 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T20:54:59.831Z] 2024-01-18 20:54:59 +0000:[INFO] [{CancelAllRunningBackupJobs}] Switching context: The context is already [/tmp/destination-config] [2024-01-18T20:54:59.831Z] 2024-01-18 20:54:59 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-58045-01-18-20h50m57s] [2024-01-18T20:55:00.088Z] 2024-01-18 20:54:59 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {CancelAllRunningBackupJobs} Cancel All Running Backup Jobs and validate [2024-01-18T20:55:00.088Z] 2024-01-18 20:54:59 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-18T20:55:00.088Z] 2024/01/18 20:54:59 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:55:05.331Z] 2024/01/18 20:55:04 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:55:10.585Z] 2024/01/18 20:55:09 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:55:15.833Z] 2024/01/18 20:55:14 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:55:20.019Z] 2024/01/18 20:55:19 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:55:25.267Z] 2024/01/18 20:55:24 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:55:30.517Z] 2024/01/18 20:55:29 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:55:35.774Z] 2024/01/18 20:55:34 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T20:55:39.945Z] 2024/01/18 20:55:39 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.64.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.8 10.240.64.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T20:55:39.946Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T20:55:39.946Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T20:55:39.946Z] {dial tcp 10.240.64.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-18T20:55:45.188Z] 2024-01-18 20:55:44 +0000:[WARNING] [{CancelAllRunningBackupJobs}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.64.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.8 10.240.64.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T20:55:45.188Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T20:55:45.188Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T20:55:45.188Z] {dial tcp 10.240.64.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T20:55:45.189Z] 2024-01-18 20:55:44 +0000:[ERROR] [{CancelAllRunningBackupJobs}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.64.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.8 10.240.64.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T20:55:45.189Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T20:55:45.189Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T20:55:45.189Z] {dial tcp 10.240.64.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T20:55:45.189Z] 2024-01-18 20:55:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T20:55:45.189Z] 2024-01-18 20:55:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] Switching context to [/tmp/source-config] [2024-01-18T20:55:45.189Z] 2024-01-18 20:55:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T20:55:45.189Z] 2024-01-18 20:55:44 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T20:55:45.189Z] 2024-01-18 20:55:45 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T20:55:45.189Z] 2024-01-18 20:55:45 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T20:55:45.189Z] 2024-01-18 20:55:45 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T20:55:45.445Z] 2024-01-18 20:55:45 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T20:55:46.006Z] 2024-01-18 20:55:45 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T20:55:46.262Z] 2024-01-18 20:55:46 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T20:55:46.821Z] 2024-01-18 20:55:46 +0000:[INFO] [{CancelAllRunningBackupJobs}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T20:55:46.821Z] 2024/01/18 20:55:46 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:55:58.983Z] 2024/01/18 20:55:56 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:56:07.074Z] 2024/01/18 20:56:06 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:56:17.019Z] 2024/01/18 20:56:16 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:56:26.984Z] 2024/01/18 20:56:26 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:56:36.930Z] 2024/01/18 20:56:36 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:56:49.097Z] 2024/01/18 20:56:46 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:56:58.044Z] 2024/01/18 20:56:56 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:57:08.027Z] 2024/01/18 20:57:06 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:57:17.988Z] 2024/01/18 20:57:16 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:57:27.968Z] 2024/01/18 20:57:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:57:37.943Z] 2024/01/18 20:57:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:57:47.886Z] 2024/01/18 20:57:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:57:57.831Z] 2024/01/18 20:57:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:58:07.767Z] 2024/01/18 20:58:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:58:17.714Z] 2024/01/18 20:58:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:58:27.667Z] 2024/01/18 20:58:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:58:37.614Z] 2024/01/18 20:58:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:58:47.558Z] 2024/01/18 20:58:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:58:57.495Z] 2024/01/18 20:58:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:59:07.446Z] 2024/01/18 20:59:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:59:19.623Z] 2024/01/18 20:59:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:59:27.717Z] 2024/01/18 20:59:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:59:37.660Z] 2024/01/18 20:59:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:59:47.629Z] 2024/01/18 20:59:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T20:59:57.577Z] 2024/01/18 20:59:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:00:09.751Z] 2024/01/18 21:00:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:00:17.864Z] 2024/01/18 21:00:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:00:27.804Z] 2024/01/18 21:00:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:00:37.759Z] 2024/01/18 21:00:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:00:47.715Z] 2024-01-18 21:00:47 +0000:[ERROR] [{CancelAllRunningBackupJobs}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.715Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.715Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.716Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.716Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.716Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.716Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.716Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.716Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.716Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.716Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.716Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.716Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.716Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.716Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.716Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.716Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.716Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.716Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.716Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.716Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.716Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.716Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.716Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.717Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.717Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.717Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.717Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.717Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.717Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-18T21:00:47.717Z] [2024-01-18T21:00:47.717Z] ------------------------------ [2024-01-18T21:00:47.717Z] • Failure [589.900 seconds] [2024-01-18T21:00:47.717Z] {CancelAllRunningBackupJobs} [2024-01-18T21:00:47.717Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_resiliency_test.go:612 [2024-01-18T21:00:47.717Z] Cancel All Running Backup Jobs and validate [] [2024-01-18T21:00:47.717Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_resiliency_test.go:648 [2024-01-18T21:00:47.717Z] [2024-01-18T21:00:47.717Z] Unexpected error: [2024-01-18T21:00:47.717Z] <*errors.errorString | 0xc0018a8d40>: { [2024-01-18T21:00:47.718Z] s: "failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{No... [2024-01-18T21:00:47.718Z] [2024-01-18T21:00:47.718Z] Gomega truncated this representation as it exceeds 'format.MaxLength'. [2024-01-18T21:00:47.718Z] Consider having the object provide a custom 'GomegaStringer' representation [2024-01-18T21:00:47.718Z] or adjust the parameters in Gomega's 'format' package. [2024-01-18T21:00:47.718Z] [2024-01-18T21:00:47.718Z] Learn more here: https://onsi.github.io/gomega/#adjusting-output [2024-01-18T21:00:47.718Z] [2024-01-18T21:00:47.718Z] failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.718Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.719Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.719Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.719Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.719Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.719Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.719Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.719Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.719Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.719Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.719Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.719Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.719Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.719Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.719Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.719Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.719Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.719Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.719Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.719Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.719Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.719Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.720Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.720Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.720Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.720Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.720Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.720Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:00:47.720Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-18T21:00:47.720Z] occurred [2024-01-18T21:00:47.720Z] [2024-01-18T21:00:47.720Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-18T21:00:47.720Z] [2024-01-18T21:00:47.720Z] Full Stack Trace [2024-01-18T21:00:47.720Z] github.com/portworx/torpedo/pkg/log.FailOnError({0x88a5060, 0xc0018a8d40}, {0x7f74bfd?, 0x6cc0320?}, {0x0?, 0x6cc0320?, 0xc00141fac0?}) [2024-01-18T21:00:47.720Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 +0x290 [2024-01-18T21:00:47.720Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest.func1() [2024-01-18T21:00:47.720Z] /go/src/github.com/portworx/torpedo/tests/common.go:7283 +0x32 [2024-01-18T21:00:47.720Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest({0xc00048c950, 0x1, 0x1}) [2024-01-18T21:00:47.720Z] /go/src/github.com/portworx/torpedo/tests/common.go:7299 +0x778 [2024-01-18T21:00:47.720Z] github.com/portworx/torpedo/tests/backup.glob..func50.3() [2024-01-18T21:00:47.720Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_resiliency_test.go:793 +0x182 [2024-01-18T21:00:47.720Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-18T21:00:47.720Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-18T21:00:47.720Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-18T21:00:47.720Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-18T21:00:47.720Z] created by testing.(*T).Run [2024-01-18T21:00:47.720Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-18T21:00:47.720Z] ------------------------------ [2024-01-18T21:00:47.720Z] STEP: schedule applications [2024-01-18T21:00:47.720Z] STEP: Validating the deployed applications [2024-01-18T21:00:47.720Z] STEP: validate applications [2024-01-18T21:00:47.720Z] STEP: validate mysql-ibm app's volumes [2024-01-18T21:00:47.721Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T21:00:47.721Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T21:00:47.721Z] STEP: get mysql-ibm app's volume: pvc-c3405f2c-a7cc-4507-84c0-cddaa7f23e92 inspected by the volume driver [2024-01-18T21:00:47.721Z] STEP: wait for mysql-ibm app to start running [2024-01-18T21:00:47.721Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T21:00:47.721Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T21:00:47.721Z] STEP: Validate Px pod restart count [2024-01-18T21:00:47.721Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T21:00:47.721Z] STEP: Adding cloud credential and backup location [2024-01-18T21:00:47.721Z] STEP: Registering source and destination clusters for backup [2024-01-18T21:00:47.721Z] STEP: Taking backup of applications [2024-01-18T21:00:47.721Z] STEP: Cancelling the ongoing backups [2024-01-18T21:00:47.721Z] STEP: Verifying if all the backup creation is cancelled [2024-01-18T21:00:47.721Z] STEP: validate apps [2024-01-18T21:00:47.721Z] STEP: validate mysql-ibm app's volumes [2024-01-18T21:00:47.721Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T21:00:47.721Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T21:00:47.721Z] STEP: get mysql-ibm app's volume: pvc-c3405f2c-a7cc-4507-84c0-cddaa7f23e92 inspected by the volume driver [2024-01-18T21:00:47.721Z] STEP: wait for mysql-ibm app to start running [2024-01-18T21:00:47.721Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T21:00:47.721Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T21:00:47.721Z] STEP: Validate Px pod restart count [2024-01-18T21:00:47.721Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T21:00:47.721Z] STEP: destroy apps [2024-01-18T21:00:47.721Z] STEP: destroy the mysql-ibm app's volumes [2024-01-18T21:00:47.721Z] STEP: start destroying mysql-ibm app [2024-01-18T21:00:47.721Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T21:00:47.721Z] 2024-01-18 21:00:47 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Log Dir: /testresults//SetUnsetNSLabelDuringScheduleBackup.log [2024-01-18T21:00:47.721Z] INFO[2024-01-18 21:00:47] --------Test Start------ [2024-01-18T21:00:47.721Z] INFO[2024-01-18 21:00:47] #Test: SetUnsetNSLabelDuringScheduleBackup [2024-01-18T21:00:47.721Z] INFO[2024-01-18 21:00:47] #Description: Create multiple namespaces and set unset namespace labels during the backup schedule [2024-01-18T21:00:47.721Z] INFO[2024-01-18 21:00:47] ------------------------ [2024-01-18T21:00:47.721Z] 2024-01-18 21:00:47 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Deploy applications [2024-01-18T21:00:47.721Z] 2024-01-18 21:00:47 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T21:00:47.721Z] 2024-01-18 21:00:47 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T21:00:47.721Z] 2024-01-18 21:00:47 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T21:00:47.979Z] 2024-01-18 21:00:47 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T21:00:47.979Z] 2024-01-18 21:00:47 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T21:00:47.979Z] 2024-01-18 21:00:47 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T21:00:47.980Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T21:00:47.980Z] 2024-01-18 21:00:47 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T21:00:47.980Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T21:00:47.980Z] 2024-01-18 21:00:47 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T21:00:47.980Z] 2024-01-18 21:00:47 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T21:00:47.980Z] 2024-01-18 21:00:47 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T21:00:47.980Z] 2024-01-18 21:00:47 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T21:00:48.236Z] 2024-01-18 21:00:47 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T21:00:48.236Z] 2024-01-18 21:00:47 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T21:00:48.236Z] 2024-01-18 21:00:48 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T21:00:48.237Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T21:00:48.237Z] 2024-01-18 21:00:48 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T21:00:48.237Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T21:00:48.237Z] 2024-01-18 21:00:48 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T21:00:48.237Z] 2024-01-18 21:00:48 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T21:00:48.238Z] 2024-01-18 21:00:48 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T21:00:48.238Z] 2024-01-18 21:00:48 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T21:00:48.494Z] 2024-01-18 21:00:48 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T21:00:48.494Z] 2024-01-18 21:00:48 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T21:00:48.494Z] 2024-01-18 21:00:48 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T21:00:48.496Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T21:00:48.496Z] 2024-01-18 21:00:48 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T21:00:48.496Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T21:00:48.496Z] 2024-01-18 21:00:48 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T21:00:48.496Z] 2024-01-18 21:00:48 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Created namespaces [mysql-ibm-pxb-0-84849-01-18-21h00m47s mysql-ibm-pxb-1-84849-01-18-21h00m47s mysql-ibm-pxb-2-84849-01-18-21h00m47s] [2024-01-18T21:00:48.496Z] 2024-01-18 21:00:48 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validate applications [2024-01-18T21:00:48.496Z] 2024-01-18 21:00:48 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validate applications [2024-01-18T21:00:48.496Z] 2024-01-18 21:00:48 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating mysql-ibm app [2024-01-18T21:00:48.496Z] 2024-01-18 21:00:48 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating mysql-ibm app's volumes [2024-01-18T21:00:48.496Z] 2024-01-18 21:00:48 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T21:00:48.752Z] 2024/01/18 21:00:48 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T21:00:58.717Z] 2024/01/18 21:00:58 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T21:01:08.660Z] 2024-01-18 21:01:08 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-84849-01-18-21h00m47s [2024-01-18T21:01:08.660Z] 2024-01-18 21:01:08 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T21:01:08.917Z] 2024-01-18 21:01:08 +0000:[WARNING] [{SetUnsetNSLabelDuringScheduleBackup}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T21:01:08.917Z] 2024-01-18 21:01:08 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] wait for mysql-ibm app to start running [2024-01-18T21:01:08.917Z] 2024-01-18 21:01:08 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T21:01:08.917Z] 2024/01/18 21:01:08 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T21:01:08.917Z] pod name:mysql-7d766cd65c-4ccws namespace:mysql-ibm-pxb-0-84849-01-18-21h00m47s running:false ready:false node:10.240.64.27 [2024-01-18T21:01:08.917Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T21:01:21.084Z] 2024/01/18 21:01:18 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T21:01:21.084Z] pod name:mysql-7d766cd65c-4ccws namespace:mysql-ibm-pxb-0-84849-01-18-21h00m47s running:false ready:false node:10.240.64.27 [2024-01-18T21:01:21.084Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T21:01:29.193Z] 2024/01/18 21:01:28 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T21:01:29.193Z] pod name:mysql-7d766cd65c-4ccws namespace:mysql-ibm-pxb-0-84849-01-18-21h00m47s running:true ready:false node:10.240.64.27 [2024-01-18T21:01:29.194Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T21:01:39.144Z] 2024/01/18 21:01:39 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T21:01:39.144Z] pod name:mysql-7d766cd65c-4ccws namespace:mysql-ibm-pxb-0-84849-01-18-21h00m47s running:true ready:false node:10.240.64.27 [2024-01-18T21:01:39.144Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T21:01:51.319Z] 2024/01/18 21:01:49 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T21:01:51.319Z] pod name:mysql-7d766cd65c-4ccws namespace:mysql-ibm-pxb-0-84849-01-18-21h00m47s running:true ready:false node:10.240.64.27 [2024-01-18T21:01:51.319Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T21:01:59.484Z] 2024/01/18 21:01:59 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T21:01:59.484Z] pod name:mysql-7d766cd65c-4ccws namespace:mysql-ibm-pxb-0-84849-01-18-21h00m47s running:true ready:false node:10.240.64.27 [2024-01-18T21:01:59.484Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T21:02:09.478Z] 2024/01/18 21:02:09 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T21:02:09.478Z] pod name:mysql-7d766cd65c-4ccws namespace:mysql-ibm-pxb-0-84849-01-18-21h00m47s running:true ready:false node:10.240.64.27 [2024-01-18T21:02:09.478Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T21:02:21.660Z] 2024/01/18 21:02:19 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T21:02:21.660Z] pod name:mysql-7d766cd65c-4ccws namespace:mysql-ibm-pxb-0-84849-01-18-21h00m47s running:true ready:false node:10.240.64.27 [2024-01-18T21:02:21.660Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T21:02:29.741Z] 2024-01-18 21:02:29 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T21:02:29.741Z] 2024-01-18 21:02:29 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T21:02:29.741Z] 2024-01-18 21:02:29 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] validate if mysql-ibm app's volumes are setup [2024-01-18T21:02:30.001Z] 2024-01-18 21:02:29 +0000:[WARNING] [{SetUnsetNSLabelDuringScheduleBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T21:02:30.001Z] 2024-01-18 21:02:29 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T21:02:30.001Z] 2024-01-18 21:02:29 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T21:02:30.001Z] 2024-01-18 21:02:29 +0000:[WARNING] [{SetUnsetNSLabelDuringScheduleBackup}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T21:02:30.001Z] 2024-01-18 21:02:29 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating mysql-ibm app [2024-01-18T21:02:30.001Z] 2024-01-18 21:02:29 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating mysql-ibm app's volumes [2024-01-18T21:02:30.001Z] 2024-01-18 21:02:29 +0000:[WARNING] [{SetUnsetNSLabelDuringScheduleBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T21:02:30.001Z] 2024-01-18 21:02:29 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T21:02:30.001Z] 2024-01-18 21:02:29 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-1-84849-01-18-21h00m47s [2024-01-18T21:02:30.001Z] 2024-01-18 21:02:29 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T21:02:30.002Z] 2024-01-18 21:02:29 +0000:[WARNING] [{SetUnsetNSLabelDuringScheduleBackup}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T21:02:30.002Z] 2024-01-18 21:02:29 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] wait for mysql-ibm app to start running [2024-01-18T21:02:30.261Z] 2024-01-18 21:02:29 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T21:02:30.261Z] 2024-01-18 21:02:30 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T21:02:30.822Z] 2024-01-18 21:02:30 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T21:02:31.749Z] 2024-01-18 21:02:31 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] validate if mysql-ibm app's volumes are setup [2024-01-18T21:02:31.749Z] 2024-01-18 21:02:31 +0000:[WARNING] [{SetUnsetNSLabelDuringScheduleBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T21:02:31.749Z] 2024-01-18 21:02:31 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T21:02:31.749Z] 2024-01-18 21:02:31 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T21:02:31.749Z] 2024-01-18 21:02:31 +0000:[WARNING] [{SetUnsetNSLabelDuringScheduleBackup}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T21:02:31.749Z] 2024-01-18 21:02:31 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating mysql-ibm app [2024-01-18T21:02:31.749Z] 2024-01-18 21:02:31 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating mysql-ibm app's volumes [2024-01-18T21:02:31.749Z] 2024-01-18 21:02:31 +0000:[WARNING] [{SetUnsetNSLabelDuringScheduleBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T21:02:31.749Z] 2024-01-18 21:02:31 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T21:02:31.750Z] 2024-01-18 21:02:31 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-2-84849-01-18-21h00m47s [2024-01-18T21:02:31.750Z] 2024-01-18 21:02:31 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T21:02:32.005Z] 2024-01-18 21:02:31 +0000:[WARNING] [{SetUnsetNSLabelDuringScheduleBackup}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T21:02:32.005Z] 2024-01-18 21:02:31 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] wait for mysql-ibm app to start running [2024-01-18T21:02:32.005Z] 2024-01-18 21:02:31 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T21:02:32.260Z] 2024-01-18 21:02:32 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T21:02:32.822Z] 2024-01-18 21:02:32 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T21:02:33.752Z] 2024-01-18 21:02:33 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] validate if mysql-ibm app's volumes are setup [2024-01-18T21:02:33.752Z] 2024-01-18 21:02:33 +0000:[WARNING] [{SetUnsetNSLabelDuringScheduleBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T21:02:33.752Z] 2024-01-18 21:02:33 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T21:02:33.752Z] 2024-01-18 21:02:33 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T21:02:33.752Z] 2024-01-18 21:02:33 +0000:[WARNING] [{SetUnsetNSLabelDuringScheduleBackup}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T21:02:33.753Z] 2024-01-18 21:02:33 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Adding labels to namespaces [2024-01-18T21:02:33.753Z] INFO[2024-01-18 21:02:33] Verifying : Description : Adding labels [map[0-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 1-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 10-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 11-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 12-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 13-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 14-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 15-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 16-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 17-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 18-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 19-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 2-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 3-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 4-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 5-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 6-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 7-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 8-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 9-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c]] to namespaces [[mysql-ibm-pxb-0-84849-01-18-21h00m47s mysql-ibm-pxb-1-84849-01-18-21h00m47s mysql-ibm-pxb-2-84849-01-18-21h00m47s]] [2024-01-18T21:02:33.753Z] INFO[2024-01-18 21:02:33] Actual:true, Expected: true [2024-01-18T21:02:33.753Z] 2024-01-18 21:02:33 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Generating namespace label string from label map for multiple namespace [2024-01-18T21:02:33.754Z] 2024-01-18 21:02:33 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.glob..func67.2.3:#2362] - labels for multiple namespace 9-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,16-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,4-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,8-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,12-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,18-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,19-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,0-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,2-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,14-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,1-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,11-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,6-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,7-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,10-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,13-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,15-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,17-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,3-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,5-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c [2024-01-18T21:02:33.754Z] 2024-01-18 21:02:33 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Creating cloud credentials and registering backup location [2024-01-18T21:02:35.117Z] 2024-01-18 21:02:34 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-1705611754] for org [default] with [aws] as provider [2024-01-18T21:02:35.117Z] 2024-01-18 21:02:34 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-18T21:02:35.117Z] 2024-01-18 21:02:35 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-1705611754] is present [2024-01-18T21:02:35.117Z] INFO[2024-01-18 21:02:35] Verifying : Description : Creating cloud credentials autogenerated-cred-1705611754 [2024-01-18T21:02:35.117Z] INFO[2024-01-18 21:02:35] Actual:true, Expected: true [2024-01-18T21:02:39.290Z] INFO[2024-01-18 21:02:38] Verifying : Description : Creating backup location autogenerated-backup-location-1705611755 [2024-01-18T21:02:39.290Z] INFO[2024-01-18 21:02:38] Actual:true, Expected: true [2024-01-18T21:02:39.290Z] 2024-01-18 21:02:38 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Configuring source and destination clusters with px-central-admin ctx [2024-01-18T21:02:39.291Z] INFO[2024-01-18 21:02:39] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T21:02:39.291Z] INFO[2024-01-18 21:02:39] Actual:true, Expected: true [2024-01-18T21:02:39.291Z] 2024-01-18 21:02:39 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T21:02:39.291Z] 2024-01-18 21:02:39 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T21:02:39.291Z] 2024-01-18 21:02:39 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T21:02:39.291Z] 2024-01-18 21:02:39 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T21:02:39.291Z] 2024-01-18 21:02:39 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Create cluster [source-cluster] in org [default] [2024-01-18T21:02:39.291Z] 2024-01-18 21:02:39 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T21:02:39.291Z] 2024-01-18 21:02:39 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T21:02:39.291Z] 2024-01-18 21:02:39 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Create cluster [destination-cluster] in org [default] [2024-01-18T21:02:39.291Z] 2024-01-18 21:02:39 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T21:02:39.291Z] 2024-01-18 21:02:39 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T21:02:39.291Z] 2024-01-18 21:02:39 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-rehqq] for IBM [2024-01-18T21:02:39.291Z] 2024-01-18 21:02:39 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-rehqq] for org [default] with [ibm] as provider [2024-01-18T21:02:39.291Z] 2024-01-18 21:02:39 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T21:02:39.291Z] 2024-01-18 21:02:39 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T21:02:39.291Z] 2024-01-18 21:02:39 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-rehqq] is present [2024-01-18T21:02:39.291Z] 2024-01-18 21:02:39 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Create cluster [source-cluster] in org [default] [2024-01-18T21:02:39.851Z] 2024-01-18 21:02:39 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-wvqmp] for IBM [2024-01-18T21:02:39.851Z] 2024-01-18 21:02:39 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-wvqmp] for org [default] with [ibm] as provider [2024-01-18T21:02:39.851Z] 2024-01-18 21:02:39 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T21:02:39.851Z] 2024-01-18 21:02:39 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T21:02:40.109Z] 2024-01-18 21:02:39 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-wvqmp] is present [2024-01-18T21:02:40.109Z] 2024-01-18 21:02:39 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Create cluster [destination-cluster] in org [default] [2024-01-18T21:02:40.110Z] INFO[2024-01-18 21:02:39] Verifying : Description : Verifying creation of source [source-cluster] and destination [destination-cluster] clusters with px-central-admin ctx [2024-01-18T21:02:40.110Z] INFO[2024-01-18 21:02:39] Actual:true, Expected: true [2024-01-18T21:02:40.110Z] INFO[2024-01-18 21:02:40] Verifying : Description : Verifying if [destination-cluster] cluster is online [2024-01-18T21:02:40.110Z] INFO[2024-01-18 21:02:40] Actual:Online, Expected: Online [2024-01-18T21:02:40.110Z] INFO[2024-01-18 21:02:40] Verifying : Description : Fetching [destination-cluster] cluster uid [2024-01-18T21:02:40.110Z] INFO[2024-01-18 21:02:40] Actual:true, Expected: true [2024-01-18T21:02:40.110Z] 2024-01-18 21:02:40 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Uid of [destination-cluster] cluster is 5f787597-2bbc-4af4-8e4a-caba411c5430 [2024-01-18T21:02:40.110Z] 2024-01-18 21:02:40 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Creating a schedule policy [2024-01-18T21:02:40.366Z] INFO[2024-01-18 21:02:40] Verifying : Description : Fetching px-central-admin ctx [2024-01-18T21:02:40.366Z] INFO[2024-01-18 21:02:40] Actual:true, Expected: true [2024-01-18T21:02:40.366Z] 2024-01-18 21:02:40 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Create Backup Schedule Policy [2024-01-18T21:02:41.312Z] INFO[2024-01-18 21:02:41] Verifying : Description : Verifying creation of periodic schedule policy of interval 15 minutes named [periodic-1705611760] [2024-01-18T21:02:41.312Z] INFO[2024-01-18 21:02:41] Actual:true, Expected: true [2024-01-18T21:02:41.312Z] INFO[2024-01-18 21:02:41] Verifying : Description : Fetching uid of periodic schedule policy named [periodic-1705611760] [2024-01-18T21:02:41.312Z] INFO[2024-01-18 21:02:41] Actual:true, Expected: true [2024-01-18T21:02:41.312Z] 2024-01-18 21:02:41 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Creating a schedule backup [2024-01-18T21:02:42.679Z] 2024-01-18 21:02:42 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-18T21:02:42.679Z] 2024-01-18 21:02:42 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Volume snapshot classes found in the cluster - [2024-01-18T21:02:42.679Z] 2024-01-18 21:02:42 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] ibmc-vpcblock-snapshot [2024-01-18T21:02:42.679Z] 2024-01-18 21:02:42 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T21:03:50.313Z] 2024-01-18 21:03:43 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] first schedule backup for schedule name [tp-backup-schedule-1705611762] is [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243] [2024-01-18T21:03:50.313Z] 2024/01/18 21:03:43 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T21:04:16.818Z] 2024-01-18 21:04:13 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243] in org [default] [2024-01-18T21:04:16.818Z] 2024-01-18 21:04:13 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243] [2024-01-18T21:04:16.818Z] 2024-01-18 21:04:13 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84849-01-18-21h00m47s] in backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243] [2024-01-18T21:04:16.818Z] 2024-01-18 21:04:13 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84849-01-18-21h00m47s] in backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243] [2024-01-18T21:04:16.819Z] 2024-01-18 21:04:13 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84849-01-18-21h00m47s] [2024-01-18T21:04:16.819Z] 2024-01-18 21:04:13 +0000:[WARNING] [{SetUnsetNSLabelDuringScheduleBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T21:04:16.819Z] 2024-01-18 21:04:13 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T21:04:16.819Z] 2024-01-18 21:04:13 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-84849-01-18-21h00m47s] are [[mysql-data]] [2024-01-18T21:04:16.819Z] 2024-01-18 21:04:13 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T21:04:16.819Z] 2024-01-18 21:04:13 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84849-01-18-21h00m47s] in backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243] [2024-01-18T21:04:16.819Z] 2024-01-18 21:04:13 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84849-01-18-21h00m47s] in backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243] [2024-01-18T21:04:16.819Z] 2024-01-18 21:04:13 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84849-01-18-21h00m47s] [2024-01-18T21:04:16.819Z] 2024-01-18 21:04:13 +0000:[WARNING] [{SetUnsetNSLabelDuringScheduleBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T21:04:16.819Z] 2024-01-18 21:04:13 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T21:04:16.819Z] 2024-01-18 21:04:13 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-84849-01-18-21h00m47s] are [[mysql-data]] [2024-01-18T21:04:16.819Z] 2024-01-18 21:04:13 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T21:04:16.819Z] 2024-01-18 21:04:13 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84849-01-18-21h00m47s] in backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243] [2024-01-18T21:04:16.819Z] 2024-01-18 21:04:13 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84849-01-18-21h00m47s] in backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243] [2024-01-18T21:04:16.819Z] 2024-01-18 21:04:13 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84849-01-18-21h00m47s] [2024-01-18T21:04:16.819Z] 2024-01-18 21:04:13 +0000:[WARNING] [{SetUnsetNSLabelDuringScheduleBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T21:04:16.819Z] 2024-01-18 21:04:13 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T21:04:16.819Z] 2024-01-18 21:04:13 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-2-84849-01-18-21h00m47s] are [[mysql-data]] [2024-01-18T21:04:16.819Z] 2024-01-18 21:04:13 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T21:04:16.819Z] 2024-01-18 21:04:13 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating CR cleanup [2024-01-18T21:04:16.820Z] 2024-01-18 21:04:13 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T21:04:16.820Z] 2024-01-18 21:04:13 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] [2024-01-18T21:04:16.820Z] 2024-01-18 21:04:13 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] under [kube-system] namespace [2024-01-18T21:04:16.820Z] 2024/01/18 21:04:13 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]}, Next try in [5s], timeout [5m0s] [2024-01-18T21:04:19.335Z] 2024-01-18 21:04:18 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] [2024-01-18T21:04:19.336Z] 2024-01-18 21:04:18 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] under [kube-system] namespace [2024-01-18T21:04:19.336Z] 2024/01/18 21:04:18 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]}, Next try in [5s], timeout [5m0s] [2024-01-18T21:04:24.578Z] 2024-01-18 21:04:23 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] [2024-01-18T21:04:24.578Z] 2024-01-18 21:04:23 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] under [kube-system] namespace [2024-01-18T21:04:24.578Z] 2024/01/18 21:04:23 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]}, Next try in [5s], timeout [5m0s] [2024-01-18T21:04:27.090Z] INFO[2024-01-18 21:04:26] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T21:04:27.090Z] INFO[2024-01-18 21:04:26] sched stats: tick interval: 19826 < 1.2s (100%) [2024-01-18T21:04:27.090Z] INFO[2024-01-18 21:04:26] sched stats: tick process durations: 19825 < 500ms (100%) [2024-01-18T21:04:28.978Z] 2024-01-18 21:04:28 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] [2024-01-18T21:04:28.978Z] 2024-01-18 21:04:28 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] under [kube-system] namespace [2024-01-18T21:04:28.978Z] 2024/01/18 21:04:28 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]}, Next try in [5s], timeout [5m0s] [2024-01-18T21:04:34.225Z] 2024-01-18 21:04:33 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] [2024-01-18T21:04:34.225Z] 2024-01-18 21:04:33 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] under [kube-system] namespace [2024-01-18T21:04:34.225Z] 2024/01/18 21:04:33 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]}, Next try in [5s], timeout [5m0s] [2024-01-18T21:04:39.471Z] 2024-01-18 21:04:39 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] [2024-01-18T21:04:39.471Z] 2024-01-18 21:04:39 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] under [kube-system] namespace [2024-01-18T21:04:39.471Z] 2024/01/18 21:04:39 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]}, Next try in [5s], timeout [5m0s] [2024-01-18T21:04:44.726Z] 2024-01-18 21:04:44 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] [2024-01-18T21:04:44.726Z] 2024-01-18 21:04:44 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] under [kube-system] namespace [2024-01-18T21:04:44.726Z] 2024/01/18 21:04:44 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]}, Next try in [5s], timeout [5m0s] [2024-01-18T21:04:49.980Z] 2024-01-18 21:04:49 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] [2024-01-18T21:04:49.980Z] 2024-01-18 21:04:49 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] under [kube-system] namespace [2024-01-18T21:04:49.980Z] 2024/01/18 21:04:49 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]}, Next try in [5s], timeout [5m0s] [2024-01-18T21:04:55.239Z] 2024-01-18 21:04:54 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] [2024-01-18T21:04:55.239Z] 2024-01-18 21:04:54 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] under [kube-system] namespace [2024-01-18T21:04:55.239Z] 2024/01/18 21:04:54 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]}, Next try in [5s], timeout [5m0s] [2024-01-18T21:04:59.402Z] 2024-01-18 21:04:59 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] [2024-01-18T21:04:59.403Z] 2024-01-18 21:04:59 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] under [kube-system] namespace [2024-01-18T21:04:59.403Z] 2024/01/18 21:04:59 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]}, Next try in [5s], timeout [5m0s] [2024-01-18T21:05:04.644Z] 2024-01-18 21:05:04 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] [2024-01-18T21:05:04.644Z] 2024-01-18 21:05:04 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] under [kube-system] namespace [2024-01-18T21:05:04.645Z] 2024/01/18 21:05:04 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]}, Next try in [5s], timeout [5m0s] [2024-01-18T21:05:09.887Z] 2024-01-18 21:05:09 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] [2024-01-18T21:05:09.887Z] 2024-01-18 21:05:09 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] under [kube-system] namespace [2024-01-18T21:05:09.887Z] 2024/01/18 21:05:09 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]}, Next try in [5s], timeout [5m0s] [2024-01-18T21:05:15.130Z] 2024-01-18 21:05:14 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] [2024-01-18T21:05:15.131Z] 2024-01-18 21:05:14 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] under [kube-system] namespace [2024-01-18T21:05:15.131Z] 2024/01/18 21:05:14 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]}, Next try in [5s], timeout [5m0s] [2024-01-18T21:05:20.377Z] 2024-01-18 21:05:19 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] [2024-01-18T21:05:20.377Z] 2024-01-18 21:05:19 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] under [kube-system] namespace [2024-01-18T21:05:20.377Z] 2024/01/18 21:05:19 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]}, Next try in [5s], timeout [5m0s] [2024-01-18T21:05:24.545Z] 2024-01-18 21:05:24 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] [2024-01-18T21:05:24.545Z] 2024-01-18 21:05:24 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] under [kube-system] namespace [2024-01-18T21:05:24.545Z] 2024/01/18 21:05:24 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]}, Next try in [5s], timeout [5m0s] [2024-01-18T21:05:29.791Z] 2024-01-18 21:05:29 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] [2024-01-18T21:05:29.791Z] 2024-01-18 21:05:29 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] under [kube-system] namespace [2024-01-18T21:05:29.791Z] 2024/01/18 21:05:29 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]}, Next try in [5s], timeout [5m0s] [2024-01-18T21:05:35.035Z] 2024-01-18 21:05:34 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] [2024-01-18T21:05:35.035Z] 2024-01-18 21:05:34 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] under [kube-system] namespace [2024-01-18T21:05:35.035Z] 2024/01/18 21:05:34 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]}, Next try in [5s], timeout [5m0s] [2024-01-18T21:05:40.278Z] 2024-01-18 21:05:39 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] [2024-01-18T21:05:40.279Z] 2024-01-18 21:05:39 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] under [kube-system] namespace [2024-01-18T21:05:40.279Z] 2024/01/18 21:05:39 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]}, Next try in [5s], timeout [5m0s] [2024-01-18T21:05:45.524Z] 2024-01-18 21:05:44 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] [2024-01-18T21:05:45.525Z] 2024-01-18 21:05:44 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]] under [kube-system] namespace [2024-01-18T21:05:45.525Z] 2024/01/18 21:05:44 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243]}, Next try in [5s], timeout [5m0s] [2024-01-18T21:05:50.780Z] 2024-01-18 21:05:49 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133]] [2024-01-18T21:05:50.780Z] INFO[2024-01-18 21:05:49] Verifying : Description : Creation and Validation of schedule backup with namespace labels, having schedule name [tp-backup-schedule-1705611762] [2024-01-18T21:05:50.780Z] INFO[2024-01-18 21:05:49] Actual:true, Expected: true [2024-01-18T21:05:50.780Z] 2024-01-18 21:05:49 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243 [2024-01-18T21:05:50.780Z] 2024-01-18 21:05:49 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [mysql-ibm-pxb-0-84849-01-18-21h00m47s mysql-ibm-pxb-1-84849-01-18-21h00m47s mysql-ibm-pxb-2-84849-01-18-21h00m47s] [2024-01-18T21:05:50.781Z] 2024-01-18 21:05:49 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are 9-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,16-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,4-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,8-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,12-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,18-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,19-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,0-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,2-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,14-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,1-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,11-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,6-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,7-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,10-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,13-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,15-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,17-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,3-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,5-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c [2024-01-18T21:05:50.781Z] INFO[2024-01-18 21:05:49] Verifying : Description : Verifying if the labeled namespace [[mysql-ibm-pxb-0-84849-01-18-21h00m47s mysql-ibm-pxb-1-84849-01-18-21h00m47s mysql-ibm-pxb-2-84849-01-18-21h00m47s]] is backed up and checks for labels [9-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,16-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,4-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,8-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,12-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,18-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,19-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,0-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,2-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,14-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,1-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,11-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,6-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,7-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,10-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,13-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,15-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,17-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,3-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,5-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c] applied to backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-210243] [2024-01-18T21:05:50.781Z] INFO[2024-01-18 21:05:49] Actual:true, Expected: true [2024-01-18T21:05:50.781Z] 2024-01-18 21:05:49 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Waiting for 15 minutes for the next schedule backup to be triggered [2024-01-18T21:14:27.947Z] INFO[2024-01-18 21:14:27] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T21:14:27.947Z] INFO[2024-01-18 21:14:27] sched stats: tick interval: 20427 < 1.2s (100%) [2024-01-18T21:14:27.947Z] INFO[2024-01-18 21:14:27] sched stats: tick process durations: 20426 < 500ms (100%) [2024-01-18T21:20:51.006Z] INFO[2024-01-18 21:20:49] Verifying : Description : Fetching the name of the second schedule backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-211746] [2024-01-18T21:20:51.006Z] INFO[2024-01-18 21:20:49] Actual:true, Expected: true [2024-01-18T21:20:51.006Z] 2024-01-18 21:20:49 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-211746] in org [default] [2024-01-18T21:20:51.006Z] 2024-01-18 21:20:49 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-211746] [2024-01-18T21:20:51.006Z] 2024-01-18 21:20:49 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84849-01-18-21h00m47s] in backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-211746] [2024-01-18T21:20:51.006Z] 2024-01-18 21:20:49 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84849-01-18-21h00m47s] in backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-211746] [2024-01-18T21:20:51.006Z] 2024-01-18 21:20:49 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84849-01-18-21h00m47s] [2024-01-18T21:20:51.006Z] 2024-01-18 21:20:49 +0000:[WARNING] [{SetUnsetNSLabelDuringScheduleBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T21:20:51.007Z] 2024-01-18 21:20:49 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T21:20:51.007Z] 2024-01-18 21:20:49 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-84849-01-18-21h00m47s] are [[mysql-data]] [2024-01-18T21:20:51.007Z] 2024-01-18 21:20:49 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T21:20:51.007Z] 2024-01-18 21:20:49 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84849-01-18-21h00m47s] in backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-211746] [2024-01-18T21:20:51.007Z] 2024-01-18 21:20:49 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84849-01-18-21h00m47s] in backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-211746] [2024-01-18T21:20:51.007Z] 2024-01-18 21:20:49 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84849-01-18-21h00m47s] [2024-01-18T21:20:51.007Z] 2024-01-18 21:20:50 +0000:[WARNING] [{SetUnsetNSLabelDuringScheduleBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T21:20:51.007Z] 2024-01-18 21:20:50 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T21:20:51.007Z] 2024-01-18 21:20:50 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-84849-01-18-21h00m47s] are [[mysql-data]] [2024-01-18T21:20:51.007Z] 2024-01-18 21:20:50 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T21:20:51.007Z] 2024-01-18 21:20:50 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84849-01-18-21h00m47s] in backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-211746] [2024-01-18T21:20:51.007Z] 2024-01-18 21:20:50 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84849-01-18-21h00m47s] in backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-211746] [2024-01-18T21:20:51.007Z] 2024-01-18 21:20:50 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84849-01-18-21h00m47s] [2024-01-18T21:20:51.007Z] 2024-01-18 21:20:50 +0000:[WARNING] [{SetUnsetNSLabelDuringScheduleBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T21:20:51.007Z] 2024-01-18 21:20:50 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T21:20:51.007Z] 2024-01-18 21:20:50 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-2-84849-01-18-21h00m47s] are [[mysql-data]] [2024-01-18T21:20:51.007Z] 2024-01-18 21:20:50 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T21:20:51.007Z] 2024-01-18 21:20:50 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating CR cleanup [2024-01-18T21:20:51.007Z] 2024-01-18 21:20:50 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T21:20:51.008Z] 2024-01-18 21:20:50 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133]] [2024-01-18T21:20:51.008Z] INFO[2024-01-18 21:20:50] Verifying : Description : Verification of success and Validation of second schedule backup named [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-211746] of schedule named [tp-backup-schedule-1705611762] [2024-01-18T21:20:51.008Z] INFO[2024-01-18 21:20:50] Actual:true, Expected: true [2024-01-18T21:20:51.008Z] 2024-01-18 21:20:50 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-211746 [2024-01-18T21:20:51.008Z] 2024-01-18 21:20:50 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [mysql-ibm-pxb-0-84849-01-18-21h00m47s mysql-ibm-pxb-1-84849-01-18-21h00m47s mysql-ibm-pxb-2-84849-01-18-21h00m47s] [2024-01-18T21:20:51.008Z] 2024-01-18 21:20:50 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are 9-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,16-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,4-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,8-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,12-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,18-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,19-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,0-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,2-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,14-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,1-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,11-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,6-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,7-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,10-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,13-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,15-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,17-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,3-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,5-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c [2024-01-18T21:20:51.009Z] INFO[2024-01-18 21:20:50] Verifying : Description : Verifying if the labeled namespace [[mysql-ibm-pxb-0-84849-01-18-21h00m47s mysql-ibm-pxb-1-84849-01-18-21h00m47s mysql-ibm-pxb-2-84849-01-18-21h00m47s]] is backed up and checks for labels [9-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,16-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,4-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,8-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,12-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,18-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,19-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,0-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,2-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,14-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,1-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,11-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,6-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,7-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,10-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,13-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,15-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,17-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,3-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,5-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c] applied to backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-211746] [2024-01-18T21:20:51.009Z] INFO[2024-01-18 21:20:50] Actual:true, Expected: true [2024-01-18T21:20:51.009Z] 2024-01-18 21:20:50 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Unset namespace label from one namespace [2024-01-18T21:20:51.009Z] INFO[2024-01-18 21:20:50] Verifying : Description : Removing namespace label map[0-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 1-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 10-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 11-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 12-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 13-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 14-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 15-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 16-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 17-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 18-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 19-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 2-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 3-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 4-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 5-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 6-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 7-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 8-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 9-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c] from namespace mysql-ibm-pxb-0-84849-01-18-21h00m47s [2024-01-18T21:20:51.009Z] INFO[2024-01-18 21:20:50] Actual:true, Expected: true [2024-01-18T21:20:51.009Z] 2024-01-18 21:20:50 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Verify namespace with removed labels is not present in next schedule backup [2024-01-18T21:20:51.266Z] 2024-01-18 21:20:51 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Waiting for [15] minutes for the next schedule backup to be triggered [2024-01-18T21:24:42.696Z] INFO[2024-01-18 21:24:28] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T21:24:42.696Z] INFO[2024-01-18 21:24:28] sched stats: tick interval: 21028 < 1.2s (100%) [2024-01-18T21:24:42.696Z] INFO[2024-01-18 21:24:28] sched stats: tick process durations: 21027 < 500ms (100%) [2024-01-18T21:34:34.296Z] INFO[2024-01-18 21:34:28] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T21:34:34.296Z] INFO[2024-01-18 21:34:28] sched stats: tick interval: 21628 < 1.2s (100%) [2024-01-18T21:34:34.296Z] INFO[2024-01-18 21:34:28] sched stats: tick process durations: 21627 < 500ms (100%) [2024-01-18T21:35:55.659Z] 2024-01-18 21:35:51 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] schedule name tp-backup-schedule-1705611762, Next schedule backup name: tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-213249 [2024-01-18T21:35:55.660Z] 2024-01-18 21:35:51 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Inspect obj backup: last_update_time: create_time_in_sec:1705613570 ownership: > > backup_info: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"3bd432f7-010e-4f1c-8212-96038c7688e1" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"dea6c915-39d6-4a23-8be8-a3c941b7ea79" > backup_path:"kube-system/tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-213249/65021167-0ac5-42c0-9fa8-f49dc29d2a01" stage:Final backup_schedule: cr_name:"tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-213249" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: cr_uid:"65021167-0ac5-42c0-9fa8-f49dc29d2a01" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"9-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,16-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,4-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,8-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,12-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,18-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,19-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,0-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,2-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,14-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,1-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,11-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,6-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,7-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,10-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,13-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,15-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,17-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,3-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,5-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c" target_namespace:"kube-system" > > [2024-01-18T21:35:55.660Z] 2024-01-18 21:35:51 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Next schedule backup name [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-213249] [2024-01-18T21:35:55.660Z] 2024-01-18 21:35:51 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-213249] in org [default] [2024-01-18T21:35:55.660Z] 2024-01-18 21:35:51 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-213249] [2024-01-18T21:35:55.660Z] 2024-01-18 21:35:51 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84849-01-18-21h00m47s] in backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-213249] [2024-01-18T21:35:55.661Z] 2024-01-18 21:35:51 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84849-01-18-21h00m47s] in backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-213249] [2024-01-18T21:35:55.661Z] 2024-01-18 21:35:51 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84849-01-18-21h00m47s] [2024-01-18T21:35:55.661Z] 2024-01-18 21:35:51 +0000:[WARNING] [{SetUnsetNSLabelDuringScheduleBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T21:35:55.661Z] 2024-01-18 21:35:51 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T21:35:55.661Z] 2024-01-18 21:35:51 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-84849-01-18-21h00m47s] are [[mysql-data]] [2024-01-18T21:35:55.661Z] 2024-01-18 21:35:51 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T21:35:55.661Z] 2024-01-18 21:35:51 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84849-01-18-21h00m47s] in backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-213249] [2024-01-18T21:35:55.661Z] 2024-01-18 21:35:51 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84849-01-18-21h00m47s] in backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-213249] [2024-01-18T21:35:55.661Z] 2024-01-18 21:35:51 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84849-01-18-21h00m47s] [2024-01-18T21:35:55.661Z] 2024-01-18 21:35:51 +0000:[WARNING] [{SetUnsetNSLabelDuringScheduleBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T21:35:55.661Z] 2024-01-18 21:35:51 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T21:35:55.661Z] 2024-01-18 21:35:51 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-2-84849-01-18-21h00m47s] are [[mysql-data]] [2024-01-18T21:35:55.661Z] 2024-01-18 21:35:51 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T21:35:55.661Z] 2024-01-18 21:35:51 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating CR cleanup [2024-01-18T21:35:55.661Z] 2024-01-18 21:35:51 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T21:35:55.661Z] 2024-01-18 21:35:51 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133]] [2024-01-18T21:35:55.661Z] INFO[2024-01-18 21:35:51] Verifying : Description : Validation of schedule backup 1 [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-213249] of schedule [tp-backup-schedule-1705611762] after getting triggered and completing [2024-01-18T21:35:55.661Z] INFO[2024-01-18 21:35:51] Actual:true, Expected: true [2024-01-18T21:35:55.661Z] 2024-01-18 21:35:51 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-213249 [2024-01-18T21:35:55.661Z] 2024-01-18 21:35:51 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [mysql-ibm-pxb-1-84849-01-18-21h00m47s mysql-ibm-pxb-2-84849-01-18-21h00m47s] [2024-01-18T21:35:55.662Z] 2024-01-18 21:35:51 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are 9-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,16-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,4-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,8-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,12-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,18-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,19-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,0-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,2-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,14-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,1-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,11-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,6-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,7-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,10-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,13-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,15-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,17-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,3-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,5-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c [2024-01-18T21:35:55.663Z] INFO[2024-01-18 21:35:51] Verifying : Description : Verifying if the labeled namespace [[mysql-ibm-pxb-0-84849-01-18-21h00m47s mysql-ibm-pxb-1-84849-01-18-21h00m47s mysql-ibm-pxb-2-84849-01-18-21h00m47s]] is backed up and checks for labels [9-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,16-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,4-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,8-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,12-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,18-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,19-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,0-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,2-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,14-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,1-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,11-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,6-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,7-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,10-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,13-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,15-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,17-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,3-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,5-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c] applied to backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-213249] [2024-01-18T21:35:55.663Z] INFO[2024-01-18 21:35:51] Actual:true, Expected: true [2024-01-18T21:35:55.663Z] 2024-01-18 21:35:51 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Set the label back to previous namespace [2024-01-18T21:35:55.663Z] INFO[2024-01-18 21:35:51] Verifying : Description : Setting the labels map[0-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 1-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 10-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 11-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 12-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 13-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 14-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 15-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 16-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 17-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 18-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 19-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 2-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 3-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 4-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 5-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 6-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 7-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 8-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c 9-effc84ef-6249-48a9-a606-c9f948cb705c:effc84ef-6249-48a9-a606-c9f948cb705c] back to namespace mysql-ibm-pxb-0-84849-01-18-21h00m47s [2024-01-18T21:35:55.663Z] INFO[2024-01-18 21:35:51] Actual:true, Expected: true [2024-01-18T21:35:55.663Z] 2024-01-18 21:35:51 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Verify namespace inclusion in next schedule backup after setting the namespace labels back [2024-01-18T21:35:55.663Z] 2024-01-18 21:35:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Waiting for [15] minutes for the next schedule backup to be triggered [2024-01-18T21:44:32.267Z] INFO[2024-01-18 21:44:29] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T21:44:32.267Z] INFO[2024-01-18 21:44:29] sched stats: tick interval: 22229 < 1.2s (100%) [2024-01-18T21:44:32.267Z] INFO[2024-01-18 21:44:29] sched stats: tick process durations: 22228 < 500ms (100%) [2024-01-18T21:50:53.754Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] schedule name tp-backup-schedule-1705611762, Next schedule backup name: tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-214753 [2024-01-18T21:50:53.756Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Inspect obj backup: last_update_time: create_time_in_sec:1705614473 ownership: > > backup_info: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c3854186-2fd2-4e80-82d4-8a7bc0e248b9" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"3bd432f7-010e-4f1c-8212-96038c7688e1" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"dea6c915-39d6-4a23-8be8-a3c941b7ea79" > backup_path:"kube-system/tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-214753/c24f3d3a-218a-49c9-b672-8faa15d02679" stage:Final backup_schedule: cr_name:"tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-214753" total_size:32212254720 resource_count:15 backup_location_ref: cloud_credential_ref: cr_uid:"c24f3d3a-218a-49c9-b672-8faa15d02679" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"9-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,16-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,4-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,8-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,12-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,18-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,19-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,0-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,2-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,14-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,1-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,11-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,6-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,7-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,10-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,13-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,15-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,17-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,3-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,5-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c" target_namespace:"kube-system" > > [2024-01-18T21:50:53.756Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Next schedule backup name [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-214753] [2024-01-18T21:50:53.756Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-214753] in org [default] [2024-01-18T21:50:53.756Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-214753] [2024-01-18T21:50:53.756Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84849-01-18-21h00m47s] in backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-214753] [2024-01-18T21:50:53.756Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84849-01-18-21h00m47s] in backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-214753] [2024-01-18T21:50:53.757Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84849-01-18-21h00m47s] [2024-01-18T21:50:53.757Z] 2024-01-18 21:50:52 +0000:[WARNING] [{SetUnsetNSLabelDuringScheduleBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T21:50:53.757Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T21:50:53.757Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-84849-01-18-21h00m47s] are [[mysql-data]] [2024-01-18T21:50:53.757Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T21:50:53.757Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84849-01-18-21h00m47s] in backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-214753] [2024-01-18T21:50:53.757Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84849-01-18-21h00m47s] in backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-214753] [2024-01-18T21:50:53.757Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84849-01-18-21h00m47s] [2024-01-18T21:50:53.757Z] 2024-01-18 21:50:52 +0000:[WARNING] [{SetUnsetNSLabelDuringScheduleBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T21:50:53.757Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T21:50:53.757Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-84849-01-18-21h00m47s] are [[mysql-data]] [2024-01-18T21:50:53.757Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T21:50:53.757Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84849-01-18-21h00m47s] in backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-214753] [2024-01-18T21:50:53.757Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84849-01-18-21h00m47s] in backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-214753] [2024-01-18T21:50:53.757Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84849-01-18-21h00m47s] [2024-01-18T21:50:53.757Z] 2024-01-18 21:50:52 +0000:[WARNING] [{SetUnsetNSLabelDuringScheduleBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T21:50:53.757Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T21:50:53.757Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-2-84849-01-18-21h00m47s] are [[mysql-data]] [2024-01-18T21:50:53.758Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T21:50:53.758Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Validating CR cleanup [2024-01-18T21:50:53.758Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T21:50:53.758Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133]] [2024-01-18T21:50:53.758Z] INFO[2024-01-18 21:50:52] Verifying : Description : Validation of schedule backup 2 [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-214753] of schedule [tp-backup-schedule-1705611762] after getting triggered and completing [2024-01-18T21:50:53.758Z] INFO[2024-01-18 21:50:52] Actual:true, Expected: true [2024-01-18T21:50:53.758Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-214753 [2024-01-18T21:50:53.758Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [mysql-ibm-pxb-0-84849-01-18-21h00m47s mysql-ibm-pxb-1-84849-01-18-21h00m47s mysql-ibm-pxb-2-84849-01-18-21h00m47s] [2024-01-18T21:50:53.758Z] 2024-01-18 21:50:52 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are 9-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,16-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,4-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,8-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,12-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,18-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,19-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,0-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,2-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,14-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,1-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,11-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,6-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,7-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,10-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,13-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,15-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,17-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,3-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,5-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c [2024-01-18T21:50:53.759Z] INFO[2024-01-18 21:50:52] Verifying : Description : Verifying if the labeled namespace [[mysql-ibm-pxb-0-84849-01-18-21h00m47s mysql-ibm-pxb-1-84849-01-18-21h00m47s mysql-ibm-pxb-2-84849-01-18-21h00m47s]] is backed up and checks for labels [9-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,16-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,4-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,8-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,12-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,18-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,19-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,0-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,2-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,14-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,1-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,11-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,6-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,7-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,10-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,13-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,15-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,17-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,3-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c,5-effc84ef-6249-48a9-a606-c9f948cb705c=effc84ef-6249-48a9-a606-c9f948cb705c] applied to backup [tp-backup-schedule-1705611762-dbe116b-interval-2024-01-18-214753] [2024-01-18T21:50:53.759Z] INFO[2024-01-18 21:50:52] Actual:true, Expected: true [2024-01-18T21:50:53.759Z] 2024-01-18 21:50:53 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Deleting schedule named [tp-backup-schedule-1705611762] along with its backups [[]] and schedule policies [[periodic-1705611760]] [2024-01-18T21:50:53.759Z] 2024/01/18 21:50:53 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705611762] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T21:51:03.714Z] 2024/01/18 21:51:03 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705611762] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T21:51:13.667Z] 2024/01/18 21:51:13 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705611762] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T21:51:23.624Z] 2024/01/18 21:51:23 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705611762] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T21:51:33.580Z] 2024/01/18 21:51:33 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705611762] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T21:51:43.541Z] 2024/01/18 21:51:43 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705611762] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T21:51:53.498Z] 2024/01/18 21:51:53 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705611762] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T21:52:05.677Z] 2024/01/18 21:52:03 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705611762] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T21:52:13.790Z] 2024/01/18 21:52:13 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705611762] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T21:52:23.731Z] 2024/01/18 21:52:23 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705611762] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T21:52:33.674Z] 2024/01/18 21:52:33 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705611762] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T21:52:43.621Z] INFO[2024-01-18 21:52:43] Verifying : Description : Verification of deleting backup schedule - tp-backup-schedule-1705611762 [2024-01-18T21:52:43.621Z] INFO[2024-01-18 21:52:43] Actual:true, Expected: true [2024-01-18T21:52:44.181Z] INFO[2024-01-18 21:52:44] Verifying : Description : Deleting backup schedule policies [periodic-1705611760] [2024-01-18T21:52:44.181Z] INFO[2024-01-18 21:52:44] Actual:true, Expected: true [2024-01-18T21:52:44.182Z] 2024-01-18 21:52:44 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Deleting deployed namespaces - [mysql-ibm-pxb-0-84849-01-18-21h00m47s mysql-ibm-pxb-1-84849-01-18-21h00m47s mysql-ibm-pxb-2-84849-01-18-21h00m47s] [2024-01-18T21:52:44.182Z] 2024-01-18 21:52:44 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Destroying apps [2024-01-18T21:52:44.182Z] 2024-01-18 21:52:44 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T21:52:44.182Z] 2024-01-18 21:52:44 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T21:52:44.182Z] 2024-01-18 21:52:44 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] start destroying mysql-ibm app [2024-01-18T21:52:44.182Z] 2024-01-18 21:52:44 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T21:52:44.437Z] 2024-01-18 21:52:44 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T21:52:44.437Z] 2024-01-18 21:52:44 +0000:[WARNING] [{SetUnsetNSLabelDuringScheduleBackup}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T21:52:44.437Z] INFO[2024-01-18 21:52:44] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T21:52:44.437Z] INFO[2024-01-18 21:52:44] Actual:true, Expected: true [2024-01-18T21:52:44.437Z] 2024-01-18 21:52:44 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T21:52:44.438Z] 2024-01-18 21:52:44 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T21:52:44.438Z] 2024-01-18 21:52:44 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] start destroying mysql-ibm app [2024-01-18T21:52:44.438Z] 2024-01-18 21:52:44 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T21:52:44.438Z] 2024-01-18 21:52:44 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T21:52:44.438Z] 2024-01-18 21:52:44 +0000:[WARNING] [{SetUnsetNSLabelDuringScheduleBackup}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T21:52:44.438Z] INFO[2024-01-18 21:52:44] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T21:52:44.438Z] INFO[2024-01-18 21:52:44] Actual:true, Expected: true [2024-01-18T21:52:44.438Z] 2024-01-18 21:52:44 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T21:52:44.694Z] 2024-01-18 21:52:44 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T21:52:44.694Z] 2024-01-18 21:52:44 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] start destroying mysql-ibm app [2024-01-18T21:52:44.949Z] 2024-01-18 21:52:44 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T21:52:44.950Z] 2024-01-18 21:52:44 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T21:52:44.950Z] 2024-01-18 21:52:44 +0000:[WARNING] [{SetUnsetNSLabelDuringScheduleBackup}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T21:52:44.950Z] INFO[2024-01-18 21:52:44] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T21:52:44.950Z] INFO[2024-01-18 21:52:44] Actual:true, Expected: true [2024-01-18T21:52:44.950Z] 2024-01-18 21:52:44 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Cleaning backup locations in map [map[58f04f42-ec7b-44fa-8ccf-2fad02dd09a7:autogenerated-backup-location-1705611755]], cloud credential [autogenerated-cred-1705611754], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T21:52:45.206Z] INFO[2024-01-18 21:52:45] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-1705611755] [2024-01-18T21:52:45.206Z] INFO[2024-01-18 21:52:45] Actual:true, Expected: true [2024-01-18T21:52:45.206Z] 2024-01-18 21:52:45 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705611755] is present [2024-01-18T21:52:45.206Z] 2024/01/18 21:52:45 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705611755 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T21:53:17.267Z] 2024-01-18 21:53:15 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705611755] is present [2024-01-18T21:53:17.267Z] 2024/01/18 21:53:15 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705611755 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T21:53:49.309Z] 2024-01-18 21:53:45 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 aws-global-aws-px-backup-system-test-4cac76dc-bl] [2024-01-18T21:53:49.309Z] INFO[2024-01-18 21:53:45] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-1705611755 [2024-01-18T21:53:49.309Z] INFO[2024-01-18 21:53:45] Actual:true, Expected: true [2024-01-18T21:53:49.309Z] 2024-01-18 21:53:45 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-1705611754] is present [2024-01-18T21:53:49.309Z] INFO[2024-01-18 21:53:45] Verifying : Description : Verifying if cloud cred [autogenerated-cred-1705611754] is present [2024-01-18T21:53:49.309Z] INFO[2024-01-18 21:53:45] Actual:true, Expected: true [2024-01-18T21:53:49.309Z] INFO[2024-01-18 21:53:45] Verifying : Description : Verifying deletion of cloud cred [autogenerated-cred-1705611754] [2024-01-18T21:53:49.309Z] INFO[2024-01-18 21:53:45] Actual:true, Expected: true [2024-01-18T21:53:49.309Z] INFO[2024-01-18 21:53:45] Verifying : Description : Deleting cloud cred autogenerated-cred-1705611754 [2024-01-18T21:53:49.309Z] INFO[2024-01-18 21:53:45] Actual:true, Expected: true [2024-01-18T21:53:49.309Z] INFO[2024-01-18 21:53:45] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T21:53:49.309Z] INFO[2024-01-18 21:53:45] Actual:true, Expected: true [2024-01-18T21:53:49.309Z] INFO[2024-01-18 21:53:45] Verifying : Description : Deleting cluster destination-cluster [2024-01-18T21:53:49.309Z] ERRO[2024-01-18 21:53:45] Actual:rpc error: code = FailedPrecondition desc = cluster [destination-cluster] in use by backupschedule [tp-backup-schedule-1705605709], cannot be deleted, Expected:  [2024-01-18T21:53:49.309Z] INFO[2024-01-18 21:53:45] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-zbbvt] [2024-01-18T21:53:49.309Z] ERRO[2024-01-18 21:53:45] Actual:rpc error: code = Unknown desc = cluster [destination-cluster] has reference to cloud credential [ibm-destination-config-cloud-cred-zbbvt:a271ea5b-dc9d-40f1-a82a-651539c7e486], Expected:  [2024-01-18T21:53:49.309Z] INFO[2024-01-18 21:53:46] Verifying : Description : Deleting cluster source-cluster [2024-01-18T21:53:49.309Z] INFO[2024-01-18 21:53:46] Actual:true, Expected: true [2024-01-18T21:53:49.309Z] INFO[2024-01-18 21:53:46] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-rehqq] [2024-01-18T21:53:49.309Z] INFO[2024-01-18 21:53:46] Actual:true, Expected: true [2024-01-18T21:53:49.309Z] INFO[2024-01-18 21:53:46] --------Test End------ [2024-01-18T21:53:49.309Z] INFO[2024-01-18 21:53:46] #Test: [2024-01-18T21:53:49.309Z] INFO[2024-01-18 21:53:46] #Description: [2024-01-18T21:53:49.309Z] INFO[2024-01-18 21:53:46] ------------------------ [2024-01-18T21:53:49.309Z] 2024-01-18 21:53:46 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-84849-01-18-21h00m47s] [2024-01-18T21:53:49.309Z] 2024-01-18 21:53:46 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-1-84849-01-18-21h00m47s] [2024-01-18T21:53:49.309Z] 2024-01-18 21:53:47 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-2-84849-01-18-21h00m47s] [2024-01-18T21:53:49.309Z] 2024-01-18 21:53:47 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T21:53:49.310Z] 2024-01-18 21:53:47 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Switching context: The context is already [/tmp/destination-config] [2024-01-18T21:53:49.310Z] 2024-01-18 21:53:47 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-84849-01-18-21h00m47s] [2024-01-18T21:53:49.310Z] 2024-01-18 21:53:47 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-1-84849-01-18-21h00m47s] [2024-01-18T21:53:49.310Z] 2024-01-18 21:53:47 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-2-84849-01-18-21h00m47s] [2024-01-18T21:53:49.310Z] 2024-01-18 21:53:48 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {SetUnsetNSLabelDuringScheduleBackup} Create multiple namespaces and set unset namespace labels [2024-01-18T21:53:49.310Z] 2024-01-18 21:53:48 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-18T21:53:49.310Z] 2024/01/18 21:53:48 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T21:53:53.415Z] 2024/01/18 21:53:53 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T21:53:58.655Z] 2024/01/18 21:53:58 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T21:54:03.901Z] 2024/01/18 21:54:03 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T21:54:09.145Z] 2024/01/18 21:54:08 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T21:54:13.312Z] 2024/01/18 21:54:13 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T21:54:18.556Z] 2024/01/18 21:54:18 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T21:54:23.799Z] 2024/01/18 21:54:23 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T21:54:29.046Z] 2024/01/18 21:54:28 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.64.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.8 10.240.64.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T21:54:29.046Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T21:54:29.046Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T21:54:29.046Z] {dial tcp 10.240.64.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-18T21:54:29.972Z] INFO[2024-01-18 21:54:29] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T21:54:29.972Z] INFO[2024-01-18 21:54:29] sched stats: tick interval: 22829 < 1.2s (100%) [2024-01-18T21:54:29.972Z] INFO[2024-01-18 21:54:29] sched stats: tick process durations: 22828 < 500ms (100%) [2024-01-18T21:54:33.249Z] 2024-01-18 21:54:33 +0000:[WARNING] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.64.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.8 10.240.64.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T21:54:33.249Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T21:54:33.249Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T21:54:33.249Z] {dial tcp 10.240.64.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T21:54:33.249Z] 2024-01-18 21:54:33 +0000:[ERROR] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.64.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.8 10.240.64.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T21:54:33.249Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T21:54:33.249Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T21:54:33.249Z] {dial tcp 10.240.64.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T21:54:33.249Z] 2024-01-18 21:54:33 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T21:54:33.249Z] 2024-01-18 21:54:33 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] Switching context to [/tmp/source-config] [2024-01-18T21:54:33.249Z] 2024-01-18 21:54:33 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T21:54:33.249Z] 2024-01-18 21:54:33 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T21:54:33.506Z] 2024-01-18 21:54:33 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T21:54:33.506Z] 2024-01-18 21:54:33 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T21:54:33.506Z] 2024-01-18 21:54:33 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T21:54:33.763Z] 2024-01-18 21:54:33 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T21:54:34.020Z] 2024-01-18 21:54:33 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T21:54:34.581Z] 2024-01-18 21:54:34 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T21:54:34.837Z] 2024-01-18 21:54:34 +0000:[INFO] [{SetUnsetNSLabelDuringScheduleBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T21:54:35.092Z] 2024/01/18 21:54:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:54:45.029Z] 2024/01/18 21:54:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:54:57.203Z] 2024/01/18 21:54:54 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:55:05.283Z] 2024/01/18 21:55:05 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:55:15.242Z] 2024/01/18 21:55:15 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:55:25.232Z] 2024/01/18 21:55:25 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:55:35.173Z] 2024/01/18 21:55:35 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:55:45.202Z] 2024/01/18 21:55:45 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:55:57.379Z] 2024/01/18 21:55:55 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:56:05.465Z] 2024/01/18 21:56:05 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:56:15.414Z] 2024/01/18 21:56:15 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:56:25.367Z] 2024/01/18 21:56:25 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:56:37.540Z] 2024/01/18 21:56:35 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:56:45.626Z] 2024/01/18 21:56:45 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:56:55.569Z] 2024/01/18 21:56:55 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:57:05.510Z] 2024/01/18 21:57:05 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:57:17.726Z] 2024/01/18 21:57:15 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:57:25.811Z] 2024/01/18 21:57:25 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:57:35.762Z] 2024/01/18 21:57:35 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:57:45.705Z] 2024/01/18 21:57:45 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:57:57.872Z] 2024/01/18 21:57:55 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:58:05.958Z] 2024/01/18 21:58:05 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:58:15.939Z] 2024/01/18 21:58:15 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:58:25.881Z] 2024/01/18 21:58:25 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:58:35.821Z] 2024/01/18 21:58:35 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:58:47.994Z] 2024/01/18 21:58:45 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:58:56.077Z] 2024/01/18 21:58:55 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:59:06.020Z] 2024/01/18 21:59:05 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:59:15.977Z] 2024/01/18 21:59:15 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:59:25.972Z] 2024/01/18 21:59:25 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T21:59:38.136Z] 2024-01-18 21:59:35 +0000:[ERROR] [{SetUnsetNSLabelDuringScheduleBackup}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.136Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.136Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.136Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.136Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.136Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.136Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.136Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.136Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.136Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.137Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.137Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.137Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.137Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.137Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.137Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.137Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.137Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.137Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.137Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.137Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.137Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.137Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.137Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.137Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.137Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.137Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.137Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.137Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.137Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-18T21:59:38.137Z] • Failure [3528.322 seconds] [2024-01-18T21:59:38.137Z] {SetUnsetNSLabelDuringScheduleBackup} [2024-01-18T21:59:38.137Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:2309 [2024-01-18T21:59:38.137Z] Create multiple namespaces and set unset namespace labels [] [2024-01-18T21:59:38.137Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:2347 [2024-01-18T21:59:38.137Z] [2024-01-18T21:59:38.137Z] Unexpected error: [2024-01-18T21:59:38.138Z] <*errors.errorString | 0xc00123c6b0>: { [2024-01-18T21:59:38.139Z] s: "failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{No... [2024-01-18T21:59:38.139Z] [2024-01-18T21:59:38.139Z] Gomega truncated this representation as it exceeds 'format.MaxLength'. [2024-01-18T21:59:38.139Z] Consider having the object provide a custom 'GomegaStringer' representation [2024-01-18T21:59:38.139Z] or adjust the parameters in Gomega's 'format' package. [2024-01-18T21:59:38.139Z] [2024-01-18T21:59:38.139Z] Learn more here: https://onsi.github.io/gomega/#adjusting-output [2024-01-18T21:59:38.139Z] [2024-01-18T21:59:38.139Z] failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.139Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.139Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.139Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.139Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.139Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.139Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.140Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.140Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.140Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.140Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.140Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.140Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.140Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.140Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.140Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.140Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.140Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.140Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.140Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.140Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.140Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.140Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.141Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.141Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.141Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.141Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.141Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.141Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T21:59:38.141Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-18T21:59:38.141Z] occurred [2024-01-18T21:59:38.141Z] [2024-01-18T21:59:38.141Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-18T21:59:38.141Z] [2024-01-18T21:59:38.141Z] Full Stack Trace [2024-01-18T21:59:38.141Z] github.com/portworx/torpedo/pkg/log.FailOnError({0x88a5060, 0xc00123c6b0}, {0x7f74bfd?, 0x6cc0320?}, {0x0?, 0x6cc0320?, 0xc000e649c0?}) [2024-01-18T21:59:38.141Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 +0x290 [2024-01-18T21:59:38.141Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest.func1() [2024-01-18T21:59:38.141Z] /go/src/github.com/portworx/torpedo/tests/common.go:7283 +0x32 [2024-01-18T21:59:38.141Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest({0xc00189f080, 0x3, 0x4}) [2024-01-18T21:59:38.141Z] /go/src/github.com/portworx/torpedo/tests/common.go:7299 +0x778 [2024-01-18T21:59:38.141Z] github.com/portworx/torpedo/tests/backup.glob..func67.3() [2024-01-18T21:59:38.141Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:2473 +0x530 [2024-01-18T21:59:38.141Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-18T21:59:38.141Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-18T21:59:38.142Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-18T21:59:38.142Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-18T21:59:38.142Z] created by testing.(*T).Run [2024-01-18T21:59:38.142Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-18T21:59:38.142Z] ------------------------------ [2024-01-18T21:59:38.142Z] STEP: schedule applications [2024-01-18T21:59:38.142Z] STEP: schedule applications [2024-01-18T21:59:38.142Z] STEP: schedule applications [2024-01-18T21:59:38.142Z] STEP: Validate applications [2024-01-18T21:59:38.142Z] STEP: validate applications [2024-01-18T21:59:38.142Z] STEP: validate mysql-ibm app's volumes [2024-01-18T21:59:38.142Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T21:59:38.142Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T21:59:38.142Z] STEP: get mysql-ibm app's volume: pvc-c3854186-2fd2-4e80-82d4-8a7bc0e248b9 inspected by the volume driver [2024-01-18T21:59:38.142Z] STEP: wait for mysql-ibm app to start running [2024-01-18T21:59:38.142Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T21:59:38.142Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T21:59:38.142Z] STEP: Validate Px pod restart count [2024-01-18T21:59:38.142Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T21:59:38.142Z] STEP: validate mysql-ibm app's volumes [2024-01-18T21:59:38.142Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T21:59:38.142Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T21:59:38.142Z] STEP: get mysql-ibm app's volume: pvc-3bd432f7-010e-4f1c-8212-96038c7688e1 inspected by the volume driver [2024-01-18T21:59:38.142Z] STEP: wait for mysql-ibm app to start running [2024-01-18T21:59:38.142Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T21:59:38.142Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T21:59:38.142Z] STEP: Validate Px pod restart count [2024-01-18T21:59:38.142Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T21:59:38.142Z] STEP: validate mysql-ibm app's volumes [2024-01-18T21:59:38.142Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T21:59:38.143Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T21:59:38.143Z] STEP: get mysql-ibm app's volume: pvc-dea6c915-39d6-4a23-8be8-a3c941b7ea79 inspected by the volume driver [2024-01-18T21:59:38.143Z] STEP: wait for mysql-ibm app to start running [2024-01-18T21:59:38.143Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T21:59:38.143Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T21:59:38.143Z] STEP: Validate Px pod restart count [2024-01-18T21:59:38.143Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T21:59:38.143Z] STEP: Adding labels to namespaces [2024-01-18T21:59:38.143Z] STEP: Generating namespace label string from label map for multiple namespace [2024-01-18T21:59:38.143Z] STEP: Creating cloud credentials and registering backup location [2024-01-18T21:59:38.143Z] STEP: Configure source and destination clusters with px-central-admin ctx [2024-01-18T21:59:38.143Z] STEP: Create schedule policy [2024-01-18T21:59:38.143Z] STEP: Creating a schedule backup [2024-01-18T21:59:38.143Z] STEP: Unset namespace label from one namespace [2024-01-18T21:59:38.143Z] STEP: Verify namespace with removed labels is not present in next schedule backup [2024-01-18T21:59:38.143Z] STEP: Set the label back to previous namespace [2024-01-18T21:59:38.143Z] STEP: Verify namespace inclusion in next schedule backup after setting the namespace labels back [2024-01-18T21:59:38.143Z] STEP: destroy apps [2024-01-18T21:59:38.143Z] STEP: destroy the mysql-ibm app's volumes [2024-01-18T21:59:38.143Z] STEP: start destroying mysql-ibm app [2024-01-18T21:59:38.143Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T21:59:38.143Z] STEP: destroy the mysql-ibm app's volumes [2024-01-18T21:59:38.143Z] STEP: start destroying mysql-ibm app [2024-01-18T21:59:38.143Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T21:59:38.143Z] STEP: destroy the mysql-ibm app's volumes [2024-01-18T21:59:38.143Z] STEP: start destroying mysql-ibm app [2024-01-18T21:59:38.143Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T21:59:38.157Z] 2024-01-18 21:59:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Log Dir: /testresults//NamespaceLabelledBackupOfEmptyNamespace.log [2024-01-18T21:59:38.157Z] INFO[2024-01-18 21:59:35] --------Test Start------ [2024-01-18T21:59:38.157Z] INFO[2024-01-18 21:59:35] #Test: NamespaceLabelledBackupOfEmptyNamespace [2024-01-18T21:59:38.157Z] INFO[2024-01-18 21:59:35] #Description: NamespaceLabelledBackupOfEmptyNamespace takes namespace labelled backup of empty namespace and restores it [2024-01-18T21:59:38.157Z] INFO[2024-01-18 21:59:35] ------------------------ [2024-01-18T21:59:38.157Z] 2024-01-18 21:59:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Deploy applications [2024-01-18T21:59:38.157Z] 2024-01-18 21:59:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T21:59:38.157Z] 2024-01-18 21:59:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T21:59:38.157Z] 2024-01-18 21:59:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T21:59:38.157Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T21:59:38.157Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T21:59:38.157Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T21:59:38.158Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T21:59:38.158Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T21:59:38.159Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T21:59:38.159Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T21:59:38.159Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T21:59:38.159Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T21:59:38.159Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T21:59:38.159Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T21:59:38.159Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T21:59:38.159Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T21:59:38.160Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T21:59:38.160Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T21:59:38.160Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T21:59:38.160Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T21:59:38.161Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T21:59:38.161Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T21:59:38.161Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T21:59:38.161Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T21:59:38.161Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T21:59:38.161Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T21:59:38.162Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T21:59:38.162Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T21:59:38.162Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T21:59:38.162Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T21:59:38.162Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Created namespaces [mysql-ibm-pxb-0-86693-01-18-21h59m35s mysql-ibm-pxb-1-86693-01-18-21h59m35s mysql-ibm-pxb-2-86693-01-18-21h59m35s] [2024-01-18T21:59:38.162Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validate applications [2024-01-18T21:59:38.162Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validate applications [2024-01-18T21:59:38.162Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validating mysql-ibm app [2024-01-18T21:59:38.162Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validating mysql-ibm app's volumes [2024-01-18T21:59:38.162Z] 2024-01-18 21:59:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T21:59:38.162Z] 2024/01/18 21:59:36 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T21:59:48.108Z] 2024/01/18 21:59:46 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T21:59:58.056Z] 2024-01-18 21:59:56 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-86693-01-18-21h59m35s [2024-01-18T21:59:58.056Z] 2024-01-18 21:59:56 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T21:59:58.056Z] 2024-01-18 21:59:57 +0000:[WARNING] [{NamespaceLabelledBackupOfEmptyNamespace}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T21:59:58.056Z] 2024-01-18 21:59:57 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] wait for mysql-ibm app to start running [2024-01-18T21:59:58.056Z] 2024-01-18 21:59:57 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T21:59:58.056Z] 2024/01/18 21:59:57 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T21:59:58.056Z] pod name:mysql-7d766cd65c-qn582 namespace:mysql-ibm-pxb-0-86693-01-18-21h59m35s running:false ready:false node:10.240.64.27 [2024-01-18T21:59:58.056Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T22:00:08.012Z] 2024/01/18 22:00:07 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T22:00:08.012Z] pod name:mysql-7d766cd65c-qn582 namespace:mysql-ibm-pxb-0-86693-01-18-21h59m35s running:true ready:false node:10.240.64.27 [2024-01-18T22:00:08.012Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T22:00:17.950Z] 2024/01/18 22:00:17 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T22:00:17.950Z] pod name:mysql-7d766cd65c-qn582 namespace:mysql-ibm-pxb-0-86693-01-18-21h59m35s running:true ready:false node:10.240.64.27 [2024-01-18T22:00:17.950Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T22:00:27.934Z] 2024/01/18 22:00:27 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T22:00:27.934Z] pod name:mysql-7d766cd65c-qn582 namespace:mysql-ibm-pxb-0-86693-01-18-21h59m35s running:true ready:false node:10.240.64.27 [2024-01-18T22:00:27.934Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T22:00:37.871Z] 2024/01/18 22:00:37 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T22:00:37.871Z] pod name:mysql-7d766cd65c-qn582 namespace:mysql-ibm-pxb-0-86693-01-18-21h59m35s running:true ready:false node:10.240.64.27 [2024-01-18T22:00:37.871Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T22:00:47.809Z] 2024/01/18 22:00:47 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T22:00:47.809Z] pod name:mysql-7d766cd65c-qn582 namespace:mysql-ibm-pxb-0-86693-01-18-21h59m35s running:true ready:false node:10.240.64.27 [2024-01-18T22:00:47.809Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T22:00:59.971Z] 2024/01/18 22:00:57 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T22:00:59.971Z] pod name:mysql-7d766cd65c-qn582 namespace:mysql-ibm-pxb-0-86693-01-18-21h59m35s running:true ready:false node:10.240.64.27 [2024-01-18T22:00:59.971Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T22:01:08.081Z] 2024/01/18 22:01:07 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T22:01:08.081Z] pod name:mysql-7d766cd65c-qn582 namespace:mysql-ibm-pxb-0-86693-01-18-21h59m35s running:true ready:false node:10.240.64.27 [2024-01-18T22:01:08.081Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T22:01:20.248Z] 2024-01-18 22:01:18 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T22:01:20.249Z] 2024-01-18 22:01:18 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T22:01:20.249Z] 2024-01-18 22:01:18 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] validate if mysql-ibm app's volumes are setup [2024-01-18T22:01:20.249Z] 2024-01-18 22:01:18 +0000:[WARNING] [{NamespaceLabelledBackupOfEmptyNamespace}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T22:01:20.249Z] 2024-01-18 22:01:18 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T22:01:20.249Z] 2024-01-18 22:01:18 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T22:01:20.249Z] 2024-01-18 22:01:18 +0000:[WARNING] [{NamespaceLabelledBackupOfEmptyNamespace}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T22:01:20.249Z] 2024-01-18 22:01:18 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validating mysql-ibm app [2024-01-18T22:01:20.249Z] 2024-01-18 22:01:18 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validating mysql-ibm app's volumes [2024-01-18T22:01:20.249Z] 2024-01-18 22:01:18 +0000:[WARNING] [{NamespaceLabelledBackupOfEmptyNamespace}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T22:01:20.249Z] 2024-01-18 22:01:18 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T22:01:20.249Z] 2024-01-18 22:01:18 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-1-86693-01-18-21h59m35s [2024-01-18T22:01:20.249Z] 2024-01-18 22:01:18 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T22:01:20.249Z] 2024-01-18 22:01:18 +0000:[WARNING] [{NamespaceLabelledBackupOfEmptyNamespace}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T22:01:20.249Z] 2024-01-18 22:01:18 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] wait for mysql-ibm app to start running [2024-01-18T22:01:20.249Z] 2024-01-18 22:01:18 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T22:01:20.249Z] 2024-01-18 22:01:18 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T22:01:20.249Z] 2024-01-18 22:01:19 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T22:01:20.249Z] 2024-01-18 22:01:20 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] validate if mysql-ibm app's volumes are setup [2024-01-18T22:01:20.249Z] 2024-01-18 22:01:20 +0000:[WARNING] [{NamespaceLabelledBackupOfEmptyNamespace}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T22:01:20.249Z] 2024-01-18 22:01:20 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T22:01:20.249Z] 2024-01-18 22:01:20 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T22:01:20.250Z] 2024-01-18 22:01:20 +0000:[WARNING] [{NamespaceLabelledBackupOfEmptyNamespace}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T22:01:20.250Z] 2024-01-18 22:01:20 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validating mysql-ibm app [2024-01-18T22:01:20.250Z] 2024-01-18 22:01:20 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validating mysql-ibm app's volumes [2024-01-18T22:01:20.250Z] 2024-01-18 22:01:20 +0000:[WARNING] [{NamespaceLabelledBackupOfEmptyNamespace}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T22:01:20.250Z] 2024-01-18 22:01:20 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T22:01:20.250Z] 2024-01-18 22:01:20 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-2-86693-01-18-21h59m35s [2024-01-18T22:01:20.506Z] 2024-01-18 22:01:20 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T22:01:20.506Z] 2024-01-18 22:01:20 +0000:[WARNING] [{NamespaceLabelledBackupOfEmptyNamespace}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T22:01:20.506Z] 2024-01-18 22:01:20 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] wait for mysql-ibm app to start running [2024-01-18T22:01:20.506Z] 2024-01-18 22:01:20 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T22:01:20.772Z] 2024-01-18 22:01:20 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T22:01:21.356Z] 2024-01-18 22:01:21 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T22:01:22.287Z] 2024-01-18 22:01:22 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] validate if mysql-ibm app's volumes are setup [2024-01-18T22:01:22.287Z] 2024-01-18 22:01:22 +0000:[WARNING] [{NamespaceLabelledBackupOfEmptyNamespace}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T22:01:22.287Z] 2024-01-18 22:01:22 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T22:01:22.287Z] 2024-01-18 22:01:22 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T22:01:22.287Z] 2024-01-18 22:01:22 +0000:[WARNING] [{NamespaceLabelledBackupOfEmptyNamespace}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T22:01:22.287Z] 2024-01-18 22:01:22 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Generating namespace label string from label map [2024-01-18T22:01:22.287Z] 2024-01-18 22:01:22 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.glob..func23.2.2:#1274] - labels for namespace are [0-8e0f81cf-0968-428f-9193-d6dec8a6c7f9=8e0f81cf-0968-428f-9193-d6dec8a6c7f9,1-8e0f81cf-0968-428f-9193-d6dec8a6c7f9=8e0f81cf-0968-428f-9193-d6dec8a6c7f9] [2024-01-18T22:01:22.287Z] 2024-01-18 22:01:22 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Creating cloud credentials and registering backup location [2024-01-18T22:01:22.543Z] 2024-01-18 22:01:22 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-oacqz] for org [default] with [aws] as provider [2024-01-18T22:01:22.543Z] 2024-01-18 22:01:22 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-18T22:01:22.543Z] 2024-01-18 22:01:22 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-oacqz] is present [2024-01-18T22:01:22.543Z] INFO[2024-01-18 22:01:22] Verifying : Description : Creating cloud credential named autogenerated-cred-oacqz [2024-01-18T22:01:22.543Z] INFO[2024-01-18 22:01:22] Actual:true, Expected: true [2024-01-18T22:01:22.543Z] 2024-01-18 22:01:22 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Created Cloud Credentials with name - autogenerated-cred-oacqz [2024-01-18T22:01:26.708Z] INFO[2024-01-18 22:01:26] Verifying : Description : Creating backup location autogenerated-backup-location-ciryj [2024-01-18T22:01:26.708Z] INFO[2024-01-18 22:01:26] Actual:true, Expected: true [2024-01-18T22:01:26.708Z] 2024-01-18 22:01:26 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Configuring source and destination clusters with px-central-admin ctx [2024-01-18T22:01:26.708Z] INFO[2024-01-18 22:01:26] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T22:01:26.708Z] INFO[2024-01-18 22:01:26] Actual:true, Expected: true [2024-01-18T22:01:26.708Z] 2024-01-18 22:01:26 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T22:01:26.708Z] 2024-01-18 22:01:26 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T22:01:26.708Z] 2024-01-18 22:01:26 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T22:01:26.708Z] 2024-01-18 22:01:26 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T22:01:26.708Z] 2024-01-18 22:01:26 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Create cluster [source-cluster] in org [default] [2024-01-18T22:01:26.708Z] 2024-01-18 22:01:26 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T22:01:26.709Z] 2024-01-18 22:01:26 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T22:01:26.709Z] 2024-01-18 22:01:26 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Create cluster [destination-cluster] in org [default] [2024-01-18T22:01:26.709Z] 2024-01-18 22:01:26 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T22:01:26.709Z] 2024-01-18 22:01:26 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T22:01:26.709Z] 2024-01-18 22:01:26 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-vsqmt] for IBM [2024-01-18T22:01:26.709Z] 2024-01-18 22:01:26 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-vsqmt] for org [default] with [ibm] as provider [2024-01-18T22:01:26.709Z] 2024-01-18 22:01:26 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T22:01:26.709Z] 2024-01-18 22:01:26 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T22:01:26.709Z] 2024-01-18 22:01:26 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-vsqmt] is present [2024-01-18T22:01:26.709Z] 2024-01-18 22:01:26 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Create cluster [source-cluster] in org [default] [2024-01-18T22:01:27.635Z] 2024-01-18 22:01:27 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-emwoc] for IBM [2024-01-18T22:01:27.635Z] 2024-01-18 22:01:27 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-emwoc] for org [default] with [ibm] as provider [2024-01-18T22:01:27.636Z] 2024-01-18 22:01:27 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T22:01:27.636Z] 2024-01-18 22:01:27 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T22:01:27.636Z] 2024-01-18 22:01:27 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-emwoc] is present [2024-01-18T22:01:27.636Z] 2024-01-18 22:01:27 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Create cluster [destination-cluster] in org [default] [2024-01-18T22:01:27.636Z] INFO[2024-01-18 22:01:27] Verifying : Description : Verifying creation of source [source-cluster] and destination [destination-cluster] clusters with px-central-admin ctx [2024-01-18T22:01:27.636Z] INFO[2024-01-18 22:01:27] Actual:true, Expected: true [2024-01-18T22:01:27.636Z] INFO[2024-01-18 22:01:27] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-18T22:01:27.636Z] INFO[2024-01-18 22:01:27] Actual:Online, Expected: Online [2024-01-18T22:01:27.636Z] INFO[2024-01-18 22:01:27] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-18T22:01:27.636Z] INFO[2024-01-18 22:01:27] Actual:true, Expected: true [2024-01-18T22:01:27.636Z] 2024-01-18 22:01:27 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Uid of [source-cluster] cluster is 429b438e-d969-4368-ac1e-63cab57cd1da [2024-01-18T22:01:27.636Z] 2024-01-18 22:01:27 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Creating schedule policy [2024-01-18T22:01:27.893Z] 2024-01-18 22:01:27 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Creating a schedule backup of empty namespace with namespace label filter [2024-01-18T22:01:28.818Z] 2024-01-18 22:01:28 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-18T22:01:28.818Z] 2024-01-18 22:01:28 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Volume snapshot classes found in the cluster - [2024-01-18T22:01:28.818Z] 2024-01-18 22:01:28 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] ibmc-vpcblock-snapshot [2024-01-18T22:01:28.818Z] 2024-01-18 22:01:28 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T22:02:36.443Z] 2024-01-18 22:02:29 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] first schedule backup for schedule name [tp-backup-schedule-sluzc] is [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129] [2024-01-18T22:02:36.443Z] 2024/01/18 22:02:29 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T22:03:02.940Z] 2024-01-18 22:02:59 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validating backup [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129] in org [default] [2024-01-18T22:03:02.940Z] 2024-01-18 22:02:59 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129] [2024-01-18T22:03:02.940Z] 2024-01-18 22:02:59 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validating CR cleanup [2024-01-18T22:03:02.940Z] 2024-01-18 22:02:59 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T22:03:02.940Z] 2024-01-18 22:02:59 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] [2024-01-18T22:03:02.941Z] 2024-01-18 22:02:59 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] under [kube-system] namespace [2024-01-18T22:03:02.941Z] 2024/01/18 22:02:59 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]}, Next try in [5s], timeout [5m0s] [2024-01-18T22:03:05.456Z] 2024-01-18 22:03:04 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] [2024-01-18T22:03:05.456Z] 2024-01-18 22:03:04 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] under [kube-system] namespace [2024-01-18T22:03:05.457Z] 2024/01/18 22:03:04 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]}, Next try in [5s], timeout [5m0s] [2024-01-18T22:03:10.700Z] 2024-01-18 22:03:09 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] [2024-01-18T22:03:10.700Z] 2024-01-18 22:03:09 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] under [kube-system] namespace [2024-01-18T22:03:10.700Z] 2024/01/18 22:03:09 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]}, Next try in [5s], timeout [5m0s] [2024-01-18T22:03:15.948Z] 2024-01-18 22:03:14 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] [2024-01-18T22:03:15.948Z] 2024-01-18 22:03:14 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] under [kube-system] namespace [2024-01-18T22:03:15.948Z] 2024/01/18 22:03:14 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]}, Next try in [5s], timeout [5m0s] [2024-01-18T22:03:20.111Z] 2024-01-18 22:03:19 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] [2024-01-18T22:03:20.111Z] 2024-01-18 22:03:19 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] under [kube-system] namespace [2024-01-18T22:03:20.111Z] 2024/01/18 22:03:19 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]}, Next try in [5s], timeout [5m0s] [2024-01-18T22:03:25.355Z] 2024-01-18 22:03:25 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] [2024-01-18T22:03:25.355Z] 2024-01-18 22:03:25 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] under [kube-system] namespace [2024-01-18T22:03:25.355Z] 2024/01/18 22:03:25 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]}, Next try in [5s], timeout [5m0s] [2024-01-18T22:03:30.605Z] 2024-01-18 22:03:30 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] [2024-01-18T22:03:30.605Z] 2024-01-18 22:03:30 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] under [kube-system] namespace [2024-01-18T22:03:30.605Z] 2024/01/18 22:03:30 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]}, Next try in [5s], timeout [5m0s] [2024-01-18T22:03:35.846Z] 2024-01-18 22:03:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] [2024-01-18T22:03:35.847Z] 2024-01-18 22:03:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] under [kube-system] namespace [2024-01-18T22:03:35.847Z] 2024/01/18 22:03:35 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]}, Next try in [5s], timeout [5m0s] [2024-01-18T22:03:41.090Z] 2024-01-18 22:03:40 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] [2024-01-18T22:03:41.090Z] 2024-01-18 22:03:40 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] under [kube-system] namespace [2024-01-18T22:03:41.090Z] 2024/01/18 22:03:40 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]}, Next try in [5s], timeout [5m0s] [2024-01-18T22:03:45.257Z] 2024-01-18 22:03:45 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] [2024-01-18T22:03:45.257Z] 2024-01-18 22:03:45 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] under [kube-system] namespace [2024-01-18T22:03:45.257Z] 2024/01/18 22:03:45 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]}, Next try in [5s], timeout [5m0s] [2024-01-18T22:03:50.505Z] 2024-01-18 22:03:50 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] [2024-01-18T22:03:50.505Z] 2024-01-18 22:03:50 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] under [kube-system] namespace [2024-01-18T22:03:50.505Z] 2024/01/18 22:03:50 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]}, Next try in [5s], timeout [5m0s] [2024-01-18T22:03:55.756Z] 2024-01-18 22:03:55 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] [2024-01-18T22:03:55.756Z] 2024-01-18 22:03:55 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] under [kube-system] namespace [2024-01-18T22:03:55.756Z] 2024/01/18 22:03:55 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]}, Next try in [5s], timeout [5m0s] [2024-01-18T22:04:01.003Z] 2024-01-18 22:04:00 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] [2024-01-18T22:04:01.003Z] 2024-01-18 22:04:00 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] under [kube-system] namespace [2024-01-18T22:04:01.004Z] 2024/01/18 22:04:00 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]}, Next try in [5s], timeout [5m0s] [2024-01-18T22:04:06.325Z] 2024-01-18 22:04:05 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] [2024-01-18T22:04:06.325Z] 2024-01-18 22:04:05 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] under [kube-system] namespace [2024-01-18T22:04:06.325Z] 2024/01/18 22:04:05 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]}, Next try in [5s], timeout [5m0s] [2024-01-18T22:04:11.569Z] 2024-01-18 22:04:10 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] [2024-01-18T22:04:11.569Z] 2024-01-18 22:04:10 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] under [kube-system] namespace [2024-01-18T22:04:11.569Z] 2024/01/18 22:04:10 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]}, Next try in [5s], timeout [5m0s] [2024-01-18T22:04:15.735Z] 2024-01-18 22:04:15 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] [2024-01-18T22:04:15.735Z] 2024-01-18 22:04:15 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] under [kube-system] namespace [2024-01-18T22:04:15.735Z] 2024/01/18 22:04:15 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]}, Next try in [5s], timeout [5m0s] [2024-01-18T22:04:20.978Z] 2024-01-18 22:04:20 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] [2024-01-18T22:04:20.978Z] 2024-01-18 22:04:20 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] under [kube-system] namespace [2024-01-18T22:04:20.979Z] 2024/01/18 22:04:20 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]}, Next try in [5s], timeout [5m0s] [2024-01-18T22:04:26.220Z] 2024-01-18 22:04:25 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] [2024-01-18T22:04:26.220Z] 2024-01-18 22:04:25 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] under [kube-system] namespace [2024-01-18T22:04:26.220Z] 2024/01/18 22:04:25 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]}, Next try in [5s], timeout [5m0s] [2024-01-18T22:04:31.462Z] INFO[2024-01-18 22:04:30] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T22:04:31.462Z] INFO[2024-01-18 22:04:30] sched stats: tick interval: 23430 < 1.2s (100%) [2024-01-18T22:04:31.462Z] INFO[2024-01-18 22:04:30] sched stats: tick process durations: 23429 < 500ms (100%) [2024-01-18T22:04:31.462Z] 2024-01-18 22:04:30 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] [2024-01-18T22:04:31.462Z] 2024-01-18 22:04:30 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] under [kube-system] namespace [2024-01-18T22:04:31.462Z] 2024/01/18 22:04:30 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]}, Next try in [5s], timeout [5m0s] [2024-01-18T22:04:36.701Z] 2024-01-18 22:04:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] [2024-01-18T22:04:36.701Z] 2024-01-18 22:04:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]] under [kube-system] namespace [2024-01-18T22:04:36.701Z] 2024/01/18 22:04:35 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129]}, Next try in [5s], timeout [5m0s] [2024-01-18T22:04:40.862Z] 2024-01-18 22:04:40 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133]] [2024-01-18T22:04:40.862Z] INFO[2024-01-18 22:04:40] Verifying : Description : Creation and Validation of empty namespace labelled schedule backup [tp-backup-schedule-sluzc] with label [0-8e0f81cf-0968-428f-9193-d6dec8a6c7f9=8e0f81cf-0968-428f-9193-d6dec8a6c7f9,1-8e0f81cf-0968-428f-9193-d6dec8a6c7f9=8e0f81cf-0968-428f-9193-d6dec8a6c7f9] [2024-01-18T22:04:40.862Z] INFO[2024-01-18 22:04:40] Actual:true, Expected: true [2024-01-18T22:04:40.862Z] 2024-01-18 22:04:40 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129 [2024-01-18T22:04:41.117Z] 2024-01-18 22:04:40 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [] [2024-01-18T22:04:41.117Z] 2024-01-18 22:04:40 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are 0-8e0f81cf-0968-428f-9193-d6dec8a6c7f9=8e0f81cf-0968-428f-9193-d6dec8a6c7f9,1-8e0f81cf-0968-428f-9193-d6dec8a6c7f9=8e0f81cf-0968-428f-9193-d6dec8a6c7f9 [2024-01-18T22:04:41.118Z] INFO[2024-01-18 22:04:40] Verifying : Description : Verifying that no namespace are included in this schedule backup tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129 as no namespaces are labelled with 0-8e0f81cf-0968-428f-9193-d6dec8a6c7f9=8e0f81cf-0968-428f-9193-d6dec8a6c7f9,1-8e0f81cf-0968-428f-9193-d6dec8a6c7f9=8e0f81cf-0968-428f-9193-d6dec8a6c7f9 label [2024-01-18T22:04:41.118Z] INFO[2024-01-18 22:04:40] Actual:true, Expected: true [2024-01-18T22:04:41.118Z] 2024-01-18 22:04:40 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Waiting for [15] minutes for the next schedule backup to be triggered [2024-01-18T22:14:32.698Z] INFO[2024-01-18 22:14:31] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T22:14:32.698Z] INFO[2024-01-18 22:14:31] sched stats: tick interval: 24031 < 1.2s (100%) [2024-01-18T22:14:32.698Z] INFO[2024-01-18 22:14:31] sched stats: tick process durations: 24030 < 500ms (100%) [2024-01-18T22:19:54.161Z] 2024-01-18 22:19:40 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] schedule name tp-backup-schedule-sluzc, Next schedule backup name: tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-221632 [2024-01-18T22:19:54.162Z] 2024-01-18 22:19:41 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Inspect obj backup: last_update_time: create_time_in_sec:1705616193 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-221632/e4e78c9a-5bc5-463f-8c84-af24de8a74f3" stage:Final backup_schedule: cr_name:"tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-221632" backup_location_ref: cloud_credential_ref: cr_uid:"e4e78c9a-5bc5-463f-8c84-af24de8a74f3" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-8e0f81cf-0968-428f-9193-d6dec8a6c7f9=8e0f81cf-0968-428f-9193-d6dec8a6c7f9,1-8e0f81cf-0968-428f-9193-d6dec8a6c7f9=8e0f81cf-0968-428f-9193-d6dec8a6c7f9" target_namespace:"kube-system" > > [2024-01-18T22:19:54.162Z] INFO[2024-01-18 22:19:41] Verifying : Description : Verifying creation of second namespace labelled schedule backup tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-221632 of emtpy namespace [2024-01-18T22:19:54.162Z] INFO[2024-01-18 22:19:41] Actual:true, Expected: true [2024-01-18T22:19:54.162Z] 2024-01-18 22:19:41 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Restoring first schedule backup of empty namespace with namespace label filter [2024-01-18T22:19:54.162Z] INFO[2024-01-18 22:19:41] Verifying : Description : Verifying restore failure of empty namespace backup [2024-01-18T22:19:54.162Z] INFO[2024-01-18 22:19:41] Actual:true, Expected: true [2024-01-18T22:19:54.162Z] 2024-01-18 22:19:41 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Deleting few schedule backup with empty namespaces [2024-01-18T22:19:54.162Z] 2024-01-18 22:19:41 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Waiting for [15] minutes for the next schedule backup to be triggered [2024-01-18T22:24:45.654Z] INFO[2024-01-18 22:24:31] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T22:24:45.654Z] INFO[2024-01-18 22:24:31] sched stats: tick interval: 24631 < 1.2s (100%) [2024-01-18T22:24:45.654Z] INFO[2024-01-18 22:24:31] sched stats: tick process durations: 24630 < 500ms (100%) [2024-01-18T22:34:37.236Z] INFO[2024-01-18 22:34:32] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T22:34:37.236Z] INFO[2024-01-18 22:34:32] sched stats: tick interval: 25232 < 1.2s (100%) [2024-01-18T22:34:37.236Z] INFO[2024-01-18 22:34:32] sched stats: tick process durations: 25231 < 500ms (100%) [2024-01-18T22:34:42.482Z] 2024-01-18 22:34:41 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] schedule name tp-backup-schedule-sluzc, Next schedule backup name: tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-223136 [2024-01-18T22:34:42.482Z] 2024-01-18 22:34:41 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Inspect obj backup: last_update_time: create_time_in_sec:1705617096 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-223136/238a21d3-15d4-4f1a-8c96-ba67b1e78435" stage:Final backup_schedule: cr_name:"tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-223136" backup_location_ref: cloud_credential_ref: cr_uid:"238a21d3-15d4-4f1a-8c96-ba67b1e78435" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-8e0f81cf-0968-428f-9193-d6dec8a6c7f9=8e0f81cf-0968-428f-9193-d6dec8a6c7f9,1-8e0f81cf-0968-428f-9193-d6dec8a6c7f9=8e0f81cf-0968-428f-9193-d6dec8a6c7f9" target_namespace:"kube-system" > > [2024-01-18T22:34:42.482Z] INFO[2024-01-18 22:34:41] Verifying : Description : Verifying creation of third schedule namespace labelled backup tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-223136 of emtpy namespace [2024-01-18T22:34:42.482Z] INFO[2024-01-18 22:34:41] Actual:true, Expected: true [2024-01-18T22:34:42.482Z] INFO[2024-01-18 22:34:42] Verifying : Description : Deleting schedule backup:tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-221632 with empty namespace [2024-01-18T22:34:42.482Z] INFO[2024-01-18 22:34:42] Actual:true, Expected: true [2024-01-18T22:34:42.482Z] INFO[2024-01-18 22:34:42] Verifying : Description : Deleting schedule backup:tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-223136 with empty namespace [2024-01-18T22:34:42.482Z] INFO[2024-01-18 22:34:42] Actual:true, Expected: true [2024-01-18T22:34:42.482Z] 2024-01-18 22:34:42 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Adding labels created above to the existing namespace and verify these namespace are included as part of next scheduled backup [2024-01-18T22:34:43.043Z] INFO[2024-01-18 22:34:42] Verifying : Description : Adding labels [map[0-8e0f81cf-0968-428f-9193-d6dec8a6c7f9:8e0f81cf-0968-428f-9193-d6dec8a6c7f9 1-8e0f81cf-0968-428f-9193-d6dec8a6c7f9:8e0f81cf-0968-428f-9193-d6dec8a6c7f9]] to multiple namespaces [[mysql-ibm-pxb-0-86693-01-18-21h59m35s mysql-ibm-pxb-1-86693-01-18-21h59m35s mysql-ibm-pxb-2-86693-01-18-21h59m35s]] [2024-01-18T22:34:43.043Z] INFO[2024-01-18 22:34:42] Actual:true, Expected: true [2024-01-18T22:34:43.043Z] 2024-01-18 22:34:42 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Waiting for [15] minutes for the next schedule backup to be triggered [2024-01-18T22:44:34.663Z] INFO[2024-01-18 22:44:32] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T22:44:34.663Z] INFO[2024-01-18 22:44:32] sched stats: tick interval: 25832 < 1.2s (100%) [2024-01-18T22:44:34.663Z] INFO[2024-01-18 22:44:32] sched stats: tick process durations: 25831 < 500ms (100%) [2024-01-18T22:49:56.172Z] 2024-01-18 22:49:42 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] schedule name tp-backup-schedule-sluzc, Next schedule backup name: tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-224639 [2024-01-18T22:49:56.172Z] 2024-01-18 22:49:42 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Inspect obj backup: last_update_time: create_time_in_sec:1705617999 ownership: > > backup_info: volumes: driver_name:"csi" options: total_size:2147483648 actual_size:2147483648 pvc_id:"79e82e89-04c5-4cf7-8000-a7eb69afbee3" > volumes: driver_name:"csi" options: total_size:2147483648 actual_size:2147483648 pvc_id:"c1f27b09-ba64-4497-8ea8-2d6bd27c7f78" > volumes: driver_name:"csi" options: total_size:2147483648 actual_size:2147483648 pvc_id:"c618ff29-4fcd-4c2f-8856-e344b5aa6aa8" > stage:Volumes backup_schedule: cr_name:"tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-224639" backup_location_ref: cloud_credential_ref: cr_uid:"8992ab26-4532-47d7-a8f4-3a4c9388e00a" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-8e0f81cf-0968-428f-9193-d6dec8a6c7f9=8e0f81cf-0968-428f-9193-d6dec8a6c7f9,1-8e0f81cf-0968-428f-9193-d6dec8a6c7f9=8e0f81cf-0968-428f-9193-d6dec8a6c7f9" target_namespace:"kube-system" > > [2024-01-18T22:49:56.173Z] INFO[2024-01-18 22:49:42] Verifying : Description : Verifying creation of next namespace labelled schedule backup tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-224639 with namespace included [2024-01-18T22:49:56.173Z] INFO[2024-01-18 22:49:42] Actual:true, Expected: true [2024-01-18T22:49:56.173Z] 2024-01-18 22:49:42 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-224639 [2024-01-18T22:49:56.173Z] 2024-01-18 22:49:42 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [mysql-ibm-pxb-0-86693-01-18-21h59m35s mysql-ibm-pxb-1-86693-01-18-21h59m35s mysql-ibm-pxb-2-86693-01-18-21h59m35s] [2024-01-18T22:49:56.173Z] 2024-01-18 22:49:42 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are 0-8e0f81cf-0968-428f-9193-d6dec8a6c7f9=8e0f81cf-0968-428f-9193-d6dec8a6c7f9,1-8e0f81cf-0968-428f-9193-d6dec8a6c7f9=8e0f81cf-0968-428f-9193-d6dec8a6c7f9 [2024-01-18T22:49:56.173Z] INFO[2024-01-18 22:49:42] Verifying : Description : Verifying if labelled namespace [mysql-ibm-pxb-0-86693-01-18-21h59m35s mysql-ibm-pxb-1-86693-01-18-21h59m35s mysql-ibm-pxb-2-86693-01-18-21h59m35s] are included in this schedule backup tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-223136 [2024-01-18T22:49:56.173Z] INFO[2024-01-18 22:49:42] Actual:true, Expected: true [2024-01-18T22:49:56.173Z] 2024/01/18 22:49:43 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-224639] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T22:50:14.264Z] 2024/01/18 22:50:13 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-224639] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T22:50:46.282Z] 2024/01/18 22:50:43 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-224639] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T22:51:18.308Z] 2024/01/18 22:51:13 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-224639] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T22:51:44.813Z] 2024-01-18 22:51:43 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validating backup [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-224639] in org [default] [2024-01-18T22:51:44.813Z] 2024-01-18 22:51:43 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-224639] [2024-01-18T22:51:44.813Z] 2024-01-18 22:51:43 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-86693-01-18-21h59m35s] in backup [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-224639] [2024-01-18T22:51:44.813Z] 2024-01-18 22:51:43 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-86693-01-18-21h59m35s] in backup [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-224639] [2024-01-18T22:51:44.813Z] 2024-01-18 22:51:43 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-86693-01-18-21h59m35s] [2024-01-18T22:51:44.813Z] 2024-01-18 22:51:43 +0000:[WARNING] [{NamespaceLabelledBackupOfEmptyNamespace}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T22:51:44.813Z] 2024-01-18 22:51:43 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T22:51:44.813Z] 2024-01-18 22:51:43 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-86693-01-18-21h59m35s] are [[mysql-data]] [2024-01-18T22:51:44.814Z] 2024-01-18 22:51:43 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T22:51:44.814Z] 2024-01-18 22:51:43 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-86693-01-18-21h59m35s] in backup [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-224639] [2024-01-18T22:51:44.814Z] 2024-01-18 22:51:43 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-86693-01-18-21h59m35s] in backup [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-224639] [2024-01-18T22:51:44.814Z] 2024-01-18 22:51:43 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-86693-01-18-21h59m35s] [2024-01-18T22:51:44.814Z] 2024-01-18 22:51:43 +0000:[WARNING] [{NamespaceLabelledBackupOfEmptyNamespace}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T22:51:44.814Z] 2024-01-18 22:51:43 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T22:51:44.814Z] 2024-01-18 22:51:43 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-86693-01-18-21h59m35s] are [[mysql-data]] [2024-01-18T22:51:44.814Z] 2024-01-18 22:51:43 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T22:51:44.814Z] 2024-01-18 22:51:43 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-86693-01-18-21h59m35s] in backup [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-224639] [2024-01-18T22:51:44.814Z] 2024-01-18 22:51:43 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-86693-01-18-21h59m35s] in backup [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-224639] [2024-01-18T22:51:44.814Z] 2024-01-18 22:51:43 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-2-86693-01-18-21h59m35s] [2024-01-18T22:51:44.814Z] 2024-01-18 22:51:43 +0000:[WARNING] [{NamespaceLabelledBackupOfEmptyNamespace}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T22:51:44.814Z] 2024-01-18 22:51:43 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T22:51:44.814Z] 2024-01-18 22:51:43 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-2-86693-01-18-21h59m35s] are [[mysql-data]] [2024-01-18T22:51:44.814Z] 2024-01-18 22:51:43 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T22:51:44.814Z] 2024-01-18 22:51:43 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validating CR cleanup [2024-01-18T22:51:44.814Z] 2024-01-18 22:51:43 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T22:51:44.814Z] 2024-01-18 22:51:43 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-224639]] [2024-01-18T22:51:44.814Z] 2024-01-18 22:51:43 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-224639]] under [kube-system] namespace [2024-01-18T22:51:44.815Z] 2024/01/18 22:51:43 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-224639]}, Next try in [5s], timeout [5m0s] [2024-01-18T22:51:49.008Z] 2024-01-18 22:51:48 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133]] [2024-01-18T22:51:49.008Z] INFO[2024-01-18 22:51:48] Verifying : Description : Verifying the success of schedule backup named [tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-224639] [2024-01-18T22:51:49.008Z] INFO[2024-01-18 22:51:48] Actual:true, Expected: true [2024-01-18T22:51:49.008Z] 2024-01-18 22:51:48 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Restoring schedule backup with namespace included [2024-01-18T22:51:49.936Z] 2024/01/18 22:51:49 DoRetryWithTimeout - Error: {restore status for [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T22:52:21.983Z] 2024/01/18 22:52:19 DoRetryWithTimeout - Error: {restore status for [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T22:52:54.009Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.CreateRestore:#936] - Restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] created successfully [2024-01-18T22:52:54.009Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Switching cluster context to cluster [destination-cluster] [2024-01-18T22:52:54.009Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Switching context: The context is already [/tmp/destination-config] [2024-01-18T22:52:54.009Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-86693-01-18-21h59m35s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-18T22:52:54.009Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-86693-01-18-21h59m35s] [2024-01-18T22:52:54.009Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T22:52:54.009Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T22:52:54.009Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-1-86693-01-18-21h59m35s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-18T22:52:54.010Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-1-86693-01-18-21h59m35s] [2024-01-18T22:52:54.010Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T22:52:54.010Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T22:52:54.010Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-2-86693-01-18-21h59m35s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-18T22:52:54.010Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-2-86693-01-18-21h59m35s] [2024-01-18T22:52:54.010Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T22:52:54.010Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T22:52:54.010Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validating restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] in org [default] [2024-01-18T22:52:54.010Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] [2024-01-18T22:52:54.010Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2110] - Checking status of restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] [2024-01-18T22:52:54.010Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-86693-01-18-21h59m35s] in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] [2024-01-18T22:52:54.010Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-86693-01-18-21h59m35s]) was found in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-86693-01-18-21h59m35s] [2024-01-18T22:52:54.010Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-86693-01-18-21h59m35s]) found in the restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] was also present on the cluster/namespace [mysql-ibm-pxb-0-86693-01-18-21h59m35s] [2024-01-18T22:52:54.010Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-86693-01-18-21h59m35s]) was found in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-86693-01-18-21h59m35s] [2024-01-18T22:52:54.011Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-86693-01-18-21h59m35s]) found in the restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] was also present on the cluster/namespace [mysql-ibm-pxb-0-86693-01-18-21h59m35s] [2024-01-18T22:52:54.011Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-86693-01-18-21h59m35s]) was found in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-86693-01-18-21h59m35s] [2024-01-18T22:52:54.011Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-86693-01-18-21h59m35s]) found in the restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] was also present on the cluster/namespace [mysql-ibm-pxb-0-86693-01-18-21h59m35s] [2024-01-18T22:52:54.011Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-86693-01-18-21h59m35s]) was found in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-86693-01-18-21h59m35s] [2024-01-18T22:52:54.011Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-86693-01-18-21h59m35s]) found in the restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] was also present on the cluster/namespace [mysql-ibm-pxb-0-86693-01-18-21h59m35s] [2024-01-18T22:52:54.011Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-86693-01-18-21h59m35s] in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] [2024-01-18T22:52:54.011Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-86693-01-18-21h59m35s] in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] [2024-01-18T22:52:54.011Z] 2024-01-18 22:52:49 +0000:[WARNING] [{NamespaceLabelledBackupOfEmptyNamespace}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T22:52:54.012Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T22:52:54.012Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-86693-01-18-21h59m35s] are [[mysql-data]] [2024-01-18T22:52:54.012Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2240] - in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces], the status of the restored volume [pvc-12f69145-19e7-46eb-99da-150260083318] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-18T22:52:54.012Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validate applications in restored namespace [mysql-ibm-pxb-0-86693-01-18-21h59m35s] due to restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] [2024-01-18T22:52:54.012Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validating mysql-ibm app [2024-01-18T22:52:54.012Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] wait for mysql-ibm app to start running [2024-01-18T22:52:54.012Z] 2024-01-18 22:52:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T22:52:54.012Z] 2024/01/18 22:52:49 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T22:52:54.012Z] pod name:mysql-7d766cd65c-ghblb namespace:mysql-ibm-pxb-0-86693-01-18-21h59m35s running:true ready:false node:10.240.64.8 [2024-01-18T22:52:54.012Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T22:53:00.565Z] 2024/01/18 22:52:59 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T22:53:00.565Z] pod name:mysql-7d766cd65c-ghblb namespace:mysql-ibm-pxb-0-86693-01-18-21h59m35s running:true ready:false node:10.240.64.8 [2024-01-18T22:53:00.565Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T22:53:10.524Z] 2024/01/18 22:53:10 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T22:53:10.524Z] pod name:mysql-7d766cd65c-ghblb namespace:mysql-ibm-pxb-0-86693-01-18-21h59m35s running:true ready:false node:10.240.64.8 [2024-01-18T22:53:10.524Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T22:53:20.465Z] 2024/01/18 22:53:20 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T22:53:20.465Z] pod name:mysql-7d766cd65c-ghblb namespace:mysql-ibm-pxb-0-86693-01-18-21h59m35s running:true ready:false node:10.240.64.8 [2024-01-18T22:53:20.465Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T22:53:30.419Z] 2024-01-18 22:53:30 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T22:53:30.675Z] 2024-01-18 22:53:30 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T22:53:30.675Z] 2024-01-18 22:53:30 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-1-86693-01-18-21h59m35s] in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] [2024-01-18T22:53:30.675Z] 2024-01-18 22:53:30 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-1-86693-01-18-21h59m35s]) was found in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-1-86693-01-18-21h59m35s] [2024-01-18T22:53:30.675Z] 2024-01-18 22:53:30 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-1-86693-01-18-21h59m35s]) found in the restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] was also present on the cluster/namespace [mysql-ibm-pxb-1-86693-01-18-21h59m35s] [2024-01-18T22:53:30.676Z] 2024-01-18 22:53:30 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-1-86693-01-18-21h59m35s]) was found in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-1-86693-01-18-21h59m35s] [2024-01-18T22:53:30.676Z] 2024-01-18 22:53:30 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-1-86693-01-18-21h59m35s]) found in the restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] was also present on the cluster/namespace [mysql-ibm-pxb-1-86693-01-18-21h59m35s] [2024-01-18T22:53:30.676Z] 2024-01-18 22:53:30 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-1-86693-01-18-21h59m35s]) was found in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-1-86693-01-18-21h59m35s] [2024-01-18T22:53:30.932Z] 2024-01-18 22:53:30 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-1-86693-01-18-21h59m35s]) found in the restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] was also present on the cluster/namespace [mysql-ibm-pxb-1-86693-01-18-21h59m35s] [2024-01-18T22:53:30.932Z] 2024-01-18 22:53:30 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-1-86693-01-18-21h59m35s]) was found in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-1-86693-01-18-21h59m35s] [2024-01-18T22:53:30.932Z] 2024-01-18 22:53:30 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-1-86693-01-18-21h59m35s]) found in the restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] was also present on the cluster/namespace [mysql-ibm-pxb-1-86693-01-18-21h59m35s] [2024-01-18T22:53:30.932Z] 2024-01-18 22:53:30 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-1-86693-01-18-21h59m35s] in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] [2024-01-18T22:53:30.932Z] 2024-01-18 22:53:30 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-1-86693-01-18-21h59m35s] in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] [2024-01-18T22:53:30.932Z] 2024-01-18 22:53:30 +0000:[WARNING] [{NamespaceLabelledBackupOfEmptyNamespace}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T22:53:30.932Z] 2024-01-18 22:53:30 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T22:53:30.932Z] 2024-01-18 22:53:30 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-86693-01-18-21h59m35s] are [[mysql-data]] [2024-01-18T22:53:30.933Z] 2024-01-18 22:53:30 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2240] - in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces], the status of the restored volume [pvc-46a11c8e-8722-4252-a239-74c9d52fcfd0] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-18T22:53:30.933Z] 2024-01-18 22:53:30 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validate applications in restored namespace [mysql-ibm-pxb-1-86693-01-18-21h59m35s] due to restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] [2024-01-18T22:53:30.933Z] 2024-01-18 22:53:30 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validating mysql-ibm app [2024-01-18T22:53:30.933Z] 2024-01-18 22:53:30 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] wait for mysql-ibm app to start running [2024-01-18T22:53:30.933Z] 2024-01-18 22:53:30 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T22:53:31.494Z] 2024-01-18 22:53:31 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T22:53:32.056Z] 2024-01-18 22:53:31 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T22:53:32.984Z] 2024-01-18 22:53:32 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-2-86693-01-18-21h59m35s] in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] [2024-01-18T22:53:32.984Z] 2024-01-18 22:53:32 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-2-86693-01-18-21h59m35s]) was found in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-2-86693-01-18-21h59m35s] [2024-01-18T22:53:32.984Z] 2024-01-18 22:53:32 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-2-86693-01-18-21h59m35s]) found in the restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] was also present on the cluster/namespace [mysql-ibm-pxb-2-86693-01-18-21h59m35s] [2024-01-18T22:53:32.984Z] 2024-01-18 22:53:32 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-2-86693-01-18-21h59m35s]) was found in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-2-86693-01-18-21h59m35s] [2024-01-18T22:53:32.985Z] 2024-01-18 22:53:32 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-2-86693-01-18-21h59m35s]) found in the restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] was also present on the cluster/namespace [mysql-ibm-pxb-2-86693-01-18-21h59m35s] [2024-01-18T22:53:32.985Z] 2024-01-18 22:53:32 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-2-86693-01-18-21h59m35s]) was found in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-2-86693-01-18-21h59m35s] [2024-01-18T22:53:32.985Z] 2024-01-18 22:53:32 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-2-86693-01-18-21h59m35s]) found in the restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] was also present on the cluster/namespace [mysql-ibm-pxb-2-86693-01-18-21h59m35s] [2024-01-18T22:53:32.985Z] 2024-01-18 22:53:32 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-2-86693-01-18-21h59m35s]) was found in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-2-86693-01-18-21h59m35s] [2024-01-18T22:53:33.243Z] 2024-01-18 22:53:33 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-2-86693-01-18-21h59m35s]) found in the restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] was also present on the cluster/namespace [mysql-ibm-pxb-2-86693-01-18-21h59m35s] [2024-01-18T22:53:33.243Z] 2024-01-18 22:53:33 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-2-86693-01-18-21h59m35s] in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] [2024-01-18T22:53:33.243Z] 2024-01-18 22:53:33 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-2-86693-01-18-21h59m35s] in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] [2024-01-18T22:53:33.244Z] 2024-01-18 22:53:33 +0000:[WARNING] [{NamespaceLabelledBackupOfEmptyNamespace}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T22:53:33.244Z] 2024-01-18 22:53:33 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T22:53:33.244Z] 2024-01-18 22:53:33 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-2-86693-01-18-21h59m35s] are [[mysql-data]] [2024-01-18T22:53:33.244Z] 2024-01-18 22:53:33 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.ValidateRestore:#2240] - in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces], the status of the restored volume [pvc-5c2f8e10-3c5f-477a-8a5a-5918addbbbdc] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-18T22:53:33.244Z] 2024-01-18 22:53:33 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validate applications in restored namespace [mysql-ibm-pxb-2-86693-01-18-21h59m35s] due to restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces] [2024-01-18T22:53:33.244Z] 2024-01-18 22:53:33 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Validating mysql-ibm app [2024-01-18T22:53:33.244Z] 2024-01-18 22:53:33 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] wait for mysql-ibm app to start running [2024-01-18T22:53:33.244Z] 2024-01-18 22:53:33 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T22:53:33.806Z] 2024-01-18 22:53:33 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T22:53:34.371Z] 2024-01-18 22:53:34 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T22:53:35.302Z] 2024-01-18 22:53:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Switching cluster context back to cluster path [/tmp/destination-config] [2024-01-18T22:53:35.302Z] 2024-01-18 22:53:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Switching context: The context is already [/tmp/destination-config] [2024-01-18T22:53:35.303Z] INFO[2024-01-18 22:53:35] Verifying : Description : Verifying restore restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces with namespaces included [2024-01-18T22:53:35.303Z] ERRO[2024-01-18 22:53:35] ValidateRestore Errors: {in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces], said restored volume [pvc-12f69145-19e7-46eb-99da-150260083318] cannot be found in the actual cluster [mysql-ibm-pxb-0-86693-01-18-21h59m35s]} [2024-01-18T22:53:35.303Z] {in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces], said restored volume [pvc-46a11c8e-8722-4252-a239-74c9d52fcfd0] cannot be found in the actual cluster [mysql-ibm-pxb-1-86693-01-18-21h59m35s]} [2024-01-18T22:53:35.303Z] {in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces], said restored volume [pvc-5c2f8e10-3c5f-477a-8a5a-5918addbbbdc] cannot be found in the actual cluster [mysql-ibm-pxb-2-86693-01-18-21h59m35s]} [2024-01-18T22:53:35.560Z] 2024-01-18 22:53:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.glob..func23.3:#1389] - Deleting the deployed applications [2024-01-18T22:53:35.560Z] 2024-01-18 22:53:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Destroying apps [2024-01-18T22:53:35.560Z] 2024-01-18 22:53:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T22:53:35.560Z] 2024-01-18 22:53:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T22:53:35.560Z] 2024-01-18 22:53:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] start destroying mysql-ibm app [2024-01-18T22:53:35.560Z] 2024-01-18 22:53:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T22:53:35.817Z] 2024-01-18 22:53:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T22:53:35.817Z] 2024-01-18 22:53:35 +0000:[WARNING] [{NamespaceLabelledBackupOfEmptyNamespace}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T22:53:35.817Z] INFO[2024-01-18 22:53:35] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T22:53:35.817Z] INFO[2024-01-18 22:53:35] Actual:true, Expected: true [2024-01-18T22:53:35.817Z] 2024-01-18 22:53:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T22:53:36.073Z] 2024-01-18 22:53:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T22:53:36.073Z] 2024-01-18 22:53:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] start destroying mysql-ibm app [2024-01-18T22:53:36.329Z] 2024-01-18 22:53:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T22:53:36.329Z] 2024-01-18 22:53:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T22:53:36.329Z] 2024-01-18 22:53:36 +0000:[WARNING] [{NamespaceLabelledBackupOfEmptyNamespace}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T22:53:36.329Z] INFO[2024-01-18 22:53:36] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T22:53:36.329Z] INFO[2024-01-18 22:53:36] Actual:true, Expected: true [2024-01-18T22:53:36.329Z] 2024-01-18 22:53:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T22:53:36.585Z] 2024-01-18 22:53:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T22:53:36.585Z] 2024-01-18 22:53:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] start destroying mysql-ibm app [2024-01-18T22:53:36.841Z] 2024-01-18 22:53:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T22:53:36.841Z] 2024-01-18 22:53:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T22:53:36.841Z] 2024-01-18 22:53:36 +0000:[WARNING] [{NamespaceLabelledBackupOfEmptyNamespace}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T22:53:36.841Z] INFO[2024-01-18 22:53:36] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T22:53:36.841Z] INFO[2024-01-18 22:53:36] Actual:true, Expected: true [2024-01-18T22:53:36.841Z] 2024-01-18 22:53:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Deleting the restores taken [2024-01-18T22:53:36.841Z] INFO[2024-01-18 22:53:36] Verifying : Description : Getting the backup driver [2024-01-18T22:53:36.841Z] INFO[2024-01-18 22:53:36] Actual:true, Expected: true [2024-01-18T22:53:36.841Z] INFO[2024-01-18 22:53:36] Verifying : Description : Deleting restore restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces [2024-01-18T22:53:36.841Z] INFO[2024-01-18 22:53:36] Actual:true, Expected: true [2024-01-18T22:53:37.097Z] 2024/01/18 22:53:36 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-sluzc] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T22:53:47.042Z] 2024/01/18 22:53:46 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-sluzc] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T22:53:56.982Z] 2024/01/18 22:53:56 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-sluzc] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T22:54:09.151Z] 2024/01/18 22:54:06 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-sluzc] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T22:54:17.275Z] 2024/01/18 22:54:16 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-sluzc] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T22:54:27.221Z] 2024/01/18 22:54:27 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-sluzc] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T22:54:33.755Z] INFO[2024-01-18 22:54:33] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T22:54:33.755Z] INFO[2024-01-18 22:54:33] sched stats: tick interval: 26433 < 1.2s (100%) [2024-01-18T22:54:33.755Z] INFO[2024-01-18 22:54:33] sched stats: tick process durations: 26432 < 500ms (100%) [2024-01-18T22:54:37.921Z] 2024/01/18 22:54:37 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-sluzc] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T22:54:47.863Z] INFO[2024-01-18 22:54:47] Verifying : Description : Verification of deleting backup schedule - tp-backup-schedule-sluzc [2024-01-18T22:54:47.863Z] INFO[2024-01-18 22:54:47] Actual:true, Expected: true [2024-01-18T22:54:47.863Z] INFO[2024-01-18 22:54:47] Verifying : Description : Deleting backup schedule policies [periodic-rbckr] [2024-01-18T22:54:47.863Z] INFO[2024-01-18 22:54:47] Actual:true, Expected: true [2024-01-18T22:54:47.863Z] 2024-01-18 22:54:47 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Cleaning backup locations in map [map[2882522f-e3ec-4f4a-b0ad-31051a6dcabc:autogenerated-backup-location-ciryj]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T22:54:47.863Z] INFO[2024-01-18 22:54:47] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-ciryj] [2024-01-18T22:54:47.863Z] INFO[2024-01-18 22:54:47] Actual:true, Expected: true [2024-01-18T22:54:47.863Z] 2024-01-18 22:54:47 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-ciryj] is present [2024-01-18T22:54:47.863Z] 2024/01/18 22:54:47 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-ciryj is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T22:55:20.471Z] 2024-01-18 22:55:17 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-ciryj] is present [2024-01-18T22:55:20.471Z] 2024/01/18 22:55:17 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-ciryj is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T22:55:52.495Z] 2024-01-18 22:55:47 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 aws-global-aws-px-backup-system-test-4cac76dc-bl] [2024-01-18T22:55:52.496Z] INFO[2024-01-18 22:55:47] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-ciryj [2024-01-18T22:55:52.496Z] INFO[2024-01-18 22:55:47] Actual:true, Expected: true [2024-01-18T22:55:52.496Z] INFO[2024-01-18 22:55:47] Verifying : Description : Verifying if cloud cred [] is present [2024-01-18T22:55:52.496Z] INFO[2024-01-18 22:55:47] Actual:true, Expected: true [2024-01-18T22:55:52.496Z] INFO[2024-01-18 22:55:47] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T22:55:52.496Z] INFO[2024-01-18 22:55:47] Actual:true, Expected: true [2024-01-18T22:55:52.496Z] INFO[2024-01-18 22:55:47] Verifying : Description : Deleting cluster destination-cluster [2024-01-18T22:55:52.496Z] ERRO[2024-01-18 22:55:47] Actual:rpc error: code = FailedPrecondition desc = cluster [destination-cluster] in use by backup [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-225545], cannot be deleted, Expected:  [2024-01-18T22:55:52.496Z] INFO[2024-01-18 22:55:48] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-zbbvt] [2024-01-18T22:55:52.496Z] ERRO[2024-01-18 22:55:48] Actual:rpc error: code = Unknown desc = cluster [destination-cluster] has reference to cloud credential [ibm-destination-config-cloud-cred-zbbvt:a271ea5b-dc9d-40f1-a82a-651539c7e486], Expected:  [2024-01-18T22:55:52.496Z] INFO[2024-01-18 22:55:48] Verifying : Description : Deleting cluster source-cluster [2024-01-18T22:55:52.496Z] INFO[2024-01-18 22:55:48] Actual:true, Expected: true [2024-01-18T22:55:52.496Z] INFO[2024-01-18 22:55:48] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-vsqmt] [2024-01-18T22:55:52.496Z] INFO[2024-01-18 22:55:48] Actual:true, Expected: true [2024-01-18T22:55:52.496Z] INFO[2024-01-18 22:55:48] --------Test End------ [2024-01-18T22:55:52.496Z] INFO[2024-01-18 22:55:48] #Test: [2024-01-18T22:55:52.496Z] INFO[2024-01-18 22:55:48] #Description: [2024-01-18T22:55:52.496Z] INFO[2024-01-18 22:55:48] ------------------------ [2024-01-18T22:55:52.496Z] 2024-01-18 22:55:48 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.EndPxBackupTorpedoTest:#7261] - >>>> FAILED TEST: {NamespaceLabelledBackupOfEmptyNamespace} Namespace labelled backup of empty namespace and restore it [2024-01-18T22:55:52.496Z] 2024-01-18 22:55:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-86693-01-18-21h59m35s] [2024-01-18T22:55:52.496Z] 2024-01-18 22:55:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-1-86693-01-18-21h59m35s] [2024-01-18T22:55:52.496Z] 2024-01-18 22:55:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-2-86693-01-18-21h59m35s] [2024-01-18T22:55:52.496Z] 2024-01-18 22:55:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T22:55:52.496Z] 2024-01-18 22:55:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Switching context: The context is already [/tmp/destination-config] [2024-01-18T22:55:52.496Z] 2024-01-18 22:55:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-86693-01-18-21h59m35s] [2024-01-18T22:55:52.496Z] 2024-01-18 22:55:49 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-1-86693-01-18-21h59m35s] [2024-01-18T22:55:52.496Z] 2024-01-18 22:55:50 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-2-86693-01-18-21h59m35s] [2024-01-18T22:55:52.497Z] 2024-01-18 22:55:50 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {NamespaceLabelledBackupOfEmptyNamespace} Namespace labelled backup of empty namespace and restore it [2024-01-18T22:55:52.497Z] 2024-01-18 22:55:50 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-18T22:55:52.497Z] 2024/01/18 22:55:50 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T22:55:55.771Z] 2024/01/18 22:55:55 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T22:56:01.023Z] 2024/01/18 22:56:00 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T22:56:06.272Z] 2024/01/18 22:56:05 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T22:56:10.452Z] 2024/01/18 22:56:10 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T22:56:15.699Z] 2024/01/18 22:56:15 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T22:56:20.943Z] 2024/01/18 22:56:20 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T22:56:26.210Z] 2024/01/18 22:56:25 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T22:56:31.467Z] 2024/01/18 22:56:30 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.128.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.7 10.240.128.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-18T22:56:31.467Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-18T22:56:31.467Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-18T22:56:31.467Z] {dial tcp 10.240.128.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-18T22:56:35.635Z] 2024-01-18 22:56:35 +0000:[WARNING] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.7 10.240.128.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-18T22:56:35.635Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-18T22:56:35.635Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-18T22:56:35.635Z] {dial tcp 10.240.128.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T22:56:35.635Z] 2024-01-18 22:56:35 +0000:[ERROR] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.7 10.240.128.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-18T22:56:35.635Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-18T22:56:35.635Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-18T22:56:35.636Z] {dial tcp 10.240.128.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T22:56:35.636Z] 2024-01-18 22:56:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T22:56:35.636Z] 2024-01-18 22:56:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] Switching context to [/tmp/source-config] [2024-01-18T22:56:35.636Z] 2024-01-18 22:56:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T22:56:35.636Z] 2024-01-18 22:56:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T22:56:35.636Z] 2024-01-18 22:56:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T22:56:35.893Z] 2024-01-18 22:56:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T22:56:35.893Z] 2024-01-18 22:56:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T22:56:35.894Z] 2024-01-18 22:56:35 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T22:56:36.456Z] 2024-01-18 22:56:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T22:56:36.714Z] 2024-01-18 22:56:36 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T22:56:37.277Z] 2024-01-18 22:56:37 +0000:[INFO] [{NamespaceLabelledBackupOfEmptyNamespace}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T22:56:37.277Z] 2024/01/18 22:56:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T22:56:49.449Z] 2024/01/18 22:56:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T22:56:57.534Z] 2024/01/18 22:56:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T22:57:07.483Z] 2024/01/18 22:57:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T22:57:17.476Z] 2024/01/18 22:57:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T22:57:27.418Z] 2024/01/18 22:57:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T22:57:39.601Z] 2024/01/18 22:57:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T22:57:47.688Z] 2024/01/18 22:57:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T22:57:57.658Z] 2024/01/18 22:57:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T22:58:07.599Z] 2024/01/18 22:58:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T22:58:17.542Z] 2024/01/18 22:58:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T22:58:29.718Z] 2024/01/18 22:58:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T22:58:37.810Z] 2024/01/18 22:58:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T22:58:47.761Z] 2024/01/18 22:58:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T22:58:57.707Z] 2024/01/18 22:58:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T22:59:09.874Z] 2024/01/18 22:59:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T22:59:17.968Z] 2024/01/18 22:59:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T22:59:27.905Z] 2024/01/18 22:59:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T22:59:37.842Z] 2024/01/18 22:59:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T22:59:50.007Z] 2024/01/18 22:59:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T22:59:58.089Z] 2024/01/18 22:59:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:00:08.030Z] 2024/01/18 23:00:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:00:17.966Z] 2024/01/18 23:00:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:00:30.149Z] 2024/01/18 23:00:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:00:38.233Z] 2024/01/18 23:00:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:00:48.174Z] 2024/01/18 23:00:48 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:00:58.113Z] 2024/01/18 23:00:58 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:01:10.307Z] 2024/01/18 23:01:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:01:18.451Z] 2024/01/18 23:01:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:01:28.419Z] 2024/01/18 23:01:28 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:01:38.359Z] 2024-01-18 23:01:38 +0000:[ERROR] [{NamespaceLabelledBackupOfEmptyNamespace}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.359Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.359Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.359Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.359Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.359Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.359Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.359Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.359Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.359Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.359Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.359Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.359Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.359Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.359Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.359Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.359Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.359Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.359Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.360Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.360Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.360Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.360Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.360Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.360Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.360Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.360Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.360Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.360Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:01:38.360Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-18T23:01:38.360Z] • Failure [3722.203 seconds] [2024-01-18T23:01:38.360Z] {NamespaceLabelledBackupOfEmptyNamespace} [2024-01-18T23:01:38.360Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_namespace_labelled_test.go:1221 [2024-01-18T23:01:38.360Z] Namespace labelled backup of empty namespace and restore it [It] [2024-01-18T23:01:38.360Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_namespace_labelled_test.go:1263 [2024-01-18T23:01:38.360Z] [2024-01-18T23:01:38.360Z] Unexpected error: [2024-01-18T23:01:38.360Z] <*errors.errorString | 0xc0011b8650>: { [2024-01-18T23:01:38.360Z] s: "error: ValidateRestore Errors: {in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces], said restored volume [pvc-12f69145-19e7-46eb-99da-150260083318] cannot be found in the actual cluster [mysql-ibm-pxb-0-86693-01-18-21h59m35s]}\n{in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces], said restored volume [pvc-46a11c8e-8722-4252-a239-74c9d52fcfd0] cannot be found in the actual cluster [mysql-ibm-pxb-1-86693-01-18-21h59m35s]}\n{in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces], said restored volume [pvc-5c2f8e10-3c5f-477a-8a5a-5918addbbbdc] cannot be found in the actual cluster [mysql-ibm-pxb-2-86693-01-18-21h59m35s]}, Description: Verifying restore restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces with namespaces included", [2024-01-18T23:01:38.360Z] } [2024-01-18T23:01:38.360Z] error: ValidateRestore Errors: {in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces], said restored volume [pvc-12f69145-19e7-46eb-99da-150260083318] cannot be found in the actual cluster [mysql-ibm-pxb-0-86693-01-18-21h59m35s]} [2024-01-18T23:01:38.361Z] {in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces], said restored volume [pvc-46a11c8e-8722-4252-a239-74c9d52fcfd0] cannot be found in the actual cluster [mysql-ibm-pxb-1-86693-01-18-21h59m35s]} [2024-01-18T23:01:38.361Z] {in restore [restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces], said restored volume [pvc-5c2f8e10-3c5f-477a-8a5a-5918addbbbdc] cannot be found in the actual cluster [mysql-ibm-pxb-2-86693-01-18-21h59m35s]}, Description: Verifying restore restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-xujgs-with-namespaces with namespaces included [2024-01-18T23:01:38.361Z] occurred [2024-01-18T23:01:38.361Z] [2024-01-18T23:01:38.361Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-18T23:01:38.361Z] [2024-01-18T23:01:38.361Z] Full Stack Trace [2024-01-18T23:01:38.361Z] github.com/portworx/torpedo/pkg/aetosutil.(*Dashboard).VerifyFatal(0x7f880ff?, {0x702af20?, 0xc0011b8490}, {0x0?, 0x0}, {0xc001d56a20, 0x84}) [2024-01-18T23:01:38.361Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 +0x143 [2024-01-18T23:01:38.361Z] github.com/portworx/torpedo/tests/backup.glob..func23.2.10() [2024-01-18T23:01:38.361Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_namespace_labelled_test.go:1382 +0x437 [2024-01-18T23:01:38.361Z] github.com/portworx/torpedo/tests/backup.glob..func23.2() [2024-01-18T23:01:38.361Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_namespace_labelled_test.go:1375 +0xab7 [2024-01-18T23:01:38.361Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-18T23:01:38.361Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-18T23:01:38.361Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-18T23:01:38.361Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-18T23:01:38.361Z] created by testing.(*T).Run [2024-01-18T23:01:38.361Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-18T23:01:38.361Z] ------------------------------ [2024-01-18T23:01:38.361Z] STEP: schedule applications [2024-01-18T23:01:38.361Z] STEP: schedule applications [2024-01-18T23:01:38.361Z] STEP: schedule applications [2024-01-18T23:01:38.361Z] STEP: Validate applications [2024-01-18T23:01:38.361Z] STEP: validate applications [2024-01-18T23:01:38.361Z] STEP: validate mysql-ibm app's volumes [2024-01-18T23:01:38.361Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T23:01:38.361Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T23:01:38.361Z] STEP: get mysql-ibm app's volume: pvc-79e82e89-04c5-4cf7-8000-a7eb69afbee3 inspected by the volume driver [2024-01-18T23:01:38.361Z] STEP: wait for mysql-ibm app to start running [2024-01-18T23:01:38.361Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T23:01:38.361Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T23:01:38.361Z] STEP: Validate Px pod restart count [2024-01-18T23:01:38.361Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T23:01:38.361Z] STEP: validate mysql-ibm app's volumes [2024-01-18T23:01:38.361Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T23:01:38.361Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T23:01:38.361Z] STEP: get mysql-ibm app's volume: pvc-c1f27b09-ba64-4497-8ea8-2d6bd27c7f78 inspected by the volume driver [2024-01-18T23:01:38.361Z] STEP: wait for mysql-ibm app to start running [2024-01-18T23:01:38.361Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T23:01:38.361Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T23:01:38.361Z] STEP: Validate Px pod restart count [2024-01-18T23:01:38.361Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T23:01:38.361Z] STEP: validate mysql-ibm app's volumes [2024-01-18T23:01:38.361Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T23:01:38.361Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T23:01:38.361Z] STEP: get mysql-ibm app's volume: pvc-c618ff29-4fcd-4c2f-8856-e344b5aa6aa8 inspected by the volume driver [2024-01-18T23:01:38.362Z] STEP: wait for mysql-ibm app to start running [2024-01-18T23:01:38.362Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T23:01:38.362Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T23:01:38.362Z] STEP: Validate Px pod restart count [2024-01-18T23:01:38.362Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T23:01:38.362Z] STEP: Generating namespace label string from label map [2024-01-18T23:01:38.362Z] STEP: Creating cloud credentials and registering backup location [2024-01-18T23:01:38.362Z] STEP: Configure source and destination clusters with px-central-admin ctx [2024-01-18T23:01:38.362Z] STEP: Create schedule policy [2024-01-18T23:01:38.362Z] STEP: Creating a schedule backup of empty namespace with namespace label filter [2024-01-18T23:01:38.362Z] STEP: Restoring first schedule backup of empty namespace with namespace label filter [2024-01-18T23:01:38.362Z] STEP: Deleting few schedule backup with empty namespaces [2024-01-18T23:01:38.362Z] STEP: Adding labels created above to the existing namespace and verify these namespace are included as part of next scheduled backup [2024-01-18T23:01:38.362Z] STEP: Restoring schedule backup with namespace included [2024-01-18T23:01:38.362Z] STEP: validate mysql-ibm app's volumes [2024-01-18T23:01:38.362Z] STEP: wait for mysql-ibm app to start running [2024-01-18T23:01:38.362Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T23:01:38.362Z] STEP: Validate Px pod restart count [2024-01-18T23:01:38.362Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T23:01:38.362Z] STEP: validate mysql-ibm app's volumes [2024-01-18T23:01:38.362Z] STEP: wait for mysql-ibm app to start running [2024-01-18T23:01:38.362Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T23:01:38.362Z] STEP: Validate Px pod restart count [2024-01-18T23:01:38.363Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T23:01:38.364Z] STEP: validate mysql-ibm app's volumes [2024-01-18T23:01:38.364Z] STEP: wait for mysql-ibm app to start running [2024-01-18T23:01:38.364Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T23:01:38.364Z] STEP: Validate Px pod restart count [2024-01-18T23:01:38.364Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T23:01:38.364Z] STEP: destroy apps [2024-01-18T23:01:38.364Z] STEP: destroy the mysql-ibm app's volumes [2024-01-18T23:01:38.364Z] STEP: start destroying mysql-ibm app [2024-01-18T23:01:38.364Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T23:01:38.364Z] STEP: destroy the mysql-ibm app's volumes [2024-01-18T23:01:38.364Z] STEP: start destroying mysql-ibm app [2024-01-18T23:01:38.364Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T23:01:38.364Z] STEP: destroy the mysql-ibm app's volumes [2024-01-18T23:01:38.364Z] STEP: start destroying mysql-ibm app [2024-01-18T23:01:38.364Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T23:01:38.364Z] SSSSS2024-01-18 23:01:38 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Log Dir: /testresults//UpdatesBackupOfUserFromAdmin.log [2024-01-18T23:01:38.364Z] INFO[2024-01-18 23:01:38] --------Test Start------ [2024-01-18T23:01:38.364Z] INFO[2024-01-18 23:01:38] #Test: UpdatesBackupOfUserFromAdmin [2024-01-18T23:01:38.364Z] INFO[2024-01-18 23:01:38] #Description: Updates backups of non admin user from px-admin with valid/in-valid account [2024-01-18T23:01:38.364Z] INFO[2024-01-18 23:01:38] ------------------------ [2024-01-18T23:01:38.364Z] 2024-01-18 23:01:38 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Deploy applications [2024-01-18T23:01:38.364Z] 2024-01-18 23:01:38 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T23:01:38.365Z] 2024-01-18 23:01:38 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T23:01:38.365Z] 2024-01-18 23:01:38 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T23:01:38.365Z] 2024-01-18 23:01:38 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T23:01:38.365Z] 2024-01-18 23:01:38 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T23:01:38.365Z] 2024-01-18 23:01:38 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T23:01:38.366Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T23:01:38.622Z] 2024-01-18 23:01:38 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T23:01:38.623Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T23:01:38.623Z] 2024-01-18 23:01:38 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T23:01:38.623Z] 2024-01-18 23:01:38 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Created namespaces [mysql-ibm-pxb-0-87568-01-18-23h01m38s] [2024-01-18T23:01:38.623Z] 2024-01-18 23:01:38 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Validate applications [2024-01-18T23:01:38.623Z] 2024-01-18 23:01:38 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Validate applications [2024-01-18T23:01:38.623Z] 2024-01-18 23:01:38 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Validating mysql-ibm app [2024-01-18T23:01:38.623Z] 2024-01-18 23:01:38 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Validating mysql-ibm app's volumes [2024-01-18T23:01:38.623Z] 2024-01-18 23:01:38 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T23:01:38.623Z] 2024/01/18 23:01:38 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T23:01:48.616Z] 2024/01/18 23:01:48 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T23:01:58.566Z] 2024-01-18 23:01:58 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-87568-01-18-23h01m38s [2024-01-18T23:01:58.824Z] 2024-01-18 23:01:58 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T23:01:58.824Z] 2024-01-18 23:01:58 +0000:[WARNING] [{UpdatesBackupOfUserFromAdmin}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T23:01:58.824Z] 2024-01-18 23:01:58 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] wait for mysql-ibm app to start running [2024-01-18T23:01:58.824Z] 2024-01-18 23:01:58 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T23:01:59.084Z] 2024/01/18 23:01:58 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:01:59.084Z] pod name:mysql-7d766cd65c-7n544 namespace:mysql-ibm-pxb-0-87568-01-18-23h01m38s running:false ready:false node:10.240.64.8 [2024-01-18T23:01:59.084Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:02:09.023Z] 2024/01/18 23:02:08 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:02:09.023Z] pod name:mysql-7d766cd65c-7n544 namespace:mysql-ibm-pxb-0-87568-01-18-23h01m38s running:true ready:false node:10.240.64.8 [2024-01-18T23:02:09.023Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:02:21.186Z] 2024/01/18 23:02:19 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:02:21.186Z] pod name:mysql-7d766cd65c-7n544 namespace:mysql-ibm-pxb-0-87568-01-18-23h01m38s running:true ready:false node:10.240.64.8 [2024-01-18T23:02:21.186Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:02:29.270Z] 2024/01/18 23:02:29 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:02:29.270Z] pod name:mysql-7d766cd65c-7n544 namespace:mysql-ibm-pxb-0-87568-01-18-23h01m38s running:true ready:false node:10.240.64.8 [2024-01-18T23:02:29.270Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:02:41.432Z] 2024/01/18 23:02:39 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:02:41.432Z] pod name:mysql-7d766cd65c-7n544 namespace:mysql-ibm-pxb-0-87568-01-18-23h01m38s running:true ready:false node:10.240.64.8 [2024-01-18T23:02:41.432Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:02:49.515Z] 2024/01/18 23:02:49 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:02:49.515Z] pod name:mysql-7d766cd65c-7n544 namespace:mysql-ibm-pxb-0-87568-01-18-23h01m38s running:true ready:false node:10.240.64.8 [2024-01-18T23:02:49.515Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:02:59.478Z] 2024/01/18 23:02:59 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:02:59.478Z] pod name:mysql-7d766cd65c-7n544 namespace:mysql-ibm-pxb-0-87568-01-18-23h01m38s running:true ready:false node:10.240.64.8 [2024-01-18T23:02:59.478Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:03:11.646Z] 2024/01/18 23:03:09 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:03:11.646Z] pod name:mysql-7d766cd65c-7n544 namespace:mysql-ibm-pxb-0-87568-01-18-23h01m38s running:true ready:false node:10.240.64.8 [2024-01-18T23:03:11.646Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:03:19.735Z] 2024-01-18 23:03:19 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T23:03:19.991Z] 2024-01-18 23:03:19 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T23:03:19.991Z] 2024-01-18 23:03:19 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] validate if mysql-ibm app's volumes are setup [2024-01-18T23:03:19.991Z] 2024-01-18 23:03:19 +0000:[WARNING] [{UpdatesBackupOfUserFromAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T23:03:19.991Z] 2024-01-18 23:03:19 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T23:03:19.991Z] 2024-01-18 23:03:19 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T23:03:19.991Z] 2024-01-18 23:03:19 +0000:[WARNING] [{UpdatesBackupOfUserFromAdmin}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T23:03:20.246Z] 2024-01-18 23:03:19 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Create a non-admin user [2024-01-18T23:03:20.246Z] 2024-01-18 23:03:19 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Creating 1 users [2024-01-18T23:03:20.501Z] 2024-01-18 23:03:20 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [backup.AddUser:#728] - Response for user [testuser1-1705618999] creation - [2024-01-18T23:03:20.501Z] 2024-01-18 23:03:20 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [backup.AddUser:#733] - User [testuser1-1705618999] created [2024-01-18T23:03:20.501Z] INFO[2024-01-18 23:03:20] Verifying : Description : Creating user - testuser1-1705618999 [2024-01-18T23:03:20.501Z] INFO[2024-01-18 23:03:20] Actual:true, Expected: true [2024-01-18T23:03:20.501Z] 2024-01-18 23:03:20 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705618999 - c9ebbb13-e7ea-4d88-9b3b-e3c540acc148 [2024-01-18T23:03:21.865Z] 2024-01-18 23:03:21 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Creating cloud credentials and backup location from non-adminuser [2024-01-18T23:03:22.427Z] 2024-01-18 23:03:22 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-testuser1-1705618999-pgvca] for org [default] with [aws] as provider [2024-01-18T23:03:22.427Z] 2024-01-18 23:03:22 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-18T23:03:22.427Z] 2024-01-18 23:03:22 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-testuser1-1705618999-pgvca] is present [2024-01-18T23:03:25.692Z] 2024-01-18 23:03:25 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Creating a schedule policy from non-admin [testuser1-1705618999] user [2024-01-18T23:03:25.692Z] INFO[2024-01-18 23:03:25] Verifying : Description : Verifying creation of periodic schedule policy of interval [15] minutes named [periodic-qntqf-testuser1-1705618999] for user [testuser1-1705618999] [2024-01-18T23:03:25.692Z] INFO[2024-01-18 23:03:25] Actual:true, Expected: true [2024-01-18T23:03:25.692Z] 2024-01-18 23:03:25 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Registering Source and Destination clusters as user : testuser1-1705618999 and verifying the status [2024-01-18T23:03:25.948Z] INFO[2024-01-18 23:03:25] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T23:03:25.948Z] INFO[2024-01-18 23:03:25] Actual:true, Expected: true [2024-01-18T23:03:25.948Z] 2024-01-18 23:03:25 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T23:03:25.948Z] 2024-01-18 23:03:25 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T23:03:25.948Z] 2024-01-18 23:03:25 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T23:03:25.948Z] 2024-01-18 23:03:25 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T23:03:25.948Z] 2024-01-18 23:03:25 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Create cluster [source-cluster] in org [default] [2024-01-18T23:03:25.948Z] 2024-01-18 23:03:25 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T23:03:25.948Z] 2024-01-18 23:03:25 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T23:03:25.948Z] 2024-01-18 23:03:25 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-18T23:03:25.948Z] 2024-01-18 23:03:25 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T23:03:25.948Z] 2024-01-18 23:03:25 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T23:03:25.948Z] 2024-01-18 23:03:25 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-cufev] for IBM [2024-01-18T23:03:25.948Z] 2024-01-18 23:03:25 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-cufev] for org [default] with [ibm] as provider [2024-01-18T23:03:25.948Z] 2024-01-18 23:03:25 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T23:03:25.949Z] 2024-01-18 23:03:25 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T23:03:25.949Z] 2024-01-18 23:03:25 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-cufev] is present [2024-01-18T23:03:25.949Z] 2024-01-18 23:03:25 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Create cluster [source-cluster] in org [default] [2024-01-18T23:03:26.873Z] 2024-01-18 23:03:26 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-bgfur] for IBM [2024-01-18T23:03:26.873Z] 2024-01-18 23:03:26 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-bgfur] for org [default] with [ibm] as provider [2024-01-18T23:03:26.873Z] 2024-01-18 23:03:26 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T23:03:26.873Z] 2024-01-18 23:03:26 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T23:03:26.873Z] 2024-01-18 23:03:26 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-bgfur] is present [2024-01-18T23:03:26.874Z] 2024-01-18 23:03:26 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-18T23:03:27.433Z] INFO[2024-01-18 23:03:27] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-18T23:03:27.433Z] INFO[2024-01-18 23:03:27] Actual:Online, Expected: Online [2024-01-18T23:03:27.433Z] INFO[2024-01-18 23:03:27] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-18T23:03:27.434Z] INFO[2024-01-18 23:03:27] Actual:true, Expected: true [2024-01-18T23:03:27.434Z] INFO[2024-01-18 23:03:27] Verifying : Description : Fetching [destination-cluster] cluster uid [2024-01-18T23:03:27.434Z] INFO[2024-01-18 23:03:27] Actual:true, Expected: true [2024-01-18T23:03:27.434Z] 2024-01-18 23:03:27 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Taking manual backup and schedule backup of applications for usertestuser1-1705618999 [2024-01-18T23:03:27.434Z] 2024-01-18 23:03:27 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T23:03:27.434Z] 2024-01-18 23:03:27 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Volume snapshot classes found in the cluster - [2024-01-18T23:03:27.434Z] 2024-01-18 23:03:27 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] ibmc-vpcblock-snapshot [2024-01-18T23:03:27.434Z] 2024-01-18 23:03:27 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T23:03:27.994Z] 2024/01/18 23:03:27 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705618999-nwmz] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T23:04:00.020Z] 2024/01/18 23:03:57 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705618999-nwmz] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:04:32.046Z] 2024/01/18 23:04:27 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705618999-nwmz] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:04:33.953Z] INFO[2024-01-18 23:04:33] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T23:04:33.953Z] INFO[2024-01-18 23:04:33] sched stats: tick interval: 27033 < 1.2s (100%) [2024-01-18T23:04:33.953Z] INFO[2024-01-18 23:04:33] sched stats: tick process durations: 27032 < 500ms (100%) [2024-01-18T23:05:00.441Z] 2024-01-18 23:04:57 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [backup.CreateBackup:#313] - Backup [tp-backup-manual-testuser1-1705618999-nwmz] created successfully [2024-01-18T23:05:00.441Z] 2024-01-18 23:04:57 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Validating backup [tp-backup-manual-testuser1-1705618999-nwmz] in org [default] [2024-01-18T23:05:00.441Z] 2024-01-18 23:04:57 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-manual-testuser1-1705618999-nwmz] [2024-01-18T23:05:00.441Z] 2024-01-18 23:04:57 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87568-01-18-23h01m38s] in backup [tp-backup-manual-testuser1-1705618999-nwmz] [2024-01-18T23:05:00.441Z] 2024-01-18 23:04:57 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87568-01-18-23h01m38s] in backup [tp-backup-manual-testuser1-1705618999-nwmz] [2024-01-18T23:05:00.441Z] 2024-01-18 23:04:57 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87568-01-18-23h01m38s] [2024-01-18T23:05:00.441Z] 2024-01-18 23:04:57 +0000:[WARNING] [{UpdatesBackupOfUserFromAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T23:05:00.441Z] 2024-01-18 23:04:57 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T23:05:00.441Z] 2024-01-18 23:04:57 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87568-01-18-23h01m38s] are [[mysql-data]] [2024-01-18T23:05:00.441Z] 2024-01-18 23:04:58 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T23:05:00.441Z] 2024-01-18 23:04:58 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Validating CR cleanup [2024-01-18T23:05:00.441Z] 2024-01-18 23:04:58 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T23:05:00.441Z] 2024-01-18 23:04:58 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] All CRs in [mysql-ibm-pxb-0-87568-01-18-23h01m38s] are [[]] [2024-01-18T23:05:00.441Z] INFO[2024-01-18 23:04:58] Verifying : Description : Verifying creation of backup [tp-backup-manual-testuser1-1705618999-nwmz] [2024-01-18T23:05:00.441Z] INFO[2024-01-18 23:04:58] Actual:true, Expected: true [2024-01-18T23:05:00.441Z] 2024-01-18 23:04:58 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Creating a schedule backup of namespace [mysql-ibm-pxb-0-87568-01-18-23h01m38s] without pre and post exec rules [2024-01-18T23:05:00.441Z] 2024-01-18 23:04:58 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-18T23:05:00.441Z] 2024-01-18 23:04:58 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Volume snapshot classes found in the cluster - [2024-01-18T23:05:00.441Z] 2024-01-18 23:04:58 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] ibmc-vpcblock-snapshot [2024-01-18T23:05:00.441Z] 2024-01-18 23:04:58 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T23:06:08.087Z] 2024-01-18 23:05:59 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] first schedule backup for schedule name [tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss] is [tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458] [2024-01-18T23:06:08.087Z] 2024-01-18 23:05:59 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Validating backup [tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458] in org [default] [2024-01-18T23:06:08.087Z] 2024-01-18 23:05:59 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458] [2024-01-18T23:06:08.087Z] 2024-01-18 23:05:59 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87568-01-18-23h01m38s] in backup [tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458] [2024-01-18T23:06:08.087Z] 2024-01-18 23:05:59 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87568-01-18-23h01m38s] in backup [tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458] [2024-01-18T23:06:08.087Z] 2024-01-18 23:05:59 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87568-01-18-23h01m38s] [2024-01-18T23:06:08.087Z] 2024-01-18 23:05:59 +0000:[WARNING] [{UpdatesBackupOfUserFromAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T23:06:08.087Z] 2024-01-18 23:05:59 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T23:06:08.088Z] 2024-01-18 23:05:59 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87568-01-18-23h01m38s] are [[mysql-data]] [2024-01-18T23:06:08.088Z] 2024-01-18 23:05:59 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T23:06:08.088Z] 2024-01-18 23:05:59 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Validating CR cleanup [2024-01-18T23:06:08.088Z] 2024-01-18 23:05:59 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T23:06:08.088Z] 2024-01-18 23:05:59 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] All CRs in [mysql-ibm-pxb-0-87568-01-18-23h01m38s] are [[tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458]] [2024-01-18T23:06:08.088Z] 2024-01-18 23:05:59 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] CR found for [[tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458]] under [mysql-ibm-pxb-0-87568-01-18-23h01m38s] namespace [2024-01-18T23:06:08.088Z] 2024/01/18 23:05:59 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:06:08.088Z] 2024-01-18 23:06:04 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] All CRs in [mysql-ibm-pxb-0-87568-01-18-23h01m38s] are [[tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458]] [2024-01-18T23:06:08.088Z] 2024-01-18 23:06:04 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] CR found for [[tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458]] under [mysql-ibm-pxb-0-87568-01-18-23h01m38s] namespace [2024-01-18T23:06:08.088Z] 2024/01/18 23:06:04 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:06:09.981Z] 2024-01-18 23:06:09 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] All CRs in [mysql-ibm-pxb-0-87568-01-18-23h01m38s] are [[tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458]] [2024-01-18T23:06:09.981Z] 2024-01-18 23:06:09 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] CR found for [[tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458]] under [mysql-ibm-pxb-0-87568-01-18-23h01m38s] namespace [2024-01-18T23:06:09.981Z] 2024/01/18 23:06:09 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:06:15.223Z] 2024-01-18 23:06:14 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] All CRs in [mysql-ibm-pxb-0-87568-01-18-23h01m38s] are [[tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458]] [2024-01-18T23:06:15.223Z] 2024-01-18 23:06:14 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] CR found for [[tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458]] under [mysql-ibm-pxb-0-87568-01-18-23h01m38s] namespace [2024-01-18T23:06:15.223Z] 2024/01/18 23:06:14 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:06:20.466Z] 2024-01-18 23:06:19 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] All CRs in [mysql-ibm-pxb-0-87568-01-18-23h01m38s] are [[]] [2024-01-18T23:06:20.466Z] INFO[2024-01-18 23:06:19] Verifying : Description : Verifying creation of backup [tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458] [2024-01-18T23:06:20.466Z] INFO[2024-01-18 23:06:19] Actual:true, Expected: true [2024-01-18T23:06:20.466Z] 2024-01-18 23:06:19 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Create invalid credential for cluster and backup object [2024-01-18T23:06:20.466Z] 2024-01-18 23:06:19 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [backup.createInvalidAWSCloudCredential:#4509] - Create cloud credential with name [invalid-autogenerated-cred-testuser1-1705618999-glndt] for org [default] [2024-01-18T23:06:20.466Z] 2024-01-18 23:06:19 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Verifying listing and update of backup of non-admin user from px-admin [2024-01-18T23:06:20.466Z] 2024-01-18 23:06:20 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705618999 - c9ebbb13-e7ea-4d88-9b3b-e3c540acc148 [2024-01-18T23:06:20.466Z] INFO[2024-01-18 23:06:20] Verifying : Description : Verification of fetching backups [[tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458 tp-backup-manual-testuser1-1705618999-nwmz]] of user [testuser1-1705618999] from px-admin user [2024-01-18T23:06:20.466Z] INFO[2024-01-18 23:06:20] Actual:true, Expected: true [2024-01-18T23:06:20.466Z] 2024-01-18 23:06:20 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [backup.glob..func33.2.8:#3164] - The list of user [testuser1-1705618999] backups from px-admin [tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458 tp-backup-manual-testuser1-1705618999-nwmz] [2024-01-18T23:06:20.466Z] INFO[2024-01-18 23:06:20] Verifying : Description : Verification of update of backup [tp-backup-manual-testuser1-1705618999-nwmz] of user [testuser1-1705618999] from px-admin user [2024-01-18T23:06:20.466Z] INFO[2024-01-18 23:06:20] Actual:true, Expected: true [2024-01-18T23:06:20.466Z] INFO[2024-01-18 23:06:20] Verifying : Description : Verification of update of backup [tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458] of user [testuser1-1705618999] from px-admin user [2024-01-18T23:06:20.466Z] INFO[2024-01-18 23:06:20] Actual:true, Expected: true [2024-01-18T23:06:20.466Z] 2024-01-18 23:06:20 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Verifying deletion of backup of non-admin user from px-admin [2024-01-18T23:06:22.981Z] 2024-01-18 23:06:22 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705618999 - c9ebbb13-e7ea-4d88-9b3b-e3c540acc148 [2024-01-18T23:06:22.981Z] INFO[2024-01-18 23:06:22] Verifying : Description : Verification of fetching backups [[tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458 tp-backup-manual-testuser1-1705618999-nwmz]] of user [testuser1-1705618999] from px-admin user [2024-01-18T23:06:22.981Z] INFO[2024-01-18 23:06:22] Actual:true, Expected: true [2024-01-18T23:06:22.981Z] 2024-01-18 23:06:22 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [backup.glob..func33.2.9:#3185] - The list of user [testuser1-1705618999] backups from px-admin [tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458 tp-backup-manual-testuser1-1705618999-nwmz] [2024-01-18T23:06:22.981Z] INFO[2024-01-18 23:06:22] Verifying : Description : Verifying deletion backup [tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458] of non-admin user [testuser1-1705618999] from px-admin user [2024-01-18T23:06:22.982Z] INFO[2024-01-18 23:06:22] Actual:true, Expected: true [2024-01-18T23:06:22.982Z] 2024/01/18 23:06:22 DoRetryWithTimeout - Error: {backupObject [tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-18T23:06:55.013Z] 2024/01/18 23:06:52 DoRetryWithTimeout - Error: {backupObject [tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-18T23:07:27.047Z] INFO[2024-01-18 23:07:22] Verifying : Description : waiting for backup [tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss-7838e2c-interval-2024-01-18-230458] deletion [2024-01-18T23:07:27.047Z] INFO[2024-01-18 23:07:22] Actual:true, Expected: true [2024-01-18T23:07:27.047Z] INFO[2024-01-18 23:07:22] Verifying : Description : Verifying deletion backup [tp-backup-manual-testuser1-1705618999-nwmz] of non-admin user [testuser1-1705618999] from px-admin user [2024-01-18T23:07:27.047Z] INFO[2024-01-18 23:07:22] Actual:true, Expected: true [2024-01-18T23:07:27.047Z] 2024/01/18 23:07:22 DoRetryWithTimeout - Error: {backupObject [tp-backup-manual-testuser1-1705618999-nwmz] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-18T23:07:53.568Z] 2024/01/18 23:07:52 DoRetryWithTimeout - Error: {backupObject [tp-backup-manual-testuser1-1705618999-nwmz] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-18T23:08:25.626Z] INFO[2024-01-18 23:08:22] Verifying : Description : waiting for backup [tp-backup-manual-testuser1-1705618999-nwmz] deletion [2024-01-18T23:08:25.626Z] INFO[2024-01-18 23:08:22] Actual:true, Expected: true [2024-01-18T23:08:25.626Z] 2024-01-18 23:08:22 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Verifying deletion of backup schedule of non-admin user from px-admin [2024-01-18T23:08:25.626Z] INFO[2024-01-18 23:08:23] Verifying : Description : Verification of fetching backups [[tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss]] of user [testuser1-1705618999] from px-admin user [2024-01-18T23:08:25.626Z] INFO[2024-01-18 23:08:23] Actual:true, Expected: true [2024-01-18T23:08:25.626Z] 2024-01-18 23:08:23 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [backup.glob..func33.2.10:#3207] - The list of user [testuser1-1705618999] backups from px-admin [tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss] [2024-01-18T23:08:25.626Z] 2024-01-18 23:08:23 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Verifying deletion of backup schedule [tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss] of non-admin user [testuser1-1705618999] from px-admin user [2024-01-18T23:08:25.626Z] 2024/01/18 23:08:23 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T23:08:33.714Z] 2024/01/18 23:08:33 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T23:08:43.668Z] INFO[2024-01-18 23:08:43] Verifying : Description : Verification of deleting backup scheudle [tp-backup-schedule-ns-testuser1-1705618999-without-rules-rsss] of user [testuser1-1705618999] from px-admin user [2024-01-18T23:08:43.668Z] INFO[2024-01-18 23:08:43] Actual:true, Expected: true [2024-01-18T23:08:43.668Z] 2024-01-18 23:08:43 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Verifying update of cluster of non-admin user from px-admin [2024-01-18T23:08:43.668Z] 2024-01-18 23:08:43 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T23:08:44.229Z] INFO[2024-01-18 23:08:44] Verifying : Description : Verification of update of cluster [source-cluster] of user [testuser1-1705618999] from px-admin user [2024-01-18T23:08:44.229Z] INFO[2024-01-18 23:08:44] Actual:true, Expected: true [2024-01-18T23:08:44.229Z] 2024-01-18 23:08:44 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T23:08:45.155Z] INFO[2024-01-18 23:08:44] Verifying : Description : Verification of update of cluster [destination-cluster] of user [testuser1-1705618999] from px-admin user [2024-01-18T23:08:45.155Z] INFO[2024-01-18 23:08:44] Actual:true, Expected: true [2024-01-18T23:08:45.155Z] 2024-01-18 23:08:44 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Verifying deletion of clusters of non-admin user from px-admin user [2024-01-18T23:08:45.411Z] INFO[2024-01-18 23:08:45] Verifying : Description : Deleting cluster source-cluster [2024-01-18T23:08:45.411Z] INFO[2024-01-18 23:08:45] Actual:true, Expected: true [2024-01-18T23:08:45.411Z] INFO[2024-01-18 23:08:45] Verifying : Description : Deleting cluster destination-cluster [2024-01-18T23:08:45.411Z] INFO[2024-01-18 23:08:45] Actual:true, Expected: true [2024-01-18T23:08:45.666Z] 2024-01-18 23:08:45 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Deleting the deployed apps after the testcase [2024-01-18T23:08:45.666Z] 2024-01-18 23:08:45 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Destroying apps [2024-01-18T23:08:45.666Z] 2024-01-18 23:08:45 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T23:08:45.666Z] 2024-01-18 23:08:45 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T23:08:45.666Z] 2024-01-18 23:08:45 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] start destroying mysql-ibm app [2024-01-18T23:08:45.666Z] 2024-01-18 23:08:45 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T23:08:45.666Z] 2024-01-18 23:08:45 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T23:08:45.666Z] 2024-01-18 23:08:45 +0000:[WARNING] [{UpdatesBackupOfUserFromAdmin}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T23:08:45.666Z] INFO[2024-01-18 23:08:45] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T23:08:45.666Z] INFO[2024-01-18 23:08:45] Actual:true, Expected: true [2024-01-18T23:08:45.667Z] 2024-01-18 23:08:45 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [backup.glob..func33.3:#3249] - Deleting backup schedule policy [2024-01-18T23:08:47.030Z] INFO[2024-01-18 23:08:46] Verifying : Description : Deleting backup schedule policies [periodic-qntqf-testuser1-1705618999] [2024-01-18T23:08:47.030Z] INFO[2024-01-18 23:08:46] Actual:true, Expected: true [2024-01-18T23:08:47.030Z] 2024-01-18 23:08:46 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Cleaning backup locations in map [map[a019b74b-3dde-4741-ad2c-0f5079612f68:autogenerated-backup-location-testuser1-1705618999-mehzf]], cloud credential [autogenerated-cred-testuser1-1705618999-pgvca], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T23:08:47.030Z] INFO[2024-01-18 23:08:46] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-testuser1-1705618999-mehzf] [2024-01-18T23:08:47.030Z] INFO[2024-01-18 23:08:46] Actual:true, Expected: true [2024-01-18T23:08:47.030Z] 2024-01-18 23:08:46 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-testuser1-1705618999-mehzf] is present [2024-01-18T23:08:47.031Z] 2024/01/18 23:08:46 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-testuser1-1705618999-mehzf is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T23:09:19.055Z] 2024-01-18 23:09:16 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-testuser1-1705618999-mehzf] is present [2024-01-18T23:09:19.055Z] 2024/01/18 23:09:16 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-testuser1-1705618999-mehzf is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T23:09:51.077Z] 2024-01-18 23:09:47 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [] [2024-01-18T23:09:51.078Z] INFO[2024-01-18 23:09:47] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-testuser1-1705618999-mehzf [2024-01-18T23:09:51.078Z] INFO[2024-01-18 23:09:47] Actual:true, Expected: true [2024-01-18T23:09:51.078Z] 2024-01-18 23:09:47 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-testuser1-1705618999-pgvca] is present [2024-01-18T23:09:51.078Z] INFO[2024-01-18 23:09:47] Verifying : Description : Verifying if cloud cred [autogenerated-cred-testuser1-1705618999-pgvca] is present [2024-01-18T23:09:51.078Z] INFO[2024-01-18 23:09:47] Actual:true, Expected: true [2024-01-18T23:09:51.078Z] INFO[2024-01-18 23:09:47] Verifying : Description : Verifying deletion of cloud cred [autogenerated-cred-testuser1-1705618999-pgvca] [2024-01-18T23:09:51.078Z] INFO[2024-01-18 23:09:47] Actual:true, Expected: true [2024-01-18T23:09:51.078Z] INFO[2024-01-18 23:09:47] Verifying : Description : Deleting cloud cred autogenerated-cred-testuser1-1705618999-pgvca [2024-01-18T23:09:51.078Z] INFO[2024-01-18 23:09:47] Actual:true, Expected: true [2024-01-18T23:09:51.078Z] INFO[2024-01-18 23:09:47] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T23:09:51.078Z] INFO[2024-01-18 23:09:47] Actual:true, Expected: true [2024-01-18T23:09:51.078Z] INFO[2024-01-18 23:09:47] --------Test End------ [2024-01-18T23:09:51.078Z] INFO[2024-01-18 23:09:47] #Test: [2024-01-18T23:09:51.078Z] INFO[2024-01-18 23:09:47] #Description: [2024-01-18T23:09:51.078Z] INFO[2024-01-18 23:09:47] ------------------------ [2024-01-18T23:09:51.078Z] 2024-01-18 23:09:47 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-87568-01-18-23h01m38s] [2024-01-18T23:09:51.078Z] 2024-01-18 23:09:47 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T23:09:51.078Z] 2024-01-18 23:09:47 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Switching context: The context is already [/tmp/destination-config] [2024-01-18T23:09:51.078Z] 2024-01-18 23:09:47 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-87568-01-18-23h01m38s] [2024-01-18T23:09:51.078Z] 2024-01-18 23:09:47 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {UpdatesBackupOfUserFromAdmin} Updates Backups and Cluster of user from px-admin [2024-01-18T23:09:51.078Z] 2024-01-18 23:09:47 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-18T23:09:51.078Z] 2024/01/18 23:09:47 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:09:52.995Z] 2024/01/18 23:09:52 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:09:58.240Z] 2024/01/18 23:09:57 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:10:03.518Z] 2024/01/18 23:10:02 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:10:07.682Z] 2024/01/18 23:10:07 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:10:12.962Z] 2024/01/18 23:10:12 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:10:18.224Z] 2024/01/18 23:10:17 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:10:23.478Z] 2024/01/18 23:10:22 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:10:27.672Z] 2024/01/18 23:10:27 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.128.20. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.20 10.240.128.20]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-18T23:10:27.672Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-18T23:10:27.672Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-18T23:10:27.672Z] {dial tcp 10.240.128.20:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-18T23:10:32.917Z] 2024-01-18 23:10:32 +0000:[WARNING] [{UpdatesBackupOfUserFromAdmin}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.20. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.20 10.240.128.20]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-18T23:10:32.918Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-18T23:10:32.918Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-18T23:10:32.918Z] {dial tcp 10.240.128.20:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T23:10:32.918Z] 2024-01-18 23:10:32 +0000:[ERROR] [{UpdatesBackupOfUserFromAdmin}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.20. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.20 10.240.128.20]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-18T23:10:32.918Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-18T23:10:32.918Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-18T23:10:32.918Z] {dial tcp 10.240.128.20:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T23:10:32.918Z] 2024-01-18 23:10:32 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T23:10:32.918Z] 2024-01-18 23:10:32 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] Switching context to [/tmp/source-config] [2024-01-18T23:10:32.918Z] 2024-01-18 23:10:32 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T23:10:32.918Z] 2024-01-18 23:10:32 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T23:10:32.918Z] 2024-01-18 23:10:32 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T23:10:32.918Z] 2024-01-18 23:10:32 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T23:10:32.918Z] 2024-01-18 23:10:32 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T23:10:33.176Z] 2024-01-18 23:10:32 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T23:10:33.433Z] 2024-01-18 23:10:33 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T23:10:33.995Z] 2024-01-18 23:10:33 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T23:10:34.251Z] 2024-01-18 23:10:34 +0000:[INFO] [{UpdatesBackupOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T23:10:34.506Z] 2024/01/18 23:10:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:10:44.479Z] 2024/01/18 23:10:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:10:54.419Z] 2024/01/18 23:10:54 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:11:06.582Z] 2024/01/18 23:11:04 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:11:14.674Z] 2024/01/18 23:11:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:11:24.617Z] 2024/01/18 23:11:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:11:34.638Z] 2024/01/18 23:11:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:11:44.578Z] 2024/01/18 23:11:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:11:56.744Z] 2024/01/18 23:11:54 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:12:04.863Z] 2024/01/18 23:12:04 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:12:14.802Z] 2024/01/18 23:12:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:12:24.773Z] 2024/01/18 23:12:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:12:34.712Z] 2024/01/18 23:12:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:12:46.877Z] 2024/01/18 23:12:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:12:54.972Z] 2024/01/18 23:12:54 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:13:04.908Z] 2024/01/18 23:13:04 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:13:14.851Z] 2024/01/18 23:13:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:13:27.022Z] 2024/01/18 23:13:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:13:35.102Z] 2024/01/18 23:13:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:13:45.052Z] 2024/01/18 23:13:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:13:54.999Z] 2024/01/18 23:13:54 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:14:04.944Z] 2024/01/18 23:14:04 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:14:17.118Z] 2024/01/18 23:14:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:14:25.234Z] 2024/01/18 23:14:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:14:35.189Z] INFO[2024-01-18 23:14:34] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T23:14:35.189Z] INFO[2024-01-18 23:14:34] sched stats: tick interval: 27634 < 1.2s (100%) [2024-01-18T23:14:35.189Z] INFO[2024-01-18 23:14:34] sched stats: tick process durations: 27633 < 500ms (100%) [2024-01-18T23:14:35.189Z] 2024/01/18 23:14:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:14:45.138Z] 2024/01/18 23:14:45 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:14:56.646Z] 2024/01/18 23:14:55 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:15:06.638Z] 2024/01/18 23:15:05 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:15:16.580Z] 2024/01/18 23:15:15 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:15:26.530Z] 2024/01/18 23:15:25 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:15:36.473Z] 2024-01-18 23:15:35 +0000:[ERROR] [{UpdatesBackupOfUserFromAdmin}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.473Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.473Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.473Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.473Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.473Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.473Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.473Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.473Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.473Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.473Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.473Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.473Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.474Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.474Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.474Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.474Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.474Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.474Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.474Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.474Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.474Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.474Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.474Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.474Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.475Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.475Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.475Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.475Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.475Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-18T23:15:36.475Z] [2024-01-18T23:15:36.475Z] ------------------------------ [2024-01-18T23:15:36.475Z] • Failure [837.013 seconds] [2024-01-18T23:15:36.475Z] {UpdatesBackupOfUserFromAdmin} [2024-01-18T23:15:36.475Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_orphaned_test.go:3028 [2024-01-18T23:15:36.475Z] Updates Backups and Cluster of user from px-admin [] [2024-01-18T23:15:36.475Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_orphaned_test.go:3068 [2024-01-18T23:15:36.475Z] [2024-01-18T23:15:36.475Z] Unexpected error: [2024-01-18T23:15:36.475Z] <*errors.errorString | 0xc000e2be80>: { [2024-01-18T23:15:36.476Z] s: "failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{No... [2024-01-18T23:15:36.476Z] [2024-01-18T23:15:36.476Z] Gomega truncated this representation as it exceeds 'format.MaxLength'. [2024-01-18T23:15:36.476Z] Consider having the object provide a custom 'GomegaStringer' representation [2024-01-18T23:15:36.476Z] or adjust the parameters in Gomega's 'format' package. [2024-01-18T23:15:36.476Z] [2024-01-18T23:15:36.476Z] Learn more here: https://onsi.github.io/gomega/#adjusting-output [2024-01-18T23:15:36.476Z] [2024-01-18T23:15:36.477Z] failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.477Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.477Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.477Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.477Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.477Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.477Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.477Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.477Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.477Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.477Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.477Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.477Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.477Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.478Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.478Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.478Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.478Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.478Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.478Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.478Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.479Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.479Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.479Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.479Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.479Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.479Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.479Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.479Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:15:36.479Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-18T23:15:36.479Z] occurred [2024-01-18T23:15:36.479Z] [2024-01-18T23:15:36.479Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-18T23:15:36.479Z] [2024-01-18T23:15:36.479Z] Full Stack Trace [2024-01-18T23:15:36.479Z] github.com/portworx/torpedo/pkg/log.FailOnError({0x88a5060, 0xc000e2be80}, {0x7f74bfd?, 0x6cc0320?}, {0x0?, 0x6cc0320?, 0xc001357bb0?}) [2024-01-18T23:15:36.479Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 +0x290 [2024-01-18T23:15:36.479Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest.func1() [2024-01-18T23:15:36.479Z] /go/src/github.com/portworx/torpedo/tests/common.go:7283 +0x32 [2024-01-18T23:15:36.479Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest({0xc000583060, 0x1, 0x1}) [2024-01-18T23:15:36.479Z] /go/src/github.com/portworx/torpedo/tests/common.go:7299 +0x778 [2024-01-18T23:15:36.479Z] github.com/portworx/torpedo/tests/backup.glob..func33.3() [2024-01-18T23:15:36.479Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_orphaned_test.go:3253 +0x30f [2024-01-18T23:15:36.479Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-18T23:15:36.479Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-18T23:15:36.479Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-18T23:15:36.479Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-18T23:15:36.479Z] created by testing.(*T).Run [2024-01-18T23:15:36.479Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-18T23:15:36.479Z] ------------------------------ [2024-01-18T23:15:36.479Z] STEP: schedule applications [2024-01-18T23:15:36.479Z] STEP: Validate applications [2024-01-18T23:15:36.479Z] STEP: validate applications [2024-01-18T23:15:36.479Z] STEP: validate mysql-ibm app's volumes [2024-01-18T23:15:36.479Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T23:15:36.479Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T23:15:36.479Z] STEP: get mysql-ibm app's volume: pvc-14966c6b-47e5-4ae9-ae8d-5873f45b3386 inspected by the volume driver [2024-01-18T23:15:36.480Z] STEP: wait for mysql-ibm app to start running [2024-01-18T23:15:36.480Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T23:15:36.480Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T23:15:36.480Z] STEP: Validate Px pod restart count [2024-01-18T23:15:36.480Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T23:15:36.480Z] STEP: Create a non-admin user [2024-01-18T23:15:36.480Z] STEP: Adding Credentials and Backup Location from non-admin user [2024-01-18T23:15:36.480Z] STEP: Create schedule policy from non-admin user [2024-01-18T23:15:36.480Z] STEP: Register source and destination cluster for backup on testuser1-1705618999 [2024-01-18T23:15:36.480Z] STEP: Taking manual backup and schedule backup of applications for user testuser1-1705618999 [2024-01-18T23:15:36.480Z] STEP: Create invalid credential for cluster and backup object [2024-01-18T23:15:36.480Z] STEP: Verifying listing and update of backup of non-admin user from px-admin [2024-01-18T23:15:36.480Z] STEP: Verifying deletion of backup of non-admin user from px-admin [2024-01-18T23:15:36.480Z] STEP: Verifying deletion of backup schedule of non-admin user from px-admin [2024-01-18T23:15:36.480Z] STEP: Verifying update of cluster of non-admin user from px-admin [2024-01-18T23:15:36.480Z] STEP: Verifying deletion of clusters of non-admin user from px-admin user [2024-01-18T23:15:36.480Z] STEP: destroy apps [2024-01-18T23:15:36.480Z] STEP: destroy the mysql-ibm app's volumes [2024-01-18T23:15:36.480Z] STEP: start destroying mysql-ibm app [2024-01-18T23:15:36.480Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T23:15:36.480Z] 2024-01-18 23:15:35 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Log Dir: /testresults//ScheduleBackupCreationAllNS.log [2024-01-18T23:15:36.480Z] INFO[2024-01-18 23:15:35] --------Test Start------ [2024-01-18T23:15:36.480Z] INFO[2024-01-18 23:15:35] #Test: ScheduleBackupCreationAllNS [2024-01-18T23:15:36.480Z] INFO[2024-01-18 23:15:35] #Description: Create schedule backup creation with all namespaces [2024-01-18T23:15:36.480Z] INFO[2024-01-18 23:15:35] ------------------------ [2024-01-18T23:15:36.480Z] 2024-01-18 23:15:35 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.glob..func58.1:#555] - Application installation [2024-01-18T23:15:36.480Z] 2024-01-18 23:15:35 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T23:15:36.480Z] 2024-01-18 23:15:35 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T23:15:36.480Z] 2024-01-18 23:15:35 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T23:15:36.480Z] 2024-01-18 23:15:35 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T23:15:36.480Z] 2024-01-18 23:15:35 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T23:15:36.480Z] 2024-01-18 23:15:35 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T23:15:36.481Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T23:15:36.481Z] 2024-01-18 23:15:35 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T23:15:36.482Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T23:15:36.483Z] 2024-01-18 23:15:35 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T23:15:36.483Z] 2024-01-18 23:15:35 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Validate applications [2024-01-18T23:15:36.483Z] 2024-01-18 23:15:35 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Validating mysql-ibm app [2024-01-18T23:15:36.483Z] 2024-01-18 23:15:35 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Validating mysql-ibm app's volumes [2024-01-18T23:15:36.483Z] 2024-01-18 23:15:35 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T23:15:36.483Z] 2024/01/18 23:15:35 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T23:15:46.422Z] 2024/01/18 23:15:45 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T23:15:56.418Z] 2024-01-18 23:15:55 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-58015-01-18-23h15m35s [2024-01-18T23:15:56.418Z] 2024-01-18 23:15:55 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T23:15:56.418Z] 2024-01-18 23:15:55 +0000:[WARNING] [{ScheduleBackupCreationAllNS}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T23:15:56.418Z] 2024-01-18 23:15:55 +0000:[INFO] [{ScheduleBackupCreationAllNS}] wait for mysql-ibm app to start running [2024-01-18T23:15:56.418Z] 2024-01-18 23:15:55 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T23:15:56.418Z] 2024/01/18 23:15:55 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:15:56.418Z] pod name:mysql-7d766cd65c-9wmvn namespace:mysql-ibm-pxb-0-58015-01-18-23h15m35s running:false ready:false node:10.240.64.8 [2024-01-18T23:15:56.418Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:16:06.355Z] 2024/01/18 23:16:05 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:16:06.355Z] pod name:mysql-7d766cd65c-9wmvn namespace:mysql-ibm-pxb-0-58015-01-18-23h15m35s running:true ready:false node:10.240.64.8 [2024-01-18T23:16:06.355Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:16:16.294Z] 2024/01/18 23:16:15 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:16:16.294Z] pod name:mysql-7d766cd65c-9wmvn namespace:mysql-ibm-pxb-0-58015-01-18-23h15m35s running:true ready:false node:10.240.64.8 [2024-01-18T23:16:16.294Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:16:26.403Z] 2024/01/18 23:16:26 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:16:26.404Z] pod name:mysql-7d766cd65c-9wmvn namespace:mysql-ibm-pxb-0-58015-01-18-23h15m35s running:true ready:false node:10.240.64.8 [2024-01-18T23:16:26.404Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:16:36.350Z] 2024/01/18 23:16:36 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:16:36.350Z] pod name:mysql-7d766cd65c-9wmvn namespace:mysql-ibm-pxb-0-58015-01-18-23h15m35s running:true ready:false node:10.240.64.8 [2024-01-18T23:16:36.350Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:16:46.294Z] 2024/01/18 23:16:46 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:16:46.294Z] pod name:mysql-7d766cd65c-9wmvn namespace:mysql-ibm-pxb-0-58015-01-18-23h15m35s running:true ready:false node:10.240.64.8 [2024-01-18T23:16:46.294Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:16:58.461Z] 2024/01/18 23:16:56 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:16:58.461Z] pod name:mysql-7d766cd65c-9wmvn namespace:mysql-ibm-pxb-0-58015-01-18-23h15m35s running:true ready:false node:10.240.64.8 [2024-01-18T23:16:58.461Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:17:06.557Z] 2024/01/18 23:17:06 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:17:06.557Z] pod name:mysql-7d766cd65c-9wmvn namespace:mysql-ibm-pxb-0-58015-01-18-23h15m35s running:true ready:false node:10.240.64.8 [2024-01-18T23:17:06.557Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:17:18.720Z] 2024-01-18 23:17:16 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T23:17:18.720Z] 2024-01-18 23:17:16 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T23:17:18.720Z] 2024-01-18 23:17:16 +0000:[INFO] [{ScheduleBackupCreationAllNS}] validate if mysql-ibm app's volumes are setup [2024-01-18T23:17:18.720Z] 2024-01-18 23:17:16 +0000:[WARNING] [{ScheduleBackupCreationAllNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T23:17:18.720Z] 2024-01-18 23:17:16 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T23:17:18.720Z] 2024-01-18 23:17:16 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T23:17:18.720Z] 2024-01-18 23:17:16 +0000:[WARNING] [{ScheduleBackupCreationAllNS}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T23:17:18.720Z] 2024-01-18 23:17:16 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Adding cloud account [2024-01-18T23:17:18.720Z] 2024-01-18 23:17:17 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [aws-1705592040] for org [default] with [aws] as provider [2024-01-18T23:17:18.720Z] 2024-01-18 23:17:17 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-18T23:17:18.720Z] 2024-01-18 23:17:17 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [aws-1705592040] is present [2024-01-18T23:17:18.720Z] INFO[2024-01-18 23:17:17] Verifying : Description : Verifying creation of cloud credential named [aws-1705592040] for org [default] with [aws] as provider [2024-01-18T23:17:18.721Z] INFO[2024-01-18 23:17:17] Actual:true, Expected: true [2024-01-18T23:17:20.608Z] INFO[2024-01-18 23:17:20] Verifying : Description : Adding Backup Location - auto-bl-1705619837 [2024-01-18T23:17:20.608Z] INFO[2024-01-18 23:17:20] Actual:true, Expected: true [2024-01-18T23:17:20.608Z] 2024-01-18 23:17:20 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Adding application clusters [2024-01-18T23:17:20.608Z] 2024-01-18 23:17:20 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Create Backup Schedule Policy [2024-01-18T23:17:21.169Z] INFO[2024-01-18 23:17:20] Verifying : Description : Verification of creating periodic schedule policy - periodic-1705592040 [2024-01-18T23:17:21.169Z] INFO[2024-01-18 23:17:20] Actual:true, Expected: true [2024-01-18T23:17:21.169Z] 2024-01-18 23:17:20 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Adding application clusters [2024-01-18T23:17:21.427Z] INFO[2024-01-18 23:17:21] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T23:17:21.427Z] INFO[2024-01-18 23:17:21] Actual:true, Expected: true [2024-01-18T23:17:21.427Z] 2024-01-18 23:17:21 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T23:17:21.684Z] 2024-01-18 23:17:21 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T23:17:21.684Z] 2024-01-18 23:17:21 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T23:17:21.684Z] 2024-01-18 23:17:21 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T23:17:21.684Z] 2024-01-18 23:17:21 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Create cluster [source-cluster] in org [default] [2024-01-18T23:17:21.684Z] 2024-01-18 23:17:21 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T23:17:21.684Z] 2024-01-18 23:17:21 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T23:17:21.684Z] 2024-01-18 23:17:21 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Create cluster [destination-cluster] in org [default] [2024-01-18T23:17:21.684Z] 2024-01-18 23:17:21 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T23:17:21.685Z] 2024-01-18 23:17:21 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T23:17:21.685Z] 2024-01-18 23:17:21 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-uclrx] for IBM [2024-01-18T23:17:21.685Z] 2024-01-18 23:17:21 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-uclrx] for org [default] with [ibm] as provider [2024-01-18T23:17:21.685Z] 2024-01-18 23:17:21 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T23:17:21.685Z] 2024-01-18 23:17:21 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T23:17:21.941Z] 2024-01-18 23:17:21 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-uclrx] is present [2024-01-18T23:17:21.941Z] 2024-01-18 23:17:21 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Create cluster [source-cluster] in org [default] [2024-01-18T23:17:23.305Z] 2024-01-18 23:17:23 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-fitnv] for IBM [2024-01-18T23:17:23.305Z] 2024-01-18 23:17:23 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-fitnv] for org [default] with [ibm] as provider [2024-01-18T23:17:23.305Z] 2024-01-18 23:17:23 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T23:17:23.305Z] 2024-01-18 23:17:23 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T23:17:23.561Z] 2024-01-18 23:17:23 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-fitnv] is present [2024-01-18T23:17:23.561Z] 2024-01-18 23:17:23 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Create cluster [destination-cluster] in org [default] [2024-01-18T23:17:23.561Z] INFO[2024-01-18 23:17:23] Verifying : Description : Verification of creating source - source-cluster and destination - destination-cluster clusters [2024-01-18T23:17:23.561Z] INFO[2024-01-18 23:17:23] Actual:true, Expected: true [2024-01-18T23:17:23.561Z] INFO[2024-01-18 23:17:23] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-18T23:17:23.561Z] INFO[2024-01-18 23:17:23] Actual:Online, Expected: Online [2024-01-18T23:17:23.561Z] 2024-01-18 23:17:23 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.glob..func58.2.5:#616] - Cluster [source-cluster] uid: [fedc5bd9-0f23-41ec-af6b-f3306a52b5bd] [2024-01-18T23:17:23.561Z] 2024-01-18 23:17:23 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Creating schedule backups [2024-01-18T23:17:23.819Z] 2024-01-18 23:17:23 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-18T23:17:23.819Z] 2024-01-18 23:17:23 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Volume snapshot classes found in the cluster - [2024-01-18T23:17:23.819Z] 2024-01-18 23:17:23 +0000:[INFO] [{ScheduleBackupCreationAllNS}] ibmc-vpcblock-snapshot [2024-01-18T23:17:23.819Z] 2024-01-18 23:17:23 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T23:18:31.469Z] 2024-01-18 23:18:24 +0000:[INFO] [{ScheduleBackupCreationAllNS}] first schedule backup for schedule name [tp-backup-schedule-1705592040] is [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724] [2024-01-18T23:18:31.469Z] 2024-01-18 23:18:24 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Validating backup [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724] in org [default] [2024-01-18T23:18:31.469Z] 2024-01-18 23:18:24 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724] [2024-01-18T23:18:31.469Z] 2024-01-18 23:18:24 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58015-01-18-23h15m35s] in backup [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724] [2024-01-18T23:18:31.469Z] 2024-01-18 23:18:24 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58015-01-18-23h15m35s] in backup [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724] [2024-01-18T23:18:31.469Z] 2024-01-18 23:18:24 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58015-01-18-23h15m35s] [2024-01-18T23:18:31.469Z] 2024-01-18 23:18:24 +0000:[WARNING] [{ScheduleBackupCreationAllNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T23:18:31.469Z] 2024-01-18 23:18:24 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T23:18:31.469Z] 2024-01-18 23:18:24 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-58015-01-18-23h15m35s] are [[mysql-data]] [2024-01-18T23:18:31.469Z] 2024-01-18 23:18:24 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T23:18:31.469Z] 2024-01-18 23:18:24 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Validating CR cleanup [2024-01-18T23:18:31.469Z] 2024-01-18 23:18:24 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T23:18:31.469Z] 2024-01-18 23:18:24 +0000:[INFO] [{ScheduleBackupCreationAllNS}] All CRs in [mysql-ibm-pxb-0-58015-01-18-23h15m35s] are [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] [2024-01-18T23:18:31.469Z] 2024-01-18 23:18:24 +0000:[INFO] [{ScheduleBackupCreationAllNS}] CR found for [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] under [mysql-ibm-pxb-0-58015-01-18-23h15m35s] namespace [2024-01-18T23:18:31.469Z] 2024/01/18 23:18:24 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:18:31.469Z] 2024-01-18 23:18:29 +0000:[INFO] [{ScheduleBackupCreationAllNS}] All CRs in [mysql-ibm-pxb-0-58015-01-18-23h15m35s] are [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] [2024-01-18T23:18:31.469Z] 2024-01-18 23:18:29 +0000:[INFO] [{ScheduleBackupCreationAllNS}] CR found for [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] under [mysql-ibm-pxb-0-58015-01-18-23h15m35s] namespace [2024-01-18T23:18:31.470Z] 2024/01/18 23:18:29 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:18:35.663Z] 2024-01-18 23:18:35 +0000:[INFO] [{ScheduleBackupCreationAllNS}] All CRs in [mysql-ibm-pxb-0-58015-01-18-23h15m35s] are [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] [2024-01-18T23:18:35.663Z] 2024-01-18 23:18:35 +0000:[INFO] [{ScheduleBackupCreationAllNS}] CR found for [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] under [mysql-ibm-pxb-0-58015-01-18-23h15m35s] namespace [2024-01-18T23:18:35.663Z] 2024/01/18 23:18:35 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:18:40.912Z] 2024-01-18 23:18:40 +0000:[INFO] [{ScheduleBackupCreationAllNS}] All CRs in [mysql-ibm-pxb-0-58015-01-18-23h15m35s] are [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] [2024-01-18T23:18:40.912Z] 2024-01-18 23:18:40 +0000:[INFO] [{ScheduleBackupCreationAllNS}] CR found for [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] under [mysql-ibm-pxb-0-58015-01-18-23h15m35s] namespace [2024-01-18T23:18:40.912Z] 2024/01/18 23:18:40 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:18:46.157Z] 2024-01-18 23:18:45 +0000:[INFO] [{ScheduleBackupCreationAllNS}] All CRs in [mysql-ibm-pxb-0-58015-01-18-23h15m35s] are [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] [2024-01-18T23:18:46.157Z] 2024-01-18 23:18:45 +0000:[INFO] [{ScheduleBackupCreationAllNS}] CR found for [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] under [mysql-ibm-pxb-0-58015-01-18-23h15m35s] namespace [2024-01-18T23:18:46.158Z] 2024/01/18 23:18:45 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:18:50.328Z] 2024-01-18 23:18:50 +0000:[INFO] [{ScheduleBackupCreationAllNS}] All CRs in [mysql-ibm-pxb-0-58015-01-18-23h15m35s] are [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] [2024-01-18T23:18:50.328Z] 2024-01-18 23:18:50 +0000:[INFO] [{ScheduleBackupCreationAllNS}] CR found for [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] under [mysql-ibm-pxb-0-58015-01-18-23h15m35s] namespace [2024-01-18T23:18:50.328Z] 2024/01/18 23:18:50 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:18:55.581Z] 2024-01-18 23:18:55 +0000:[INFO] [{ScheduleBackupCreationAllNS}] All CRs in [mysql-ibm-pxb-0-58015-01-18-23h15m35s] are [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] [2024-01-18T23:18:55.581Z] 2024-01-18 23:18:55 +0000:[INFO] [{ScheduleBackupCreationAllNS}] CR found for [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] under [mysql-ibm-pxb-0-58015-01-18-23h15m35s] namespace [2024-01-18T23:18:55.581Z] 2024/01/18 23:18:55 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:19:00.833Z] 2024-01-18 23:19:00 +0000:[INFO] [{ScheduleBackupCreationAllNS}] All CRs in [mysql-ibm-pxb-0-58015-01-18-23h15m35s] are [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] [2024-01-18T23:19:00.833Z] 2024-01-18 23:19:00 +0000:[INFO] [{ScheduleBackupCreationAllNS}] CR found for [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] under [mysql-ibm-pxb-0-58015-01-18-23h15m35s] namespace [2024-01-18T23:19:00.833Z] 2024/01/18 23:19:00 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:19:06.083Z] 2024-01-18 23:19:05 +0000:[INFO] [{ScheduleBackupCreationAllNS}] All CRs in [mysql-ibm-pxb-0-58015-01-18-23h15m35s] are [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] [2024-01-18T23:19:06.083Z] 2024-01-18 23:19:05 +0000:[INFO] [{ScheduleBackupCreationAllNS}] CR found for [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] under [mysql-ibm-pxb-0-58015-01-18-23h15m35s] namespace [2024-01-18T23:19:06.083Z] 2024/01/18 23:19:05 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:19:11.370Z] 2024-01-18 23:19:10 +0000:[INFO] [{ScheduleBackupCreationAllNS}] All CRs in [mysql-ibm-pxb-0-58015-01-18-23h15m35s] are [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] [2024-01-18T23:19:11.370Z] 2024-01-18 23:19:10 +0000:[INFO] [{ScheduleBackupCreationAllNS}] CR found for [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] under [mysql-ibm-pxb-0-58015-01-18-23h15m35s] namespace [2024-01-18T23:19:11.370Z] 2024/01/18 23:19:10 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:19:15.533Z] 2024-01-18 23:19:15 +0000:[INFO] [{ScheduleBackupCreationAllNS}] All CRs in [mysql-ibm-pxb-0-58015-01-18-23h15m35s] are [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] [2024-01-18T23:19:15.533Z] 2024-01-18 23:19:15 +0000:[INFO] [{ScheduleBackupCreationAllNS}] CR found for [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] under [mysql-ibm-pxb-0-58015-01-18-23h15m35s] namespace [2024-01-18T23:19:15.533Z] 2024/01/18 23:19:15 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:19:20.776Z] 2024-01-18 23:19:20 +0000:[INFO] [{ScheduleBackupCreationAllNS}] All CRs in [mysql-ibm-pxb-0-58015-01-18-23h15m35s] are [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] [2024-01-18T23:19:20.776Z] 2024-01-18 23:19:20 +0000:[INFO] [{ScheduleBackupCreationAllNS}] CR found for [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] under [mysql-ibm-pxb-0-58015-01-18-23h15m35s] namespace [2024-01-18T23:19:20.776Z] 2024/01/18 23:19:20 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:19:26.053Z] 2024-01-18 23:19:25 +0000:[INFO] [{ScheduleBackupCreationAllNS}] All CRs in [mysql-ibm-pxb-0-58015-01-18-23h15m35s] are [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] [2024-01-18T23:19:26.053Z] 2024-01-18 23:19:25 +0000:[INFO] [{ScheduleBackupCreationAllNS}] CR found for [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] under [mysql-ibm-pxb-0-58015-01-18-23h15m35s] namespace [2024-01-18T23:19:26.053Z] 2024/01/18 23:19:25 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:19:31.297Z] 2024-01-18 23:19:30 +0000:[INFO] [{ScheduleBackupCreationAllNS}] All CRs in [mysql-ibm-pxb-0-58015-01-18-23h15m35s] are [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] [2024-01-18T23:19:31.297Z] 2024-01-18 23:19:30 +0000:[INFO] [{ScheduleBackupCreationAllNS}] CR found for [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] under [mysql-ibm-pxb-0-58015-01-18-23h15m35s] namespace [2024-01-18T23:19:31.297Z] 2024/01/18 23:19:30 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:19:36.558Z] 2024-01-18 23:19:35 +0000:[INFO] [{ScheduleBackupCreationAllNS}] All CRs in [mysql-ibm-pxb-0-58015-01-18-23h15m35s] are [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] [2024-01-18T23:19:36.558Z] 2024-01-18 23:19:35 +0000:[INFO] [{ScheduleBackupCreationAllNS}] CR found for [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] under [mysql-ibm-pxb-0-58015-01-18-23h15m35s] namespace [2024-01-18T23:19:36.558Z] 2024/01/18 23:19:35 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:19:40.836Z] 2024-01-18 23:19:40 +0000:[INFO] [{ScheduleBackupCreationAllNS}] All CRs in [mysql-ibm-pxb-0-58015-01-18-23h15m35s] are [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] [2024-01-18T23:19:40.836Z] 2024-01-18 23:19:40 +0000:[INFO] [{ScheduleBackupCreationAllNS}] CR found for [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] under [mysql-ibm-pxb-0-58015-01-18-23h15m35s] namespace [2024-01-18T23:19:40.836Z] 2024/01/18 23:19:40 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:19:46.088Z] 2024-01-18 23:19:45 +0000:[INFO] [{ScheduleBackupCreationAllNS}] All CRs in [mysql-ibm-pxb-0-58015-01-18-23h15m35s] are [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] [2024-01-18T23:19:46.088Z] 2024-01-18 23:19:45 +0000:[INFO] [{ScheduleBackupCreationAllNS}] CR found for [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] under [mysql-ibm-pxb-0-58015-01-18-23h15m35s] namespace [2024-01-18T23:19:46.088Z] 2024/01/18 23:19:45 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:19:51.347Z] 2024-01-18 23:19:50 +0000:[INFO] [{ScheduleBackupCreationAllNS}] All CRs in [mysql-ibm-pxb-0-58015-01-18-23h15m35s] are [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] [2024-01-18T23:19:51.347Z] 2024-01-18 23:19:50 +0000:[INFO] [{ScheduleBackupCreationAllNS}] CR found for [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] under [mysql-ibm-pxb-0-58015-01-18-23h15m35s] namespace [2024-01-18T23:19:51.347Z] 2024/01/18 23:19:50 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:19:56.595Z] 2024-01-18 23:19:55 +0000:[INFO] [{ScheduleBackupCreationAllNS}] All CRs in [mysql-ibm-pxb-0-58015-01-18-23h15m35s] are [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] [2024-01-18T23:19:56.595Z] 2024-01-18 23:19:55 +0000:[INFO] [{ScheduleBackupCreationAllNS}] CR found for [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] under [mysql-ibm-pxb-0-58015-01-18-23h15m35s] namespace [2024-01-18T23:19:56.595Z] 2024/01/18 23:19:55 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:20:00.759Z] 2024-01-18 23:20:00 +0000:[INFO] [{ScheduleBackupCreationAllNS}] All CRs in [mysql-ibm-pxb-0-58015-01-18-23h15m35s] are [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] [2024-01-18T23:20:00.759Z] 2024-01-18 23:20:00 +0000:[INFO] [{ScheduleBackupCreationAllNS}] CR found for [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] under [mysql-ibm-pxb-0-58015-01-18-23h15m35s] namespace [2024-01-18T23:20:00.759Z] 2024/01/18 23:20:00 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:20:06.015Z] 2024-01-18 23:20:05 +0000:[INFO] [{ScheduleBackupCreationAllNS}] All CRs in [mysql-ibm-pxb-0-58015-01-18-23h15m35s] are [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] [2024-01-18T23:20:06.015Z] 2024-01-18 23:20:05 +0000:[INFO] [{ScheduleBackupCreationAllNS}] CR found for [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] under [mysql-ibm-pxb-0-58015-01-18-23h15m35s] namespace [2024-01-18T23:20:06.015Z] 2024/01/18 23:20:05 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:20:11.259Z] 2024-01-18 23:20:10 +0000:[INFO] [{ScheduleBackupCreationAllNS}] All CRs in [mysql-ibm-pxb-0-58015-01-18-23h15m35s] are [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] [2024-01-18T23:20:11.259Z] 2024-01-18 23:20:10 +0000:[INFO] [{ScheduleBackupCreationAllNS}] CR found for [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] under [mysql-ibm-pxb-0-58015-01-18-23h15m35s] namespace [2024-01-18T23:20:11.259Z] 2024/01/18 23:20:10 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:20:16.510Z] 2024-01-18 23:20:15 +0000:[INFO] [{ScheduleBackupCreationAllNS}] All CRs in [mysql-ibm-pxb-0-58015-01-18-23h15m35s] are [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] [2024-01-18T23:20:16.510Z] 2024-01-18 23:20:15 +0000:[INFO] [{ScheduleBackupCreationAllNS}] CR found for [[tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]] under [mysql-ibm-pxb-0-58015-01-18-23h15m35s] namespace [2024-01-18T23:20:16.510Z] 2024/01/18 23:20:15 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:20:21.770Z] 2024-01-18 23:20:20 +0000:[INFO] [{ScheduleBackupCreationAllNS}] All CRs in [mysql-ibm-pxb-0-58015-01-18-23h15m35s] are [[]] [2024-01-18T23:20:21.770Z] INFO[2024-01-18 23:20:20] Verifying : Description : Creation and Validation of schedule backup with schedule name [tp-backup-schedule-1705592040] [2024-01-18T23:20:21.770Z] INFO[2024-01-18 23:20:20] Actual:true, Expected: true [2024-01-18T23:20:21.770Z] INFO[2024-01-18 23:20:21] Verifying : Description : Suspending Backup Schedule [tp-backup-schedule-1705592040] [2024-01-18T23:20:21.770Z] INFO[2024-01-18 23:20:21] Actual:true, Expected: true [2024-01-18T23:20:21.770Z] 2024-01-18 23:20:21 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Restoring scheduled backups [2024-01-18T23:20:22.026Z] 2024/01/18 23:20:22 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T23:20:55.163Z] 2024/01/18 23:20:52 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:21:27.181Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.CreateRestore:#936] - Restore [tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724] created successfully [2024-01-18T23:21:27.181Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Switching cluster context to cluster [destination-cluster] [2024-01-18T23:21:27.181Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Switching context: The context is already [/tmp/destination-config] [2024-01-18T23:21:27.181Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-58015-01-18-23h15m35s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-18T23:21:27.181Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-58015-01-18-23h15m35s] [2024-01-18T23:21:27.181Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T23:21:27.181Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T23:21:27.181Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Validating restore [tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724] in org [default] [2024-01-18T23:21:27.181Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724] [2024-01-18T23:21:27.181Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724] [2024-01-18T23:21:27.181Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-58015-01-18-23h15m35s] in restore [tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724] [2024-01-18T23:21:27.181Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-58015-01-18-23h15m35s]) was found in restore [tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-58015-01-18-23h15m35s] [2024-01-18T23:21:27.181Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-58015-01-18-23h15m35s]) found in the restore [tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724] was also present on the cluster/namespace [mysql-ibm-pxb-0-58015-01-18-23h15m35s] [2024-01-18T23:21:27.181Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-58015-01-18-23h15m35s]) was found in restore [tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-58015-01-18-23h15m35s] [2024-01-18T23:21:27.182Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-58015-01-18-23h15m35s]) found in the restore [tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724] was also present on the cluster/namespace [mysql-ibm-pxb-0-58015-01-18-23h15m35s] [2024-01-18T23:21:27.182Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-58015-01-18-23h15m35s]) was found in restore [tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-58015-01-18-23h15m35s] [2024-01-18T23:21:27.182Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-58015-01-18-23h15m35s]) found in the restore [tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724] was also present on the cluster/namespace [mysql-ibm-pxb-0-58015-01-18-23h15m35s] [2024-01-18T23:21:27.182Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-58015-01-18-23h15m35s]) was found in restore [tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-58015-01-18-23h15m35s] [2024-01-18T23:21:27.182Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-58015-01-18-23h15m35s]) found in the restore [tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724] was also present on the cluster/namespace [mysql-ibm-pxb-0-58015-01-18-23h15m35s] [2024-01-18T23:21:27.182Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-58015-01-18-23h15m35s] in restore [tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724] [2024-01-18T23:21:27.182Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-58015-01-18-23h15m35s] in restore [tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724] [2024-01-18T23:21:27.182Z] 2024-01-18 23:21:22 +0000:[WARNING] [{ScheduleBackupCreationAllNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T23:21:27.182Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T23:21:27.182Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-58015-01-18-23h15m35s] are [[mysql-data]] [2024-01-18T23:21:27.182Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.ValidateRestore:#2240] - in restore [tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724], the status of the restored volume [pvc-c1cfe9bb-8463-4533-9cb7-8178e6c82fde] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-18T23:21:27.182Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Validate applications in restored namespace [mysql-ibm-pxb-0-58015-01-18-23h15m35s] due to restore [tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724] [2024-01-18T23:21:27.182Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Validating mysql-ibm app [2024-01-18T23:21:27.182Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] wait for mysql-ibm app to start running [2024-01-18T23:21:27.182Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T23:21:27.182Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T23:21:27.183Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T23:21:27.183Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Switching cluster context back to cluster path [/tmp/destination-config] [2024-01-18T23:21:27.183Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Switching context: The context is already [/tmp/destination-config] [2024-01-18T23:21:27.183Z] INFO[2024-01-18 23:21:22] Verifying : Description : Verifying creation of restore tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724 of backup tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724 [2024-01-18T23:21:27.183Z] ERRO[2024-01-18 23:21:22] ValidateRestore Errors: {in restore [tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724], said restored volume [pvc-c1cfe9bb-8463-4533-9cb7-8178e6c82fde] cannot be found in the actual cluster [mysql-ibm-pxb-0-58015-01-18-23h15m35s]} [2024-01-18T23:21:27.183Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Clean up objects after test execution [2024-01-18T23:21:27.183Z] 2024-01-18 23:21:22 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.glob..func58.3:#758] - Deleting backup schedules [2024-01-18T23:21:27.183Z] 2024/01/18 23:21:23 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705592040] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T23:21:33.717Z] 2024/01/18 23:21:33 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705592040] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T23:21:43.652Z] 2024/01/18 23:21:43 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705592040] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T23:21:53.594Z] 2024/01/18 23:21:53 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705592040] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T23:22:03.531Z] 2024/01/18 23:22:03 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705592040] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T23:22:13.469Z] 2024/01/18 23:22:13 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705592040] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T23:22:23.407Z] 2024/01/18 23:22:23 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705592040] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T23:22:33.379Z] 2024/01/18 23:22:33 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705592040] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T23:22:43.355Z] 2024/01/18 23:22:43 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705592040] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-18T23:22:53.290Z] INFO[2024-01-18 23:22:53] Verifying : Description : Verification of deleting backup schedule - tp-backup-schedule-1705592040 [2024-01-18T23:22:53.290Z] INFO[2024-01-18 23:22:53] Actual:true, Expected: true [2024-01-18T23:22:53.290Z] 2024-01-18 23:22:53 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.glob..func58.3:#763] - Deleting backup schedule policy [2024-01-18T23:22:53.851Z] INFO[2024-01-18 23:22:53] Verifying : Description : Deleting backup schedule policies [periodic-1705592040] [2024-01-18T23:22:53.851Z] INFO[2024-01-18 23:22:53] Actual:true, Expected: true [2024-01-18T23:22:53.851Z] 2024-01-18 23:22:53 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.glob..func58.3:#767] - Deleting the deployed applications after test execution [2024-01-18T23:22:53.851Z] 2024-01-18 23:22:53 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Destroying apps [2024-01-18T23:22:53.851Z] 2024-01-18 23:22:53 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T23:22:53.851Z] 2024-01-18 23:22:53 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T23:22:53.851Z] 2024-01-18 23:22:53 +0000:[INFO] [{ScheduleBackupCreationAllNS}] start destroying mysql-ibm app [2024-01-18T23:22:53.851Z] 2024-01-18 23:22:53 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T23:22:53.851Z] 2024-01-18 23:22:53 +0000:[INFO] [{ScheduleBackupCreationAllNS}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T23:22:53.852Z] 2024-01-18 23:22:53 +0000:[WARNING] [{ScheduleBackupCreationAllNS}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T23:22:53.852Z] INFO[2024-01-18 23:22:53] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T23:22:53.852Z] INFO[2024-01-18 23:22:53] Actual:true, Expected: true [2024-01-18T23:22:53.852Z] 2024-01-18 23:22:53 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Cleaning backup locations in map [map[d5907d09-9fe9-459a-a0ad-482db93f361c:auto-bl-1705619837]], cloud credential [aws-1705592040], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T23:22:54.108Z] INFO[2024-01-18 23:22:53] Verifying : Description : Verifying deletion of backup location [auto-bl-1705619837] [2024-01-18T23:22:54.108Z] INFO[2024-01-18 23:22:53] Actual:true, Expected: true [2024-01-18T23:22:54.108Z] 2024-01-18 23:22:54 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.IsBackupLocationPresent:#2442] - Backup location [auto-bl-1705619837] is present [2024-01-18T23:22:54.108Z] 2024/01/18 23:22:54 DoRetryWithTimeout - Error: {backup location auto-bl-1705619837 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T23:23:26.235Z] 2024-01-18 23:23:24 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.IsBackupLocationPresent:#2442] - Backup location [auto-bl-1705619837] is present [2024-01-18T23:23:26.235Z] 2024/01/18 23:23:24 DoRetryWithTimeout - Error: {backup location auto-bl-1705619837 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T23:23:58.302Z] 2024-01-18 23:23:54 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 aws-global-aws-px-backup-system-test-4cac76dc-bl] [2024-01-18T23:23:58.302Z] INFO[2024-01-18 23:23:54] Verifying : Description : Verifying backup location deletion status auto-bl-1705619837 [2024-01-18T23:23:58.302Z] INFO[2024-01-18 23:23:54] Actual:true, Expected: true [2024-01-18T23:23:58.302Z] 2024-01-18 23:23:54 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [aws-1705592040] is present [2024-01-18T23:23:58.302Z] INFO[2024-01-18 23:23:54] Verifying : Description : Verifying if cloud cred [aws-1705592040] is present [2024-01-18T23:23:58.302Z] INFO[2024-01-18 23:23:54] Actual:true, Expected: true [2024-01-18T23:23:58.302Z] INFO[2024-01-18 23:23:54] Verifying : Description : Verifying deletion of cloud cred [aws-1705592040] [2024-01-18T23:23:58.302Z] INFO[2024-01-18 23:23:54] Actual:true, Expected: true [2024-01-18T23:23:58.303Z] INFO[2024-01-18 23:23:54] Verifying : Description : Deleting cloud cred aws-1705592040 [2024-01-18T23:23:58.303Z] INFO[2024-01-18 23:23:54] Actual:true, Expected: true [2024-01-18T23:23:58.303Z] INFO[2024-01-18 23:23:54] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T23:23:58.303Z] INFO[2024-01-18 23:23:54] Actual:true, Expected: true [2024-01-18T23:23:58.303Z] INFO[2024-01-18 23:23:54] Verifying : Description : Deleting cluster destination-cluster [2024-01-18T23:23:58.303Z] ERRO[2024-01-18 23:23:54] Actual:rpc error: code = FailedPrecondition desc = cluster [destination-cluster] in use by backupschedule [tp-backup-schedule-1705605709], cannot be deleted, Expected:  [2024-01-18T23:23:58.303Z] INFO[2024-01-18 23:23:54] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-zbbvt] [2024-01-18T23:23:58.303Z] ERRO[2024-01-18 23:23:54] Actual:rpc error: code = Unknown desc = cluster [destination-cluster] has reference to cloud credential [ibm-destination-config-cloud-cred-zbbvt:a271ea5b-dc9d-40f1-a82a-651539c7e486], Expected:  [2024-01-18T23:23:58.303Z] INFO[2024-01-18 23:23:55] Verifying : Description : Deleting cluster source-cluster [2024-01-18T23:23:58.303Z] INFO[2024-01-18 23:23:55] Actual:true, Expected: true [2024-01-18T23:23:58.303Z] INFO[2024-01-18 23:23:55] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-uclrx] [2024-01-18T23:23:58.303Z] INFO[2024-01-18 23:23:55] Actual:true, Expected: true [2024-01-18T23:23:58.303Z] INFO[2024-01-18 23:23:55] --------Test End------ [2024-01-18T23:23:58.303Z] INFO[2024-01-18 23:23:55] #Test: [2024-01-18T23:23:58.303Z] INFO[2024-01-18 23:23:55] #Description: [2024-01-18T23:23:58.303Z] INFO[2024-01-18 23:23:55] ------------------------ [2024-01-18T23:23:58.303Z] 2024-01-18 23:23:55 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.EndPxBackupTorpedoTest:#7261] - >>>> FAILED TEST: {ScheduleBackupCreationAllNS} Schedule Backup Creation with all namespaces [2024-01-18T23:23:58.303Z] 2024-01-18 23:23:55 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-58015-01-18-23h15m35s] [2024-01-18T23:23:58.303Z] 2024-01-18 23:23:55 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T23:23:58.303Z] 2024-01-18 23:23:55 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Switching context: The context is already [/tmp/destination-config] [2024-01-18T23:23:58.303Z] 2024-01-18 23:23:56 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-58015-01-18-23h15m35s] [2024-01-18T23:23:58.303Z] 2024-01-18 23:23:56 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {ScheduleBackupCreationAllNS} Schedule Backup Creation with all namespaces [2024-01-18T23:23:58.303Z] 2024-01-18 23:23:56 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-18T23:23:58.303Z] 2024/01/18 23:23:56 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:24:01.597Z] 2024/01/18 23:24:01 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:24:06.838Z] 2024/01/18 23:24:06 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:24:12.078Z] 2024/01/18 23:24:11 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:24:16.241Z] 2024/01/18 23:24:16 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:24:21.486Z] 2024/01/18 23:24:21 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:24:26.726Z] 2024/01/18 23:24:26 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:24:31.973Z] 2024/01/18 23:24:31 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:24:36.135Z] INFO[2024-01-18 23:24:35] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T23:24:36.135Z] INFO[2024-01-18 23:24:35] sched stats: tick interval: 28235 < 1.2s (100%) [2024-01-18T23:24:36.135Z] INFO[2024-01-18 23:24:35] sched stats: tick process durations: 28234 < 500ms (100%) [2024-01-18T23:24:36.391Z] 2024/01/18 23:24:36 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.128.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.7 10.240.128.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-18T23:24:36.391Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-18T23:24:36.391Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-18T23:24:36.391Z] {dial tcp 10.240.128.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-18T23:24:41.634Z] 2024-01-18 23:24:41 +0000:[WARNING] [{ScheduleBackupCreationAllNS}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.7 10.240.128.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-18T23:24:41.634Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-18T23:24:41.634Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-18T23:24:41.635Z] {dial tcp 10.240.128.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T23:24:41.635Z] 2024-01-18 23:24:41 +0000:[ERROR] [{ScheduleBackupCreationAllNS}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.7 10.240.128.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-18T23:24:41.635Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-18T23:24:41.635Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-18T23:24:41.635Z] {dial tcp 10.240.128.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T23:24:41.635Z] 2024-01-18 23:24:41 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T23:24:41.635Z] 2024-01-18 23:24:41 +0000:[INFO] [{ScheduleBackupCreationAllNS}] Switching context to [/tmp/source-config] [2024-01-18T23:24:41.635Z] 2024-01-18 23:24:41 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T23:24:41.635Z] 2024-01-18 23:24:41 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T23:24:41.635Z] 2024-01-18 23:24:41 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T23:24:41.635Z] 2024-01-18 23:24:41 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T23:24:41.635Z] 2024-01-18 23:24:41 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T23:24:41.635Z] 2024-01-18 23:24:41 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T23:24:42.195Z] 2024-01-18 23:24:41 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T23:24:42.453Z] 2024-01-18 23:24:42 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T23:24:43.015Z] 2024-01-18 23:24:42 +0000:[INFO] [{ScheduleBackupCreationAllNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T23:24:43.015Z] 2024/01/18 23:24:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:24:55.177Z] 2024/01/18 23:24:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:25:03.258Z] 2024/01/18 23:25:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:25:13.207Z] 2024/01/18 23:25:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:25:23.155Z] 2024/01/18 23:25:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:25:35.326Z] 2024/01/18 23:25:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:25:43.407Z] 2024/01/18 23:25:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:25:53.350Z] 2024/01/18 23:25:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:26:03.311Z] 2024/01/18 23:26:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:26:15.475Z] 2024/01/18 23:26:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:26:23.569Z] 2024/01/18 23:26:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:26:33.521Z] 2024/01/18 23:26:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:26:43.469Z] 2024/01/18 23:26:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:26:53.420Z] 2024/01/18 23:26:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:27:03.387Z] 2024/01/18 23:27:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:27:15.557Z] 2024/01/18 23:27:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:27:23.640Z] 2024/01/18 23:27:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:27:33.600Z] 2024/01/18 23:27:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:27:43.564Z] 2024/01/18 23:27:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:27:55.722Z] 2024/01/18 23:27:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:28:04.245Z] 2024/01/18 23:28:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:28:14.179Z] 2024/01/18 23:28:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:28:24.119Z] 2024/01/18 23:28:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:28:34.063Z] 2024/01/18 23:28:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:28:44.008Z] 2024/01/18 23:28:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:28:53.982Z] 2024/01/18 23:28:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:29:03.922Z] 2024/01/18 23:29:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:29:13.860Z] 2024/01/18 23:29:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:29:23.800Z] 2024/01/18 23:29:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:29:35.963Z] 2024/01/18 23:29:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:29:44.060Z] 2024-01-18 23:29:43 +0000:[ERROR] [{ScheduleBackupCreationAllNS}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.060Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.060Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.060Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.060Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.060Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.060Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.060Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.061Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.061Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.061Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.061Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.061Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.061Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.061Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.061Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.061Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.061Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.061Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.061Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.061Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.061Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.061Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.061Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.061Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.061Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.061Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.061Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.061Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:29:44.061Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-18T23:29:44.061Z] • Failure [848.623 seconds] [2024-01-18T23:29:44.061Z] {ScheduleBackupCreationAllNS} [2024-01-18T23:29:44.061Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:520 [2024-01-18T23:29:44.061Z] Schedule Backup Creation with all namespaces [It] [2024-01-18T23:29:44.062Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:569 [2024-01-18T23:29:44.062Z] [2024-01-18T23:29:44.062Z] Unexpected error: [2024-01-18T23:29:44.062Z] <*errors.errorString | 0xc0010fb300>: { [2024-01-18T23:29:44.062Z] s: "error: ValidateRestore Errors: {in restore [tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724], said restored volume [pvc-c1cfe9bb-8463-4533-9cb7-8178e6c82fde] cannot be found in the actual cluster [mysql-ibm-pxb-0-58015-01-18-23h15m35s]}, Description: Verifying creation of restore tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724 of backup tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724", [2024-01-18T23:29:44.062Z] } [2024-01-18T23:29:44.062Z] error: ValidateRestore Errors: {in restore [tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724], said restored volume [pvc-c1cfe9bb-8463-4533-9cb7-8178e6c82fde] cannot be found in the actual cluster [mysql-ibm-pxb-0-58015-01-18-23h15m35s]}, Description: Verifying creation of restore tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724 of backup tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724 [2024-01-18T23:29:44.062Z] occurred [2024-01-18T23:29:44.062Z] [2024-01-18T23:29:44.062Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-18T23:29:44.062Z] [2024-01-18T23:29:44.062Z] Full Stack Trace [2024-01-18T23:29:44.062Z] github.com/portworx/torpedo/pkg/aetosutil.(*Dashboard).VerifyFatal(0x7f8804b?, {0x702af20?, 0xc0010fb130}, {0x0?, 0x0}, {0xc001406240, 0xb4}) [2024-01-18T23:29:44.062Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 +0x143 [2024-01-18T23:29:44.062Z] github.com/portworx/torpedo/tests/backup.glob..func58.2.7() [2024-01-18T23:29:44.062Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:640 +0x41f [2024-01-18T23:29:44.062Z] github.com/portworx/torpedo/tests/backup.glob..func58.2() [2024-01-18T23:29:44.062Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:633 +0x8c9 [2024-01-18T23:29:44.062Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-18T23:29:44.062Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-18T23:29:44.062Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-18T23:29:44.062Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-18T23:29:44.062Z] created by testing.(*T).Run [2024-01-18T23:29:44.062Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-18T23:29:44.062Z] ------------------------------ [2024-01-18T23:29:44.062Z] STEP: schedule applications [2024-01-18T23:29:44.062Z] STEP: Validate deployed applications [2024-01-18T23:29:44.062Z] STEP: validate applications [2024-01-18T23:29:44.062Z] STEP: validate mysql-ibm app's volumes [2024-01-18T23:29:44.062Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T23:29:44.062Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T23:29:44.062Z] STEP: get mysql-ibm app's volume: pvc-4d5300c5-6763-4e28-8adc-5b134d4a91cc inspected by the volume driver [2024-01-18T23:29:44.062Z] STEP: wait for mysql-ibm app to start running [2024-01-18T23:29:44.062Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T23:29:44.062Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T23:29:44.062Z] STEP: Validate Px pod restart count [2024-01-18T23:29:44.062Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T23:29:44.062Z] STEP: Adding Cloud Account [2024-01-18T23:29:44.062Z] STEP: Adding Backup Location [2024-01-18T23:29:44.062Z] STEP: Creating Schedule Policies [2024-01-18T23:29:44.062Z] STEP: Adding Clusters for backup [2024-01-18T23:29:44.062Z] STEP: Creating schedule backups [2024-01-18T23:29:44.062Z] STEP: Restoring scheduled backups [2024-01-18T23:29:44.062Z] STEP: validate mysql-ibm app's volumes [2024-01-18T23:29:44.062Z] STEP: wait for mysql-ibm app to start running [2024-01-18T23:29:44.062Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T23:29:44.062Z] STEP: Validate Px pod restart count [2024-01-18T23:29:44.062Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T23:29:44.062Z] STEP: destroy apps [2024-01-18T23:29:44.062Z] STEP: destroy the mysql-ibm app's volumes [2024-01-18T23:29:44.062Z] STEP: start destroying mysql-ibm app [2024-01-18T23:29:44.062Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T23:29:44.063Z] 2024-01-18 23:29:43 +0000:[INFO] [{CustomResourceRestore}] Log Dir: /testresults//CustomResourceRestore.log [2024-01-18T23:29:44.063Z] INFO[2024-01-18 23:29:43] --------Test Start------ [2024-01-18T23:29:44.063Z] INFO[2024-01-18 23:29:43] #Test: CustomResourceRestore [2024-01-18T23:29:44.063Z] INFO[2024-01-18 23:29:43] #Description: Create custom resource restore [2024-01-18T23:29:44.063Z] INFO[2024-01-18 23:29:43] ------------------------ [2024-01-18T23:29:44.063Z] 2024-01-18 23:29:43 +0000:[INFO] [{CustomResourceRestore}] Deploy applications [2024-01-18T23:29:44.063Z] 2024-01-18 23:29:43 +0000:[INFO] [{CustomResourceRestore}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T23:29:44.063Z] 2024-01-18 23:29:43 +0000:[INFO] [{CustomResourceRestore}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T23:29:44.063Z] 2024-01-18 23:29:43 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T23:29:44.063Z] 2024-01-18 23:29:43 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T23:29:44.063Z] 2024-01-18 23:29:43 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T23:29:44.063Z] 2024-01-18 23:29:43 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T23:29:44.064Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T23:29:44.064Z] 2024-01-18 23:29:43 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T23:29:44.064Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T23:29:44.064Z] 2024-01-18 23:29:43 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T23:29:44.064Z] 2024-01-18 23:29:43 +0000:[INFO] [{CustomResourceRestore}] Validate applications [2024-01-18T23:29:44.064Z] 2024-01-18 23:29:43 +0000:[INFO] [{CustomResourceRestore}] Validate applications [2024-01-18T23:29:44.064Z] 2024-01-18 23:29:43 +0000:[INFO] [{CustomResourceRestore}] Validating mysql-ibm app [2024-01-18T23:29:44.064Z] 2024-01-18 23:29:43 +0000:[INFO] [{CustomResourceRestore}] Validating mysql-ibm app's volumes [2024-01-18T23:29:44.064Z] 2024-01-18 23:29:44 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T23:29:44.320Z] 2024/01/18 23:29:44 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T23:29:54.259Z] 2024/01/18 23:29:54 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T23:30:04.227Z] 2024-01-18 23:30:04 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-58041-01-18-23h29m43s [2024-01-18T23:30:04.227Z] 2024-01-18 23:30:04 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T23:30:04.227Z] 2024-01-18 23:30:04 +0000:[WARNING] [{CustomResourceRestore}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T23:30:04.227Z] 2024-01-18 23:30:04 +0000:[INFO] [{CustomResourceRestore}] wait for mysql-ibm app to start running [2024-01-18T23:30:04.227Z] 2024-01-18 23:30:04 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T23:30:04.483Z] 2024/01/18 23:30:04 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:30:04.483Z] pod name:mysql-7d766cd65c-mgsmf namespace:mysql-ibm-pxb-0-58041-01-18-23h29m43s running:false ready:false node:10.240.64.8 [2024-01-18T23:30:04.483Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:30:14.423Z] 2024/01/18 23:30:14 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:30:14.423Z] pod name:mysql-7d766cd65c-mgsmf namespace:mysql-ibm-pxb-0-58041-01-18-23h29m43s running:true ready:false node:10.240.64.8 [2024-01-18T23:30:14.423Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:30:26.590Z] 2024/01/18 23:30:24 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:30:26.590Z] pod name:mysql-7d766cd65c-mgsmf namespace:mysql-ibm-pxb-0-58041-01-18-23h29m43s running:true ready:false node:10.240.64.8 [2024-01-18T23:30:26.590Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:30:34.681Z] 2024/01/18 23:30:34 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:30:34.681Z] pod name:mysql-7d766cd65c-mgsmf namespace:mysql-ibm-pxb-0-58041-01-18-23h29m43s running:true ready:false node:10.240.64.8 [2024-01-18T23:30:34.681Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:30:46.845Z] 2024/01/18 23:30:44 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:30:46.845Z] pod name:mysql-7d766cd65c-mgsmf namespace:mysql-ibm-pxb-0-58041-01-18-23h29m43s running:true ready:false node:10.240.64.8 [2024-01-18T23:30:46.845Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:30:54.931Z] 2024/01/18 23:30:54 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:30:54.931Z] pod name:mysql-7d766cd65c-mgsmf namespace:mysql-ibm-pxb-0-58041-01-18-23h29m43s running:true ready:false node:10.240.64.8 [2024-01-18T23:30:54.931Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:31:04.878Z] 2024/01/18 23:31:04 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:31:04.878Z] pod name:mysql-7d766cd65c-mgsmf namespace:mysql-ibm-pxb-0-58041-01-18-23h29m43s running:true ready:false node:10.240.64.8 [2024-01-18T23:31:04.878Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:31:17.052Z] 2024/01/18 23:31:14 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:31:17.053Z] pod name:mysql-7d766cd65c-mgsmf namespace:mysql-ibm-pxb-0-58041-01-18-23h29m43s running:true ready:false node:10.240.64.8 [2024-01-18T23:31:17.053Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:31:25.212Z] 2024-01-18 23:31:24 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T23:31:25.212Z] 2024-01-18 23:31:25 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T23:31:25.468Z] 2024-01-18 23:31:25 +0000:[INFO] [{CustomResourceRestore}] validate if mysql-ibm app's volumes are setup [2024-01-18T23:31:25.468Z] 2024-01-18 23:31:25 +0000:[WARNING] [{CustomResourceRestore}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T23:31:25.468Z] 2024-01-18 23:31:25 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T23:31:25.468Z] 2024-01-18 23:31:25 +0000:[INFO] [{CustomResourceRestore}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T23:31:25.468Z] 2024-01-18 23:31:25 +0000:[WARNING] [{CustomResourceRestore}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T23:31:25.468Z] 2024-01-18 23:31:25 +0000:[INFO] [{CustomResourceRestore}] Creating cloud credentials and backup location [2024-01-18T23:31:26.028Z] 2024-01-18 23:31:25 +0000:[INFO] [{CustomResourceRestore}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-1705620685] for org [default] with [aws] as provider [2024-01-18T23:31:26.028Z] 2024-01-18 23:31:25 +0000:[INFO] [{CustomResourceRestore}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-18T23:31:26.028Z] 2024-01-18 23:31:25 +0000:[INFO] [{CustomResourceRestore}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-1705620685] is present [2024-01-18T23:31:26.028Z] INFO[2024-01-18 23:31:25] Verifying : Description : Verifying creation of cloud credential named [autogenerated-cred-1705620685] for org [default] with [aws] as provider [2024-01-18T23:31:26.028Z] INFO[2024-01-18 23:31:25] Actual:true, Expected: true [2024-01-18T23:31:26.028Z] 2024-01-18 23:31:25 +0000:[INFO] [{CustomResourceRestore}] Created Cloud Credentials with name - autogenerated-cred-1705620685 [2024-01-18T23:31:30.199Z] 2024-01-18 23:31:30 +0000:[INFO] [{CustomResourceRestore}] Created Backup Location with name - autogenerated-backup-location-1705620685 [2024-01-18T23:31:30.199Z] 2024-01-18 23:31:30 +0000:[INFO] [{CustomResourceRestore}] Register source and destination cluster for backup [2024-01-18T23:31:30.454Z] INFO[2024-01-18 23:31:30] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T23:31:30.454Z] INFO[2024-01-18 23:31:30] Actual:true, Expected: true [2024-01-18T23:31:30.454Z] 2024-01-18 23:31:30 +0000:[INFO] [{CustomResourceRestore}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T23:31:30.454Z] 2024-01-18 23:31:30 +0000:[INFO] [{CustomResourceRestore}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T23:31:30.454Z] 2024-01-18 23:31:30 +0000:[INFO] [{CustomResourceRestore}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T23:31:30.454Z] 2024-01-18 23:31:30 +0000:[INFO] [{CustomResourceRestore}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T23:31:30.454Z] 2024-01-18 23:31:30 +0000:[INFO] [{CustomResourceRestore}] Create cluster [source-cluster] in org [default] [2024-01-18T23:31:30.454Z] 2024-01-18 23:31:30 +0000:[INFO] [{CustomResourceRestore}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T23:31:30.454Z] 2024-01-18 23:31:30 +0000:[INFO] [{CustomResourceRestore}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T23:31:30.454Z] 2024-01-18 23:31:30 +0000:[INFO] [{CustomResourceRestore}] Create cluster [destination-cluster] in org [default] [2024-01-18T23:31:30.454Z] 2024-01-18 23:31:30 +0000:[INFO] [{CustomResourceRestore}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T23:31:30.454Z] 2024-01-18 23:31:30 +0000:[INFO] [{CustomResourceRestore}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T23:31:30.454Z] 2024-01-18 23:31:30 +0000:[INFO] [{CustomResourceRestore}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-umthn] for IBM [2024-01-18T23:31:30.454Z] 2024-01-18 23:31:30 +0000:[INFO] [{CustomResourceRestore}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-umthn] for org [default] with [ibm] as provider [2024-01-18T23:31:30.455Z] 2024-01-18 23:31:30 +0000:[INFO] [{CustomResourceRestore}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T23:31:30.455Z] 2024-01-18 23:31:30 +0000:[INFO] [{CustomResourceRestore}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T23:31:30.712Z] 2024-01-18 23:31:30 +0000:[INFO] [{CustomResourceRestore}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-umthn] is present [2024-01-18T23:31:30.712Z] 2024-01-18 23:31:30 +0000:[INFO] [{CustomResourceRestore}] Create cluster [source-cluster] in org [default] [2024-01-18T23:31:31.275Z] 2024-01-18 23:31:31 +0000:[INFO] [{CustomResourceRestore}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-jfzok] for IBM [2024-01-18T23:31:31.275Z] 2024-01-18 23:31:31 +0000:[INFO] [{CustomResourceRestore}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-jfzok] for org [default] with [ibm] as provider [2024-01-18T23:31:31.275Z] 2024-01-18 23:31:31 +0000:[INFO] [{CustomResourceRestore}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T23:31:31.275Z] 2024-01-18 23:31:31 +0000:[INFO] [{CustomResourceRestore}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T23:31:31.537Z] 2024-01-18 23:31:31 +0000:[INFO] [{CustomResourceRestore}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-jfzok] is present [2024-01-18T23:31:31.538Z] 2024-01-18 23:31:31 +0000:[INFO] [{CustomResourceRestore}] Create cluster [destination-cluster] in org [default] [2024-01-18T23:31:31.538Z] INFO[2024-01-18 23:31:31] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-18T23:31:31.538Z] INFO[2024-01-18 23:31:31] Actual:Online, Expected: Online [2024-01-18T23:31:31.538Z] INFO[2024-01-18 23:31:31] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-18T23:31:31.538Z] INFO[2024-01-18 23:31:31] Actual:true, Expected: true [2024-01-18T23:31:31.538Z] 2024-01-18 23:31:31 +0000:[INFO] [{CustomResourceRestore}] Taking backup of applications [2024-01-18T23:31:31.794Z] 2024-01-18 23:31:31 +0000:[INFO] [{CustomResourceRestore}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T23:31:31.794Z] 2024-01-18 23:31:31 +0000:[INFO] [{CustomResourceRestore}] Volume snapshot classes found in the cluster - [2024-01-18T23:31:31.794Z] 2024-01-18 23:31:31 +0000:[INFO] [{CustomResourceRestore}] ibmc-vpcblock-snapshot [2024-01-18T23:31:31.794Z] 2024-01-18 23:31:31 +0000:[INFO] [{CustomResourceRestore}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T23:31:32.356Z] 2024/01/18 23:31:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58041-01-18-23h29m43s-1705620691] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T23:32:04.377Z] 2024/01/18 23:32:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58041-01-18-23h29m43s-1705620691] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:32:36.421Z] 2024/01/18 23:32:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58041-01-18-23h29m43s-1705620691] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:33:02.921Z] 2024-01-18 23:33:02 +0000:[INFO] [{CustomResourceRestore}] [backup.CreateBackup:#313] - Backup [tp-backup-mysql-ibm-pxb-0-58041-01-18-23h29m43s-1705620691] created successfully [2024-01-18T23:33:02.921Z] 2024-01-18 23:33:02 +0000:[INFO] [{CustomResourceRestore}] Validating backup [tp-backup-mysql-ibm-pxb-0-58041-01-18-23h29m43s-1705620691] in org [default] [2024-01-18T23:33:02.921Z] 2024-01-18 23:33:02 +0000:[INFO] [{CustomResourceRestore}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-mysql-ibm-pxb-0-58041-01-18-23h29m43s-1705620691] [2024-01-18T23:33:02.921Z] 2024-01-18 23:33:02 +0000:[INFO] [{CustomResourceRestore}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58041-01-18-23h29m43s] in backup [tp-backup-mysql-ibm-pxb-0-58041-01-18-23h29m43s-1705620691] [2024-01-18T23:33:02.921Z] 2024-01-18 23:33:02 +0000:[INFO] [{CustomResourceRestore}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58041-01-18-23h29m43s] in backup [tp-backup-mysql-ibm-pxb-0-58041-01-18-23h29m43s-1705620691] [2024-01-18T23:33:02.921Z] 2024-01-18 23:33:02 +0000:[INFO] [{CustomResourceRestore}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58041-01-18-23h29m43s] [2024-01-18T23:33:02.921Z] 2024-01-18 23:33:02 +0000:[WARNING] [{CustomResourceRestore}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T23:33:02.921Z] 2024-01-18 23:33:02 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T23:33:02.921Z] 2024-01-18 23:33:02 +0000:[INFO] [{CustomResourceRestore}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-58041-01-18-23h29m43s] are [[mysql-data]] [2024-01-18T23:33:02.921Z] 2024-01-18 23:33:02 +0000:[INFO] [{CustomResourceRestore}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T23:33:02.921Z] 2024-01-18 23:33:02 +0000:[INFO] [{CustomResourceRestore}] Validating CR cleanup [2024-01-18T23:33:02.921Z] 2024-01-18 23:33:02 +0000:[INFO] [{CustomResourceRestore}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T23:33:02.921Z] 2024-01-18 23:33:02 +0000:[INFO] [{CustomResourceRestore}] All CRs in [mysql-ibm-pxb-0-58041-01-18-23h29m43s] are [[]] [2024-01-18T23:33:02.921Z] INFO[2024-01-18 23:33:02] Verifying : Description : Creation and Validation of backup [tp-backup-mysql-ibm-pxb-0-58041-01-18-23h29m43s-1705620691] [2024-01-18T23:33:02.921Z] INFO[2024-01-18 23:33:02] Actual:true, Expected: true [2024-01-18T23:33:03.178Z] 2024-01-18 23:33:02 +0000:[INFO] [{CustomResourceRestore}] Restoring backed up applications [2024-01-18T23:33:03.178Z] 2024-01-18 23:33:02 +0000:[INFO] [{CustomResourceRestore}] [backup.CreateCustomRestoreWithPVCs:#2459] - Getting the UID of the backup needed to be restored [2024-01-18T23:33:03.178Z] 2024-01-18 23:33:03 +0000:[INFO] [{CustomResourceRestore}] [backup.CreateCustomRestoreWithPVCs:#2483] - selected deployment mysql [2024-01-18T23:33:03.740Z] 2024/01/18 23:33:03 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-mysql-ibm-pxb-0-58041-01-18-23h29m43s-1705620691-1705620782] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T23:33:35.792Z] 2024/01/18 23:33:33 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-mysql-ibm-pxb-0-58041-01-18-23h29m43s-1705620691-1705620782] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:34:07.822Z] 2024-01-18 23:34:03 +0000:[INFO] [{CustomResourceRestore}] Validating restored resources [2024-01-18T23:34:07.822Z] INFO[2024-01-18 23:34:03] Verifying : Description : checking the deployment status for mysql in namespace mysql-ibm-pxb-0-58041-01-18-23h29m43s-restored [2024-01-18T23:34:07.822Z] INFO[2024-01-18 23:34:03] Actual:True, Expected: True [2024-01-18T23:34:07.822Z] 2024-01-18 23:34:03 +0000:[INFO] [{CustomResourceRestore}] Deleting backup [2024-01-18T23:34:07.822Z] INFO[2024-01-18 23:34:03] Verifying : Description : trying to get backup UID for backup tp-backup-mysql-ibm-pxb-0-58041-01-18-23h29m43s-1705620691 [2024-01-18T23:34:07.822Z] INFO[2024-01-18 23:34:03] Actual:true, Expected: true [2024-01-18T23:34:07.822Z] 2024-01-18 23:34:03 +0000:[INFO] [{CustomResourceRestore}] [backup.glob..func59.3:#905] - About to delete backup - tp-backup-mysql-ibm-pxb-0-58041-01-18-23h29m43s-1705620691 [2024-01-18T23:34:07.822Z] INFO[2024-01-18 23:34:04] Verifying : Description : Verifying backup tp-backup-mysql-ibm-pxb-0-58041-01-18-23h29m43s-1705620691 deletion is successful [2024-01-18T23:34:07.822Z] INFO[2024-01-18 23:34:04] Actual:true, Expected: true [2024-01-18T23:34:07.823Z] 2024-01-18 23:34:04 +0000:[INFO] [{CustomResourceRestore}] Deleting restore [2024-01-18T23:34:07.823Z] INFO[2024-01-18 23:34:04] Verifying : Description : Getting the backup driver [2024-01-18T23:34:07.823Z] INFO[2024-01-18 23:34:04] Actual:true, Expected: true [2024-01-18T23:34:07.823Z] INFO[2024-01-18 23:34:04] Verifying : Description : Deleting user restore tp-restore-tp-backup-mysql-ibm-pxb-0-58041-01-18-23h29m43s-1705620691-1705620782 [2024-01-18T23:34:07.823Z] INFO[2024-01-18 23:34:04] Actual:true, Expected: true [2024-01-18T23:34:07.823Z] 2024-01-18 23:34:04 +0000:[INFO] [{CustomResourceRestore}] [backup.glob..func59.3:#915] - Deleting the deployed apps after the testcase [2024-01-18T23:34:07.823Z] 2024-01-18 23:34:04 +0000:[INFO] [{CustomResourceRestore}] Destroying apps [2024-01-18T23:34:07.823Z] 2024-01-18 23:34:04 +0000:[INFO] [{CustomResourceRestore}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T23:34:07.823Z] 2024-01-18 23:34:04 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T23:34:07.823Z] 2024-01-18 23:34:04 +0000:[INFO] [{CustomResourceRestore}] start destroying mysql-ibm app [2024-01-18T23:34:07.823Z] 2024-01-18 23:34:04 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T23:34:07.823Z] 2024-01-18 23:34:04 +0000:[INFO] [{CustomResourceRestore}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T23:34:07.823Z] 2024-01-18 23:34:04 +0000:[WARNING] [{CustomResourceRestore}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T23:34:07.823Z] INFO[2024-01-18 23:34:04] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T23:34:07.823Z] INFO[2024-01-18 23:34:04] Actual:true, Expected: true [2024-01-18T23:34:07.823Z] 2024-01-18 23:34:04 +0000:[INFO] [{CustomResourceRestore}] Cleaning backup locations in map [map[3c1e193b-4da0-49c5-ab4c-d660d0261d7c:autogenerated-backup-location-1705620685]], cloud credential [autogenerated-cred-1705620685], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T23:34:07.823Z] INFO[2024-01-18 23:34:04] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-1705620685] [2024-01-18T23:34:07.823Z] INFO[2024-01-18 23:34:04] Actual:true, Expected: true [2024-01-18T23:34:07.823Z] 2024-01-18 23:34:04 +0000:[INFO] [{CustomResourceRestore}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705620685] is present [2024-01-18T23:34:07.823Z] 2024/01/18 23:34:04 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705620685 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T23:34:07.823Z] Iteration: 61 [2024-01-18T23:34:08.385Z] Logs from iteration: 61 [2024-01-18T23:34:08.947Z] 2024/01/18 23:33:33 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-mysql-ibm-pxb-0-58041-01-18-23h29m43s-1705620691-1705620782] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:34:08.947Z] 2024-01-18 23:34:03 +0000:[INFO] [{CustomResourceRestore}] Validating restored resources [2024-01-18T23:34:08.947Z] INFO[2024-01-18 23:34:03] Verifying : Description : checking the deployment status for mysql in namespace mysql-ibm-pxb-0-58041-01-18-23h29m43s-restored [2024-01-18T23:34:08.947Z] INFO[2024-01-18 23:34:03] Actual:True, Expected: True [2024-01-18T23:34:08.947Z] 2024-01-18 23:34:03 +0000:[INFO] [{CustomResourceRestore}] Deleting backup [2024-01-18T23:34:08.947Z] INFO[2024-01-18 23:34:03] Verifying : Description : trying to get backup UID for backup tp-backup-mysql-ibm-pxb-0-58041-01-18-23h29m43s-1705620691 [2024-01-18T23:34:08.947Z] INFO[2024-01-18 23:34:03] Actual:true, Expected: true [2024-01-18T23:34:08.947Z] 2024-01-18 23:34:03 +0000:[INFO] [{CustomResourceRestore}] [backup.glob..func59.3:#905] - About to delete backup - tp-backup-mysql-ibm-pxb-0-58041-01-18-23h29m43s-1705620691 [2024-01-18T23:34:08.947Z] INFO[2024-01-18 23:34:04] Verifying : Description : Verifying backup tp-backup-mysql-ibm-pxb-0-58041-01-18-23h29m43s-1705620691 deletion is successful [2024-01-18T23:34:08.947Z] INFO[2024-01-18 23:34:04] Actual:true, Expected: true [2024-01-18T23:34:08.947Z] 2024-01-18 23:34:04 +0000:[INFO] [{CustomResourceRestore}] Deleting restore [2024-01-18T23:34:08.947Z] INFO[2024-01-18 23:34:04] Verifying : Description : Getting the backup driver [2024-01-18T23:34:08.947Z] INFO[2024-01-18 23:34:04] Actual:true, Expected: true [2024-01-18T23:34:08.947Z] INFO[2024-01-18 23:34:04] Verifying : Description : Deleting user restore tp-restore-tp-backup-mysql-ibm-pxb-0-58041-01-18-23h29m43s-1705620691-1705620782 [2024-01-18T23:34:08.947Z] INFO[2024-01-18 23:34:04] Actual:true, Expected: true [2024-01-18T23:34:08.947Z] 2024-01-18 23:34:04 +0000:[INFO] [{CustomResourceRestore}] [backup.glob..func59.3:#915] - Deleting the deployed apps after the testcase [2024-01-18T23:34:08.947Z] 2024-01-18 23:34:04 +0000:[INFO] [{CustomResourceRestore}] Destroying apps [2024-01-18T23:34:08.947Z] 2024-01-18 23:34:04 +0000:[INFO] [{CustomResourceRestore}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T23:34:08.947Z] 2024-01-18 23:34:04 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T23:34:08.947Z] 2024-01-18 23:34:04 +0000:[INFO] [{CustomResourceRestore}] start destroying mysql-ibm app [2024-01-18T23:34:08.947Z] 2024-01-18 23:34:04 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T23:34:08.947Z] 2024-01-18 23:34:04 +0000:[INFO] [{CustomResourceRestore}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T23:34:08.947Z] 2024-01-18 23:34:04 +0000:[WARNING] [{CustomResourceRestore}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T23:34:08.947Z] INFO[2024-01-18 23:34:04] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T23:34:08.947Z] INFO[2024-01-18 23:34:04] Actual:true, Expected: true [2024-01-18T23:34:08.947Z] 2024-01-18 23:34:04 +0000:[INFO] [{CustomResourceRestore}] Cleaning backup locations in map [map[3c1e193b-4da0-49c5-ab4c-d660d0261d7c:autogenerated-backup-location-1705620685]], cloud credential [autogenerated-cred-1705620685], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T23:34:08.947Z] INFO[2024-01-18 23:34:04] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-1705620685] [2024-01-18T23:34:08.947Z] INFO[2024-01-18 23:34:04] Actual:true, Expected: true [2024-01-18T23:34:08.948Z] 2024-01-18 23:34:04 +0000:[INFO] [{CustomResourceRestore}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705620685] is present [2024-01-18T23:34:08.948Z] 2024/01/18 23:34:04 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705620685 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T23:34:35.438Z] 2024-01-18 23:34:34 +0000:[INFO] [{CustomResourceRestore}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705620685] is present [2024-01-18T23:34:35.438Z] 2024/01/18 23:34:34 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705620685 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T23:34:35.694Z] INFO[2024-01-18 23:34:35] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T23:34:35.694Z] INFO[2024-01-18 23:34:35] sched stats: tick interval: 28835 < 1.2s (100%) [2024-01-18T23:34:35.694Z] INFO[2024-01-18 23:34:35] sched stats: tick process durations: 28834 < 500ms (100%) [2024-01-18T23:35:07.718Z] 2024-01-18 23:35:04 +0000:[INFO] [{CustomResourceRestore}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 aws-global-aws-px-backup-system-test-4cac76dc-bl] [2024-01-18T23:35:07.718Z] INFO[2024-01-18 23:35:04] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-1705620685 [2024-01-18T23:35:07.718Z] INFO[2024-01-18 23:35:04] Actual:true, Expected: true [2024-01-18T23:35:07.718Z] 2024-01-18 23:35:04 +0000:[INFO] [{CustomResourceRestore}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-1705620685] is present [2024-01-18T23:35:07.718Z] INFO[2024-01-18 23:35:04] Verifying : Description : Verifying if cloud cred [autogenerated-cred-1705620685] is present [2024-01-18T23:35:07.718Z] INFO[2024-01-18 23:35:04] Actual:true, Expected: true [2024-01-18T23:35:07.718Z] INFO[2024-01-18 23:35:04] Verifying : Description : Verifying deletion of cloud cred [autogenerated-cred-1705620685] [2024-01-18T23:35:07.718Z] INFO[2024-01-18 23:35:04] Actual:true, Expected: true [2024-01-18T23:35:07.718Z] INFO[2024-01-18 23:35:04] Verifying : Description : Deleting cloud cred autogenerated-cred-1705620685 [2024-01-18T23:35:07.718Z] INFO[2024-01-18 23:35:04] Actual:true, Expected: true [2024-01-18T23:35:07.718Z] INFO[2024-01-18 23:35:04] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T23:35:07.718Z] INFO[2024-01-18 23:35:04] Actual:true, Expected: true [2024-01-18T23:35:07.718Z] INFO[2024-01-18 23:35:04] Verifying : Description : Deleting cluster destination-cluster [2024-01-18T23:35:07.718Z] ERRO[2024-01-18 23:35:04] Actual:rpc error: code = FailedPrecondition desc = cluster [destination-cluster] in use by backup [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-232558], cannot be deleted, Expected:  [2024-01-18T23:35:07.718Z] INFO[2024-01-18 23:35:05] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-zbbvt] [2024-01-18T23:35:07.718Z] ERRO[2024-01-18 23:35:05] Actual:rpc error: code = Unknown desc = cluster [destination-cluster] has reference to cloud credential [ibm-destination-config-cloud-cred-zbbvt:a271ea5b-dc9d-40f1-a82a-651539c7e486], Expected:  [2024-01-18T23:35:07.718Z] INFO[2024-01-18 23:35:05] Verifying : Description : Deleting cluster source-cluster [2024-01-18T23:35:07.718Z] INFO[2024-01-18 23:35:05] Actual:true, Expected: true [2024-01-18T23:35:07.718Z] INFO[2024-01-18 23:35:06] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-umthn] [2024-01-18T23:35:07.718Z] INFO[2024-01-18 23:35:06] Actual:true, Expected: true [2024-01-18T23:35:07.718Z] INFO[2024-01-18 23:35:06] --------Test End------ [2024-01-18T23:35:07.718Z] INFO[2024-01-18 23:35:06] #Test: [2024-01-18T23:35:07.718Z] INFO[2024-01-18 23:35:06] #Description: [2024-01-18T23:35:07.718Z] INFO[2024-01-18 23:35:06] ------------------------ [2024-01-18T23:35:07.718Z] 2024-01-18 23:35:06 +0000:[INFO] [{CustomResourceRestore}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-58041-01-18-23h29m43s] [2024-01-18T23:35:07.718Z] 2024-01-18 23:35:06 +0000:[INFO] [{CustomResourceRestore}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-58041-01-18-23h29m43s-restored] [2024-01-18T23:35:07.718Z] 2024-01-18 23:35:06 +0000:[INFO] [{CustomResourceRestore}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T23:35:07.719Z] 2024-01-18 23:35:06 +0000:[INFO] [{CustomResourceRestore}] Switching context: The context is already [/tmp/destination-config] [2024-01-18T23:35:07.719Z] 2024-01-18 23:35:06 +0000:[INFO] [{CustomResourceRestore}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-58041-01-18-23h29m43s] [2024-01-18T23:35:07.719Z] 2024-01-18 23:35:06 +0000:[INFO] [{CustomResourceRestore}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-58041-01-18-23h29m43s-restored] [2024-01-18T23:35:07.719Z] 2024-01-18 23:35:07 +0000:[INFO] [{CustomResourceRestore}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {CustomResourceRestore} Create custom resource restore [2024-01-18T23:35:07.719Z] 2024-01-18 23:35:07 +0000:[INFO] [{CustomResourceRestore}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-18T23:35:07.719Z] 2024/01/18 23:35:07 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:35:12.970Z] 2024/01/18 23:35:12 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:35:17.137Z] 2024/01/18 23:35:17 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:35:22.397Z] 2024/01/18 23:35:22 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:35:27.666Z] 2024/01/18 23:35:27 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:35:32.914Z] 2024/01/18 23:35:32 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:35:37.080Z] 2024/01/18 23:35:37 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:35:42.321Z] 2024/01/18 23:35:42 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:35:47.567Z] 2024/01/18 23:35:47 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.0.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.8 10.240.0.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-18T23:35:47.567Z] {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-18T23:35:47.567Z] {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-18T23:35:47.567Z] {dial tcp 10.240.0.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-18T23:35:52.817Z] 2024-01-18 23:35:52 +0000:[WARNING] [{CustomResourceRestore}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.8 10.240.0.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-18T23:35:52.817Z] {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-18T23:35:52.817Z] {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-18T23:35:52.817Z] {dial tcp 10.240.0.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T23:35:52.817Z] 2024-01-18 23:35:52 +0000:[ERROR] [{CustomResourceRestore}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.8 10.240.0.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-18T23:35:52.817Z] {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-18T23:35:52.817Z] {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-18T23:35:52.817Z] {dial tcp 10.240.0.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T23:35:52.817Z] 2024-01-18 23:35:52 +0000:[INFO] [{CustomResourceRestore}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T23:35:52.817Z] 2024-01-18 23:35:52 +0000:[INFO] [{CustomResourceRestore}] Switching context to [/tmp/source-config] [2024-01-18T23:35:52.817Z] 2024-01-18 23:35:52 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T23:35:52.817Z] 2024-01-18 23:35:52 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T23:35:52.817Z] 2024-01-18 23:35:52 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T23:35:52.817Z] 2024-01-18 23:35:52 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T23:35:52.817Z] 2024-01-18 23:35:52 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T23:35:52.817Z] 2024-01-18 23:35:52 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T23:35:53.074Z] 2024-01-18 23:35:52 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T23:35:53.333Z] 2024-01-18 23:35:53 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T23:35:53.894Z] 2024-01-18 23:35:53 +0000:[INFO] [{CustomResourceRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T23:35:54.150Z] 2024/01/18 23:35:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:36:04.096Z] 2024/01/18 23:36:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:36:14.036Z] 2024/01/18 23:36:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:36:26.205Z] 2024/01/18 23:36:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:36:34.291Z] 2024/01/18 23:36:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:36:44.228Z] 2024/01/18 23:36:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:36:54.200Z] 2024/01/18 23:36:54 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:37:04.158Z] 2024/01/18 23:37:04 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:37:16.342Z] 2024/01/18 23:37:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:37:24.501Z] 2024/01/18 23:37:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:37:34.438Z] 2024/01/18 23:37:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:37:44.383Z] 2024/01/18 23:37:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:37:54.319Z] 2024/01/18 23:37:54 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:38:04.294Z] 2024/01/18 23:38:04 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:38:16.508Z] 2024/01/18 23:38:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:38:24.593Z] 2024/01/18 23:38:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:38:34.536Z] 2024/01/18 23:38:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:38:44.487Z] 2024/01/18 23:38:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:38:54.471Z] 2024/01/18 23:38:54 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:39:06.639Z] 2024/01/18 23:39:04 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:39:14.747Z] 2024/01/18 23:39:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:39:25.976Z] 2024/01/18 23:39:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:39:35.921Z] 2024/01/18 23:39:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:39:45.860Z] 2024/01/18 23:39:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:39:55.810Z] 2024/01/18 23:39:54 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:40:05.753Z] 2024/01/18 23:40:04 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:40:15.689Z] 2024/01/18 23:40:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:40:25.631Z] 2024/01/18 23:40:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:40:35.576Z] 2024/01/18 23:40:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:40:45.518Z] 2024/01/18 23:40:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:40:55.465Z] 2024-01-18 23:40:54 +0000:[ERROR] [{CustomResourceRestore}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.465Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.465Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.465Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.465Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.466Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.466Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.466Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.466Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.466Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.466Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.466Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.466Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.466Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.466Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.466Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.466Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.466Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.466Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.466Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.466Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.466Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.466Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.466Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.467Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.467Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.467Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.467Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.467Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.467Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-18T23:40:55.467Z] • Failure [670.931 seconds] [2024-01-18T23:40:55.467Z] {CustomResourceRestore} [2024-01-18T23:40:55.467Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:775 [2024-01-18T23:40:55.467Z] Create custom resource restore [] [2024-01-18T23:40:55.467Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:815 [2024-01-18T23:40:55.467Z] [2024-01-18T23:40:55.467Z] Unexpected error: [2024-01-18T23:40:55.467Z] <*errors.errorString | 0xc0012e90d0>: { [2024-01-18T23:40:55.468Z] s: "failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{No... [2024-01-18T23:40:55.468Z] [2024-01-18T23:40:55.468Z] Gomega truncated this representation as it exceeds 'format.MaxLength'. [2024-01-18T23:40:55.468Z] Consider having the object provide a custom 'GomegaStringer' representation [2024-01-18T23:40:55.468Z] or adjust the parameters in Gomega's 'format' package. [2024-01-18T23:40:55.468Z] [2024-01-18T23:40:55.468Z] Learn more here: https://onsi.github.io/gomega/#adjusting-output [2024-01-18T23:40:55.468Z] [2024-01-18T23:40:55.468Z] failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.468Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.468Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.468Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.469Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.469Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.469Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.469Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.469Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.469Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.469Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.469Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.469Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.469Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.469Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.469Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.469Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.469Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.469Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.469Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.469Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.469Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.469Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.470Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.470Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.470Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.470Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.470Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.470Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:40:55.470Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-18T23:40:55.470Z] occurred [2024-01-18T23:40:55.470Z] [2024-01-18T23:40:55.470Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-18T23:40:55.470Z] [2024-01-18T23:40:55.470Z] Full Stack Trace [2024-01-18T23:40:55.470Z] github.com/portworx/torpedo/pkg/log.FailOnError({0x88a5060, 0xc0012e90d0}, {0x7f74bfd?, 0x6cc0320?}, {0x0?, 0x6cc0320?, 0xc0017784b0?}) [2024-01-18T23:40:55.470Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 +0x290 [2024-01-18T23:40:55.470Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest.func1() [2024-01-18T23:40:55.470Z] /go/src/github.com/portworx/torpedo/tests/common.go:7283 +0x32 [2024-01-18T23:40:55.470Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest({0xc000582ee0, 0x1, 0x1}) [2024-01-18T23:40:55.470Z] /go/src/github.com/portworx/torpedo/tests/common.go:7299 +0x778 [2024-01-18T23:40:55.470Z] github.com/portworx/torpedo/tests/backup.glob..func59.3() [2024-01-18T23:40:55.470Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:921 +0x259 [2024-01-18T23:40:55.470Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-18T23:40:55.470Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-18T23:40:55.470Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-18T23:40:55.470Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-18T23:40:55.470Z] created by testing.(*T).Run [2024-01-18T23:40:55.470Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-18T23:40:55.470Z] ------------------------------ [2024-01-18T23:40:55.470Z] STEP: schedule applications [2024-01-18T23:40:55.470Z] STEP: Validate applications [2024-01-18T23:40:55.470Z] STEP: validate applications [2024-01-18T23:40:55.470Z] STEP: validate mysql-ibm app's volumes [2024-01-18T23:40:55.470Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T23:40:55.470Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T23:40:55.470Z] STEP: get mysql-ibm app's volume: pvc-1bb49e4e-aa79-48c6-986f-4400a5a577a2 inspected by the volume driver [2024-01-18T23:40:55.470Z] STEP: wait for mysql-ibm app to start running [2024-01-18T23:40:55.470Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T23:40:55.470Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T23:40:55.470Z] STEP: Validate Px pod restart count [2024-01-18T23:40:55.470Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T23:40:55.470Z] STEP: Creating credentials and backup location [2024-01-18T23:40:55.470Z] STEP: Register source and destination cluster for backup [2024-01-18T23:40:55.470Z] STEP: Taking backup of applications [2024-01-18T23:40:55.470Z] STEP: Restoring the backed up application [2024-01-18T23:40:55.470Z] STEP: Validating restored resources [2024-01-18T23:40:55.470Z] STEP: destroy apps [2024-01-18T23:40:55.471Z] STEP: destroy the mysql-ibm app's volumes [2024-01-18T23:40:55.471Z] STEP: start destroying mysql-ibm app [2024-01-18T23:40:55.471Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T23:40:55.471Z] 2024-01-18 23:40:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Log Dir: /testresults//DeleteSameNameObjectsByMultipleUsersFromAdmin.log [2024-01-18T23:40:55.471Z] INFO[2024-01-18 23:40:54] --------Test Start------ [2024-01-18T23:40:55.471Z] INFO[2024-01-18 23:40:54] #Test: DeleteSameNameObjectsByMultipleUsersFromAdmin [2024-01-18T23:40:55.471Z] INFO[2024-01-18 23:40:54] #Description: Delete backups, backup schedules, restore and cluster objects created by multiple user with same name from the admin [2024-01-18T23:40:55.471Z] INFO[2024-01-18 23:40:54] ------------------------ [2024-01-18T23:40:55.471Z] 2024-01-18 23:40:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T23:40:55.471Z] 2024-01-18 23:40:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T23:40:55.471Z] 2024-01-18 23:40:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T23:40:55.471Z] 2024-01-18 23:40:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T23:40:55.471Z] 2024-01-18 23:40:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T23:40:55.471Z] 2024-01-18 23:40:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T23:40:55.472Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T23:40:55.472Z] 2024-01-18 23:40:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T23:40:55.472Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T23:40:55.472Z] 2024-01-18 23:40:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T23:40:55.472Z] 2024-01-18 23:40:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Scheduled application namespaces: [mysql-ibm-pxb-0-87563-01-18-23h40m54s] [2024-01-18T23:40:55.472Z] 2024-01-18 23:40:55 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating applications [2024-01-18T23:40:55.472Z] 2024-01-18 23:40:55 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validate applications [2024-01-18T23:40:55.472Z] 2024-01-18 23:40:55 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating mysql-ibm app [2024-01-18T23:40:55.472Z] 2024-01-18 23:40:55 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating mysql-ibm app's volumes [2024-01-18T23:40:55.472Z] 2024-01-18 23:40:55 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T23:40:55.473Z] 2024/01/18 23:40:55 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T23:41:05.412Z] 2024/01/18 23:41:05 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T23:41:17.579Z] 2024-01-18 23:41:15 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-87563-01-18-23h40m54s [2024-01-18T23:41:17.579Z] 2024-01-18 23:41:15 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T23:41:17.579Z] 2024-01-18 23:41:15 +0000:[WARNING] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T23:41:17.579Z] 2024-01-18 23:41:15 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] wait for mysql-ibm app to start running [2024-01-18T23:41:17.579Z] 2024-01-18 23:41:15 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T23:41:17.579Z] 2024/01/18 23:41:15 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:41:17.579Z] pod name:mysql-7d766cd65c-xfznl namespace:mysql-ibm-pxb-0-87563-01-18-23h40m54s running:false ready:false node:10.240.64.8 [2024-01-18T23:41:17.579Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:41:27.518Z] 2024/01/18 23:41:25 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:41:27.518Z] pod name:mysql-7d766cd65c-xfznl namespace:mysql-ibm-pxb-0-87563-01-18-23h40m54s running:true ready:false node:10.240.64.8 [2024-01-18T23:41:27.518Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:41:37.458Z] 2024/01/18 23:41:35 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:41:37.458Z] pod name:mysql-7d766cd65c-xfznl namespace:mysql-ibm-pxb-0-87563-01-18-23h40m54s running:true ready:false node:10.240.64.8 [2024-01-18T23:41:37.458Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:41:47.398Z] 2024/01/18 23:41:45 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:41:47.398Z] pod name:mysql-7d766cd65c-xfznl namespace:mysql-ibm-pxb-0-87563-01-18-23h40m54s running:true ready:false node:10.240.64.8 [2024-01-18T23:41:47.398Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:41:57.335Z] 2024/01/18 23:41:56 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:41:57.335Z] pod name:mysql-7d766cd65c-xfznl namespace:mysql-ibm-pxb-0-87563-01-18-23h40m54s running:true ready:false node:10.240.64.8 [2024-01-18T23:41:57.335Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:42:07.289Z] 2024/01/18 23:42:06 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:42:07.289Z] pod name:mysql-7d766cd65c-xfznl namespace:mysql-ibm-pxb-0-87563-01-18-23h40m54s running:true ready:false node:10.240.64.8 [2024-01-18T23:42:07.290Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:42:17.257Z] 2024/01/18 23:42:16 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:42:17.257Z] pod name:mysql-7d766cd65c-xfznl namespace:mysql-ibm-pxb-0-87563-01-18-23h40m54s running:true ready:false node:10.240.64.8 [2024-01-18T23:42:17.258Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:42:27.198Z] 2024/01/18 23:42:26 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:42:27.198Z] pod name:mysql-7d766cd65c-xfznl namespace:mysql-ibm-pxb-0-87563-01-18-23h40m54s running:true ready:false node:10.240.64.8 [2024-01-18T23:42:27.198Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:42:37.140Z] 2024-01-18 23:42:36 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-18T23:42:37.140Z] 2024-01-18 23:42:36 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-18T23:42:37.140Z] 2024-01-18 23:42:36 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] validate if mysql-ibm app's volumes are setup [2024-01-18T23:42:37.141Z] 2024-01-18 23:42:36 +0000:[WARNING] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T23:42:37.141Z] 2024-01-18 23:42:36 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T23:42:37.141Z] 2024-01-18 23:42:36 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T23:42:37.141Z] 2024-01-18 23:42:36 +0000:[WARNING] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-18T23:42:37.141Z] 2024-01-18 23:42:36 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Creating 3 users with px-backup-infra.admin role [2024-01-18T23:42:37.141Z] 2024-01-18 23:42:36 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Creating 3 users [2024-01-18T23:42:37.702Z] 2024-01-18 23:42:37 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.AddUser:#728] - Response for user [testuser3-1705621356] creation - [2024-01-18T23:42:37.702Z] 2024-01-18 23:42:37 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.AddUser:#733] - User [testuser3-1705621356] created [2024-01-18T23:42:37.702Z] INFO[2024-01-18 23:42:37] Verifying : Description : Creating user - testuser3-1705621356 [2024-01-18T23:42:37.702Z] INFO[2024-01-18 23:42:37] Actual:true, Expected: true [2024-01-18T23:42:37.702Z] 2024-01-18 23:42:37 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.AddUser:#728] - Response for user [testuser1-1705621356] creation - [2024-01-18T23:42:37.702Z] 2024-01-18 23:42:37 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.AddUser:#733] - User [testuser1-1705621356] created [2024-01-18T23:42:37.702Z] INFO[2024-01-18 23:42:37] Verifying : Description : Creating user - testuser1-1705621356 [2024-01-18T23:42:37.702Z] INFO[2024-01-18 23:42:37] Actual:true, Expected: true [2024-01-18T23:42:37.957Z] 2024-01-18 23:42:37 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.AddUser:#728] - Response for user [testuser2-1705621356] creation - [2024-01-18T23:42:37.957Z] 2024-01-18 23:42:37 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.AddUser:#733] - User [testuser2-1705621356] created [2024-01-18T23:42:37.957Z] INFO[2024-01-18 23:42:37] Verifying : Description : Creating user - testuser2-1705621356 [2024-01-18T23:42:37.957Z] INFO[2024-01-18 23:42:37] Actual:true, Expected: true [2024-01-18T23:42:38.885Z] 2024-01-18 23:42:38 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser3-1705621356 - 91842096-c915-473d-a79f-a0c2b68c1e6e [2024-01-18T23:42:41.401Z] 2024-01-18 23:42:40 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705621356 - 23ad8145-bcb6-4851-8525-1262e96ea046 [2024-01-18T23:42:43.917Z] 2024-01-18 23:42:43 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2-1705621356 - 0aa307b0-cec7-44b2-998e-7c610c260910 [2024-01-18T23:42:45.284Z] 2024-01-18 23:42:44 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Creating cloud credential and backup location from the user testuser2-1705621356 [2024-01-18T23:42:45.284Z] 2024-01-18 23:42:44 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Creating cloud credential and backup location from the user testuser3-1705621356 [2024-01-18T23:42:45.284Z] 2024-01-18 23:42:44 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Creating cloud credential and backup location from the user testuser1-1705621356 [2024-01-18T23:42:45.846Z] 2024-01-18 23:42:45 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-cegb] for org [default] with [aws] as provider [2024-01-18T23:42:45.846Z] 2024-01-18 23:42:45 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-18T23:42:46.102Z] 2024-01-18 23:42:45 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-cegb] is present [2024-01-18T23:42:46.102Z] 2024-01-18 23:42:45 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-cegb] for org [default] with [aws] as provider [2024-01-18T23:42:46.102Z] 2024-01-18 23:42:45 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-18T23:42:46.103Z] 2024-01-18 23:42:46 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-cegb] is present [2024-01-18T23:42:46.358Z] 2024-01-18 23:42:46 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-cegb] for org [default] with [aws] as provider [2024-01-18T23:42:46.358Z] 2024-01-18 23:42:46 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-18T23:42:46.358Z] 2024-01-18 23:42:46 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-cegb] is present [2024-01-18T23:42:48.244Z] 2024-01-18 23:42:47 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Creating source and destination cluster from the user testuser3-1705621356 [2024-01-18T23:42:48.244Z] INFO[2024-01-18 23:42:48] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T23:42:48.244Z] INFO[2024-01-18 23:42:48] Actual:true, Expected: true [2024-01-18T23:42:48.244Z] 2024-01-18 23:42:48 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T23:42:48.244Z] 2024-01-18 23:42:48 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T23:42:48.244Z] 2024-01-18 23:42:48 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T23:42:48.244Z] 2024-01-18 23:42:48 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T23:42:48.244Z] 2024-01-18 23:42:48 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Create cluster [source-cluster] in org [default] [2024-01-18T23:42:48.244Z] 2024-01-18 23:42:48 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T23:42:48.244Z] 2024-01-18 23:42:48 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T23:42:48.244Z] 2024-01-18 23:42:48 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-18T23:42:48.244Z] 2024-01-18 23:42:48 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T23:42:48.244Z] 2024-01-18 23:42:48 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T23:42:48.244Z] 2024-01-18 23:42:48 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-bimhn] for IBM [2024-01-18T23:42:48.244Z] 2024-01-18 23:42:48 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-bimhn] for org [default] with [ibm] as provider [2024-01-18T23:42:48.244Z] 2024-01-18 23:42:48 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T23:42:48.245Z] 2024-01-18 23:42:48 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T23:42:48.245Z] 2024-01-18 23:42:48 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-bimhn] is present [2024-01-18T23:42:48.245Z] 2024-01-18 23:42:48 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Create cluster [source-cluster] in org [default] [2024-01-18T23:42:48.808Z] 2024-01-18 23:42:48 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-ddvlx] for IBM [2024-01-18T23:42:48.808Z] 2024-01-18 23:42:48 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-ddvlx] for org [default] with [ibm] as provider [2024-01-18T23:42:48.808Z] 2024-01-18 23:42:48 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T23:42:48.809Z] 2024-01-18 23:42:48 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T23:42:49.064Z] 2024-01-18 23:42:48 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-ddvlx] is present [2024-01-18T23:42:49.064Z] 2024-01-18 23:42:48 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-18T23:42:49.625Z] INFO[2024-01-18 23:42:49] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-18T23:42:49.625Z] INFO[2024-01-18 23:42:49] Actual:Online, Expected: Online [2024-01-18T23:42:49.625Z] INFO[2024-01-18 23:42:49] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-18T23:42:49.625Z] INFO[2024-01-18 23:42:49] Actual:true, Expected: true [2024-01-18T23:42:49.625Z] INFO[2024-01-18 23:42:49] Verifying : Description : Fetching [destination-cluster] cluster uid [2024-01-18T23:42:49.625Z] INFO[2024-01-18 23:42:49] Actual:true, Expected: true [2024-01-18T23:42:49.625Z] 2024-01-18 23:42:49 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Taking backup of applications from the user testuser3-1705621356 [2024-01-18T23:42:49.880Z] 2024-01-18 23:42:49 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T23:42:49.880Z] 2024-01-18 23:42:49 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Volume snapshot classes found in the cluster - [2024-01-18T23:42:49.880Z] 2024-01-18 23:42:49 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] ibmc-vpcblock-snapshot [2024-01-18T23:42:49.880Z] 2024-01-18 23:42:49 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T23:42:50.442Z] 2024/01/18 23:42:50 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T23:42:51.004Z] 2024-01-18 23:42:50 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Creating source and destination cluster from the user testuser1-1705621356 [2024-01-18T23:42:51.004Z] INFO[2024-01-18 23:42:50] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T23:42:51.005Z] INFO[2024-01-18 23:42:50] Actual:true, Expected: true [2024-01-18T23:42:51.005Z] 2024-01-18 23:42:50 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T23:42:51.005Z] 2024-01-18 23:42:50 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T23:42:51.005Z] 2024-01-18 23:42:50 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T23:42:51.005Z] 2024-01-18 23:42:50 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T23:42:51.005Z] 2024-01-18 23:42:50 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Create cluster [source-cluster] in org [default] [2024-01-18T23:42:51.005Z] 2024-01-18 23:42:50 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T23:42:51.005Z] 2024-01-18 23:42:50 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T23:42:51.005Z] 2024-01-18 23:42:50 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-18T23:42:51.005Z] 2024-01-18 23:42:50 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T23:42:51.005Z] 2024-01-18 23:42:50 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T23:42:51.005Z] 2024-01-18 23:42:50 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-gzmjh] for IBM [2024-01-18T23:42:51.005Z] 2024-01-18 23:42:50 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-gzmjh] for org [default] with [ibm] as provider [2024-01-18T23:42:51.005Z] 2024-01-18 23:42:50 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T23:42:51.005Z] 2024-01-18 23:42:50 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T23:42:51.261Z] 2024-01-18 23:42:51 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-gzmjh] is present [2024-01-18T23:42:51.261Z] 2024-01-18 23:42:51 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Create cluster [source-cluster] in org [default] [2024-01-18T23:42:51.822Z] 2024-01-18 23:42:51 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-ffjdm] for IBM [2024-01-18T23:42:51.822Z] 2024-01-18 23:42:51 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-ffjdm] for org [default] with [ibm] as provider [2024-01-18T23:42:51.823Z] 2024-01-18 23:42:51 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T23:42:51.823Z] 2024-01-18 23:42:51 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T23:42:51.823Z] 2024-01-18 23:42:51 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-ffjdm] is present [2024-01-18T23:42:51.823Z] 2024-01-18 23:42:51 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-18T23:42:52.757Z] INFO[2024-01-18 23:42:52] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-18T23:42:52.758Z] INFO[2024-01-18 23:42:52] Actual:Online, Expected: Online [2024-01-18T23:42:52.758Z] INFO[2024-01-18 23:42:52] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-18T23:42:52.758Z] INFO[2024-01-18 23:42:52] Actual:true, Expected: true [2024-01-18T23:42:52.758Z] INFO[2024-01-18 23:42:52] Verifying : Description : Fetching [destination-cluster] cluster uid [2024-01-18T23:42:52.758Z] INFO[2024-01-18 23:42:52] Actual:true, Expected: true [2024-01-18T23:42:52.758Z] 2024-01-18 23:42:52 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Taking backup of applications from the user testuser1-1705621356 [2024-01-18T23:42:52.758Z] 2024-01-18 23:42:52 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T23:42:52.758Z] 2024-01-18 23:42:52 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Volume snapshot classes found in the cluster - [2024-01-18T23:42:52.758Z] 2024-01-18 23:42:52 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] ibmc-vpcblock-snapshot [2024-01-18T23:42:52.758Z] 2024-01-18 23:42:52 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T23:42:53.320Z] 2024/01/18 23:42:53 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T23:42:53.320Z] 2024-01-18 23:42:53 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Creating source and destination cluster from the user testuser2-1705621356 [2024-01-18T23:42:53.320Z] INFO[2024-01-18 23:42:53] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-18T23:42:53.320Z] INFO[2024-01-18 23:42:53] Actual:true, Expected: true [2024-01-18T23:42:53.320Z] 2024-01-18 23:42:53 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-18T23:42:53.320Z] 2024-01-18 23:42:53 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-18T23:42:53.320Z] 2024-01-18 23:42:53 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-18T23:42:53.320Z] 2024-01-18 23:42:53 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-18T23:42:53.320Z] 2024-01-18 23:42:53 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Create cluster [source-cluster] in org [default] [2024-01-18T23:42:53.320Z] 2024-01-18 23:42:53 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T23:42:53.320Z] 2024-01-18 23:42:53 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-18T23:42:53.320Z] 2024-01-18 23:42:53 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-18T23:42:53.320Z] 2024-01-18 23:42:53 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T23:42:53.320Z] 2024-01-18 23:42:53 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-18T23:42:53.320Z] 2024-01-18 23:42:53 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-nvnow] for IBM [2024-01-18T23:42:53.320Z] 2024-01-18 23:42:53 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-nvnow] for org [default] with [ibm] as provider [2024-01-18T23:42:53.320Z] 2024-01-18 23:42:53 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T23:42:53.320Z] 2024-01-18 23:42:53 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T23:42:53.578Z] 2024-01-18 23:42:53 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-nvnow] is present [2024-01-18T23:42:53.578Z] 2024-01-18 23:42:53 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Create cluster [source-cluster] in org [default] [2024-01-18T23:42:54.503Z] 2024-01-18 23:42:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-ziypr] for IBM [2024-01-18T23:42:54.503Z] 2024-01-18 23:42:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-ziypr] for org [default] with [ibm] as provider [2024-01-18T23:42:54.503Z] 2024-01-18 23:42:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-18T23:42:54.504Z] 2024-01-18 23:42:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-18T23:42:54.504Z] 2024-01-18 23:42:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-ziypr] is present [2024-01-18T23:42:54.504Z] 2024-01-18 23:42:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-18T23:42:55.444Z] INFO[2024-01-18 23:42:55] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-18T23:42:55.444Z] INFO[2024-01-18 23:42:55] Actual:Online, Expected: Online [2024-01-18T23:42:55.444Z] INFO[2024-01-18 23:42:55] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-18T23:42:55.444Z] INFO[2024-01-18 23:42:55] Actual:true, Expected: true [2024-01-18T23:42:55.444Z] INFO[2024-01-18 23:42:55] Verifying : Description : Fetching [destination-cluster] cluster uid [2024-01-18T23:42:55.444Z] INFO[2024-01-18 23:42:55] Actual:true, Expected: true [2024-01-18T23:42:55.444Z] 2024-01-18 23:42:55 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Taking backup of applications from the user testuser2-1705621356 [2024-01-18T23:42:55.444Z] 2024-01-18 23:42:55 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-18T23:42:55.444Z] 2024-01-18 23:42:55 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Volume snapshot classes found in the cluster - [2024-01-18T23:42:55.444Z] 2024-01-18 23:42:55 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] ibmc-vpcblock-snapshot [2024-01-18T23:42:55.444Z] 2024-01-18 23:42:55 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T23:42:56.005Z] 2024/01/18 23:42:55 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T23:43:22.571Z] 2024/01/18 23:43:20 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:43:23.133Z] 2024/01/18 23:43:23 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:43:26.409Z] 2024/01/18 23:43:25 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:43:52.954Z] 2024/01/18 23:43:50 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:43:53.210Z] 2024/01/18 23:43:53 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:43:56.476Z] 2024/01/18 23:43:55 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:44:22.964Z] 2024/01/18 23:44:20 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:44:23.219Z] 2024/01/18 23:44:23 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:44:26.493Z] 2024/01/18 23:44:25 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:44:36.435Z] INFO[2024-01-18 23:44:35] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T23:44:36.435Z] INFO[2024-01-18 23:44:35] sched stats: tick interval: 29435 < 1.2s (100%) [2024-01-18T23:44:36.435Z] INFO[2024-01-18 23:44:35] sched stats: tick process durations: 29434 < 500ms (100%) [2024-01-18T23:44:51.290Z] 2024/01/18 23:44:50 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:44:53.177Z] 2024/01/18 23:44:53 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:44:56.440Z] 2024/01/18 23:44:55 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:45:22.936Z] 2024-01-18 23:45:20 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.CreateBackup:#313] - Backup [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] created successfully [2024-01-18T23:45:22.937Z] 2024-01-18 23:45:20 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating backup [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] in org [default] [2024-01-18T23:45:22.937Z] 2024-01-18 23:45:20 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] [2024-01-18T23:45:22.937Z] 2024-01-18 23:45:20 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87563-01-18-23h40m54s] in backup [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] [2024-01-18T23:45:22.937Z] 2024-01-18 23:45:20 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87563-01-18-23h40m54s] in backup [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] [2024-01-18T23:45:22.937Z] 2024-01-18 23:45:20 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87563-01-18-23h40m54s] [2024-01-18T23:45:22.937Z] 2024-01-18 23:45:20 +0000:[WARNING] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T23:45:22.937Z] 2024-01-18 23:45:20 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T23:45:22.937Z] 2024-01-18 23:45:20 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[mysql-data]] [2024-01-18T23:45:22.937Z] 2024-01-18 23:45:20 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T23:45:22.937Z] 2024-01-18 23:45:20 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating CR cleanup [2024-01-18T23:45:22.937Z] 2024-01-18 23:45:20 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T23:45:22.937Z] 2024-01-18 23:45:20 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[]] [2024-01-18T23:45:22.937Z] INFO[2024-01-18 23:45:20] Verifying : Description : Verifying creation and validation of backup [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] of namespace (scheduled Context) [mysql-ibm-pxb-0-87563-01-18-23h40m54s] [2024-01-18T23:45:22.937Z] INFO[2024-01-18 23:45:20] Actual:true, Expected: true [2024-01-18T23:45:22.937Z] 2024-01-18 23:45:20 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.glob..func26.2.3.3:#136] - The list of user backups taken are: map[testuser1-1705621356:map[] testuser2-1705621356:map[] testuser3-1705621356:map[tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb:mysql-ibm-pxb-0-87563-01-18-23h40m54s]] [2024-01-18T23:45:22.937Z] 2024-01-18 23:45:20 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Creating schedule policy from the user testuser3-1705621356 [2024-01-18T23:45:22.937Z] INFO[2024-01-18 23:45:20] Verifying : Description : Verifying creation schedule policy periodic-cegb [2024-01-18T23:45:22.937Z] INFO[2024-01-18 23:45:20] Actual:true, Expected: true [2024-01-18T23:45:22.937Z] 2024-01-18 23:45:20 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Taking schedule backup of applications from the user testuser3-1705621356 [2024-01-18T23:45:22.937Z] 2024-01-18 23:45:20 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-18T23:45:22.937Z] 2024-01-18 23:45:20 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Volume snapshot classes found in the cluster - [2024-01-18T23:45:22.937Z] 2024-01-18 23:45:20 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] ibmc-vpcblock-snapshot [2024-01-18T23:45:22.937Z] 2024-01-18 23:45:20 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T23:45:23.196Z] 2024-01-18 23:45:23 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.CreateBackup:#313] - Backup [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] created successfully [2024-01-18T23:45:23.196Z] 2024-01-18 23:45:23 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating backup [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] in org [default] [2024-01-18T23:45:23.196Z] 2024-01-18 23:45:23 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] [2024-01-18T23:45:23.196Z] 2024-01-18 23:45:23 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87563-01-18-23h40m54s] in backup [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] [2024-01-18T23:45:23.196Z] 2024-01-18 23:45:23 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87563-01-18-23h40m54s] in backup [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] [2024-01-18T23:45:23.196Z] 2024-01-18 23:45:23 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87563-01-18-23h40m54s] [2024-01-18T23:45:23.196Z] 2024-01-18 23:45:23 +0000:[WARNING] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T23:45:23.196Z] 2024-01-18 23:45:23 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T23:45:23.197Z] 2024-01-18 23:45:23 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[mysql-data]] [2024-01-18T23:45:23.453Z] 2024-01-18 23:45:23 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T23:45:23.453Z] 2024-01-18 23:45:23 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating CR cleanup [2024-01-18T23:45:23.453Z] 2024-01-18 23:45:23 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T23:45:23.453Z] 2024-01-18 23:45:23 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:45:23.453Z] INFO[2024-01-18 23:45:23] Verifying : Description : Verifying creation and validation of backup [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] of namespace (scheduled Context) [mysql-ibm-pxb-0-87563-01-18-23h40m54s] [2024-01-18T23:45:23.453Z] INFO[2024-01-18 23:45:23] Actual:true, Expected: true [2024-01-18T23:45:23.453Z] 2024-01-18 23:45:23 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.glob..func26.2.3.3:#136] - The list of user backups taken are: map[testuser1-1705621356:map[tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb:mysql-ibm-pxb-0-87563-01-18-23h40m54s] testuser2-1705621356:map[] testuser3-1705621356:map[tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb:mysql-ibm-pxb-0-87563-01-18-23h40m54s]] [2024-01-18T23:45:23.453Z] 2024-01-18 23:45:23 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Creating schedule policy from the user testuser1-1705621356 [2024-01-18T23:45:23.709Z] INFO[2024-01-18 23:45:23] Verifying : Description : Verifying creation schedule policy periodic-cegb [2024-01-18T23:45:23.709Z] INFO[2024-01-18 23:45:23] Actual:true, Expected: true [2024-01-18T23:45:23.709Z] 2024-01-18 23:45:23 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Taking schedule backup of applications from the user testuser1-1705621356 [2024-01-18T23:45:23.709Z] 2024-01-18 23:45:23 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-18T23:45:23.965Z] 2024-01-18 23:45:23 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Volume snapshot classes found in the cluster - [2024-01-18T23:45:23.965Z] 2024-01-18 23:45:23 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] ibmc-vpcblock-snapshot [2024-01-18T23:45:23.965Z] 2024-01-18 23:45:23 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T23:45:25.858Z] 2024-01-18 23:45:25 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.CreateBackup:#313] - Backup [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] created successfully [2024-01-18T23:45:25.858Z] 2024-01-18 23:45:25 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating backup [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] in org [default] [2024-01-18T23:45:25.858Z] 2024-01-18 23:45:25 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] [2024-01-18T23:45:26.113Z] 2024-01-18 23:45:25 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87563-01-18-23h40m54s] in backup [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] [2024-01-18T23:45:26.113Z] 2024-01-18 23:45:25 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87563-01-18-23h40m54s] in backup [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] [2024-01-18T23:45:26.113Z] 2024-01-18 23:45:25 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87563-01-18-23h40m54s] [2024-01-18T23:45:26.113Z] 2024-01-18 23:45:25 +0000:[WARNING] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T23:45:26.113Z] 2024-01-18 23:45:25 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T23:45:26.113Z] 2024-01-18 23:45:25 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[mysql-data]] [2024-01-18T23:45:26.113Z] 2024-01-18 23:45:25 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T23:45:26.113Z] 2024-01-18 23:45:25 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating CR cleanup [2024-01-18T23:45:26.114Z] 2024-01-18 23:45:26 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T23:45:26.369Z] 2024-01-18 23:45:26 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:45:26.369Z] INFO[2024-01-18 23:45:26] Verifying : Description : Verifying creation and validation of backup [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] of namespace (scheduled Context) [mysql-ibm-pxb-0-87563-01-18-23h40m54s] [2024-01-18T23:45:26.369Z] INFO[2024-01-18 23:45:26] Actual:true, Expected: true [2024-01-18T23:45:26.369Z] 2024-01-18 23:45:26 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.glob..func26.2.3.3:#136] - The list of user backups taken are: map[testuser1-1705621356:map[tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb:mysql-ibm-pxb-0-87563-01-18-23h40m54s] testuser2-1705621356:map[tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb:mysql-ibm-pxb-0-87563-01-18-23h40m54s] testuser3-1705621356:map[tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb:mysql-ibm-pxb-0-87563-01-18-23h40m54s]] [2024-01-18T23:45:26.369Z] 2024-01-18 23:45:26 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Creating schedule policy from the user testuser2-1705621356 [2024-01-18T23:45:26.369Z] INFO[2024-01-18 23:45:26] Verifying : Description : Verifying creation schedule policy periodic-cegb [2024-01-18T23:45:26.369Z] INFO[2024-01-18 23:45:26] Actual:true, Expected: true [2024-01-18T23:45:26.369Z] 2024-01-18 23:45:26 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Taking schedule backup of applications from the user testuser2-1705621356 [2024-01-18T23:45:26.627Z] 2024-01-18 23:45:26 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-18T23:45:26.627Z] 2024-01-18 23:45:26 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Volume snapshot classes found in the cluster - [2024-01-18T23:45:26.627Z] 2024-01-18 23:45:26 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] ibmc-vpcblock-snapshot [2024-01-18T23:45:26.627Z] 2024-01-18 23:45:26 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-18T23:46:22.805Z] 2024-01-18 23:46:22 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] first schedule backup for schedule name [backup-schedule-cegb] is [backup-schedule-cegb-d94312a-interval-2024-01-18-234521] [2024-01-18T23:46:22.805Z] 2024/01/18 23:46:22 DoRetryWithTimeout - Error: {backup status for [backup-schedule-cegb-d94312a-interval-2024-01-18-234521] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:46:24.693Z] 2024-01-18 23:46:24 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] first schedule backup for schedule name [backup-schedule-cegb] is [backup-schedule-cegb-9a77205-interval-2024-01-18-234524] [2024-01-18T23:46:24.693Z] 2024/01/18 23:46:24 DoRetryWithTimeout - Error: {backup status for [backup-schedule-cegb-9a77205-interval-2024-01-18-234524] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:46:27.214Z] 2024-01-18 23:46:27 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] first schedule backup for schedule name [backup-schedule-cegb] is [backup-schedule-cegb-5d614c2-interval-2024-01-18-234526] [2024-01-18T23:46:27.214Z] 2024/01/18 23:46:27 DoRetryWithTimeout - Error: {backup status for [backup-schedule-cegb-5d614c2-interval-2024-01-18-234526] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:46:53.713Z] 2024/01/18 23:46:52 DoRetryWithTimeout - Error: {backup status for [backup-schedule-cegb-d94312a-interval-2024-01-18-234521] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:46:54.638Z] 2024/01/18 23:46:54 DoRetryWithTimeout - Error: {backup status for [backup-schedule-cegb-9a77205-interval-2024-01-18-234524] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:46:57.153Z] 2024/01/18 23:46:57 DoRetryWithTimeout - Error: {backup status for [backup-schedule-cegb-5d614c2-interval-2024-01-18-234526] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:47:23.657Z] 2024/01/18 23:47:22 DoRetryWithTimeout - Error: {backup status for [backup-schedule-cegb-d94312a-interval-2024-01-18-234521] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:47:24.584Z] 2024/01/18 23:47:24 DoRetryWithTimeout - Error: {backup status for [backup-schedule-cegb-9a77205-interval-2024-01-18-234524] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:47:27.876Z] 2024/01/18 23:47:27 DoRetryWithTimeout - Error: {backup status for [backup-schedule-cegb-5d614c2-interval-2024-01-18-234526] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:47:54.378Z] 2024-01-18 23:47:52 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating backup [backup-schedule-cegb-d94312a-interval-2024-01-18-234521] in org [default] [2024-01-18T23:47:54.378Z] 2024-01-18 23:47:52 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [backup-schedule-cegb-d94312a-interval-2024-01-18-234521] [2024-01-18T23:47:54.378Z] 2024-01-18 23:47:52 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87563-01-18-23h40m54s] in backup [backup-schedule-cegb-d94312a-interval-2024-01-18-234521] [2024-01-18T23:47:54.378Z] 2024-01-18 23:47:52 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87563-01-18-23h40m54s] in backup [backup-schedule-cegb-d94312a-interval-2024-01-18-234521] [2024-01-18T23:47:54.378Z] 2024-01-18 23:47:52 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87563-01-18-23h40m54s] [2024-01-18T23:47:54.378Z] 2024-01-18 23:47:52 +0000:[WARNING] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T23:47:54.378Z] 2024-01-18 23:47:52 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T23:47:54.379Z] 2024-01-18 23:47:52 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[mysql-data]] [2024-01-18T23:47:54.379Z] 2024-01-18 23:47:52 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T23:47:54.379Z] 2024-01-18 23:47:52 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating CR cleanup [2024-01-18T23:47:54.379Z] 2024-01-18 23:47:52 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T23:47:54.379Z] 2024-01-18 23:47:52 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:47:54.379Z] 2024-01-18 23:47:52 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:47:54.379Z] 2024/01/18 23:47:52 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-d94312a-interval-2024-01-18-234521]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:47:54.642Z] 2024-01-18 23:47:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating backup [backup-schedule-cegb-9a77205-interval-2024-01-18-234524] in org [default] [2024-01-18T23:47:54.642Z] 2024-01-18 23:47:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [backup-schedule-cegb-9a77205-interval-2024-01-18-234524] [2024-01-18T23:47:54.642Z] 2024-01-18 23:47:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87563-01-18-23h40m54s] in backup [backup-schedule-cegb-9a77205-interval-2024-01-18-234524] [2024-01-18T23:47:54.642Z] 2024-01-18 23:47:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87563-01-18-23h40m54s] in backup [backup-schedule-cegb-9a77205-interval-2024-01-18-234524] [2024-01-18T23:47:54.642Z] 2024-01-18 23:47:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87563-01-18-23h40m54s] [2024-01-18T23:47:54.642Z] 2024-01-18 23:47:54 +0000:[WARNING] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T23:47:54.643Z] 2024-01-18 23:47:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T23:47:54.643Z] 2024-01-18 23:47:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[mysql-data]] [2024-01-18T23:47:54.643Z] 2024-01-18 23:47:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T23:47:54.643Z] 2024-01-18 23:47:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating CR cleanup [2024-01-18T23:47:54.898Z] 2024-01-18 23:47:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T23:47:54.898Z] 2024-01-18 23:47:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:47:54.898Z] 2024-01-18 23:47:54 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:47:54.898Z] 2024/01/18 23:47:54 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-9a77205-interval-2024-01-18-234524]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:47:57.410Z] 2024-01-18 23:47:57 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating backup [backup-schedule-cegb-5d614c2-interval-2024-01-18-234526] in org [default] [2024-01-18T23:47:57.411Z] 2024-01-18 23:47:57 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [backup-schedule-cegb-5d614c2-interval-2024-01-18-234526] [2024-01-18T23:47:57.411Z] 2024-01-18 23:47:57 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87563-01-18-23h40m54s] in backup [backup-schedule-cegb-5d614c2-interval-2024-01-18-234526] [2024-01-18T23:47:57.411Z] 2024-01-18 23:47:57 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87563-01-18-23h40m54s] in backup [backup-schedule-cegb-5d614c2-interval-2024-01-18-234526] [2024-01-18T23:47:57.411Z] 2024-01-18 23:47:57 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87563-01-18-23h40m54s] [2024-01-18T23:47:57.411Z] 2024-01-18 23:47:57 +0000:[WARNING] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-18T23:47:57.411Z] 2024-01-18 23:47:57 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T23:47:57.411Z] 2024-01-18 23:47:57 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[mysql-data]] [2024-01-18T23:47:57.411Z] 2024-01-18 23:47:57 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-18T23:47:57.411Z] 2024-01-18 23:47:57 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating CR cleanup [2024-01-18T23:47:57.411Z] 2024-01-18 23:47:57 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-18T23:47:57.666Z] 2024-01-18 23:47:57 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:47:57.667Z] 2024-01-18 23:47:57 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:47:57.667Z] 2024/01/18 23:47:57 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-5d614c2-interval-2024-01-18-234526]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:47:57.667Z] 2024-01-18 23:47:57 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:47:57.667Z] 2024-01-18 23:47:57 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:47:57.667Z] 2024/01/18 23:47:57 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-d94312a-interval-2024-01-18-234521]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:48:00.187Z] 2024-01-18 23:47:59 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:00.187Z] 2024-01-18 23:47:59 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:48:00.187Z] 2024/01/18 23:47:59 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-9a77205-interval-2024-01-18-234524]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:48:02.705Z] 2024-01-18 23:48:02 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:02.705Z] 2024-01-18 23:48:02 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:48:02.705Z] 2024/01/18 23:48:02 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-5d614c2-interval-2024-01-18-234526]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:48:02.705Z] 2024-01-18 23:48:02 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:02.705Z] 2024-01-18 23:48:02 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:48:02.705Z] 2024/01/18 23:48:02 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-d94312a-interval-2024-01-18-234521]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:48:05.224Z] 2024-01-18 23:48:04 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:05.224Z] 2024-01-18 23:48:04 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:48:05.224Z] 2024/01/18 23:48:04 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-9a77205-interval-2024-01-18-234524]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:48:07.737Z] 2024-01-18 23:48:07 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:07.737Z] 2024-01-18 23:48:07 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:48:07.737Z] 2024/01/18 23:48:07 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-5d614c2-interval-2024-01-18-234526]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:48:07.738Z] 2024-01-18 23:48:07 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:07.738Z] 2024-01-18 23:48:07 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:48:07.738Z] 2024/01/18 23:48:07 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-d94312a-interval-2024-01-18-234521]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:48:10.255Z] 2024-01-18 23:48:09 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:10.255Z] 2024-01-18 23:48:09 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:48:10.255Z] 2024/01/18 23:48:09 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-9a77205-interval-2024-01-18-234524]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:48:12.774Z] 2024-01-18 23:48:12 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:12.774Z] 2024-01-18 23:48:12 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:48:12.774Z] 2024/01/18 23:48:12 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-5d614c2-interval-2024-01-18-234526]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:48:12.774Z] 2024-01-18 23:48:12 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:12.774Z] 2024-01-18 23:48:12 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:48:12.774Z] 2024/01/18 23:48:12 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-d94312a-interval-2024-01-18-234521]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:48:15.287Z] 2024-01-18 23:48:14 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:15.287Z] 2024-01-18 23:48:14 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:48:15.287Z] 2024/01/18 23:48:14 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-9a77205-interval-2024-01-18-234524]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:48:17.805Z] 2024-01-18 23:48:17 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:17.805Z] 2024-01-18 23:48:17 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:48:17.805Z] 2024-01-18 23:48:17 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:17.805Z] 2024-01-18 23:48:17 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:48:17.805Z] 2024/01/18 23:48:17 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-5d614c2-interval-2024-01-18-234526]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:48:17.805Z] 2024/01/18 23:48:17 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-d94312a-interval-2024-01-18-234521]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:48:20.324Z] 2024-01-18 23:48:19 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:20.324Z] 2024-01-18 23:48:19 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:48:20.324Z] 2024/01/18 23:48:19 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-9a77205-interval-2024-01-18-234524]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:48:22.839Z] 2024-01-18 23:48:22 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:22.839Z] 2024-01-18 23:48:22 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:48:22.839Z] 2024-01-18 23:48:22 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:22.839Z] 2024/01/18 23:48:22 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-d94312a-interval-2024-01-18-234521]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:48:22.839Z] 2024-01-18 23:48:22 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-5d614c2-interval-2024-01-18-234526 backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:48:22.839Z] 2024/01/18 23:48:22 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-5d614c2-interval-2024-01-18-234526]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:48:25.354Z] 2024-01-18 23:48:24 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:25.355Z] 2024-01-18 23:48:24 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:48:25.355Z] 2024/01/18 23:48:24 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-9a77205-interval-2024-01-18-234524]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:48:27.872Z] 2024-01-18 23:48:27 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:27.872Z] INFO[2024-01-18 23:48:27] Verifying : Description : Verifying creation and validation of schedule backup with schedule name [periodic-cegb] [2024-01-18T23:48:27.872Z] INFO[2024-01-18 23:48:27] Actual:true, Expected: true [2024-01-18T23:48:27.872Z] 2024-01-18 23:48:27 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:27.872Z] 2024-01-18 23:48:27 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:48:27.873Z] 2024/01/18 23:48:27 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-d94312a-interval-2024-01-18-234521]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:48:30.429Z] 2024-01-18 23:48:29 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:30.429Z] 2024-01-18 23:48:29 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:48:30.429Z] 2024/01/18 23:48:29 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-9a77205-interval-2024-01-18-234524]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:48:32.958Z] 2024-01-18 23:48:32 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:32.958Z] 2024-01-18 23:48:32 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:48:32.958Z] 2024/01/18 23:48:32 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-d94312a-interval-2024-01-18-234521]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:48:35.529Z] 2024-01-18 23:48:34 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:35.529Z] 2024-01-18 23:48:34 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:48:35.529Z] 2024/01/18 23:48:34 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-9a77205-interval-2024-01-18-234524]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:48:38.054Z] 2024-01-18 23:48:37 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:38.054Z] 2024-01-18 23:48:37 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:48:38.054Z] 2024/01/18 23:48:37 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-d94312a-interval-2024-01-18-234521]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:48:40.575Z] 2024-01-18 23:48:39 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:40.575Z] 2024-01-18 23:48:39 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:48:40.575Z] 2024/01/18 23:48:39 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-9a77205-interval-2024-01-18-234524]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:48:43.088Z] 2024-01-18 23:48:42 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:43.088Z] 2024-01-18 23:48:42 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-9a77205-interval-2024-01-18-234524 backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:48:43.088Z] 2024/01/18 23:48:42 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-d94312a-interval-2024-01-18-234521]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:48:45.599Z] 2024-01-18 23:48:45 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:45.599Z] INFO[2024-01-18 23:48:45] Verifying : Description : Verifying creation and validation of schedule backup with schedule name [periodic-cegb] [2024-01-18T23:48:45.599Z] INFO[2024-01-18 23:48:45] Actual:true, Expected: true [2024-01-18T23:48:48.113Z] 2024-01-18 23:48:47 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:48.113Z] 2024-01-18 23:48:47 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:48:48.113Z] 2024/01/18 23:48:47 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-d94312a-interval-2024-01-18-234521]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:48:53.375Z] 2024-01-18 23:48:52 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:53.375Z] 2024-01-18 23:48:52 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:48:53.375Z] 2024/01/18 23:48:52 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-d94312a-interval-2024-01-18-234521]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:48:58.616Z] 2024-01-18 23:48:57 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:48:58.616Z] 2024-01-18 23:48:57 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:48:58.616Z] 2024/01/18 23:48:57 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-d94312a-interval-2024-01-18-234521]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:49:03.881Z] 2024-01-18 23:49:02 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] [2024-01-18T23:49:03.881Z] 2024-01-18 23:49:02 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] CR found for [[backup-schedule-cegb-d94312a-interval-2024-01-18-234521]] under [mysql-ibm-pxb-0-87563-01-18-23h40m54s] namespace [2024-01-18T23:49:03.881Z] 2024/01/18 23:49:02 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-cegb-d94312a-interval-2024-01-18-234521]}, Next try in [5s], timeout [5m0s] [2024-01-18T23:49:08.043Z] 2024-01-18 23:49:07 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87563-01-18-23h40m54s] are [[]] [2024-01-18T23:49:08.043Z] INFO[2024-01-18 23:49:07] Verifying : Description : Verifying creation and validation of schedule backup with schedule name [periodic-cegb] [2024-01-18T23:49:08.043Z] INFO[2024-01-18 23:49:07] Actual:true, Expected: true [2024-01-18T23:49:08.043Z] 2024-01-18 23:49:07 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Taking restore of backups from the user testuser3-1705621356 [2024-01-18T23:49:08.605Z] 2024/01/18 23:49:08 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T23:49:40.636Z] 2024/01/18 23:49:38 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-18T23:50:12.658Z] 2024-01-18 23:50:08 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.CreateRestore:#936] - Restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] created successfully [2024-01-18T23:50:12.658Z] 2024-01-18 23:50:08 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Switching cluster context to cluster [destination-cluster] [2024-01-18T23:50:12.658Z] 2024-01-18 23:50:08 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Switching context: The context is already [/tmp/destination-config] [2024-01-18T23:50:12.658Z] 2024-01-18 23:50:08 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-87563-01-18-23h40m54s] with namespace mapping [map[mysql-ibm-pxb-0-87563-01-18-23h40m54s:custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]] and storage Class Mapping [map[]] [2024-01-18T23:50:12.659Z] 2024-01-18 23:50:08 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-87563-01-18-23h40m54s] [2024-01-18T23:50:12.659Z] 2024-01-18 23:50:08 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T23:50:12.659Z] 2024-01-18 23:50:08 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T23:50:12.659Z] 2024-01-18 23:50:08 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] in org [default] [2024-01-18T23:50:12.659Z] 2024-01-18 23:50:08 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] [2024-01-18T23:50:12.659Z] 2024-01-18 23:50:08 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] [2024-01-18T23:50:12.659Z] 2024-01-18 23:50:08 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating specs for the namespace (restoredAppContext) [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] [2024-01-18T23:50:12.659Z] 2024-01-18 23:50:08 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) was found in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb], as expected by presence in expectedRestoredAppContext [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] [2024-01-18T23:50:12.659Z] 2024-01-18 23:50:08 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) was found in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb], as expected by presence in expectedRestoredAppContext [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] [2024-01-18T23:50:12.659Z] 2024-01-18 23:50:08 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) was found in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb], as expected by presence in expectedRestoredAppContext [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] [2024-01-18T23:50:12.659Z] 2024-01-18 23:50:08 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) found in the restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] was also present on the cluster/namespace [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] [2024-01-18T23:50:12.659Z] 2024-01-18 23:50:08 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) was found in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb], as expected by presence in expectedRestoredAppContext [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] [2024-01-18T23:50:12.659Z] 2024-01-18 23:50:08 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) found in the restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] was also present on the cluster/namespace [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] [2024-01-18T23:50:12.660Z] 2024-01-18 23:50:08 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating Restored Volumes for the namespace (restoredAppContext) [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] [2024-01-18T23:50:12.660Z] 2024-01-18 23:50:08 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] [2024-01-18T23:50:12.660Z] 2024-01-18 23:50:08 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Switching cluster context back to cluster path [/tmp/destination-config] [2024-01-18T23:50:12.660Z] 2024-01-18 23:50:08 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Switching context: The context is already [/tmp/destination-config] [2024-01-18T23:50:12.660Z] INFO[2024-01-18 23:50:08] Verifying : Description : Verifying creation of restore tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb of backup tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb [2024-01-18T23:50:12.660Z] ERRO[2024-01-18 23:50:08] ValidateRestore Errors: {presence of object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) found in the restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] on the cluster/namespace [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] could not be verified as scheduler is not K8s} [2024-01-18T23:50:12.660Z] {presence of object (name: [mysql], kind: [Service], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) found in the restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] on the cluster/namespace [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] could not be verified as scheduler is not K8s} [2024-01-18T23:50:12.660Z] {error getting volumes for namespace (expectedRestoredAppContext) [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb], hence skipping volume validation. Error in Inst().S.GetVolumes: [error getting pvc: mysql-data, namespace: custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb. Err: persistentvolumeclaims "mysql-data" not found]} [2024-01-18T23:50:12.660Z] 2024-01-18 23:50:08 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.glob..func26.2.4:#202] - The list of user restores taken are: map[testuser3-1705621356:map[]] [2024-01-18T23:50:12.660Z] 2024-01-18 23:50:08 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Verifying backups of the user testuser3-1705621356 from the admin [2024-01-18T23:50:12.660Z] 2024-01-18 23:50:08 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Verifying backup schedules of the user testuser3-1705621356 from the admin [2024-01-18T23:50:12.660Z] 2024-01-18 23:50:08 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Verifying restores of the user testuser3-1705621356 from the admin [2024-01-18T23:50:12.660Z] 2024-01-18 23:50:09 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Taking restore of backups from the user testuser1-1705621356 [2024-01-18T23:50:12.660Z] 2024/01/18 23:50:09 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T23:50:44.682Z] 2024-01-18 23:50:39 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.CreateRestore:#936] - Restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] created successfully [2024-01-18T23:50:44.682Z] 2024-01-18 23:50:39 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Switching cluster context to cluster [destination-cluster] [2024-01-18T23:50:44.682Z] 2024-01-18 23:50:39 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Switching context: The context is already [/tmp/destination-config] [2024-01-18T23:50:44.682Z] 2024-01-18 23:50:39 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-87563-01-18-23h40m54s] with namespace mapping [map[mysql-ibm-pxb-0-87563-01-18-23h40m54s:custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]] and storage Class Mapping [map[]] [2024-01-18T23:50:44.682Z] 2024-01-18 23:50:39 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-87563-01-18-23h40m54s] [2024-01-18T23:50:44.682Z] 2024-01-18 23:50:39 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T23:50:44.682Z] 2024-01-18 23:50:39 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T23:50:44.682Z] 2024-01-18 23:50:39 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] in org [default] [2024-01-18T23:50:44.682Z] 2024-01-18 23:50:39 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] [2024-01-18T23:50:44.682Z] 2024-01-18 23:50:39 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] [2024-01-18T23:50:44.682Z] 2024-01-18 23:50:39 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating specs for the namespace (restoredAppContext) [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] [2024-01-18T23:50:44.682Z] 2024-01-18 23:50:39 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) was found in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb], as expected by presence in expectedRestoredAppContext [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] [2024-01-18T23:50:44.683Z] 2024-01-18 23:50:39 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) was found in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb], as expected by presence in expectedRestoredAppContext [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] [2024-01-18T23:50:44.683Z] 2024-01-18 23:50:39 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) was found in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb], as expected by presence in expectedRestoredAppContext [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] [2024-01-18T23:50:44.683Z] 2024-01-18 23:50:39 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) found in the restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] was also present on the cluster/namespace [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] [2024-01-18T23:50:44.683Z] 2024-01-18 23:50:39 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) was found in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb], as expected by presence in expectedRestoredAppContext [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] [2024-01-18T23:50:44.683Z] 2024-01-18 23:50:39 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) found in the restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] was also present on the cluster/namespace [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] [2024-01-18T23:50:44.683Z] 2024-01-18 23:50:39 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating Restored Volumes for the namespace (restoredAppContext) [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] [2024-01-18T23:50:44.683Z] 2024-01-18 23:50:39 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] [2024-01-18T23:50:44.683Z] 2024-01-18 23:50:39 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Switching cluster context back to cluster path [/tmp/destination-config] [2024-01-18T23:50:44.683Z] 2024-01-18 23:50:39 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Switching context: The context is already [/tmp/destination-config] [2024-01-18T23:50:44.683Z] INFO[2024-01-18 23:50:39] Verifying : Description : Verifying creation of restore tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb of backup tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb [2024-01-18T23:50:44.683Z] ERRO[2024-01-18 23:50:39] ValidateRestore Errors: {presence of object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) found in the restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] on the cluster/namespace [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] could not be verified as scheduler is not K8s} [2024-01-18T23:50:44.683Z] {presence of object (name: [mysql], kind: [Service], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) found in the restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] on the cluster/namespace [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] could not be verified as scheduler is not K8s} [2024-01-18T23:50:44.684Z] {error getting volumes for namespace (expectedRestoredAppContext) [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb], hence skipping volume validation. Error in Inst().S.GetVolumes: [error getting pvc: mysql-data, namespace: custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb. Err: persistentvolumeclaims "mysql-data" not found]} [2024-01-18T23:50:44.684Z] 2024-01-18 23:50:39 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.glob..func26.2.4:#202] - The list of user restores taken are: map[testuser1-1705621356:map[] testuser3-1705621356:map[]] [2024-01-18T23:50:44.684Z] 2024-01-18 23:50:39 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Verifying backups of the user testuser1-1705621356 from the admin [2024-01-18T23:50:44.684Z] 2024-01-18 23:50:39 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Verifying backup schedules of the user testuser1-1705621356 from the admin [2024-01-18T23:50:44.684Z] 2024-01-18 23:50:40 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Verifying restores of the user testuser1-1705621356 from the admin [2024-01-18T23:50:44.684Z] 2024-01-18 23:50:40 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Taking restore of backups from the user testuser2-1705621356 [2024-01-18T23:50:44.684Z] 2024/01/18 23:50:40 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-18T23:51:11.180Z] 2024-01-18 23:51:10 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.CreateRestore:#936] - Restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] created successfully [2024-01-18T23:51:11.180Z] 2024-01-18 23:51:10 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Switching cluster context to cluster [destination-cluster] [2024-01-18T23:51:11.180Z] 2024-01-18 23:51:10 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Switching context: The context is already [/tmp/destination-config] [2024-01-18T23:51:11.180Z] 2024-01-18 23:51:10 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-87563-01-18-23h40m54s] with namespace mapping [map[mysql-ibm-pxb-0-87563-01-18-23h40m54s:custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]] and storage Class Mapping [map[]] [2024-01-18T23:51:11.181Z] 2024-01-18 23:51:10 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-87563-01-18-23h40m54s] [2024-01-18T23:51:11.181Z] 2024-01-18 23:51:10 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T23:51:11.181Z] 2024-01-18 23:51:10 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T23:51:11.181Z] 2024-01-18 23:51:10 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] in org [default] [2024-01-18T23:51:11.181Z] 2024-01-18 23:51:10 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] [2024-01-18T23:51:11.181Z] 2024-01-18 23:51:10 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] [2024-01-18T23:51:11.181Z] 2024-01-18 23:51:10 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating specs for the namespace (restoredAppContext) [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] [2024-01-18T23:51:11.181Z] 2024-01-18 23:51:10 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) was found in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb], as expected by presence in expectedRestoredAppContext [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] [2024-01-18T23:51:11.181Z] 2024-01-18 23:51:10 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) was found in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb], as expected by presence in expectedRestoredAppContext [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] [2024-01-18T23:51:11.181Z] 2024-01-18 23:51:10 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) was found in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb], as expected by presence in expectedRestoredAppContext [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] [2024-01-18T23:51:11.181Z] 2024-01-18 23:51:10 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) found in the restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] was also present on the cluster/namespace [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] [2024-01-18T23:51:11.181Z] 2024-01-18 23:51:10 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) was found in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb], as expected by presence in expectedRestoredAppContext [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] [2024-01-18T23:51:11.182Z] 2024-01-18 23:51:10 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) found in the restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] was also present on the cluster/namespace [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] [2024-01-18T23:51:11.182Z] 2024-01-18 23:51:10 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Validating Restored Volumes for the namespace (restoredAppContext) [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] [2024-01-18T23:51:11.182Z] 2024-01-18 23:51:10 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] in restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] [2024-01-18T23:51:11.182Z] 2024-01-18 23:51:10 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Switching cluster context back to cluster path [/tmp/destination-config] [2024-01-18T23:51:11.182Z] 2024-01-18 23:51:10 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Switching context: The context is already [/tmp/destination-config] [2024-01-18T23:51:11.182Z] INFO[2024-01-18 23:51:10] Verifying : Description : Verifying creation of restore tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb of backup tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb [2024-01-18T23:51:11.182Z] ERRO[2024-01-18 23:51:10] ValidateRestore Errors: {presence of object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) found in the restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] on the cluster/namespace [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] could not be verified as scheduler is not K8s} [2024-01-18T23:51:11.182Z] {presence of object (name: [mysql], kind: [Service], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) found in the restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] on the cluster/namespace [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] could not be verified as scheduler is not K8s} [2024-01-18T23:51:11.182Z] {error getting volumes for namespace (expectedRestoredAppContext) [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb], hence skipping volume validation. Error in Inst().S.GetVolumes: [error getting pvc: mysql-data, namespace: custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb. Err: persistentvolumeclaims "mysql-data" not found]} [2024-01-18T23:51:11.182Z] 2024-01-18 23:51:10 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.glob..func26.2.4:#202] - The list of user restores taken are: map[testuser1-1705621356:map[] testuser2-1705621356:map[] testuser3-1705621356:map[]] [2024-01-18T23:51:11.182Z] 2024-01-18 23:51:10 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Verifying backups of the user testuser2-1705621356 from the admin [2024-01-18T23:51:11.182Z] 2024-01-18 23:51:11 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Verifying backup schedules of the user testuser2-1705621356 from the admin [2024-01-18T23:51:11.440Z] 2024-01-18 23:51:11 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Verifying restores of the user testuser2-1705621356 from the admin [2024-01-18T23:51:11.441Z] 2024-01-18 23:51:11 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Deleting user testuser2-1705621356 schedule backups, backup schedule and schedule policy from the admin [2024-01-18T23:51:11.441Z] 2024-01-18 23:51:11 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Deleting user testuser3-1705621356 schedule backups, backup schedule and schedule policy from the admin [2024-01-18T23:51:11.441Z] 2024-01-18 23:51:11 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Deleting user testuser1-1705621356 schedule backups, backup schedule and schedule policy from the admin [2024-01-18T23:51:11.697Z] 2024-01-18 23:51:11 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Deleting user testuser2-1705621356 backups from the admin [2024-01-18T23:51:11.954Z] 2024-01-18 23:51:11 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Deleting user testuser3-1705621356 backups from the admin [2024-01-18T23:51:12.210Z] 2024-01-18 23:51:12 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Deleting user testuser1-1705621356 backups from the admin [2024-01-18T23:51:12.772Z] 2024-01-18 23:51:12 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Waiting for the backups and backup schedule to be deleted [2024-01-18T23:51:13.028Z] 2024-01-18 23:51:12 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Waiting for the backups and backup schedule to be deleted [2024-01-18T23:51:13.284Z] 2024-01-18 23:51:13 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Waiting for the backups and backup schedule to be deleted [2024-01-18T23:51:14.212Z] 2024/01/18 23:51:13 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-cegb] is in [DeletePending] state}, Next try in [30s], timeout [1h0m0s] [2024-01-18T23:51:14.212Z] 2024/01/18 23:51:13 DoRetryWithTimeout - Error: {backup [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] is in [Deleting] state}, Next try in [30s], timeout [1h0m0s] [2024-01-18T23:51:14.212Z] 2024/01/18 23:51:14 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-cegb] is in [DeletePending] state}, Next try in [30s], timeout [1h0m0s] [2024-01-18T23:51:14.212Z] 2024/01/18 23:51:14 DoRetryWithTimeout - Error: {backup [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] is in [Deleting] state}, Next try in [30s], timeout [1h0m0s] [2024-01-18T23:51:14.469Z] 2024/01/18 23:51:14 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-cegb] is in [DeletePending] state}, Next try in [30s], timeout [1h0m0s] [2024-01-18T23:51:14.469Z] 2024/01/18 23:51:14 DoRetryWithTimeout - Error: {backup [tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] is in [Deleting] state}, Next try in [30s], timeout [1h0m0s] [2024-01-18T23:51:46.500Z] 2024/01/18 23:51:43 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-cegb] is in [DeletePending] state}, Next try in [30s], timeout [1h0m0s] [2024-01-18T23:51:46.500Z] 2024/01/18 23:51:44 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-cegb] is in [DeletePending] state}, Next try in [30s], timeout [1h0m0s] [2024-01-18T23:51:46.500Z] 2024/01/18 23:51:44 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-cegb] is in [DeletePending] state}, Next try in [30s], timeout [1h0m0s] [2024-01-18T23:52:18.528Z] 2024-01-18 23:52:14 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Deleting user testuser2-1705621356 restores from the admin [2024-01-18T23:52:18.528Z] 2024-01-18 23:52:14 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Deleting user testuser2-1705621356 source and destination cluster from the admin [2024-01-18T23:52:18.528Z] 2024-01-18 23:52:14 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Deleting user testuser3-1705621356 restores from the admin [2024-01-18T23:52:18.528Z] 2024-01-18 23:52:14 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Deleting user testuser3-1705621356 source and destination cluster from the admin [2024-01-18T23:52:18.528Z] INFO[2024-01-18 23:52:14] Verifying : Description : Verifying deletion of cluster [source-cluster] of the user testuser2-1705621356 [2024-01-18T23:52:18.528Z] INFO[2024-01-18 23:52:14] Actual:true, Expected: true [2024-01-18T23:52:18.528Z] 2024-01-18 23:52:14 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Deleting user testuser1-1705621356 restores from the admin [2024-01-18T23:52:18.528Z] 2024-01-18 23:52:14 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Deleting user testuser1-1705621356 source and destination cluster from the admin [2024-01-18T23:52:18.528Z] INFO[2024-01-18 23:52:14] Verifying : Description : Verifying deletion of cluster [source-cluster] of the user testuser3-1705621356 [2024-01-18T23:52:18.528Z] INFO[2024-01-18 23:52:14] Actual:true, Expected: true [2024-01-18T23:52:18.528Z] INFO[2024-01-18 23:52:14] Verifying : Description : Verifying deletion of cluster [destination-cluster] of the user testuser2-1705621356 [2024-01-18T23:52:18.528Z] INFO[2024-01-18 23:52:14] Actual:true, Expected: true [2024-01-18T23:52:18.528Z] INFO[2024-01-18 23:52:14] Verifying : Description : Verifying deletion of cluster [source-cluster] of the user testuser1-1705621356 [2024-01-18T23:52:18.528Z] INFO[2024-01-18 23:52:14] Actual:true, Expected: true [2024-01-18T23:52:18.528Z] INFO[2024-01-18 23:52:14] Verifying : Description : Verifying deletion of cluster [destination-cluster] of the user testuser3-1705621356 [2024-01-18T23:52:18.528Z] INFO[2024-01-18 23:52:14] Actual:true, Expected: true [2024-01-18T23:52:18.528Z] INFO[2024-01-18 23:52:14] Verifying : Description : Verifying deletion of cluster [destination-cluster] of the user testuser1-1705621356 [2024-01-18T23:52:18.528Z] INFO[2024-01-18 23:52:14] Actual:true, Expected: true [2024-01-18T23:52:18.528Z] 2024-01-18 23:52:14 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Destroying the scheduled applications [2024-01-18T23:52:18.528Z] 2024-01-18 23:52:14 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Destroying apps [2024-01-18T23:52:18.528Z] 2024-01-18 23:52:14 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-18T23:52:18.528Z] 2024-01-18 23:52:14 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-18T23:52:18.528Z] 2024-01-18 23:52:14 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] start destroying mysql-ibm app [2024-01-18T23:52:18.528Z] 2024-01-18 23:52:15 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-18T23:52:18.528Z] 2024-01-18 23:52:15 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T23:52:18.529Z] 2024-01-18 23:52:15 +0000:[WARNING] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-18T23:52:18.529Z] INFO[2024-01-18 23:52:15] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-18T23:52:18.529Z] INFO[2024-01-18 23:52:15] Actual:true, Expected: true [2024-01-18T23:52:18.529Z] 2024-01-18 23:52:15 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Cleaning backup locations in map [map[c5b86e41-3385-47b9-ae7f-79804298bbaf:autogenerated-backup-location-cegb]], cloud credential [autogenerated-cred-cegb], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T23:52:18.529Z] 2024-01-18 23:52:15 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Cleaning backup locations in map [map[fe41668a-fbcb-4c94-a1a9-758b0882f0c6:autogenerated-backup-location-cegb]], cloud credential [autogenerated-cred-cegb], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T23:52:18.529Z] 2024-01-18 23:52:15 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Cleaning backup locations in map [map[d10ae85a-d1f3-4e9f-9340-4a6e79d3ac7e:autogenerated-backup-location-cegb]], cloud credential [autogenerated-cred-cegb], source [source-cluster] and destination [destination-cluster] cluster [2024-01-18T23:52:18.529Z] INFO[2024-01-18 23:52:15] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-cegb] [2024-01-18T23:52:18.529Z] INFO[2024-01-18 23:52:15] Actual:true, Expected: true [2024-01-18T23:52:18.529Z] INFO[2024-01-18 23:52:15] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-cegb] [2024-01-18T23:52:18.529Z] INFO[2024-01-18 23:52:15] Actual:true, Expected: true [2024-01-18T23:52:18.529Z] INFO[2024-01-18 23:52:15] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-cegb] [2024-01-18T23:52:18.529Z] INFO[2024-01-18 23:52:15] Actual:true, Expected: true [2024-01-18T23:52:18.529Z] 2024-01-18 23:52:15 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-cegb] is present [2024-01-18T23:52:18.529Z] 2024-01-18 23:52:15 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-cegb] is present [2024-01-18T23:52:18.529Z] 2024-01-18 23:52:15 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-cegb] is present [2024-01-18T23:52:18.529Z] 2024/01/18 23:52:15 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-cegb is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T23:52:18.529Z] 2024/01/18 23:52:15 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-cegb is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T23:52:18.529Z] 2024/01/18 23:52:15 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-cegb is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T23:52:50.557Z] 2024-01-18 23:52:45 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-cegb] is present [2024-01-18T23:52:50.557Z] 2024-01-18 23:52:45 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-cegb] is present [2024-01-18T23:52:50.557Z] 2024-01-18 23:52:45 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-cegb] is present [2024-01-18T23:52:50.557Z] 2024/01/18 23:52:45 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-cegb is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T23:52:50.557Z] 2024/01/18 23:52:45 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-cegb is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T23:52:50.557Z] 2024/01/18 23:52:45 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-cegb is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-18T23:53:17.053Z] 2024-01-18 23:53:15 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [] [2024-01-18T23:53:17.053Z] INFO[2024-01-18 23:53:15] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-cegb [2024-01-18T23:53:17.053Z] INFO[2024-01-18 23:53:15] Actual:true, Expected: true [2024-01-18T23:53:17.053Z] 2024-01-18 23:53:15 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [] [2024-01-18T23:53:17.053Z] INFO[2024-01-18 23:53:15] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-cegb [2024-01-18T23:53:17.053Z] INFO[2024-01-18 23:53:15] Actual:true, Expected: true [2024-01-18T23:53:17.053Z] 2024-01-18 23:53:15 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [] [2024-01-18T23:53:17.053Z] INFO[2024-01-18 23:53:15] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-cegb [2024-01-18T23:53:17.053Z] INFO[2024-01-18 23:53:15] Actual:true, Expected: true [2024-01-18T23:53:17.053Z] 2024-01-18 23:53:15 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-cegb] is present [2024-01-18T23:53:17.053Z] INFO[2024-01-18 23:53:15] Verifying : Description : Verifying if cloud cred [autogenerated-cred-cegb] is present [2024-01-18T23:53:17.053Z] INFO[2024-01-18 23:53:15] Actual:true, Expected: true [2024-01-18T23:53:17.053Z] 2024-01-18 23:53:15 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-cegb] is present [2024-01-18T23:53:17.053Z] INFO[2024-01-18 23:53:15] Verifying : Description : Verifying if cloud cred [autogenerated-cred-cegb] is present [2024-01-18T23:53:17.053Z] INFO[2024-01-18 23:53:15] Actual:true, Expected: true [2024-01-18T23:53:17.053Z] 2024-01-18 23:53:15 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-cegb] is present [2024-01-18T23:53:17.054Z] INFO[2024-01-18 23:53:15] Verifying : Description : Verifying if cloud cred [autogenerated-cred-cegb] is present [2024-01-18T23:53:17.054Z] INFO[2024-01-18 23:53:15] Actual:true, Expected: true [2024-01-18T23:53:17.054Z] INFO[2024-01-18 23:53:15] Verifying : Description : Verifying deletion of cloud cred [autogenerated-cred-cegb] [2024-01-18T23:53:17.054Z] INFO[2024-01-18 23:53:15] Actual:true, Expected: true [2024-01-18T23:53:17.054Z] INFO[2024-01-18 23:53:15] Verifying : Description : Verifying deletion of cloud cred [autogenerated-cred-cegb] [2024-01-18T23:53:17.054Z] INFO[2024-01-18 23:53:15] Actual:true, Expected: true [2024-01-18T23:53:17.054Z] INFO[2024-01-18 23:53:15] Verifying : Description : Verifying deletion of cloud cred [autogenerated-cred-cegb] [2024-01-18T23:53:17.054Z] INFO[2024-01-18 23:53:15] Actual:true, Expected: true [2024-01-18T23:53:17.054Z] INFO[2024-01-18 23:53:15] Verifying : Description : Deleting cloud cred autogenerated-cred-cegb [2024-01-18T23:53:17.054Z] INFO[2024-01-18 23:53:15] Actual:true, Expected: true [2024-01-18T23:53:17.054Z] INFO[2024-01-18 23:53:15] Verifying : Description : Deleting cloud cred autogenerated-cred-cegb [2024-01-18T23:53:17.054Z] INFO[2024-01-18 23:53:15] Actual:true, Expected: true [2024-01-18T23:53:17.054Z] INFO[2024-01-18 23:53:15] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T23:53:17.054Z] INFO[2024-01-18 23:53:15] Actual:true, Expected: true [2024-01-18T23:53:17.054Z] INFO[2024-01-18 23:53:15] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T23:53:17.054Z] INFO[2024-01-18 23:53:15] Actual:true, Expected: true [2024-01-18T23:53:17.054Z] INFO[2024-01-18 23:53:15] Verifying : Description : Deleting cloud cred autogenerated-cred-cegb [2024-01-18T23:53:17.054Z] INFO[2024-01-18 23:53:15] Actual:true, Expected: true [2024-01-18T23:53:17.054Z] INFO[2024-01-18 23:53:15] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-18T23:53:17.054Z] INFO[2024-01-18 23:53:15] Actual:true, Expected: true [2024-01-18T23:53:17.054Z] 2024-01-18 23:53:16 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser3-1705621356 - 91842096-c915-473d-a79f-a0c2b68c1e6e [2024-01-18T23:53:17.054Z] 2024-01-18 23:53:16 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705621356 - 23ad8145-bcb6-4851-8525-1262e96ea046 [2024-01-18T23:53:17.984Z] 2024-01-18 23:53:17 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2-1705621356 - 0aa307b0-cec7-44b2-998e-7c610c260910 [2024-01-18T23:53:18.271Z] 2024-01-18 23:53:18 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.DeleteUser:#755] - Response for user [testuser3-1705621356] deletion - [2024-01-18T23:53:18.271Z] 2024-01-18 23:53:18 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.DeleteUser:#759] - Deleted User - testuser3-1705621356 [2024-01-18T23:53:18.271Z] 2024-01-18 23:53:18 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.DeleteUser:#755] - Response for user [testuser1-1705621356] deletion - [2024-01-18T23:53:18.271Z] 2024-01-18 23:53:18 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.DeleteUser:#759] - Deleted User - testuser1-1705621356 [2024-01-18T23:53:18.839Z] 2024-01-18 23:53:18 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.DeleteUser:#755] - Response for user [testuser2-1705621356] deletion - [2024-01-18T23:53:18.839Z] 2024-01-18 23:53:18 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [backup.DeleteUser:#759] - Deleted User - testuser2-1705621356 [2024-01-18T23:53:18.839Z] INFO[2024-01-18 23:53:18] --------Test End------ [2024-01-18T23:53:18.839Z] INFO[2024-01-18 23:53:18] #Test: [2024-01-18T23:53:18.839Z] INFO[2024-01-18 23:53:18] #Description: [2024-01-18T23:53:18.839Z] INFO[2024-01-18 23:53:18] ------------------------ [2024-01-18T23:53:18.839Z] 2024-01-18 23:53:18 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.EndPxBackupTorpedoTest:#7261] - >>>> FAILED TEST: {DeleteSameNameObjectsByMultipleUsersFromAdmin} Deletes backups, backup schedules, restore and cluster objects created by multiple user with same name from the admin [2024-01-18T23:53:18.839Z] 2024-01-18 23:53:18 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-87563-01-18-23h40m54s] [2024-01-18T23:53:19.095Z] 2024-01-18 23:53:18 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-18T23:53:19.095Z] 2024-01-18 23:53:18 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Switching context: The context is already [/tmp/destination-config] [2024-01-18T23:53:19.350Z] 2024-01-18 23:53:19 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-87563-01-18-23h40m54s] [2024-01-18T23:53:19.350Z] 2024-01-18 23:53:19 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {DeleteSameNameObjectsByMultipleUsersFromAdmin} Deletes backups, backup schedules, restore and cluster objects created by multiple user with same name from the admin [2024-01-18T23:53:19.351Z] 2024-01-18 23:53:19 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-18T23:53:19.351Z] 2024/01/18 23:53:19 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:53:24.608Z] 2024/01/18 23:53:24 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:53:29.856Z] 2024/01/18 23:53:29 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:53:35.105Z] 2024/01/18 23:53:34 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:53:40.351Z] 2024/01/18 23:53:39 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:53:44.517Z] 2024/01/18 23:53:44 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:53:49.762Z] 2024/01/18 23:53:49 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:53:55.008Z] 2024/01/18 23:53:54 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-18T23:54:00.249Z] 2024/01/18 23:53:59 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.64.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.8 10.240.64.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T23:54:00.249Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T23:54:00.249Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T23:54:00.250Z] {dial tcp 10.240.64.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-18T23:54:04.416Z] 2024-01-18 23:54:04 +0000:[WARNING] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.64.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.8 10.240.64.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T23:54:04.416Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T23:54:04.416Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T23:54:04.416Z] {dial tcp 10.240.64.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T23:54:04.416Z] 2024-01-18 23:54:04 +0000:[ERROR] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.64.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.8 10.240.64.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T23:54:04.416Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T23:54:04.416Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-18T23:54:04.416Z] {dial tcp 10.240.64.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-18T23:54:04.416Z] 2024-01-18 23:54:04 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-18T23:54:04.416Z] 2024-01-18 23:54:04 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] Switching context to [/tmp/source-config] [2024-01-18T23:54:04.673Z] 2024-01-18 23:54:04 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T23:54:04.673Z] 2024-01-18 23:54:04 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T23:54:04.673Z] 2024-01-18 23:54:04 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-18T23:54:04.673Z] 2024-01-18 23:54:04 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T23:54:04.673Z] 2024-01-18 23:54:04 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T23:54:04.928Z] 2024-01-18 23:54:04 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-18T23:54:05.491Z] 2024-01-18 23:54:05 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T23:54:05.813Z] 2024-01-18 23:54:05 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T23:54:06.071Z] 2024-01-18 23:54:05 +0000:[INFO] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-18T23:54:06.328Z] 2024/01/18 23:54:06 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:54:16.266Z] 2024/01/18 23:54:16 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:54:28.453Z] 2024/01/18 23:54:26 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:54:36.538Z] 2024/01/18 23:54:36 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:54:36.795Z] INFO[2024-01-18 23:54:36] sched stats: workers: current=10, started=10, exited=0 [2024-01-18T23:54:36.795Z] INFO[2024-01-18 23:54:36] sched stats: tick interval: 30036 < 1.2s (100%) [2024-01-18T23:54:36.795Z] INFO[2024-01-18 23:54:36] sched stats: tick process durations: 30035 < 500ms (100%) [2024-01-18T23:54:46.755Z] 2024/01/18 23:54:46 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:54:56.699Z] 2024/01/18 23:54:56 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:55:06.638Z] 2024/01/18 23:55:06 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:55:16.580Z] 2024/01/18 23:55:16 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:55:26.526Z] 2024/01/18 23:55:26 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:55:36.472Z] 2024/01/18 23:55:36 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:55:48.637Z] 2024/01/18 23:55:46 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:55:56.732Z] 2024/01/18 23:55:56 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:56:06.680Z] 2024/01/18 23:56:06 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:56:16.621Z] 2024/01/18 23:56:16 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:56:28.791Z] 2024/01/18 23:56:26 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:56:36.876Z] 2024/01/18 23:56:36 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:56:46.812Z] 2024/01/18 23:56:46 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:56:56.754Z] 2024/01/18 23:56:56 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:57:06.738Z] 2024/01/18 23:57:06 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:57:18.920Z] 2024/01/18 23:57:16 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:57:27.003Z] 2024/01/18 23:57:26 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:57:36.941Z] 2024/01/18 23:57:36 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:57:46.881Z] 2024/01/18 23:57:46 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:57:59.048Z] 2024/01/18 23:57:56 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:58:07.127Z] 2024/01/18 23:58:06 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:58:17.065Z] 2024/01/18 23:58:16 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:58:27.003Z] 2024/01/18 23:58:26 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:58:39.188Z] 2024/01/18 23:58:36 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:58:47.276Z] 2024/01/18 23:58:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:58:57.229Z] 2024/01/18 23:58:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-18T23:59:07.177Z] 2024-01-18 23:59:07 +0000:[ERROR] [{DeleteSameNameObjectsByMultipleUsersFromAdmin}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.177Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.177Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.177Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.177Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.177Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.177Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.177Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.177Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.177Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.178Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.178Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.178Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.178Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.178Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.178Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.178Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.178Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.178Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.178Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.178Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.178Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.178Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.178Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.178Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.178Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.178Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.178Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.178Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-18T23:59:07.178Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-18T23:59:07.178Z] • Failure [1092.341 seconds] [2024-01-18T23:59:07.178Z] {DeleteSameNameObjectsByMultipleUsersFromAdmin} [2024-01-18T23:59:07.178Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_orphaned_test.go:21 [2024-01-18T23:59:07.178Z] Deletes backups, backup schedules, restore and cluster objects created by multiple user with same name from the admin [It] [2024-01-18T23:59:07.178Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_orphaned_test.go:59 [2024-01-18T23:59:07.178Z] [2024-01-18T23:59:07.178Z] Unexpected error: [2024-01-18T23:59:07.178Z] <*errors.errorString | 0xc001635150>: { [2024-01-18T23:59:07.179Z] s: "error: ValidateRestore Errors: {presence of object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) found in the restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] on the cluster/namespace [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] could not be verified as scheduler is not K8s}\n{presence of object (name: [mysql], kind: [Service], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) found in the restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] on the cluster/namespace [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] could not be verified as scheduler is not K8s}\n{error getting volumes for namespace (expectedRestoredAppContext) [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb], hence skipping volume validation. Error in Inst().S.GetVolumes: [error getting pvc: mysql-data, namespace: custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb. Err: persistentvolumeclaims \"mysql-data\" not found]}, Description: Verifying creation of restore tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb of backup tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb", [2024-01-18T23:59:07.179Z] } [2024-01-18T23:59:07.179Z] error: ValidateRestore Errors: {presence of object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) found in the restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] on the cluster/namespace [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] could not be verified as scheduler is not K8s} [2024-01-18T23:59:07.179Z] {presence of object (name: [mysql], kind: [Service], namespace: [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb]) found in the restore [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb] on the cluster/namespace [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb] could not be verified as scheduler is not K8s} [2024-01-18T23:59:07.179Z] {error getting volumes for namespace (expectedRestoredAppContext) [custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb], hence skipping volume validation. Error in Inst().S.GetVolumes: [error getting pvc: mysql-data, namespace: custom-mysql-ibm-pxb-0-87563-01-18-23h40m54scegb. Err: persistentvolumeclaims "mysql-data" not found]}, Description: Verifying creation of restore tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb of backup tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb [2024-01-18T23:59:07.179Z] occurred [2024-01-18T23:59:07.179Z] [2024-01-18T23:59:07.179Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-18T23:59:07.179Z] [2024-01-18T23:59:07.179Z] Full Stack Trace [2024-01-18T23:59:07.179Z] github.com/portworx/torpedo/pkg/aetosutil.(*Dashboard).VerifyFatal(0x7f8804b?, {0x702af20?, 0xc001634f70}, {0x0?, 0x0}, {0xc002018d20, 0xa0}) [2024-01-18T23:59:07.179Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 +0x143 [2024-01-18T23:59:07.179Z] github.com/portworx/torpedo/tests/backup.glob..func26.2.4.1({0xc0019aec40, 0x36}, {0xc0014de230, 0x41}, {0xc001c1a990, 0x25}) [2024-01-18T23:59:07.179Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_orphaned_test.go:189 +0x31d [2024-01-18T23:59:07.179Z] created by github.com/portworx/torpedo/tests/backup.glob..func26.2.4 [2024-01-18T23:59:07.179Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_orphaned_test.go:199 +0x3ae [2024-01-18T23:59:07.179Z] ------------------------------ [2024-01-18T23:59:07.179Z] STEP: schedule applications [2024-01-18T23:59:07.179Z] STEP: Validate applications [2024-01-18T23:59:07.179Z] STEP: validate applications [2024-01-18T23:59:07.179Z] STEP: validate mysql-ibm app's volumes [2024-01-18T23:59:07.179Z] STEP: inspect mysql-ibm app's volumes [2024-01-18T23:59:07.179Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-18T23:59:07.179Z] STEP: get mysql-ibm app's volume: pvc-605d74f5-62c2-461e-96c7-129b0085b35a inspected by the volume driver [2024-01-18T23:59:07.179Z] STEP: wait for mysql-ibm app to start running [2024-01-18T23:59:07.179Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-18T23:59:07.179Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-18T23:59:07.180Z] STEP: Validate Px pod restart count [2024-01-18T23:59:07.180Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-18T23:59:07.180Z] STEP: Create 3 users with px-backup-infra.admin role [2024-01-18T23:59:07.180Z] STEP: Create cloud credential and backup location from the user testuser3-1705621356 [2024-01-18T23:59:07.180Z] STEP: Create cloud credential and backup location from the user testuser2-1705621356 [2024-01-18T23:59:07.180Z] STEP: Create cloud credential and backup location from the user testuser1-1705621356 [2024-01-18T23:59:07.180Z] STEP: Create source and destination cluster from the user testuser3-1705621356 [2024-01-18T23:59:07.180Z] STEP: Take backup of applications from the user testuser3-1705621356 [2024-01-18T23:59:07.180Z] STEP: Create source and destination cluster from the user testuser1-1705621356 [2024-01-18T23:59:07.180Z] STEP: Take backup of applications from the user testuser1-1705621356 [2024-01-18T23:59:07.180Z] STEP: Create source and destination cluster from the user testuser2-1705621356 [2024-01-18T23:59:07.180Z] STEP: Take backup of applications from the user testuser2-1705621356 [2024-01-18T23:59:07.180Z] STEP: Create schedule policy from the user testuser3-1705621356 [2024-01-18T23:59:07.180Z] STEP: Take schedule backup of applications from the user testuser3-1705621356 [2024-01-18T23:59:07.180Z] STEP: Create schedule policy from the user testuser1-1705621356 [2024-01-18T23:59:07.180Z] STEP: Take schedule backup of applications from the user testuser1-1705621356 [2024-01-18T23:59:07.180Z] STEP: Create schedule policy from the user testuser2-1705621356 [2024-01-18T23:59:07.180Z] STEP: Take schedule backup of applications from the user testuser2-1705621356 [2024-01-18T23:59:07.180Z] STEP: Take restore of backups from the user testuser3-1705621356 [2024-01-18T23:59:07.180Z] STEP: Verify backups of the user testuser3-1705621356 from the admin [2024-01-18T23:59:07.180Z] STEP: Verify backup schedules of the user testuser3-1705621356 from the admin [2024-01-18T23:59:07.180Z] STEP: Verify restores of the user testuser3-1705621356 from the admin [2024-01-18T23:59:07.180Z] STEP: Take restore of backups from the user testuser1-1705621356 [2024-01-18T23:59:07.180Z] STEP: Verify backups of the user testuser1-1705621356 from the admin [2024-01-18T23:59:07.180Z] STEP: Verify backup schedules of the user testuser1-1705621356 from the admin [2024-01-18T23:59:07.180Z] STEP: Verify restores of the user testuser1-1705621356 from the admin [2024-01-18T23:59:07.180Z] STEP: Take restore of backups from the user testuser2-1705621356 [2024-01-18T23:59:07.180Z] STEP: Verify backups of the user testuser2-1705621356 from the admin [2024-01-18T23:59:07.180Z] STEP: Verify backup schedules of the user testuser2-1705621356 from the admin [2024-01-18T23:59:07.180Z] STEP: Verify restores of the user testuser2-1705621356 from the admin [2024-01-18T23:59:07.180Z] STEP: Delete user testuser2-1705621356 schedule backups, backup schedule and schedule policy from the admin [2024-01-18T23:59:07.180Z] STEP: Delete user testuser1-1705621356 schedule backups, backup schedule and schedule policy from the admin [2024-01-18T23:59:07.180Z] STEP: Delete user testuser3-1705621356 schedule backups, backup schedule and schedule policy from the admin [2024-01-18T23:59:07.180Z] STEP: Delete user testuser2-1705621356 backups from the admin [2024-01-18T23:59:07.180Z] STEP: Delete user testuser3-1705621356 backups from the admin [2024-01-18T23:59:07.180Z] STEP: Delete user testuser1-1705621356 backups from the admin [2024-01-18T23:59:07.180Z] STEP: Wait for the backups and backup schedule to be deleted [2024-01-18T23:59:07.180Z] STEP: Wait for the backups and backup schedule to be deleted [2024-01-18T23:59:07.180Z] STEP: Wait for the backups and backup schedule to be deleted [2024-01-18T23:59:07.180Z] STEP: Delete user testuser2-1705621356 restores from the admin [2024-01-18T23:59:07.180Z] STEP: Delete user testuser2-1705621356 source and destination cluster from the admin [2024-01-18T23:59:07.180Z] STEP: Delete user testuser3-1705621356 restores from the admin [2024-01-18T23:59:07.180Z] STEP: Delete user testuser3-1705621356 source and destination cluster from the admin [2024-01-18T23:59:07.180Z] STEP: Delete user testuser1-1705621356 restores from the admin [2024-01-18T23:59:07.180Z] STEP: Delete user testuser1-1705621356 source and destination cluster from the admin [2024-01-18T23:59:07.180Z] STEP: destroy apps [2024-01-18T23:59:07.180Z] STEP: destroy the mysql-ibm app's volumes [2024-01-18T23:59:07.180Z] STEP: start destroying mysql-ibm app [2024-01-18T23:59:07.181Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-18T23:59:07.181Z] SS2024-01-18 23:59:07 +0000:[INFO] [{SharedBackupDelete}] Log Dir: /testresults//SharedBackupDelete.log [2024-01-18T23:59:07.181Z] INFO[2024-01-18 23:59:07] --------Test Start------ [2024-01-18T23:59:07.181Z] INFO[2024-01-18 23:59:07] #Test: SharedBackupDelete [2024-01-18T23:59:07.181Z] INFO[2024-01-18 23:59:07] #Description: Share backup with multiple users and delete the backup [2024-01-18T23:59:07.181Z] INFO[2024-01-18 23:59:07] ------------------------ [2024-01-18T23:59:07.181Z] 2024-01-18 23:59:07 +0000:[INFO] [{SharedBackupDelete}] Deploy applications [2024-01-18T23:59:07.181Z] 2024-01-18 23:59:07 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-18T23:59:07.181Z] 2024-01-18 23:59:07 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-18T23:59:07.181Z] 2024-01-18 23:59:07 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-18T23:59:07.181Z] 2024-01-18 23:59:07 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-18T23:59:07.437Z] 2024-01-18 23:59:07 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-18T23:59:07.437Z] 2024-01-18 23:59:07 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-18T23:59:07.438Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T23:59:07.438Z] 2024-01-18 23:59:07 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-18T23:59:07.438Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-18T23:59:07.438Z] 2024-01-18 23:59:07 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-18T23:59:07.438Z] 2024-01-18 23:59:07 +0000:[INFO] [{SharedBackupDelete}] Validate applications [2024-01-18T23:59:07.438Z] 2024-01-18 23:59:07 +0000:[INFO] [{SharedBackupDelete}] Validate applications [2024-01-18T23:59:07.438Z] 2024-01-18 23:59:07 +0000:[INFO] [{SharedBackupDelete}] Validating mysql-ibm app [2024-01-18T23:59:07.438Z] 2024-01-18 23:59:07 +0000:[INFO] [{SharedBackupDelete}] Validating mysql-ibm app's volumes [2024-01-18T23:59:07.438Z] 2024-01-18 23:59:07 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-18T23:59:07.438Z] 2024/01/18 23:59:07 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T23:59:18.165Z] 2024/01/18 23:59:17 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-18T23:59:28.104Z] 2024-01-18 23:59:27 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-82946-01-18-23h59m07s [2024-01-18T23:59:28.104Z] 2024-01-18 23:59:27 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-18T23:59:28.104Z] 2024-01-18 23:59:27 +0000:[WARNING] [{SharedBackupDelete}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-18T23:59:28.104Z] 2024-01-18 23:59:27 +0000:[INFO] [{SharedBackupDelete}] wait for mysql-ibm app to start running [2024-01-18T23:59:28.105Z] 2024-01-18 23:59:27 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-18T23:59:28.105Z] 2024/01/18 23:59:27 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:59:28.105Z] pod name:mysql-7d766cd65c-4z8q6 namespace:mysql-ibm-pxb-0-82946-01-18-23h59m07s running:false ready:false node:10.240.64.8 [2024-01-18T23:59:28.105Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:59:38.046Z] 2024/01/18 23:59:37 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:59:38.046Z] pod name:mysql-7d766cd65c-4z8q6 namespace:mysql-ibm-pxb-0-82946-01-18-23h59m07s running:true ready:false node:10.240.64.8 [2024-01-18T23:59:38.046Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:59:47.988Z] 2024/01/18 23:59:47 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:59:47.988Z] pod name:mysql-7d766cd65c-4z8q6 namespace:mysql-ibm-pxb-0-82946-01-18-23h59m07s running:true ready:false node:10.240.64.8 [2024-01-18T23:59:47.988Z] }, Next try in [10s], timeout [10m0s] [2024-01-18T23:59:57.988Z] 2024/01/18 23:59:57 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-18T23:59:57.988Z] pod name:mysql-7d766cd65c-4z8q6 namespace:mysql-ibm-pxb-0-82946-01-18-23h59m07s running:true ready:false node:10.240.64.8 [2024-01-18T23:59:57.988Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:00:08.046Z] 2024/01/19 00:00:07 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:00:08.046Z] pod name:mysql-7d766cd65c-4z8q6 namespace:mysql-ibm-pxb-0-82946-01-18-23h59m07s running:true ready:false node:10.240.64.8 [2024-01-19T00:00:08.046Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:00:20.267Z] 2024/01/19 00:00:18 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:00:20.267Z] pod name:mysql-7d766cd65c-4z8q6 namespace:mysql-ibm-pxb-0-82946-01-18-23h59m07s running:true ready:false node:10.240.64.8 [2024-01-19T00:00:20.267Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:00:28.478Z] 2024/01/19 00:00:28 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:00:28.478Z] pod name:mysql-7d766cd65c-4z8q6 namespace:mysql-ibm-pxb-0-82946-01-18-23h59m07s running:true ready:false node:10.240.64.8 [2024-01-19T00:00:28.478Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:00:38.435Z] 2024-01-19 00:00:38 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T00:00:38.435Z] 2024-01-19 00:00:38 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T00:00:38.692Z] 2024-01-19 00:00:38 +0000:[INFO] [{SharedBackupDelete}] validate if mysql-ibm app's volumes are setup [2024-01-19T00:00:38.692Z] 2024-01-19 00:00:38 +0000:[WARNING] [{SharedBackupDelete}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:00:38.692Z] 2024-01-19 00:00:38 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:00:38.692Z] 2024-01-19 00:00:38 +0000:[INFO] [{SharedBackupDelete}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T00:00:38.692Z] 2024-01-19 00:00:38 +0000:[WARNING] [{SharedBackupDelete}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T00:00:38.692Z] 2024-01-19 00:00:38 +0000:[INFO] [{SharedBackupDelete}] Creating 10 users [2024-01-19T00:00:40.065Z] I0119 00:00:39.621040 14 request.go:682] Waited for 1.061245064s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-19T00:00:44.234Z] 2024-01-19 00:00:43 +0000:[INFO] [{SharedBackupDelete}] [backup.AddUser:#728] - Response for user [testuser3-1705622438] creation - [2024-01-19T00:00:44.234Z] 2024-01-19 00:00:43 +0000:[INFO] [{SharedBackupDelete}] [backup.AddUser:#733] - User [testuser3-1705622438] created [2024-01-19T00:00:44.234Z] INFO[2024-01-19 00:00:43] Verifying : Description : Creating user - testuser3-1705622438 [2024-01-19T00:00:44.234Z] INFO[2024-01-19 00:00:43] Actual:true, Expected: true [2024-01-19T00:00:44.234Z] 2024-01-19 00:00:43 +0000:[INFO] [{SharedBackupDelete}] [backup.AddUser:#728] - Response for user [testuser10-1705622438] creation - [2024-01-19T00:00:44.234Z] 2024-01-19 00:00:43 +0000:[INFO] [{SharedBackupDelete}] [backup.AddUser:#733] - User [testuser10-1705622438] created [2024-01-19T00:00:44.234Z] INFO[2024-01-19 00:00:43] Verifying : Description : Creating user - testuser10-1705622438 [2024-01-19T00:00:44.234Z] INFO[2024-01-19 00:00:43] Actual:true, Expected: true [2024-01-19T00:00:44.234Z] 2024-01-19 00:00:43 +0000:[INFO] [{SharedBackupDelete}] [backup.AddUser:#728] - Response for user [testuser4-1705622438] creation - [2024-01-19T00:00:44.234Z] 2024-01-19 00:00:43 +0000:[INFO] [{SharedBackupDelete}] [backup.AddUser:#733] - User [testuser4-1705622438] created [2024-01-19T00:00:44.234Z] INFO[2024-01-19 00:00:43] Verifying : Description : Creating user - testuser4-1705622438 [2024-01-19T00:00:44.234Z] INFO[2024-01-19 00:00:43] Actual:true, Expected: true [2024-01-19T00:00:44.234Z] 2024-01-19 00:00:43 +0000:[INFO] [{SharedBackupDelete}] [backup.AddUser:#728] - Response for user [testuser5-1705622438] creation - [2024-01-19T00:00:44.234Z] 2024-01-19 00:00:43 +0000:[INFO] [{SharedBackupDelete}] [backup.AddUser:#733] - User [testuser5-1705622438] created [2024-01-19T00:00:44.234Z] INFO[2024-01-19 00:00:43] Verifying : Description : Creating user - testuser5-1705622438 [2024-01-19T00:00:44.234Z] INFO[2024-01-19 00:00:43] Actual:true, Expected: true [2024-01-19T00:00:44.234Z] 2024-01-19 00:00:44 +0000:[INFO] [{SharedBackupDelete}] [backup.AddUser:#728] - Response for user [testuser2-1705622438] creation - [2024-01-19T00:00:44.234Z] 2024-01-19 00:00:44 +0000:[INFO] [{SharedBackupDelete}] [backup.AddUser:#733] - User [testuser2-1705622438] created [2024-01-19T00:00:44.234Z] INFO[2024-01-19 00:00:44] Verifying : Description : Creating user - testuser2-1705622438 [2024-01-19T00:00:44.234Z] INFO[2024-01-19 00:00:44] Actual:true, Expected: true [2024-01-19T00:00:44.491Z] 2024-01-19 00:00:44 +0000:[INFO] [{SharedBackupDelete}] [backup.AddUser:#728] - Response for user [testuser6-1705622438] creation - [2024-01-19T00:00:44.491Z] 2024-01-19 00:00:44 +0000:[INFO] [{SharedBackupDelete}] [backup.AddUser:#733] - User [testuser6-1705622438] created [2024-01-19T00:00:44.491Z] INFO[2024-01-19 00:00:44] Verifying : Description : Creating user - testuser6-1705622438 [2024-01-19T00:00:44.491Z] INFO[2024-01-19 00:00:44] Actual:true, Expected: true [2024-01-19T00:00:44.747Z] 2024-01-19 00:00:44 +0000:[INFO] [{SharedBackupDelete}] [backup.AddUser:#728] - Response for user [testuser1-1705622438] creation - [2024-01-19T00:00:44.747Z] 2024-01-19 00:00:44 +0000:[INFO] [{SharedBackupDelete}] [backup.AddUser:#733] - User [testuser1-1705622438] created [2024-01-19T00:00:44.747Z] INFO[2024-01-19 00:00:44] Verifying : Description : Creating user - testuser1-1705622438 [2024-01-19T00:00:44.747Z] INFO[2024-01-19 00:00:44] Actual:true, Expected: true [2024-01-19T00:00:45.003Z] 2024-01-19 00:00:44 +0000:[INFO] [{SharedBackupDelete}] [backup.AddUser:#728] - Response for user [testuser7-1705622438] creation - [2024-01-19T00:00:45.003Z] 2024-01-19 00:00:44 +0000:[INFO] [{SharedBackupDelete}] [backup.AddUser:#733] - User [testuser7-1705622438] created [2024-01-19T00:00:45.003Z] INFO[2024-01-19 00:00:44] Verifying : Description : Creating user - testuser7-1705622438 [2024-01-19T00:00:45.003Z] INFO[2024-01-19 00:00:44] Actual:true, Expected: true [2024-01-19T00:00:45.003Z] 2024-01-19 00:00:44 +0000:[INFO] [{SharedBackupDelete}] [backup.AddUser:#728] - Response for user [testuser8-1705622438] creation - [2024-01-19T00:00:45.003Z] 2024-01-19 00:00:44 +0000:[INFO] [{SharedBackupDelete}] [backup.AddUser:#733] - User [testuser8-1705622438] created [2024-01-19T00:00:45.003Z] INFO[2024-01-19 00:00:44] Verifying : Description : Creating user - testuser8-1705622438 [2024-01-19T00:00:45.003Z] INFO[2024-01-19 00:00:44] Actual:true, Expected: true [2024-01-19T00:00:45.258Z] 2024-01-19 00:00:45 +0000:[INFO] [{SharedBackupDelete}] [backup.AddUser:#728] - Response for user [testuser9-1705622438] creation - [2024-01-19T00:00:45.259Z] 2024-01-19 00:00:45 +0000:[INFO] [{SharedBackupDelete}] [backup.AddUser:#733] - User [testuser9-1705622438] created [2024-01-19T00:00:45.259Z] INFO[2024-01-19 00:00:45] Verifying : Description : Creating user - testuser9-1705622438 [2024-01-19T00:00:45.259Z] INFO[2024-01-19 00:00:45] Actual:true, Expected: true [2024-01-19T00:00:45.259Z] 2024-01-19 00:00:45 +0000:[INFO] [{SharedBackupDelete}] Creating cloud credentials and backup location [2024-01-19T00:00:46.631Z] 2024-01-19 00:00:46 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-1705622446] for org [default] with [aws] as provider [2024-01-19T00:00:46.631Z] 2024-01-19 00:00:46 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T00:00:46.631Z] 2024-01-19 00:00:46 +0000:[INFO] [{SharedBackupDelete}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-1705622446] is present [2024-01-19T00:00:46.631Z] INFO[2024-01-19 00:00:46] Verifying : Description : Verifying creation of cloud credential named [autogenerated-cred-1705622446] for org [default] with [aws] as provider [2024-01-19T00:00:46.631Z] INFO[2024-01-19 00:00:46] Actual:true, Expected: true [2024-01-19T00:00:46.631Z] 2024-01-19 00:00:46 +0000:[INFO] [{SharedBackupDelete}] Created Cloud Credentials with name - autogenerated-cred-1705622446 [2024-01-19T00:00:50.795Z] INFO[2024-01-19 00:00:49] Verifying : Description : Creating backup location autogenerated-backup-location-1705622446 [2024-01-19T00:00:50.795Z] INFO[2024-01-19 00:00:49] Actual:true, Expected: true [2024-01-19T00:00:50.795Z] 2024-01-19 00:00:49 +0000:[INFO] [{SharedBackupDelete}] Created Backup Location with name - autogenerated-backup-location-1705622446 [2024-01-19T00:00:50.795Z] 2024-01-19 00:00:49 +0000:[INFO] [{SharedBackupDelete}] Registering Source and Destination clusters and verifying the status [2024-01-19T00:00:50.795Z] INFO[2024-01-19 00:00:50] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T00:00:50.795Z] INFO[2024-01-19 00:00:50] Actual:true, Expected: true [2024-01-19T00:00:50.795Z] 2024-01-19 00:00:50 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T00:00:50.795Z] 2024-01-19 00:00:50 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T00:00:50.795Z] 2024-01-19 00:00:50 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T00:00:50.795Z] 2024-01-19 00:00:50 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T00:00:50.795Z] 2024-01-19 00:00:50 +0000:[INFO] [{SharedBackupDelete}] Create cluster [source-cluster] in org [default] [2024-01-19T00:00:50.796Z] 2024-01-19 00:00:50 +0000:[INFO] [{SharedBackupDelete}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T00:00:50.796Z] 2024-01-19 00:00:50 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T00:00:50.796Z] 2024-01-19 00:00:50 +0000:[INFO] [{SharedBackupDelete}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:00:50.796Z] 2024-01-19 00:00:50 +0000:[INFO] [{SharedBackupDelete}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T00:00:50.796Z] 2024-01-19 00:00:50 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T00:00:50.796Z] 2024-01-19 00:00:50 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-pntez] for IBM [2024-01-19T00:00:50.796Z] 2024-01-19 00:00:50 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-pntez] for org [default] with [ibm] as provider [2024-01-19T00:00:50.796Z] 2024-01-19 00:00:50 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:00:50.796Z] 2024-01-19 00:00:50 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:00:50.796Z] 2024-01-19 00:00:50 +0000:[INFO] [{SharedBackupDelete}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-pntez] is present [2024-01-19T00:00:50.796Z] 2024-01-19 00:00:50 +0000:[INFO] [{SharedBackupDelete}] Create cluster [source-cluster] in org [default] [2024-01-19T00:00:51.052Z] 2024-01-19 00:00:50 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-zrovm] for IBM [2024-01-19T00:00:51.052Z] 2024-01-19 00:00:50 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-zrovm] for org [default] with [ibm] as provider [2024-01-19T00:00:51.052Z] 2024-01-19 00:00:50 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:00:51.052Z] 2024-01-19 00:00:50 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:00:51.314Z] 2024-01-19 00:00:51 +0000:[INFO] [{SharedBackupDelete}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-zrovm] is present [2024-01-19T00:00:51.314Z] 2024-01-19 00:00:51 +0000:[INFO] [{SharedBackupDelete}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:00:51.314Z] INFO[2024-01-19 00:00:51] Verifying : Description : Creating source and destination cluster [2024-01-19T00:00:51.314Z] INFO[2024-01-19 00:00:51] Actual:true, Expected: true [2024-01-19T00:00:51.314Z] INFO[2024-01-19 00:00:51] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T00:00:51.314Z] INFO[2024-01-19 00:00:51] Actual:Online, Expected: Online [2024-01-19T00:00:51.314Z] INFO[2024-01-19 00:00:51] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T00:00:51.314Z] INFO[2024-01-19 00:00:51] Actual:true, Expected: true [2024-01-19T00:00:51.314Z] 2024-01-19 00:00:51 +0000:[INFO] [{SharedBackupDelete}] Taking backup of applications [2024-01-19T00:01:03.513Z] 2024-01-19 00:01:01 +0000:[INFO] [{SharedBackupDelete}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T00:01:03.513Z] 2024-01-19 00:01:01 +0000:[INFO] [{SharedBackupDelete}] Volume snapshot classes found in the cluster - [2024-01-19T00:01:03.513Z] 2024-01-19 00:01:01 +0000:[INFO] [{SharedBackupDelete}] ibmc-vpcblock-snapshot [2024-01-19T00:01:03.513Z] 2024-01-19 00:01:01 +0000:[INFO] [{SharedBackupDelete}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T00:01:03.513Z] 2024/01/19 00:01:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622461] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T00:01:11.633Z] 2024-01-19 00:01:11 +0000:[INFO] [{SharedBackupDelete}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T00:01:11.633Z] 2024-01-19 00:01:11 +0000:[INFO] [{SharedBackupDelete}] Volume snapshot classes found in the cluster - [2024-01-19T00:01:11.633Z] 2024-01-19 00:01:11 +0000:[INFO] [{SharedBackupDelete}] ibmc-vpcblock-snapshot [2024-01-19T00:01:11.633Z] 2024-01-19 00:01:11 +0000:[INFO] [{SharedBackupDelete}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T00:01:12.195Z] 2024/01/19 00:01:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622471] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T00:01:22.136Z] 2024-01-19 00:01:21 +0000:[INFO] [{SharedBackupDelete}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T00:01:22.136Z] 2024-01-19 00:01:21 +0000:[INFO] [{SharedBackupDelete}] Volume snapshot classes found in the cluster - [2024-01-19T00:01:22.136Z] 2024-01-19 00:01:21 +0000:[INFO] [{SharedBackupDelete}] ibmc-vpcblock-snapshot [2024-01-19T00:01:22.136Z] 2024-01-19 00:01:21 +0000:[INFO] [{SharedBackupDelete}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T00:01:22.136Z] 2024/01/19 00:01:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622481] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T00:01:32.098Z] 2024-01-19 00:01:31 +0000:[INFO] [{SharedBackupDelete}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T00:01:32.098Z] 2024-01-19 00:01:31 +0000:[INFO] [{SharedBackupDelete}] Volume snapshot classes found in the cluster - [2024-01-19T00:01:32.098Z] 2024-01-19 00:01:31 +0000:[INFO] [{SharedBackupDelete}] ibmc-vpcblock-snapshot [2024-01-19T00:01:32.098Z] 2024-01-19 00:01:31 +0000:[INFO] [{SharedBackupDelete}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T00:01:32.354Z] 2024/01/19 00:01:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622461] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:01:32.354Z] 2024/01/19 00:01:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622491] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T00:01:42.294Z] 2024-01-19 00:01:41 +0000:[INFO] [{SharedBackupDelete}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T00:01:42.294Z] 2024-01-19 00:01:41 +0000:[INFO] [{SharedBackupDelete}] Volume snapshot classes found in the cluster - [2024-01-19T00:01:42.294Z] 2024-01-19 00:01:41 +0000:[INFO] [{SharedBackupDelete}] ibmc-vpcblock-snapshot [2024-01-19T00:01:42.294Z] 2024-01-19 00:01:41 +0000:[INFO] [{SharedBackupDelete}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T00:01:42.294Z] 2024/01/19 00:01:42 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622501] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T00:01:42.294Z] 2024/01/19 00:01:42 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622471] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:01:52.229Z] 2024-01-19 00:01:51 +0000:[INFO] [{SharedBackupDelete}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T00:01:52.229Z] 2024-01-19 00:01:51 +0000:[INFO] [{SharedBackupDelete}] Volume snapshot classes found in the cluster - [2024-01-19T00:01:52.229Z] 2024-01-19 00:01:51 +0000:[INFO] [{SharedBackupDelete}] ibmc-vpcblock-snapshot [2024-01-19T00:01:52.229Z] 2024-01-19 00:01:51 +0000:[INFO] [{SharedBackupDelete}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T00:01:52.229Z] 2024/01/19 00:01:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622481] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:01:52.799Z] 2024/01/19 00:01:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622511] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T00:02:02.735Z] 2024-01-19 00:02:01 +0000:[INFO] [{SharedBackupDelete}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T00:02:02.735Z] 2024-01-19 00:02:01 +0000:[INFO] [{SharedBackupDelete}] Volume snapshot classes found in the cluster - [2024-01-19T00:02:02.735Z] 2024-01-19 00:02:01 +0000:[INFO] [{SharedBackupDelete}] ibmc-vpcblock-snapshot [2024-01-19T00:02:02.736Z] 2024-01-19 00:02:01 +0000:[INFO] [{SharedBackupDelete}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T00:02:02.736Z] 2024/01/19 00:02:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622461] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:02:02.736Z] 2024/01/19 00:02:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622491] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:02:02.736Z] 2024/01/19 00:02:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622521] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T00:02:12.673Z] 2024-01-19 00:02:11 +0000:[INFO] [{SharedBackupDelete}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T00:02:12.673Z] 2024-01-19 00:02:11 +0000:[INFO] [{SharedBackupDelete}] Volume snapshot classes found in the cluster - [2024-01-19T00:02:12.673Z] 2024-01-19 00:02:11 +0000:[INFO] [{SharedBackupDelete}] ibmc-vpcblock-snapshot [2024-01-19T00:02:12.673Z] 2024-01-19 00:02:11 +0000:[INFO] [{SharedBackupDelete}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T00:02:12.673Z] 2024/01/19 00:02:11 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622531] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T00:02:12.673Z] 2024/01/19 00:02:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622501] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:02:12.673Z] 2024/01/19 00:02:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622471] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:02:22.609Z] 2024-01-19 00:02:21 +0000:[INFO] [{SharedBackupDelete}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T00:02:22.609Z] 2024-01-19 00:02:21 +0000:[INFO] [{SharedBackupDelete}] Volume snapshot classes found in the cluster - [2024-01-19T00:02:22.609Z] 2024-01-19 00:02:21 +0000:[INFO] [{SharedBackupDelete}] ibmc-vpcblock-snapshot [2024-01-19T00:02:22.609Z] 2024-01-19 00:02:21 +0000:[INFO] [{SharedBackupDelete}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T00:02:22.609Z] 2024/01/19 00:02:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622481] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:02:22.609Z] 2024/01/19 00:02:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622541] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T00:02:22.869Z] 2024/01/19 00:02:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622511] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:02:32.809Z] 2024-01-19 00:02:31 +0000:[INFO] [{SharedBackupDelete}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T00:02:32.809Z] 2024-01-19 00:02:31 +0000:[INFO] [{SharedBackupDelete}] Volume snapshot classes found in the cluster - [2024-01-19T00:02:32.809Z] 2024-01-19 00:02:31 +0000:[INFO] [{SharedBackupDelete}] ibmc-vpcblock-snapshot [2024-01-19T00:02:32.809Z] 2024-01-19 00:02:31 +0000:[INFO] [{SharedBackupDelete}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T00:02:32.809Z] 2024/01/19 00:02:31 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622551] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T00:02:32.809Z] 2024/01/19 00:02:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622461] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:02:32.809Z] 2024/01/19 00:02:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622491] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:02:32.809Z] 2024/01/19 00:02:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622521] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:02:42.751Z] 2024/01/19 00:02:41 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622531] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:02:42.751Z] 2024/01/19 00:02:42 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622501] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:02:42.751Z] 2024/01/19 00:02:42 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622471] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:02:52.707Z] 2024/01/19 00:02:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622481] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:02:52.707Z] 2024/01/19 00:02:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622541] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:02:52.965Z] 2024/01/19 00:02:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622511] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:03:02.907Z] 2024/01/19 00:03:01 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622551] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:03:02.907Z] 2024/01/19 00:03:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622461] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:03:02.907Z] 2024/01/19 00:03:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622491] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:03:02.907Z] 2024/01/19 00:03:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622521] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:03:12.850Z] 2024/01/19 00:03:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622531] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:03:12.850Z] 2024/01/19 00:03:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622501] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:03:12.850Z] 2024/01/19 00:03:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622471] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:03:22.789Z] 2024/01/19 00:03:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622481] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:03:22.789Z] 2024/01/19 00:03:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622541] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:03:23.047Z] 2024/01/19 00:03:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622511] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:03:32.991Z] 2024/01/19 00:03:31 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622551] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:03:32.991Z] 2024/01/19 00:03:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622461] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:03:32.991Z] 2024/01/19 00:03:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622491] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:03:32.991Z] 2024/01/19 00:03:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622521] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:03:42.926Z] 2024/01/19 00:03:42 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622531] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:03:42.926Z] 2024/01/19 00:03:42 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622501] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:03:42.926Z] 2024/01/19 00:03:42 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622471] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:03:52.865Z] 2024/01/19 00:03:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622481] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:03:52.865Z] 2024/01/19 00:03:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622541] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:03:52.865Z] 2024/01/19 00:03:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622511] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:04:02.806Z] 2024/01/19 00:04:01 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622551] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:04:02.806Z] 2024/01/19 00:04:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622461] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:04:02.806Z] 2024/01/19 00:04:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622491] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:04:02.806Z] 2024/01/19 00:04:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622521] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:04:12.802Z] 2024/01/19 00:04:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622531] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:04:12.802Z] 2024/01/19 00:04:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622501] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:04:12.802Z] 2024/01/19 00:04:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622471] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:04:22.740Z] 2024/01/19 00:04:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622481] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:04:22.741Z] 2024/01/19 00:04:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622541] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:04:22.996Z] 2024/01/19 00:04:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622511] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:04:32.948Z] 2024/01/19 00:04:31 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622551] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:04:32.948Z] 2024/01/19 00:04:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622461] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:04:32.949Z] 2024/01/19 00:04:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622491] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:04:32.949Z] 2024/01/19 00:04:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622521] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:04:38.191Z] INFO[2024-01-19 00:04:37] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T00:04:38.191Z] INFO[2024-01-19 00:04:37] sched stats: tick interval: 30637 < 1.2s (100%) [2024-01-19T00:04:38.191Z] INFO[2024-01-19 00:04:37] sched stats: tick process durations: 30636 < 500ms (100%) [2024-01-19T00:04:42.358Z] 2024/01/19 00:04:42 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622531] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:04:42.358Z] 2024/01/19 00:04:42 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622501] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:04:42.358Z] 2024/01/19 00:04:42 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622471] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:04:52.302Z] 2024/01/19 00:04:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622481] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:04:52.302Z] 2024/01/19 00:04:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622541] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:04:52.917Z] 2024/01/19 00:04:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622511] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:05:02.854Z] 2024/01/19 00:05:01 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622551] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:05:02.854Z] 2024/01/19 00:05:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622461] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:05:02.854Z] 2024/01/19 00:05:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622491] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:05:02.854Z] 2024/01/19 00:05:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622521] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:05:12.796Z] 2024/01/19 00:05:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622531] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:05:12.797Z] 2024/01/19 00:05:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622501] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:05:12.797Z] 2024/01/19 00:05:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622471] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:05:22.734Z] 2024/01/19 00:05:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622481] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:05:22.734Z] 2024/01/19 00:05:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622541] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:05:22.992Z] 2024/01/19 00:05:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622511] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:05:32.943Z] 2024/01/19 00:05:31 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622551] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:05:32.944Z] 2024/01/19 00:05:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622461] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:05:32.944Z] 2024/01/19 00:05:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622491] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:05:32.944Z] 2024/01/19 00:05:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622521] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:05:42.888Z] 2024/01/19 00:05:42 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622531] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:05:42.888Z] 2024/01/19 00:05:42 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622501] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:05:42.889Z] 2024/01/19 00:05:42 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622471] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:05:52.831Z] 2024/01/19 00:05:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622481] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:05:52.831Z] 2024/01/19 00:05:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622541] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:05:53.087Z] 2024/01/19 00:05:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622511] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:06:03.030Z] 2024/01/19 00:06:01 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622551] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:06:03.030Z] 2024/01/19 00:06:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622461] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:06:03.030Z] 2024/01/19 00:06:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622491] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:06:03.030Z] 2024/01/19 00:06:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622521] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:06:12.978Z] 2024/01/19 00:06:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622531] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:06:12.978Z] 2024/01/19 00:06:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622501] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:06:12.978Z] 2024/01/19 00:06:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622471] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:06:22.949Z] 2024/01/19 00:06:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622541] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:06:22.949Z] 2024/01/19 00:06:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622481] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:06:22.949Z] 2024/01/19 00:06:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622511] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:06:32.891Z] 2024/01/19 00:06:31 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622551] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:06:32.891Z] 2024/01/19 00:06:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622461] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:06:32.891Z] 2024/01/19 00:06:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622491] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:06:32.891Z] 2024/01/19 00:06:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622521] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:06:42.835Z] 2024/01/19 00:06:42 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622531] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:06:42.835Z] 2024/01/19 00:06:42 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622501] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:06:42.835Z] 2024/01/19 00:06:42 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622471] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:06:52.783Z] 2024/01/19 00:06:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622541] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:06:52.783Z] 2024/01/19 00:06:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622481] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:06:53.039Z] 2024/01/19 00:06:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622511] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:07:02.976Z] 2024/01/19 00:07:01 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622551] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:07:02.976Z] 2024/01/19 00:07:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622461] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:07:02.976Z] 2024/01/19 00:07:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622491] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:07:02.976Z] 2024/01/19 00:07:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622521] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:07:12.916Z] 2024/01/19 00:07:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622531] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:07:12.916Z] 2024/01/19 00:07:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622501] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:07:12.916Z] 2024/01/19 00:07:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622471] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:07:22.854Z] 2024/01/19 00:07:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622541] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:07:22.854Z] 2024/01/19 00:07:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622481] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:07:23.112Z] 2024/01/19 00:07:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622511] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:07:33.047Z] 2024/01/19 00:07:31 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622551] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:07:33.047Z] 2024-01-19 00:07:32 +0000:[INFO] [{SharedBackupDelete}] [backup.CreateBackup:#313] - Backup [tp-backup-1705622461] created successfully [2024-01-19T00:07:33.047Z] 2024-01-19 00:07:32 +0000:[INFO] [{SharedBackupDelete}] Validating backup [tp-backup-1705622461] in org [default] [2024-01-19T00:07:33.047Z] 2024-01-19 00:07:32 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705622461] [2024-01-19T00:07:33.047Z] 2024-01-19 00:07:32 +0000:[INFO] [{SharedBackupDelete}] [backup.CreateBackup:#313] - Backup [tp-backup-1705622491] created successfully [2024-01-19T00:07:33.047Z] 2024-01-19 00:07:32 +0000:[INFO] [{SharedBackupDelete}] Validating backup [tp-backup-1705622491] in org [default] [2024-01-19T00:07:33.047Z] 2024-01-19 00:07:32 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705622491] [2024-01-19T00:07:33.048Z] 2024-01-19 00:07:32 +0000:[INFO] [{SharedBackupDelete}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] in backup [tp-backup-1705622461] [2024-01-19T00:07:33.048Z] 2024-01-19 00:07:32 +0000:[INFO] [{SharedBackupDelete}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] in backup [tp-backup-1705622461] [2024-01-19T00:07:33.048Z] 2024-01-19 00:07:32 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] [2024-01-19T00:07:33.048Z] 2024-01-19 00:07:32 +0000:[INFO] [{SharedBackupDelete}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] in backup [tp-backup-1705622491] [2024-01-19T00:07:33.048Z] 2024-01-19 00:07:32 +0000:[INFO] [{SharedBackupDelete}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] in backup [tp-backup-1705622491] [2024-01-19T00:07:33.048Z] 2024-01-19 00:07:32 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] [2024-01-19T00:07:33.048Z] 2024-01-19 00:07:32 +0000:[WARNING] [{SharedBackupDelete}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:07:33.048Z] 2024-01-19 00:07:32 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:07:33.048Z] 2024-01-19 00:07:32 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82946-01-18-23h59m07s] are [[mysql-data]] [2024-01-19T00:07:33.049Z] 2024/01/19 00:07:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622521] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:07:33.049Z] 2024-01-19 00:07:32 +0000:[WARNING] [{SharedBackupDelete}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:07:33.049Z] 2024-01-19 00:07:32 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:07:33.049Z] 2024-01-19 00:07:32 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82946-01-18-23h59m07s] are [[mysql-data]] [2024-01-19T00:07:33.049Z] 2024-01-19 00:07:32 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T00:07:33.049Z] 2024-01-19 00:07:32 +0000:[INFO] [{SharedBackupDelete}] Validating CR cleanup [2024-01-19T00:07:33.049Z] 2024-01-19 00:07:32 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T00:07:33.049Z] 2024-01-19 00:07:32 +0000:[INFO] [{SharedBackupDelete}] Validating CR cleanup [2024-01-19T00:07:33.049Z] 2024-01-19 00:07:32 +0000:[INFO] [{SharedBackupDelete}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T00:07:33.049Z] 2024-01-19 00:07:32 +0000:[INFO] [{SharedBackupDelete}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T00:07:33.049Z] 2024-01-19 00:07:32 +0000:[INFO] [{SharedBackupDelete}] All CRs in [mysql-ibm-pxb-0-82946-01-18-23h59m07s] are [[tp-backup-1705622481-0fed22c tp-backup-1705622491-1419bfc tp-backup-1705622501-2f375b2 tp-backup-1705622511-6469100 tp-backup-1705622521-b87e233 tp-backup-1705622531-bdb0364 tp-backup-1705622541-786599f tp-backup-1705622551-88e113a]] [2024-01-19T00:07:33.049Z] INFO[2024-01-19 00:07:32] Verifying : Description : Creation and Validation of backup [tp-backup-1705622461] [2024-01-19T00:07:33.049Z] INFO[2024-01-19 00:07:32] Actual:true, Expected: true [2024-01-19T00:07:33.049Z] 2024-01-19 00:07:32 +0000:[INFO] [{SharedBackupDelete}] All CRs in [mysql-ibm-pxb-0-82946-01-18-23h59m07s] are [[tp-backup-1705622481-0fed22c tp-backup-1705622491-1419bfc tp-backup-1705622501-2f375b2 tp-backup-1705622511-6469100 tp-backup-1705622521-b87e233 tp-backup-1705622531-bdb0364 tp-backup-1705622541-786599f tp-backup-1705622551-88e113a]] [2024-01-19T00:07:33.049Z] 2024-01-19 00:07:32 +0000:[INFO] [{SharedBackupDelete}] CR found for [[tp-backup-1705622481-0fed22c tp-backup-1705622491-1419bfc tp-backup-1705622501-2f375b2 tp-backup-1705622511-6469100 tp-backup-1705622521-b87e233 tp-backup-1705622531-bdb0364 tp-backup-1705622541-786599f tp-backup-1705622551-88e113a]] under [mysql-ibm-pxb-0-82946-01-18-23h59m07s] namespace [2024-01-19T00:07:33.049Z] 2024/01/19 00:07:32 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-1705622491]}, Next try in [5s], timeout [5m0s] [2024-01-19T00:07:38.294Z] 2024-01-19 00:07:38 +0000:[INFO] [{SharedBackupDelete}] All CRs in [mysql-ibm-pxb-0-82946-01-18-23h59m07s] are [[tp-backup-1705622501-2f375b2 tp-backup-1705622511-6469100 tp-backup-1705622521-b87e233 tp-backup-1705622531-bdb0364 tp-backup-1705622541-786599f tp-backup-1705622551-88e113a]] [2024-01-19T00:07:38.294Z] INFO[2024-01-19 00:07:38] Verifying : Description : Creation and Validation of backup [tp-backup-1705622491] [2024-01-19T00:07:38.294Z] INFO[2024-01-19 00:07:38] Actual:true, Expected: true [2024-01-19T00:07:42.461Z] 2024/01/19 00:07:42 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622531] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:07:42.461Z] 2024/01/19 00:07:42 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622501] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:07:42.461Z] 2024-01-19 00:07:42 +0000:[INFO] [{SharedBackupDelete}] [backup.CreateBackup:#313] - Backup [tp-backup-1705622471] created successfully [2024-01-19T00:07:42.461Z] 2024-01-19 00:07:42 +0000:[INFO] [{SharedBackupDelete}] Validating backup [tp-backup-1705622471] in org [default] [2024-01-19T00:07:42.461Z] 2024-01-19 00:07:42 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705622471] [2024-01-19T00:07:42.461Z] 2024-01-19 00:07:42 +0000:[INFO] [{SharedBackupDelete}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] in backup [tp-backup-1705622471] [2024-01-19T00:07:42.461Z] 2024-01-19 00:07:42 +0000:[INFO] [{SharedBackupDelete}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] in backup [tp-backup-1705622471] [2024-01-19T00:07:42.461Z] 2024-01-19 00:07:42 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] [2024-01-19T00:07:42.717Z] 2024-01-19 00:07:42 +0000:[WARNING] [{SharedBackupDelete}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:07:42.717Z] 2024-01-19 00:07:42 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:07:42.717Z] 2024-01-19 00:07:42 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82946-01-18-23h59m07s] are [[mysql-data]] [2024-01-19T00:07:42.717Z] 2024-01-19 00:07:42 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T00:07:42.717Z] 2024-01-19 00:07:42 +0000:[INFO] [{SharedBackupDelete}] Validating CR cleanup [2024-01-19T00:07:42.717Z] 2024-01-19 00:07:42 +0000:[INFO] [{SharedBackupDelete}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T00:07:42.973Z] 2024-01-19 00:07:42 +0000:[INFO] [{SharedBackupDelete}] All CRs in [mysql-ibm-pxb-0-82946-01-18-23h59m07s] are [[tp-backup-1705622501-2f375b2 tp-backup-1705622511-6469100 tp-backup-1705622521-b87e233 tp-backup-1705622531-bdb0364 tp-backup-1705622541-786599f tp-backup-1705622551-88e113a]] [2024-01-19T00:07:42.973Z] INFO[2024-01-19 00:07:42] Verifying : Description : Creation and Validation of backup [tp-backup-1705622471] [2024-01-19T00:07:42.973Z] INFO[2024-01-19 00:07:42] Actual:true, Expected: true [2024-01-19T00:07:52.909Z] 2024/01/19 00:07:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622541] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:07:52.909Z] 2024-01-19 00:07:52 +0000:[INFO] [{SharedBackupDelete}] [backup.CreateBackup:#313] - Backup [tp-backup-1705622481] created successfully [2024-01-19T00:07:52.909Z] 2024-01-19 00:07:52 +0000:[INFO] [{SharedBackupDelete}] Validating backup [tp-backup-1705622481] in org [default] [2024-01-19T00:07:52.909Z] 2024-01-19 00:07:52 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705622481] [2024-01-19T00:07:52.909Z] 2024-01-19 00:07:52 +0000:[INFO] [{SharedBackupDelete}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] in backup [tp-backup-1705622481] [2024-01-19T00:07:52.909Z] 2024-01-19 00:07:52 +0000:[INFO] [{SharedBackupDelete}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] in backup [tp-backup-1705622481] [2024-01-19T00:07:52.909Z] 2024-01-19 00:07:52 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] [2024-01-19T00:07:52.910Z] 2024-01-19 00:07:52 +0000:[WARNING] [{SharedBackupDelete}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:07:52.910Z] 2024-01-19 00:07:52 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:07:52.910Z] 2024-01-19 00:07:52 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82946-01-18-23h59m07s] are [[mysql-data]] [2024-01-19T00:07:52.910Z] 2024-01-19 00:07:52 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T00:07:52.910Z] 2024-01-19 00:07:52 +0000:[INFO] [{SharedBackupDelete}] Validating CR cleanup [2024-01-19T00:07:52.910Z] 2024-01-19 00:07:52 +0000:[INFO] [{SharedBackupDelete}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T00:07:52.910Z] 2024-01-19 00:07:52 +0000:[INFO] [{SharedBackupDelete}] All CRs in [mysql-ibm-pxb-0-82946-01-18-23h59m07s] are [[tp-backup-1705622501-2f375b2 tp-backup-1705622511-6469100 tp-backup-1705622521-b87e233 tp-backup-1705622531-bdb0364 tp-backup-1705622541-786599f tp-backup-1705622551-88e113a]] [2024-01-19T00:07:52.910Z] INFO[2024-01-19 00:07:52] Verifying : Description : Creation and Validation of backup [tp-backup-1705622481] [2024-01-19T00:07:52.910Z] INFO[2024-01-19 00:07:52] Actual:true, Expected: true [2024-01-19T00:07:53.168Z] 2024/01/19 00:07:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622511] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:08:03.658Z] 2024/01/19 00:08:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622551] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:08:03.658Z] 2024-01-19 00:08:02 +0000:[INFO] [{SharedBackupDelete}] [backup.CreateBackup:#313] - Backup [tp-backup-1705622521] created successfully [2024-01-19T00:08:03.658Z] 2024-01-19 00:08:02 +0000:[INFO] [{SharedBackupDelete}] Validating backup [tp-backup-1705622521] in org [default] [2024-01-19T00:08:03.658Z] 2024-01-19 00:08:02 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705622521] [2024-01-19T00:08:03.658Z] 2024-01-19 00:08:02 +0000:[INFO] [{SharedBackupDelete}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] in backup [tp-backup-1705622521] [2024-01-19T00:08:03.658Z] 2024-01-19 00:08:02 +0000:[INFO] [{SharedBackupDelete}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] in backup [tp-backup-1705622521] [2024-01-19T00:08:03.658Z] 2024-01-19 00:08:02 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] [2024-01-19T00:08:03.658Z] 2024-01-19 00:08:02 +0000:[WARNING] [{SharedBackupDelete}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:08:03.658Z] 2024-01-19 00:08:02 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:08:03.658Z] 2024-01-19 00:08:02 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82946-01-18-23h59m07s] are [[mysql-data]] [2024-01-19T00:08:03.658Z] 2024-01-19 00:08:02 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T00:08:03.658Z] 2024-01-19 00:08:02 +0000:[INFO] [{SharedBackupDelete}] Validating CR cleanup [2024-01-19T00:08:03.658Z] 2024-01-19 00:08:02 +0000:[INFO] [{SharedBackupDelete}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T00:08:03.658Z] 2024-01-19 00:08:02 +0000:[INFO] [{SharedBackupDelete}] All CRs in [mysql-ibm-pxb-0-82946-01-18-23h59m07s] are [[tp-backup-1705622501-2f375b2 tp-backup-1705622511-6469100 tp-backup-1705622521-b87e233 tp-backup-1705622531-bdb0364 tp-backup-1705622541-786599f tp-backup-1705622551-88e113a]] [2024-01-19T00:08:03.658Z] 2024-01-19 00:08:02 +0000:[INFO] [{SharedBackupDelete}] CR found for [[tp-backup-1705622501-2f375b2 tp-backup-1705622511-6469100 tp-backup-1705622521-b87e233 tp-backup-1705622531-bdb0364 tp-backup-1705622541-786599f tp-backup-1705622551-88e113a]] under [mysql-ibm-pxb-0-82946-01-18-23h59m07s] namespace [2024-01-19T00:08:03.658Z] 2024/01/19 00:08:02 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-1705622521]}, Next try in [5s], timeout [5m0s] [2024-01-19T00:08:08.902Z] 2024-01-19 00:08:07 +0000:[INFO] [{SharedBackupDelete}] All CRs in [mysql-ibm-pxb-0-82946-01-18-23h59m07s] are [[tp-backup-1705622501-2f375b2 tp-backup-1705622531-bdb0364 tp-backup-1705622541-786599f tp-backup-1705622551-88e113a]] [2024-01-19T00:08:08.903Z] INFO[2024-01-19 00:08:07] Verifying : Description : Creation and Validation of backup [tp-backup-1705622521] [2024-01-19T00:08:08.903Z] INFO[2024-01-19 00:08:07] Actual:true, Expected: true [2024-01-19T00:08:13.071Z] 2024/01/19 00:08:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622531] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:08:13.071Z] 2024/01/19 00:08:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622501] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:08:23.006Z] 2024/01/19 00:08:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705622541] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:08:23.006Z] 2024-01-19 00:08:22 +0000:[INFO] [{SharedBackupDelete}] [backup.CreateBackup:#313] - Backup [tp-backup-1705622511] created successfully [2024-01-19T00:08:23.006Z] 2024-01-19 00:08:22 +0000:[INFO] [{SharedBackupDelete}] Validating backup [tp-backup-1705622511] in org [default] [2024-01-19T00:08:23.006Z] 2024-01-19 00:08:22 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705622511] [2024-01-19T00:08:23.261Z] 2024-01-19 00:08:23 +0000:[INFO] [{SharedBackupDelete}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] in backup [tp-backup-1705622511] [2024-01-19T00:08:23.261Z] 2024-01-19 00:08:23 +0000:[INFO] [{SharedBackupDelete}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] in backup [tp-backup-1705622511] [2024-01-19T00:08:23.261Z] 2024-01-19 00:08:23 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] [2024-01-19T00:08:23.261Z] 2024-01-19 00:08:23 +0000:[WARNING] [{SharedBackupDelete}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:08:23.261Z] 2024-01-19 00:08:23 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:08:23.261Z] 2024-01-19 00:08:23 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82946-01-18-23h59m07s] are [[mysql-data]] [2024-01-19T00:08:23.261Z] 2024-01-19 00:08:23 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T00:08:23.262Z] 2024-01-19 00:08:23 +0000:[INFO] [{SharedBackupDelete}] Validating CR cleanup [2024-01-19T00:08:23.262Z] 2024-01-19 00:08:23 +0000:[INFO] [{SharedBackupDelete}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T00:08:23.517Z] 2024-01-19 00:08:23 +0000:[INFO] [{SharedBackupDelete}] All CRs in [mysql-ibm-pxb-0-82946-01-18-23h59m07s] are [[tp-backup-1705622531-bdb0364 tp-backup-1705622541-786599f tp-backup-1705622551-88e113a]] [2024-01-19T00:08:23.517Z] INFO[2024-01-19 00:08:23] Verifying : Description : Creation and Validation of backup [tp-backup-1705622511] [2024-01-19T00:08:23.517Z] INFO[2024-01-19 00:08:23] Actual:true, Expected: true [2024-01-19T00:08:33.456Z] 2024-01-19 00:08:32 +0000:[INFO] [{SharedBackupDelete}] [backup.CreateBackup:#313] - Backup [tp-backup-1705622551] created successfully [2024-01-19T00:08:33.456Z] 2024-01-19 00:08:32 +0000:[INFO] [{SharedBackupDelete}] Validating backup [tp-backup-1705622551] in org [default] [2024-01-19T00:08:33.456Z] 2024-01-19 00:08:32 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705622551] [2024-01-19T00:08:33.456Z] 2024-01-19 00:08:32 +0000:[INFO] [{SharedBackupDelete}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] in backup [tp-backup-1705622551] [2024-01-19T00:08:33.456Z] 2024-01-19 00:08:32 +0000:[INFO] [{SharedBackupDelete}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] in backup [tp-backup-1705622551] [2024-01-19T00:08:33.456Z] 2024-01-19 00:08:32 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] [2024-01-19T00:08:33.456Z] 2024-01-19 00:08:32 +0000:[WARNING] [{SharedBackupDelete}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:08:33.456Z] 2024-01-19 00:08:32 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:08:33.456Z] 2024-01-19 00:08:32 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82946-01-18-23h59m07s] are [[mysql-data]] [2024-01-19T00:08:33.456Z] 2024-01-19 00:08:32 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T00:08:33.456Z] 2024-01-19 00:08:32 +0000:[INFO] [{SharedBackupDelete}] Validating CR cleanup [2024-01-19T00:08:33.456Z] 2024-01-19 00:08:32 +0000:[INFO] [{SharedBackupDelete}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T00:08:33.456Z] 2024-01-19 00:08:32 +0000:[INFO] [{SharedBackupDelete}] All CRs in [mysql-ibm-pxb-0-82946-01-18-23h59m07s] are [[]] [2024-01-19T00:08:33.456Z] INFO[2024-01-19 00:08:32] Verifying : Description : Creation and Validation of backup [tp-backup-1705622551] [2024-01-19T00:08:33.456Z] INFO[2024-01-19 00:08:32] Actual:true, Expected: true [2024-01-19T00:08:43.395Z] 2024-01-19 00:08:42 +0000:[INFO] [{SharedBackupDelete}] [backup.CreateBackup:#313] - Backup [tp-backup-1705622531] created successfully [2024-01-19T00:08:43.395Z] 2024-01-19 00:08:42 +0000:[INFO] [{SharedBackupDelete}] Validating backup [tp-backup-1705622531] in org [default] [2024-01-19T00:08:43.395Z] 2024-01-19 00:08:42 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705622531] [2024-01-19T00:08:43.395Z] 2024-01-19 00:08:42 +0000:[INFO] [{SharedBackupDelete}] [backup.CreateBackup:#313] - Backup [tp-backup-1705622501] created successfully [2024-01-19T00:08:43.395Z] 2024-01-19 00:08:42 +0000:[INFO] [{SharedBackupDelete}] Validating backup [tp-backup-1705622501] in org [default] [2024-01-19T00:08:43.395Z] 2024-01-19 00:08:42 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705622501] [2024-01-19T00:08:43.395Z] 2024-01-19 00:08:42 +0000:[INFO] [{SharedBackupDelete}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] in backup [tp-backup-1705622531] [2024-01-19T00:08:43.395Z] 2024-01-19 00:08:42 +0000:[INFO] [{SharedBackupDelete}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] in backup [tp-backup-1705622531] [2024-01-19T00:08:43.395Z] 2024-01-19 00:08:42 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] [2024-01-19T00:08:43.395Z] 2024-01-19 00:08:42 +0000:[WARNING] [{SharedBackupDelete}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:08:43.395Z] 2024-01-19 00:08:42 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:08:43.395Z] 2024-01-19 00:08:42 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82946-01-18-23h59m07s] are [[mysql-data]] [2024-01-19T00:08:43.395Z] 2024-01-19 00:08:42 +0000:[INFO] [{SharedBackupDelete}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] in backup [tp-backup-1705622501] [2024-01-19T00:08:43.395Z] 2024-01-19 00:08:42 +0000:[INFO] [{SharedBackupDelete}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] in backup [tp-backup-1705622501] [2024-01-19T00:08:43.395Z] 2024-01-19 00:08:42 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] [2024-01-19T00:08:43.395Z] 2024-01-19 00:08:42 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T00:08:43.395Z] 2024-01-19 00:08:42 +0000:[INFO] [{SharedBackupDelete}] Validating CR cleanup [2024-01-19T00:08:43.395Z] 2024-01-19 00:08:42 +0000:[WARNING] [{SharedBackupDelete}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:08:43.395Z] 2024-01-19 00:08:42 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:08:43.396Z] 2024-01-19 00:08:42 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82946-01-18-23h59m07s] are [[mysql-data]] [2024-01-19T00:08:43.396Z] 2024-01-19 00:08:42 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T00:08:43.396Z] 2024-01-19 00:08:42 +0000:[INFO] [{SharedBackupDelete}] Validating CR cleanup [2024-01-19T00:08:43.396Z] 2024-01-19 00:08:42 +0000:[INFO] [{SharedBackupDelete}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T00:08:43.396Z] 2024-01-19 00:08:42 +0000:[INFO] [{SharedBackupDelete}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T00:08:43.396Z] 2024-01-19 00:08:42 +0000:[INFO] [{SharedBackupDelete}] All CRs in [mysql-ibm-pxb-0-82946-01-18-23h59m07s] are [[]] [2024-01-19T00:08:43.396Z] INFO[2024-01-19 00:08:42] Verifying : Description : Creation and Validation of backup [tp-backup-1705622531] [2024-01-19T00:08:43.396Z] INFO[2024-01-19 00:08:42] Actual:true, Expected: true [2024-01-19T00:08:43.396Z] 2024-01-19 00:08:42 +0000:[INFO] [{SharedBackupDelete}] All CRs in [mysql-ibm-pxb-0-82946-01-18-23h59m07s] are [[]] [2024-01-19T00:08:43.396Z] INFO[2024-01-19 00:08:42] Verifying : Description : Creation and Validation of backup [tp-backup-1705622501] [2024-01-19T00:08:43.396Z] INFO[2024-01-19 00:08:42] Actual:true, Expected: true [2024-01-19T00:08:53.335Z] 2024-01-19 00:08:52 +0000:[INFO] [{SharedBackupDelete}] [backup.CreateBackup:#313] - Backup [tp-backup-1705622541] created successfully [2024-01-19T00:08:53.335Z] 2024-01-19 00:08:52 +0000:[INFO] [{SharedBackupDelete}] Validating backup [tp-backup-1705622541] in org [default] [2024-01-19T00:08:53.335Z] 2024-01-19 00:08:52 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705622541] [2024-01-19T00:08:53.335Z] 2024-01-19 00:08:52 +0000:[INFO] [{SharedBackupDelete}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] in backup [tp-backup-1705622541] [2024-01-19T00:08:53.335Z] 2024-01-19 00:08:52 +0000:[INFO] [{SharedBackupDelete}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] in backup [tp-backup-1705622541] [2024-01-19T00:08:53.335Z] 2024-01-19 00:08:52 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82946-01-18-23h59m07s] [2024-01-19T00:08:53.335Z] 2024-01-19 00:08:52 +0000:[WARNING] [{SharedBackupDelete}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:08:53.335Z] 2024-01-19 00:08:52 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:08:53.335Z] 2024-01-19 00:08:52 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82946-01-18-23h59m07s] are [[mysql-data]] [2024-01-19T00:08:53.335Z] 2024-01-19 00:08:52 +0000:[INFO] [{SharedBackupDelete}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T00:08:53.335Z] 2024-01-19 00:08:52 +0000:[INFO] [{SharedBackupDelete}] Validating CR cleanup [2024-01-19T00:08:53.335Z] 2024-01-19 00:08:52 +0000:[INFO] [{SharedBackupDelete}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T00:08:53.335Z] 2024-01-19 00:08:52 +0000:[INFO] [{SharedBackupDelete}] All CRs in [mysql-ibm-pxb-0-82946-01-18-23h59m07s] are [[]] [2024-01-19T00:08:53.335Z] INFO[2024-01-19 00:08:52] Verifying : Description : Creation and Validation of backup [tp-backup-1705622541] [2024-01-19T00:08:53.335Z] INFO[2024-01-19 00:08:52] Actual:true, Expected: true [2024-01-19T00:08:53.335Z] 2024-01-19 00:08:52 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.5:#2121] - List of backups - [tp-backup-1705622461 tp-backup-1705622471 tp-backup-1705622481 tp-backup-1705622491 tp-backup-1705622501 tp-backup-1705622511 tp-backup-1705622521 tp-backup-1705622531 tp-backup-1705622541 tp-backup-1705622551] [2024-01-19T00:08:53.335Z] 2024-01-19 00:08:52 +0000:[INFO] [{SharedBackupDelete}] Share backup with multiple users [2024-01-19T00:08:53.335Z] 2024-01-19 00:08:53 +0000:[INFO] [{SharedBackupDelete}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705622461 - d4b48047-71e8-4719-9689-3e19572f4a88 [2024-01-19T00:08:53.591Z] 2024-01-19 00:08:53 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser3-1705622438 - cdc67b42-871e-4ca1-8d95-9ba7dea58b08 [2024-01-19T00:08:54.153Z] 2024-01-19 00:08:54 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser10-1705622438 - a57151ec-69ca-4f7c-ad7b-7048c5513968 [2024-01-19T00:08:55.080Z] 2024-01-19 00:08:54 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser4-1705622438 - 95ebe0ab-2852-448c-a013-75d46766a6eb [2024-01-19T00:08:56.006Z] 2024-01-19 00:08:55 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser5-1705622438 - a4c4106c-2fa5-494f-adb2-f36bb650cc3a [2024-01-19T00:08:56.567Z] 2024-01-19 00:08:56 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2-1705622438 - c161acfd-6047-4d40-93ba-9f88c22efdf5 [2024-01-19T00:08:57.495Z] 2024-01-19 00:08:57 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser6-1705622438 - 88f8d071-50ff-4aad-a907-c3cd4ea9027b [2024-01-19T00:08:58.422Z] 2024-01-19 00:08:58 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705622438 - e7b7bdc4-893c-434d-852f-21edd74993e1 [2024-01-19T00:08:58.983Z] 2024-01-19 00:08:58 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser7-1705622438 - 1a84069f-7604-48f6-8a17-90dfb3e1769f [2024-01-19T00:08:59.909Z] 2024-01-19 00:08:59 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser8-1705622438 - d76ecda7-a2f7-4a32-96c7-0986ec23ef07 [2024-01-19T00:09:00.837Z] 2024-01-19 00:09:00 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser9-1705622438 - e253b80a-8c31-4ad2-9a21-702b0c386ec1 [2024-01-19T00:09:00.837Z] 2024-01-19 00:09:00 +0000:[INFO] [{SharedBackupDelete}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705622471 - cf27783e-4e16-4376-9696-48856e1fc24c [2024-01-19T00:09:01.399Z] 2024-01-19 00:09:01 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser3-1705622438 - cdc67b42-871e-4ca1-8d95-9ba7dea58b08 [2024-01-19T00:09:02.325Z] 2024-01-19 00:09:02 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser10-1705622438 - a57151ec-69ca-4f7c-ad7b-7048c5513968 [2024-01-19T00:09:03.251Z] 2024-01-19 00:09:02 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser4-1705622438 - 95ebe0ab-2852-448c-a013-75d46766a6eb [2024-01-19T00:09:03.811Z] 2024-01-19 00:09:03 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser5-1705622438 - a4c4106c-2fa5-494f-adb2-f36bb650cc3a [2024-01-19T00:09:04.736Z] 2024-01-19 00:09:04 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2-1705622438 - c161acfd-6047-4d40-93ba-9f88c22efdf5 [2024-01-19T00:09:05.662Z] 2024-01-19 00:09:05 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser6-1705622438 - 88f8d071-50ff-4aad-a907-c3cd4ea9027b [2024-01-19T00:09:06.222Z] 2024-01-19 00:09:06 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705622438 - e7b7bdc4-893c-434d-852f-21edd74993e1 [2024-01-19T00:09:07.148Z] 2024-01-19 00:09:06 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser7-1705622438 - 1a84069f-7604-48f6-8a17-90dfb3e1769f [2024-01-19T00:09:08.078Z] 2024-01-19 00:09:07 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser8-1705622438 - d76ecda7-a2f7-4a32-96c7-0986ec23ef07 [2024-01-19T00:09:08.639Z] 2024-01-19 00:09:08 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser9-1705622438 - e253b80a-8c31-4ad2-9a21-702b0c386ec1 [2024-01-19T00:09:08.895Z] 2024-01-19 00:09:08 +0000:[INFO] [{SharedBackupDelete}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705622481 - 0fed22cb-ddb2-44f3-865c-fc068845430a [2024-01-19T00:09:09.455Z] 2024-01-19 00:09:09 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser3-1705622438 - cdc67b42-871e-4ca1-8d95-9ba7dea58b08 [2024-01-19T00:09:10.382Z] 2024-01-19 00:09:10 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser10-1705622438 - a57151ec-69ca-4f7c-ad7b-7048c5513968 [2024-01-19T00:09:10.943Z] 2024-01-19 00:09:10 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser4-1705622438 - 95ebe0ab-2852-448c-a013-75d46766a6eb [2024-01-19T00:09:11.869Z] 2024-01-19 00:09:11 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser5-1705622438 - a4c4106c-2fa5-494f-adb2-f36bb650cc3a [2024-01-19T00:09:12.798Z] 2024-01-19 00:09:12 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2-1705622438 - c161acfd-6047-4d40-93ba-9f88c22efdf5 [2024-01-19T00:09:13.359Z] 2024-01-19 00:09:13 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser6-1705622438 - 88f8d071-50ff-4aad-a907-c3cd4ea9027b [2024-01-19T00:09:14.286Z] 2024-01-19 00:09:14 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705622438 - e7b7bdc4-893c-434d-852f-21edd74993e1 [2024-01-19T00:09:15.212Z] 2024-01-19 00:09:14 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser7-1705622438 - 1a84069f-7604-48f6-8a17-90dfb3e1769f [2024-01-19T00:09:15.773Z] 2024-01-19 00:09:15 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser8-1705622438 - d76ecda7-a2f7-4a32-96c7-0986ec23ef07 [2024-01-19T00:09:16.699Z] 2024-01-19 00:09:16 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser9-1705622438 - e253b80a-8c31-4ad2-9a21-702b0c386ec1 [2024-01-19T00:09:16.699Z] 2024-01-19 00:09:16 +0000:[INFO] [{SharedBackupDelete}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705622491 - 1419bfc6-272d-44d6-908c-cb61c0a3705d [2024-01-19T00:09:17.625Z] 2024-01-19 00:09:17 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser3-1705622438 - cdc67b42-871e-4ca1-8d95-9ba7dea58b08 [2024-01-19T00:09:18.187Z] 2024-01-19 00:09:18 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser10-1705622438 - a57151ec-69ca-4f7c-ad7b-7048c5513968 [2024-01-19T00:09:19.113Z] 2024-01-19 00:09:18 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser4-1705622438 - 95ebe0ab-2852-448c-a013-75d46766a6eb [2024-01-19T00:09:20.042Z] 2024-01-19 00:09:19 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser5-1705622438 - a4c4106c-2fa5-494f-adb2-f36bb650cc3a [2024-01-19T00:09:20.603Z] 2024-01-19 00:09:20 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2-1705622438 - c161acfd-6047-4d40-93ba-9f88c22efdf5 [2024-01-19T00:09:21.529Z] 2024-01-19 00:09:21 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser6-1705622438 - 88f8d071-50ff-4aad-a907-c3cd4ea9027b [2024-01-19T00:09:22.454Z] 2024-01-19 00:09:22 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705622438 - e7b7bdc4-893c-434d-852f-21edd74993e1 [2024-01-19T00:09:23.014Z] 2024-01-19 00:09:22 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser7-1705622438 - 1a84069f-7604-48f6-8a17-90dfb3e1769f [2024-01-19T00:09:23.941Z] 2024-01-19 00:09:23 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser8-1705622438 - d76ecda7-a2f7-4a32-96c7-0986ec23ef07 [2024-01-19T00:09:24.547Z] 2024-01-19 00:09:24 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser9-1705622438 - e253b80a-8c31-4ad2-9a21-702b0c386ec1 [2024-01-19T00:09:24.803Z] 2024-01-19 00:09:24 +0000:[INFO] [{SharedBackupDelete}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705622501 - 2f375b21-e24d-4cc9-b2d2-d671dd141896 [2024-01-19T00:09:25.364Z] 2024-01-19 00:09:25 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser3-1705622438 - cdc67b42-871e-4ca1-8d95-9ba7dea58b08 [2024-01-19T00:09:26.290Z] 2024-01-19 00:09:26 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser10-1705622438 - a57151ec-69ca-4f7c-ad7b-7048c5513968 [2024-01-19T00:09:27.225Z] 2024-01-19 00:09:26 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser4-1705622438 - 95ebe0ab-2852-448c-a013-75d46766a6eb [2024-01-19T00:09:27.790Z] 2024-01-19 00:09:27 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser5-1705622438 - a4c4106c-2fa5-494f-adb2-f36bb650cc3a [2024-01-19T00:09:28.716Z] 2024-01-19 00:09:28 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2-1705622438 - c161acfd-6047-4d40-93ba-9f88c22efdf5 [2024-01-19T00:09:29.642Z] 2024-01-19 00:09:29 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser6-1705622438 - 88f8d071-50ff-4aad-a907-c3cd4ea9027b [2024-01-19T00:09:30.202Z] 2024-01-19 00:09:30 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705622438 - e7b7bdc4-893c-434d-852f-21edd74993e1 [2024-01-19T00:09:31.128Z] 2024-01-19 00:09:30 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser7-1705622438 - 1a84069f-7604-48f6-8a17-90dfb3e1769f [2024-01-19T00:09:32.054Z] 2024-01-19 00:09:31 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser8-1705622438 - d76ecda7-a2f7-4a32-96c7-0986ec23ef07 [2024-01-19T00:09:32.615Z] 2024-01-19 00:09:32 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser9-1705622438 - e253b80a-8c31-4ad2-9a21-702b0c386ec1 [2024-01-19T00:09:32.871Z] 2024-01-19 00:09:32 +0000:[INFO] [{SharedBackupDelete}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705622511 - 64691009-2b1c-4d95-baf4-1ae6276ed3b4 [2024-01-19T00:09:33.432Z] 2024-01-19 00:09:33 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser3-1705622438 - cdc67b42-871e-4ca1-8d95-9ba7dea58b08 [2024-01-19T00:09:34.359Z] 2024-01-19 00:09:34 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser10-1705622438 - a57151ec-69ca-4f7c-ad7b-7048c5513968 [2024-01-19T00:09:35.288Z] 2024-01-19 00:09:34 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser4-1705622438 - 95ebe0ab-2852-448c-a013-75d46766a6eb [2024-01-19T00:09:35.906Z] 2024-01-19 00:09:35 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser5-1705622438 - a4c4106c-2fa5-494f-adb2-f36bb650cc3a [2024-01-19T00:09:36.836Z] 2024-01-19 00:09:36 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2-1705622438 - c161acfd-6047-4d40-93ba-9f88c22efdf5 [2024-01-19T00:09:37.398Z] 2024-01-19 00:09:37 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser6-1705622438 - 88f8d071-50ff-4aad-a907-c3cd4ea9027b [2024-01-19T00:09:38.323Z] 2024-01-19 00:09:38 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705622438 - e7b7bdc4-893c-434d-852f-21edd74993e1 [2024-01-19T00:09:39.251Z] 2024-01-19 00:09:38 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser7-1705622438 - 1a84069f-7604-48f6-8a17-90dfb3e1769f [2024-01-19T00:09:39.812Z] 2024-01-19 00:09:39 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser8-1705622438 - d76ecda7-a2f7-4a32-96c7-0986ec23ef07 [2024-01-19T00:09:40.740Z] 2024-01-19 00:09:40 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser9-1705622438 - e253b80a-8c31-4ad2-9a21-702b0c386ec1 [2024-01-19T00:09:40.740Z] 2024-01-19 00:09:40 +0000:[INFO] [{SharedBackupDelete}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705622521 - b87e233c-7aea-4853-823b-c23a2c47e963 [2024-01-19T00:09:41.670Z] 2024-01-19 00:09:41 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser3-1705622438 - cdc67b42-871e-4ca1-8d95-9ba7dea58b08 [2024-01-19T00:09:42.235Z] 2024-01-19 00:09:42 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser10-1705622438 - a57151ec-69ca-4f7c-ad7b-7048c5513968 [2024-01-19T00:09:43.165Z] 2024-01-19 00:09:42 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser4-1705622438 - 95ebe0ab-2852-448c-a013-75d46766a6eb [2024-01-19T00:09:44.091Z] 2024-01-19 00:09:43 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser5-1705622438 - a4c4106c-2fa5-494f-adb2-f36bb650cc3a [2024-01-19T00:09:44.652Z] 2024-01-19 00:09:44 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2-1705622438 - c161acfd-6047-4d40-93ba-9f88c22efdf5 [2024-01-19T00:09:45.578Z] 2024-01-19 00:09:45 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser6-1705622438 - 88f8d071-50ff-4aad-a907-c3cd4ea9027b [2024-01-19T00:09:46.506Z] 2024-01-19 00:09:46 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705622438 - e7b7bdc4-893c-434d-852f-21edd74993e1 [2024-01-19T00:09:47.069Z] 2024-01-19 00:09:46 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser7-1705622438 - 1a84069f-7604-48f6-8a17-90dfb3e1769f [2024-01-19T00:09:47.997Z] 2024-01-19 00:09:47 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser8-1705622438 - d76ecda7-a2f7-4a32-96c7-0986ec23ef07 [2024-01-19T00:09:48.560Z] 2024-01-19 00:09:48 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser9-1705622438 - e253b80a-8c31-4ad2-9a21-702b0c386ec1 [2024-01-19T00:09:48.815Z] 2024-01-19 00:09:48 +0000:[INFO] [{SharedBackupDelete}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705622531 - bdb03649-e4b2-4db2-9943-d8d1840d0055 [2024-01-19T00:09:49.376Z] 2024-01-19 00:09:49 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser3-1705622438 - cdc67b42-871e-4ca1-8d95-9ba7dea58b08 [2024-01-19T00:09:50.305Z] 2024-01-19 00:09:50 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser10-1705622438 - a57151ec-69ca-4f7c-ad7b-7048c5513968 [2024-01-19T00:09:51.262Z] 2024-01-19 00:09:50 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser4-1705622438 - 95ebe0ab-2852-448c-a013-75d46766a6eb [2024-01-19T00:09:51.824Z] 2024-01-19 00:09:51 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser5-1705622438 - a4c4106c-2fa5-494f-adb2-f36bb650cc3a [2024-01-19T00:09:52.754Z] 2024-01-19 00:09:52 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2-1705622438 - c161acfd-6047-4d40-93ba-9f88c22efdf5 [2024-01-19T00:09:53.681Z] 2024-01-19 00:09:53 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser6-1705622438 - 88f8d071-50ff-4aad-a907-c3cd4ea9027b [2024-01-19T00:09:54.242Z] 2024-01-19 00:09:54 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705622438 - e7b7bdc4-893c-434d-852f-21edd74993e1 [2024-01-19T00:09:55.178Z] 2024-01-19 00:09:54 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser7-1705622438 - 1a84069f-7604-48f6-8a17-90dfb3e1769f [2024-01-19T00:09:55.738Z] 2024-01-19 00:09:55 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser8-1705622438 - d76ecda7-a2f7-4a32-96c7-0986ec23ef07 [2024-01-19T00:09:56.664Z] 2024-01-19 00:09:56 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser9-1705622438 - e253b80a-8c31-4ad2-9a21-702b0c386ec1 [2024-01-19T00:09:56.920Z] 2024-01-19 00:09:56 +0000:[INFO] [{SharedBackupDelete}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705622541 - 786599f5-988f-4ed1-ba76-11043038c352 [2024-01-19T00:09:57.480Z] 2024-01-19 00:09:57 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser3-1705622438 - cdc67b42-871e-4ca1-8d95-9ba7dea58b08 [2024-01-19T00:09:58.407Z] 2024-01-19 00:09:58 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser10-1705622438 - a57151ec-69ca-4f7c-ad7b-7048c5513968 [2024-01-19T00:09:58.971Z] 2024-01-19 00:09:58 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser4-1705622438 - 95ebe0ab-2852-448c-a013-75d46766a6eb [2024-01-19T00:09:59.897Z] 2024-01-19 00:09:59 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser5-1705622438 - a4c4106c-2fa5-494f-adb2-f36bb650cc3a [2024-01-19T00:10:00.826Z] 2024-01-19 00:10:00 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2-1705622438 - c161acfd-6047-4d40-93ba-9f88c22efdf5 [2024-01-19T00:10:01.388Z] 2024-01-19 00:10:01 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser6-1705622438 - 88f8d071-50ff-4aad-a907-c3cd4ea9027b [2024-01-19T00:10:02.318Z] 2024-01-19 00:10:02 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705622438 - e7b7bdc4-893c-434d-852f-21edd74993e1 [2024-01-19T00:10:03.245Z] 2024-01-19 00:10:02 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser7-1705622438 - 1a84069f-7604-48f6-8a17-90dfb3e1769f [2024-01-19T00:10:03.808Z] 2024-01-19 00:10:03 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser8-1705622438 - d76ecda7-a2f7-4a32-96c7-0986ec23ef07 [2024-01-19T00:10:04.732Z] 2024-01-19 00:10:04 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser9-1705622438 - e253b80a-8c31-4ad2-9a21-702b0c386ec1 [2024-01-19T00:10:04.732Z] 2024-01-19 00:10:04 +0000:[INFO] [{SharedBackupDelete}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705622551 - 88e113ad-daf4-408c-ba66-c96fe6910cb0 [2024-01-19T00:10:05.665Z] 2024-01-19 00:10:05 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser3-1705622438 - cdc67b42-871e-4ca1-8d95-9ba7dea58b08 [2024-01-19T00:10:06.225Z] 2024-01-19 00:10:06 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser10-1705622438 - a57151ec-69ca-4f7c-ad7b-7048c5513968 [2024-01-19T00:10:07.157Z] 2024-01-19 00:10:06 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser4-1705622438 - 95ebe0ab-2852-448c-a013-75d46766a6eb [2024-01-19T00:10:08.086Z] 2024-01-19 00:10:07 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser5-1705622438 - a4c4106c-2fa5-494f-adb2-f36bb650cc3a [2024-01-19T00:10:08.649Z] 2024-01-19 00:10:08 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2-1705622438 - c161acfd-6047-4d40-93ba-9f88c22efdf5 [2024-01-19T00:10:09.575Z] 2024-01-19 00:10:09 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser6-1705622438 - 88f8d071-50ff-4aad-a907-c3cd4ea9027b [2024-01-19T00:10:10.504Z] 2024-01-19 00:10:10 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705622438 - e7b7bdc4-893c-434d-852f-21edd74993e1 [2024-01-19T00:10:11.076Z] 2024-01-19 00:10:10 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser7-1705622438 - 1a84069f-7604-48f6-8a17-90dfb3e1769f [2024-01-19T00:10:12.004Z] 2024-01-19 00:10:11 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser8-1705622438 - d76ecda7-a2f7-4a32-96c7-0986ec23ef07 [2024-01-19T00:10:12.564Z] 2024-01-19 00:10:12 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser9-1705622438 - e253b80a-8c31-4ad2-9a21-702b0c386ec1 [2024-01-19T00:10:13.128Z] 2024-01-19 00:10:12 +0000:[INFO] [{SharedBackupDelete}] Registering Source and Destination clusters from user context for user -testuser3-1705622438 [2024-01-19T00:10:13.128Z] INFO[2024-01-19 00:10:12] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T00:10:13.128Z] INFO[2024-01-19 00:10:12] Actual:true, Expected: true [2024-01-19T00:10:13.128Z] 2024-01-19 00:10:12 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T00:10:13.383Z] 2024-01-19 00:10:13 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T00:10:13.383Z] 2024-01-19 00:10:13 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T00:10:13.383Z] 2024-01-19 00:10:13 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T00:10:13.383Z] 2024-01-19 00:10:13 +0000:[INFO] [{SharedBackupDelete}] Create cluster [source-cluster] in org [default] [2024-01-19T00:10:13.384Z] 2024-01-19 00:10:13 +0000:[INFO] [{SharedBackupDelete}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T00:10:13.384Z] 2024-01-19 00:10:13 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T00:10:13.384Z] 2024-01-19 00:10:13 +0000:[INFO] [{SharedBackupDelete}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:10:13.384Z] 2024-01-19 00:10:13 +0000:[INFO] [{SharedBackupDelete}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T00:10:13.384Z] 2024-01-19 00:10:13 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T00:10:13.384Z] 2024-01-19 00:10:13 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-gsmhp] for IBM [2024-01-19T00:10:13.384Z] 2024-01-19 00:10:13 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-gsmhp] for org [default] with [ibm] as provider [2024-01-19T00:10:13.384Z] 2024-01-19 00:10:13 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:13.384Z] 2024-01-19 00:10:13 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:13.384Z] 2024-01-19 00:10:13 +0000:[WARNING] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-gsmhp] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName3 LastName3 with roles: [px-backup-app.user]] [2024-01-19T00:10:13.384Z] 2024-01-19 00:10:13 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName3 LastName3 with roles: [px-backup-app.user] [2024-01-19T00:10:14.747Z] 2024-01-19 00:10:14 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-gsmhp from admin context and sharing with all the users [2024-01-19T00:10:14.747Z] 2024-01-19 00:10:14 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-gsmhp] for org [default] with [ibm] as provider [2024-01-19T00:10:14.747Z] 2024-01-19 00:10:14 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:14.747Z] 2024-01-19 00:10:14 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:14.747Z] 2024-01-19 00:10:14 +0000:[INFO] [{SharedBackupDelete}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-gsmhp] is present [2024-01-19T00:10:15.003Z] 2024-01-19 00:10:14 +0000:[INFO] [{SharedBackupDelete}] Create cluster [source-cluster] in org [default] [2024-01-19T00:10:15.569Z] 2024-01-19 00:10:15 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-vizom] for IBM [2024-01-19T00:10:15.569Z] 2024-01-19 00:10:15 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-vizom] for org [default] with [ibm] as provider [2024-01-19T00:10:15.569Z] 2024-01-19 00:10:15 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:15.569Z] 2024-01-19 00:10:15 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:15.569Z] 2024-01-19 00:10:15 +0000:[WARNING] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-vizom] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName3 LastName3 with roles: [px-backup-app.user]] [2024-01-19T00:10:15.569Z] 2024-01-19 00:10:15 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName3 LastName3 with roles: [px-backup-app.user] [2024-01-19T00:10:16.131Z] 2024-01-19 00:10:15 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-vizom from admin context and sharing with all the users [2024-01-19T00:10:16.131Z] 2024-01-19 00:10:15 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-vizom] for org [default] with [ibm] as provider [2024-01-19T00:10:16.131Z] 2024-01-19 00:10:15 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:16.131Z] 2024-01-19 00:10:15 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:16.131Z] 2024-01-19 00:10:16 +0000:[INFO] [{SharedBackupDelete}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-vizom] is present [2024-01-19T00:10:16.391Z] 2024-01-19 00:10:16 +0000:[INFO] [{SharedBackupDelete}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:10:16.953Z] INFO[2024-01-19 00:10:16] Verifying : Description : Creating source and destination cluster [2024-01-19T00:10:16.953Z] INFO[2024-01-19 00:10:16] Actual:true, Expected: true [2024-01-19T00:10:17.211Z] INFO[2024-01-19 00:10:17] Verifying : Description : Verifying backup restore [tp-restore-1705623016] is not possible for backup [tp-backup-1705622461] with user [testuser3-1705622438] [2024-01-19T00:10:17.211Z] INFO[2024-01-19 00:10:17] Actual:true, Expected: true [2024-01-19T00:10:17.211Z] 2024-01-19 00:10:17 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [cdc67b42-871e-4ca1-8d95-9ba7dea58b08] doesn't have permission to delete backup [tp-backup-1705622461] [2024-01-19T00:10:17.211Z] INFO[2024-01-19 00:10:17] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622461] with user [testuser3-1705622438] [2024-01-19T00:10:17.212Z] INFO[2024-01-19 00:10:17] Actual:true, Expected: true [2024-01-19T00:10:17.469Z] INFO[2024-01-19 00:10:17] Verifying : Description : Verifying backup restore [tp-restore-1705623017] is not possible for backup [tp-backup-1705622471] with user [testuser3-1705622438] [2024-01-19T00:10:17.469Z] INFO[2024-01-19 00:10:17] Actual:true, Expected: true [2024-01-19T00:10:17.469Z] 2024-01-19 00:10:17 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [cdc67b42-871e-4ca1-8d95-9ba7dea58b08] doesn't have permission to delete backup [tp-backup-1705622471] [2024-01-19T00:10:17.469Z] INFO[2024-01-19 00:10:17] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622471] with user [testuser3-1705622438] [2024-01-19T00:10:17.469Z] INFO[2024-01-19 00:10:17] Actual:true, Expected: true [2024-01-19T00:10:17.725Z] INFO[2024-01-19 00:10:17] Verifying : Description : Verifying backup restore [tp-restore-1705623017] is not possible for backup [tp-backup-1705622481] with user [testuser3-1705622438] [2024-01-19T00:10:17.725Z] INFO[2024-01-19 00:10:17] Actual:true, Expected: true [2024-01-19T00:10:17.725Z] 2024-01-19 00:10:17 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [cdc67b42-871e-4ca1-8d95-9ba7dea58b08] doesn't have permission to delete backup [tp-backup-1705622481] [2024-01-19T00:10:17.725Z] INFO[2024-01-19 00:10:17] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622481] with user [testuser3-1705622438] [2024-01-19T00:10:17.725Z] INFO[2024-01-19 00:10:17] Actual:true, Expected: true [2024-01-19T00:10:17.725Z] INFO[2024-01-19 00:10:17] Verifying : Description : Verifying backup restore [tp-restore-1705623017] is not possible for backup [tp-backup-1705622491] with user [testuser3-1705622438] [2024-01-19T00:10:17.725Z] INFO[2024-01-19 00:10:17] Actual:true, Expected: true [2024-01-19T00:10:17.981Z] 2024-01-19 00:10:17 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [cdc67b42-871e-4ca1-8d95-9ba7dea58b08] doesn't have permission to delete backup [tp-backup-1705622491] [2024-01-19T00:10:17.981Z] INFO[2024-01-19 00:10:17] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622491] with user [testuser3-1705622438] [2024-01-19T00:10:17.981Z] INFO[2024-01-19 00:10:17] Actual:true, Expected: true [2024-01-19T00:10:17.981Z] INFO[2024-01-19 00:10:17] Verifying : Description : Verifying backup restore [tp-restore-1705623017] is not possible for backup [tp-backup-1705622501] with user [testuser3-1705622438] [2024-01-19T00:10:17.981Z] INFO[2024-01-19 00:10:17] Actual:true, Expected: true [2024-01-19T00:10:17.981Z] 2024-01-19 00:10:17 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [cdc67b42-871e-4ca1-8d95-9ba7dea58b08] doesn't have permission to delete backup [tp-backup-1705622501] [2024-01-19T00:10:17.981Z] INFO[2024-01-19 00:10:17] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622501] with user [testuser3-1705622438] [2024-01-19T00:10:17.981Z] INFO[2024-01-19 00:10:17] Actual:true, Expected: true [2024-01-19T00:10:18.237Z] INFO[2024-01-19 00:10:18] Verifying : Description : Verifying backup restore [tp-restore-1705623017] is not possible for backup [tp-backup-1705622511] with user [testuser3-1705622438] [2024-01-19T00:10:18.237Z] INFO[2024-01-19 00:10:18] Actual:true, Expected: true [2024-01-19T00:10:18.237Z] 2024-01-19 00:10:18 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [cdc67b42-871e-4ca1-8d95-9ba7dea58b08] doesn't have permission to delete backup [tp-backup-1705622511] [2024-01-19T00:10:18.237Z] INFO[2024-01-19 00:10:18] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622511] with user [testuser3-1705622438] [2024-01-19T00:10:18.237Z] INFO[2024-01-19 00:10:18] Actual:true, Expected: true [2024-01-19T00:10:18.495Z] INFO[2024-01-19 00:10:18] Verifying : Description : Verifying backup restore [tp-restore-1705623018] is not possible for backup [tp-backup-1705622521] with user [testuser3-1705622438] [2024-01-19T00:10:18.495Z] INFO[2024-01-19 00:10:18] Actual:true, Expected: true [2024-01-19T00:10:18.495Z] 2024-01-19 00:10:18 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [cdc67b42-871e-4ca1-8d95-9ba7dea58b08] doesn't have permission to delete backup [tp-backup-1705622521] [2024-01-19T00:10:18.496Z] INFO[2024-01-19 00:10:18] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622521] with user [testuser3-1705622438] [2024-01-19T00:10:18.496Z] INFO[2024-01-19 00:10:18] Actual:true, Expected: true [2024-01-19T00:10:18.496Z] INFO[2024-01-19 00:10:18] Verifying : Description : Verifying backup restore [tp-restore-1705623018] is not possible for backup [tp-backup-1705622531] with user [testuser3-1705622438] [2024-01-19T00:10:18.496Z] INFO[2024-01-19 00:10:18] Actual:true, Expected: true [2024-01-19T00:10:18.496Z] 2024-01-19 00:10:18 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [cdc67b42-871e-4ca1-8d95-9ba7dea58b08] doesn't have permission to delete backup [tp-backup-1705622531] [2024-01-19T00:10:18.496Z] INFO[2024-01-19 00:10:18] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622531] with user [testuser3-1705622438] [2024-01-19T00:10:18.496Z] INFO[2024-01-19 00:10:18] Actual:true, Expected: true [2024-01-19T00:10:18.753Z] INFO[2024-01-19 00:10:18] Verifying : Description : Verifying backup restore [tp-restore-1705623018] is not possible for backup [tp-backup-1705622541] with user [testuser3-1705622438] [2024-01-19T00:10:18.753Z] INFO[2024-01-19 00:10:18] Actual:true, Expected: true [2024-01-19T00:10:18.753Z] 2024-01-19 00:10:18 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [cdc67b42-871e-4ca1-8d95-9ba7dea58b08] doesn't have permission to delete backup [tp-backup-1705622541] [2024-01-19T00:10:18.754Z] INFO[2024-01-19 00:10:18] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622541] with user [testuser3-1705622438] [2024-01-19T00:10:18.754Z] INFO[2024-01-19 00:10:18] Actual:true, Expected: true [2024-01-19T00:10:19.009Z] INFO[2024-01-19 00:10:18] Verifying : Description : Verifying backup restore [tp-restore-1705623018] is not possible for backup [tp-backup-1705622551] with user [testuser3-1705622438] [2024-01-19T00:10:19.009Z] INFO[2024-01-19 00:10:18] Actual:true, Expected: true [2024-01-19T00:10:19.009Z] 2024-01-19 00:10:18 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [cdc67b42-871e-4ca1-8d95-9ba7dea58b08] doesn't have permission to delete backup [tp-backup-1705622551] [2024-01-19T00:10:19.009Z] INFO[2024-01-19 00:10:18] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622551] with user [testuser3-1705622438] [2024-01-19T00:10:19.009Z] INFO[2024-01-19 00:10:18] Actual:true, Expected: true [2024-01-19T00:10:19.009Z] 2024-01-19 00:10:18 +0000:[INFO] [{SharedBackupDelete}] Registering Source and Destination clusters from user context for user -testuser10-1705622438 [2024-01-19T00:10:19.009Z] INFO[2024-01-19 00:10:18] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T00:10:19.009Z] INFO[2024-01-19 00:10:18] Actual:true, Expected: true [2024-01-19T00:10:19.010Z] 2024-01-19 00:10:18 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T00:10:19.010Z] 2024-01-19 00:10:18 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T00:10:19.010Z] 2024-01-19 00:10:18 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T00:10:19.010Z] 2024-01-19 00:10:18 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T00:10:19.010Z] 2024-01-19 00:10:18 +0000:[INFO] [{SharedBackupDelete}] Create cluster [source-cluster] in org [default] [2024-01-19T00:10:19.010Z] 2024-01-19 00:10:18 +0000:[INFO] [{SharedBackupDelete}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T00:10:19.010Z] 2024-01-19 00:10:18 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T00:10:19.010Z] 2024-01-19 00:10:18 +0000:[INFO] [{SharedBackupDelete}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:10:19.010Z] 2024-01-19 00:10:18 +0000:[INFO] [{SharedBackupDelete}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T00:10:19.010Z] 2024-01-19 00:10:18 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T00:10:19.010Z] 2024-01-19 00:10:18 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-wfyfe] for IBM [2024-01-19T00:10:19.010Z] 2024-01-19 00:10:18 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-wfyfe] for org [default] with [ibm] as provider [2024-01-19T00:10:19.010Z] 2024-01-19 00:10:18 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:19.010Z] 2024-01-19 00:10:18 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:19.266Z] 2024-01-19 00:10:18 +0000:[WARNING] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-wfyfe] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName10 LastName10 with roles: [px-backup-app.user]] [2024-01-19T00:10:19.266Z] 2024-01-19 00:10:18 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName10 LastName10 with roles: [px-backup-app.user] [2024-01-19T00:10:19.521Z] 2024-01-19 00:10:19 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-wfyfe from admin context and sharing with all the users [2024-01-19T00:10:19.521Z] 2024-01-19 00:10:19 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-wfyfe] for org [default] with [ibm] as provider [2024-01-19T00:10:19.521Z] 2024-01-19 00:10:19 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:19.521Z] 2024-01-19 00:10:19 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:19.521Z] 2024-01-19 00:10:19 +0000:[INFO] [{SharedBackupDelete}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-wfyfe] is present [2024-01-19T00:10:19.777Z] 2024-01-19 00:10:19 +0000:[INFO] [{SharedBackupDelete}] Create cluster [source-cluster] in org [default] [2024-01-19T00:10:20.378Z] 2024-01-19 00:10:20 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-lfzhw] for IBM [2024-01-19T00:10:20.378Z] 2024-01-19 00:10:20 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-lfzhw] for org [default] with [ibm] as provider [2024-01-19T00:10:20.378Z] 2024-01-19 00:10:20 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:20.378Z] 2024-01-19 00:10:20 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:20.378Z] 2024-01-19 00:10:20 +0000:[WARNING] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-lfzhw] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName10 LastName10 with roles: [px-backup-app.user]] [2024-01-19T00:10:20.378Z] 2024-01-19 00:10:20 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName10 LastName10 with roles: [px-backup-app.user] [2024-01-19T00:10:20.639Z] 2024-01-19 00:10:20 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-lfzhw from admin context and sharing with all the users [2024-01-19T00:10:20.639Z] 2024-01-19 00:10:20 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-lfzhw] for org [default] with [ibm] as provider [2024-01-19T00:10:20.639Z] 2024-01-19 00:10:20 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:20.639Z] 2024-01-19 00:10:20 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:20.639Z] 2024-01-19 00:10:20 +0000:[INFO] [{SharedBackupDelete}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-lfzhw] is present [2024-01-19T00:10:20.895Z] 2024-01-19 00:10:20 +0000:[INFO] [{SharedBackupDelete}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:10:21.456Z] INFO[2024-01-19 00:10:21] Verifying : Description : Creating source and destination cluster [2024-01-19T00:10:21.456Z] INFO[2024-01-19 00:10:21] Actual:true, Expected: true [2024-01-19T00:10:21.712Z] INFO[2024-01-19 00:10:21] Verifying : Description : Verifying backup restore [tp-restore-1705623021] is not possible for backup [tp-backup-1705622461] with user [testuser10-1705622438] [2024-01-19T00:10:21.712Z] INFO[2024-01-19 00:10:21] Actual:true, Expected: true [2024-01-19T00:10:21.712Z] 2024-01-19 00:10:21 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [a57151ec-69ca-4f7c-ad7b-7048c5513968] doesn't have permission to delete backup [tp-backup-1705622461] [2024-01-19T00:10:21.713Z] INFO[2024-01-19 00:10:21] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622461] with user [testuser10-1705622438] [2024-01-19T00:10:21.713Z] INFO[2024-01-19 00:10:21] Actual:true, Expected: true [2024-01-19T00:10:21.973Z] INFO[2024-01-19 00:10:21] Verifying : Description : Verifying backup restore [tp-restore-1705623021] is not possible for backup [tp-backup-1705622471] with user [testuser10-1705622438] [2024-01-19T00:10:21.973Z] INFO[2024-01-19 00:10:21] Actual:true, Expected: true [2024-01-19T00:10:21.973Z] 2024-01-19 00:10:21 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [a57151ec-69ca-4f7c-ad7b-7048c5513968] doesn't have permission to delete backup [tp-backup-1705622471] [2024-01-19T00:10:21.973Z] INFO[2024-01-19 00:10:21] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622471] with user [testuser10-1705622438] [2024-01-19T00:10:21.973Z] INFO[2024-01-19 00:10:21] Actual:true, Expected: true [2024-01-19T00:10:21.973Z] INFO[2024-01-19 00:10:21] Verifying : Description : Verifying backup restore [tp-restore-1705623021] is not possible for backup [tp-backup-1705622481] with user [testuser10-1705622438] [2024-01-19T00:10:21.973Z] INFO[2024-01-19 00:10:21] Actual:true, Expected: true [2024-01-19T00:10:21.973Z] 2024-01-19 00:10:21 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [a57151ec-69ca-4f7c-ad7b-7048c5513968] doesn't have permission to delete backup [tp-backup-1705622481] [2024-01-19T00:10:21.973Z] INFO[2024-01-19 00:10:21] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622481] with user [testuser10-1705622438] [2024-01-19T00:10:21.973Z] INFO[2024-01-19 00:10:21] Actual:true, Expected: true [2024-01-19T00:10:22.229Z] INFO[2024-01-19 00:10:22] Verifying : Description : Verifying backup restore [tp-restore-1705623021] is not possible for backup [tp-backup-1705622491] with user [testuser10-1705622438] [2024-01-19T00:10:22.229Z] INFO[2024-01-19 00:10:22] Actual:true, Expected: true [2024-01-19T00:10:22.229Z] 2024-01-19 00:10:22 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [a57151ec-69ca-4f7c-ad7b-7048c5513968] doesn't have permission to delete backup [tp-backup-1705622491] [2024-01-19T00:10:22.229Z] INFO[2024-01-19 00:10:22] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622491] with user [testuser10-1705622438] [2024-01-19T00:10:22.229Z] INFO[2024-01-19 00:10:22] Actual:true, Expected: true [2024-01-19T00:10:22.485Z] INFO[2024-01-19 00:10:22] Verifying : Description : Verifying backup restore [tp-restore-1705623022] is not possible for backup [tp-backup-1705622501] with user [testuser10-1705622438] [2024-01-19T00:10:22.485Z] INFO[2024-01-19 00:10:22] Actual:true, Expected: true [2024-01-19T00:10:22.485Z] 2024-01-19 00:10:22 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [a57151ec-69ca-4f7c-ad7b-7048c5513968] doesn't have permission to delete backup [tp-backup-1705622501] [2024-01-19T00:10:22.485Z] INFO[2024-01-19 00:10:22] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622501] with user [testuser10-1705622438] [2024-01-19T00:10:22.485Z] INFO[2024-01-19 00:10:22] Actual:true, Expected: true [2024-01-19T00:10:22.485Z] INFO[2024-01-19 00:10:22] Verifying : Description : Verifying backup restore [tp-restore-1705623022] is not possible for backup [tp-backup-1705622511] with user [testuser10-1705622438] [2024-01-19T00:10:22.485Z] INFO[2024-01-19 00:10:22] Actual:true, Expected: true [2024-01-19T00:10:22.485Z] 2024-01-19 00:10:22 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [a57151ec-69ca-4f7c-ad7b-7048c5513968] doesn't have permission to delete backup [tp-backup-1705622511] [2024-01-19T00:10:22.485Z] INFO[2024-01-19 00:10:22] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622511] with user [testuser10-1705622438] [2024-01-19T00:10:22.485Z] INFO[2024-01-19 00:10:22] Actual:true, Expected: true [2024-01-19T00:10:22.741Z] INFO[2024-01-19 00:10:22] Verifying : Description : Verifying backup restore [tp-restore-1705623022] is not possible for backup [tp-backup-1705622521] with user [testuser10-1705622438] [2024-01-19T00:10:22.741Z] INFO[2024-01-19 00:10:22] Actual:true, Expected: true [2024-01-19T00:10:22.741Z] 2024-01-19 00:10:22 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [a57151ec-69ca-4f7c-ad7b-7048c5513968] doesn't have permission to delete backup [tp-backup-1705622521] [2024-01-19T00:10:22.741Z] INFO[2024-01-19 00:10:22] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622521] with user [testuser10-1705622438] [2024-01-19T00:10:22.741Z] INFO[2024-01-19 00:10:22] Actual:true, Expected: true [2024-01-19T00:10:22.997Z] INFO[2024-01-19 00:10:22] Verifying : Description : Verifying backup restore [tp-restore-1705623022] is not possible for backup [tp-backup-1705622531] with user [testuser10-1705622438] [2024-01-19T00:10:22.997Z] INFO[2024-01-19 00:10:22] Actual:true, Expected: true [2024-01-19T00:10:22.997Z] 2024-01-19 00:10:22 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [a57151ec-69ca-4f7c-ad7b-7048c5513968] doesn't have permission to delete backup [tp-backup-1705622531] [2024-01-19T00:10:22.997Z] INFO[2024-01-19 00:10:22] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622531] with user [testuser10-1705622438] [2024-01-19T00:10:22.997Z] INFO[2024-01-19 00:10:22] Actual:true, Expected: true [2024-01-19T00:10:23.259Z] INFO[2024-01-19 00:10:22] Verifying : Description : Verifying backup restore [tp-restore-1705623022] is not possible for backup [tp-backup-1705622541] with user [testuser10-1705622438] [2024-01-19T00:10:23.259Z] INFO[2024-01-19 00:10:22] Actual:true, Expected: true [2024-01-19T00:10:23.259Z] 2024-01-19 00:10:23 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [a57151ec-69ca-4f7c-ad7b-7048c5513968] doesn't have permission to delete backup [tp-backup-1705622541] [2024-01-19T00:10:23.259Z] INFO[2024-01-19 00:10:23] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622541] with user [testuser10-1705622438] [2024-01-19T00:10:23.259Z] INFO[2024-01-19 00:10:23] Actual:true, Expected: true [2024-01-19T00:10:23.259Z] INFO[2024-01-19 00:10:23] Verifying : Description : Verifying backup restore [tp-restore-1705623023] is not possible for backup [tp-backup-1705622551] with user [testuser10-1705622438] [2024-01-19T00:10:23.259Z] INFO[2024-01-19 00:10:23] Actual:true, Expected: true [2024-01-19T00:10:23.259Z] 2024-01-19 00:10:23 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [a57151ec-69ca-4f7c-ad7b-7048c5513968] doesn't have permission to delete backup [tp-backup-1705622551] [2024-01-19T00:10:23.259Z] INFO[2024-01-19 00:10:23] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622551] with user [testuser10-1705622438] [2024-01-19T00:10:23.259Z] INFO[2024-01-19 00:10:23] Actual:true, Expected: true [2024-01-19T00:10:23.515Z] 2024-01-19 00:10:23 +0000:[INFO] [{SharedBackupDelete}] Registering Source and Destination clusters from user context for user -testuser4-1705622438 [2024-01-19T00:10:23.515Z] INFO[2024-01-19 00:10:23] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T00:10:23.515Z] INFO[2024-01-19 00:10:23] Actual:true, Expected: true [2024-01-19T00:10:23.516Z] 2024-01-19 00:10:23 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T00:10:23.516Z] 2024-01-19 00:10:23 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T00:10:23.516Z] 2024-01-19 00:10:23 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T00:10:23.516Z] 2024-01-19 00:10:23 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T00:10:23.516Z] 2024-01-19 00:10:23 +0000:[INFO] [{SharedBackupDelete}] Create cluster [source-cluster] in org [default] [2024-01-19T00:10:23.516Z] 2024-01-19 00:10:23 +0000:[INFO] [{SharedBackupDelete}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T00:10:23.516Z] 2024-01-19 00:10:23 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T00:10:23.516Z] 2024-01-19 00:10:23 +0000:[INFO] [{SharedBackupDelete}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:10:23.516Z] 2024-01-19 00:10:23 +0000:[INFO] [{SharedBackupDelete}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T00:10:23.516Z] 2024-01-19 00:10:23 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T00:10:23.516Z] 2024-01-19 00:10:23 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-mtvko] for IBM [2024-01-19T00:10:23.516Z] 2024-01-19 00:10:23 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-mtvko] for org [default] with [ibm] as provider [2024-01-19T00:10:23.516Z] 2024-01-19 00:10:23 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:23.516Z] 2024-01-19 00:10:23 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:23.516Z] 2024-01-19 00:10:23 +0000:[WARNING] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-mtvko] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName4 LastName4 with roles: [px-backup-app.user]] [2024-01-19T00:10:23.516Z] 2024-01-19 00:10:23 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName4 LastName4 with roles: [px-backup-app.user] [2024-01-19T00:10:23.784Z] 2024-01-19 00:10:23 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-mtvko from admin context and sharing with all the users [2024-01-19T00:10:23.784Z] 2024-01-19 00:10:23 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-mtvko] for org [default] with [ibm] as provider [2024-01-19T00:10:23.784Z] 2024-01-19 00:10:23 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:23.784Z] 2024-01-19 00:10:23 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:23.784Z] 2024-01-19 00:10:23 +0000:[INFO] [{SharedBackupDelete}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-mtvko] is present [2024-01-19T00:10:24.047Z] 2024-01-19 00:10:23 +0000:[INFO] [{SharedBackupDelete}] Create cluster [source-cluster] in org [default] [2024-01-19T00:10:24.608Z] 2024-01-19 00:10:24 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-dggka] for IBM [2024-01-19T00:10:24.608Z] 2024-01-19 00:10:24 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-dggka] for org [default] with [ibm] as provider [2024-01-19T00:10:24.608Z] 2024-01-19 00:10:24 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:24.608Z] 2024-01-19 00:10:24 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:24.608Z] 2024-01-19 00:10:24 +0000:[WARNING] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-dggka] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName4 LastName4 with roles: [px-backup-app.user]] [2024-01-19T00:10:24.608Z] 2024-01-19 00:10:24 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName4 LastName4 with roles: [px-backup-app.user] [2024-01-19T00:10:24.864Z] 2024-01-19 00:10:24 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-dggka from admin context and sharing with all the users [2024-01-19T00:10:24.864Z] 2024-01-19 00:10:24 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-dggka] for org [default] with [ibm] as provider [2024-01-19T00:10:24.864Z] 2024-01-19 00:10:24 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:24.864Z] 2024-01-19 00:10:24 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:25.121Z] 2024-01-19 00:10:24 +0000:[INFO] [{SharedBackupDelete}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-dggka] is present [2024-01-19T00:10:25.121Z] 2024-01-19 00:10:24 +0000:[INFO] [{SharedBackupDelete}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:10:26.053Z] INFO[2024-01-19 00:10:25] Verifying : Description : Creating source and destination cluster [2024-01-19T00:10:26.053Z] INFO[2024-01-19 00:10:25] Actual:true, Expected: true [2024-01-19T00:10:26.053Z] INFO[2024-01-19 00:10:25] Verifying : Description : Verifying backup restore [tp-restore-1705623025] is not possible for backup [tp-backup-1705622461] with user [testuser4-1705622438] [2024-01-19T00:10:26.053Z] INFO[2024-01-19 00:10:25] Actual:true, Expected: true [2024-01-19T00:10:26.054Z] 2024-01-19 00:10:25 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [95ebe0ab-2852-448c-a013-75d46766a6eb] doesn't have permission to delete backup [tp-backup-1705622461] [2024-01-19T00:10:26.054Z] INFO[2024-01-19 00:10:25] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622461] with user [testuser4-1705622438] [2024-01-19T00:10:26.054Z] INFO[2024-01-19 00:10:25] Actual:true, Expected: true [2024-01-19T00:10:26.310Z] INFO[2024-01-19 00:10:26] Verifying : Description : Verifying backup restore [tp-restore-1705623025] is not possible for backup [tp-backup-1705622471] with user [testuser4-1705622438] [2024-01-19T00:10:26.310Z] INFO[2024-01-19 00:10:26] Actual:true, Expected: true [2024-01-19T00:10:26.310Z] 2024-01-19 00:10:26 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [95ebe0ab-2852-448c-a013-75d46766a6eb] doesn't have permission to delete backup [tp-backup-1705622471] [2024-01-19T00:10:26.310Z] INFO[2024-01-19 00:10:26] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622471] with user [testuser4-1705622438] [2024-01-19T00:10:26.310Z] INFO[2024-01-19 00:10:26] Actual:true, Expected: true [2024-01-19T00:10:26.310Z] INFO[2024-01-19 00:10:26] Verifying : Description : Verifying backup restore [tp-restore-1705623026] is not possible for backup [tp-backup-1705622481] with user [testuser4-1705622438] [2024-01-19T00:10:26.310Z] INFO[2024-01-19 00:10:26] Actual:true, Expected: true [2024-01-19T00:10:26.310Z] 2024-01-19 00:10:26 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [95ebe0ab-2852-448c-a013-75d46766a6eb] doesn't have permission to delete backup [tp-backup-1705622481] [2024-01-19T00:10:26.311Z] INFO[2024-01-19 00:10:26] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622481] with user [testuser4-1705622438] [2024-01-19T00:10:26.311Z] INFO[2024-01-19 00:10:26] Actual:true, Expected: true [2024-01-19T00:10:26.566Z] INFO[2024-01-19 00:10:26] Verifying : Description : Verifying backup restore [tp-restore-1705623026] is not possible for backup [tp-backup-1705622491] with user [testuser4-1705622438] [2024-01-19T00:10:26.566Z] INFO[2024-01-19 00:10:26] Actual:true, Expected: true [2024-01-19T00:10:26.566Z] 2024-01-19 00:10:26 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [95ebe0ab-2852-448c-a013-75d46766a6eb] doesn't have permission to delete backup [tp-backup-1705622491] [2024-01-19T00:10:26.566Z] INFO[2024-01-19 00:10:26] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622491] with user [testuser4-1705622438] [2024-01-19T00:10:26.566Z] INFO[2024-01-19 00:10:26] Actual:true, Expected: true [2024-01-19T00:10:26.822Z] INFO[2024-01-19 00:10:26] Verifying : Description : Verifying backup restore [tp-restore-1705623026] is not possible for backup [tp-backup-1705622501] with user [testuser4-1705622438] [2024-01-19T00:10:26.822Z] INFO[2024-01-19 00:10:26] Actual:true, Expected: true [2024-01-19T00:10:26.822Z] 2024-01-19 00:10:26 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [95ebe0ab-2852-448c-a013-75d46766a6eb] doesn't have permission to delete backup [tp-backup-1705622501] [2024-01-19T00:10:26.822Z] INFO[2024-01-19 00:10:26] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622501] with user [testuser4-1705622438] [2024-01-19T00:10:26.822Z] INFO[2024-01-19 00:10:26] Actual:true, Expected: true [2024-01-19T00:10:26.822Z] INFO[2024-01-19 00:10:26] Verifying : Description : Verifying backup restore [tp-restore-1705623026] is not possible for backup [tp-backup-1705622511] with user [testuser4-1705622438] [2024-01-19T00:10:26.822Z] INFO[2024-01-19 00:10:26] Actual:true, Expected: true [2024-01-19T00:10:27.078Z] 2024-01-19 00:10:26 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [95ebe0ab-2852-448c-a013-75d46766a6eb] doesn't have permission to delete backup [tp-backup-1705622511] [2024-01-19T00:10:27.078Z] INFO[2024-01-19 00:10:26] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622511] with user [testuser4-1705622438] [2024-01-19T00:10:27.078Z] INFO[2024-01-19 00:10:26] Actual:true, Expected: true [2024-01-19T00:10:27.078Z] INFO[2024-01-19 00:10:26] Verifying : Description : Verifying backup restore [tp-restore-1705623026] is not possible for backup [tp-backup-1705622521] with user [testuser4-1705622438] [2024-01-19T00:10:27.078Z] INFO[2024-01-19 00:10:26] Actual:true, Expected: true [2024-01-19T00:10:27.078Z] 2024-01-19 00:10:26 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [95ebe0ab-2852-448c-a013-75d46766a6eb] doesn't have permission to delete backup [tp-backup-1705622521] [2024-01-19T00:10:27.078Z] INFO[2024-01-19 00:10:26] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622521] with user [testuser4-1705622438] [2024-01-19T00:10:27.078Z] INFO[2024-01-19 00:10:26] Actual:true, Expected: true [2024-01-19T00:10:27.339Z] INFO[2024-01-19 00:10:27] Verifying : Description : Verifying backup restore [tp-restore-1705623027] is not possible for backup [tp-backup-1705622531] with user [testuser4-1705622438] [2024-01-19T00:10:27.339Z] INFO[2024-01-19 00:10:27] Actual:true, Expected: true [2024-01-19T00:10:27.339Z] 2024-01-19 00:10:27 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [95ebe0ab-2852-448c-a013-75d46766a6eb] doesn't have permission to delete backup [tp-backup-1705622531] [2024-01-19T00:10:27.339Z] INFO[2024-01-19 00:10:27] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622531] with user [testuser4-1705622438] [2024-01-19T00:10:27.339Z] INFO[2024-01-19 00:10:27] Actual:true, Expected: true [2024-01-19T00:10:27.339Z] INFO[2024-01-19 00:10:27] Verifying : Description : Verifying backup restore [tp-restore-1705623027] is not possible for backup [tp-backup-1705622541] with user [testuser4-1705622438] [2024-01-19T00:10:27.339Z] INFO[2024-01-19 00:10:27] Actual:true, Expected: true [2024-01-19T00:10:27.595Z] 2024-01-19 00:10:27 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [95ebe0ab-2852-448c-a013-75d46766a6eb] doesn't have permission to delete backup [tp-backup-1705622541] [2024-01-19T00:10:27.595Z] INFO[2024-01-19 00:10:27] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622541] with user [testuser4-1705622438] [2024-01-19T00:10:27.595Z] INFO[2024-01-19 00:10:27] Actual:true, Expected: true [2024-01-19T00:10:27.595Z] INFO[2024-01-19 00:10:27] Verifying : Description : Verifying backup restore [tp-restore-1705623027] is not possible for backup [tp-backup-1705622551] with user [testuser4-1705622438] [2024-01-19T00:10:27.595Z] INFO[2024-01-19 00:10:27] Actual:true, Expected: true [2024-01-19T00:10:27.595Z] 2024-01-19 00:10:27 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [95ebe0ab-2852-448c-a013-75d46766a6eb] doesn't have permission to delete backup [tp-backup-1705622551] [2024-01-19T00:10:27.595Z] INFO[2024-01-19 00:10:27] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622551] with user [testuser4-1705622438] [2024-01-19T00:10:27.595Z] INFO[2024-01-19 00:10:27] Actual:true, Expected: true [2024-01-19T00:10:27.852Z] 2024-01-19 00:10:27 +0000:[INFO] [{SharedBackupDelete}] Registering Source and Destination clusters from user context for user -testuser5-1705622438 [2024-01-19T00:10:27.852Z] INFO[2024-01-19 00:10:27] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T00:10:27.852Z] INFO[2024-01-19 00:10:27] Actual:true, Expected: true [2024-01-19T00:10:27.852Z] 2024-01-19 00:10:27 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T00:10:27.852Z] 2024-01-19 00:10:27 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T00:10:27.852Z] 2024-01-19 00:10:27 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T00:10:27.852Z] 2024-01-19 00:10:27 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T00:10:27.852Z] 2024-01-19 00:10:27 +0000:[INFO] [{SharedBackupDelete}] Create cluster [source-cluster] in org [default] [2024-01-19T00:10:27.852Z] 2024-01-19 00:10:27 +0000:[INFO] [{SharedBackupDelete}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T00:10:27.852Z] 2024-01-19 00:10:27 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T00:10:27.852Z] 2024-01-19 00:10:27 +0000:[INFO] [{SharedBackupDelete}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:10:27.852Z] 2024-01-19 00:10:27 +0000:[INFO] [{SharedBackupDelete}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T00:10:27.852Z] 2024-01-19 00:10:27 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T00:10:27.852Z] 2024-01-19 00:10:27 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-pksdz] for IBM [2024-01-19T00:10:27.852Z] 2024-01-19 00:10:27 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-pksdz] for org [default] with [ibm] as provider [2024-01-19T00:10:27.852Z] 2024-01-19 00:10:27 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:27.852Z] 2024-01-19 00:10:27 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:27.853Z] 2024-01-19 00:10:27 +0000:[WARNING] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-pksdz] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName5 LastName5 with roles: [px-backup-app.user]] [2024-01-19T00:10:27.853Z] 2024-01-19 00:10:27 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName5 LastName5 with roles: [px-backup-app.user] [2024-01-19T00:10:28.110Z] 2024-01-19 00:10:27 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-pksdz from admin context and sharing with all the users [2024-01-19T00:10:28.110Z] 2024-01-19 00:10:27 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-pksdz] for org [default] with [ibm] as provider [2024-01-19T00:10:28.110Z] 2024-01-19 00:10:27 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:28.110Z] 2024-01-19 00:10:27 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:28.365Z] 2024-01-19 00:10:28 +0000:[INFO] [{SharedBackupDelete}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-pksdz] is present [2024-01-19T00:10:28.365Z] 2024-01-19 00:10:28 +0000:[INFO] [{SharedBackupDelete}] Create cluster [source-cluster] in org [default] [2024-01-19T00:10:28.961Z] 2024-01-19 00:10:28 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-qtxon] for IBM [2024-01-19T00:10:28.961Z] 2024-01-19 00:10:28 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-qtxon] for org [default] with [ibm] as provider [2024-01-19T00:10:28.961Z] 2024-01-19 00:10:28 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:28.961Z] 2024-01-19 00:10:28 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:28.961Z] 2024-01-19 00:10:28 +0000:[WARNING] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-qtxon] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName5 LastName5 with roles: [px-backup-app.user]] [2024-01-19T00:10:28.961Z] 2024-01-19 00:10:28 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName5 LastName5 with roles: [px-backup-app.user] [2024-01-19T00:10:29.522Z] 2024-01-19 00:10:29 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-qtxon from admin context and sharing with all the users [2024-01-19T00:10:29.522Z] 2024-01-19 00:10:29 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-qtxon] for org [default] with [ibm] as provider [2024-01-19T00:10:29.522Z] 2024-01-19 00:10:29 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:29.522Z] 2024-01-19 00:10:29 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:29.522Z] 2024-01-19 00:10:29 +0000:[INFO] [{SharedBackupDelete}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-qtxon] is present [2024-01-19T00:10:29.780Z] 2024-01-19 00:10:29 +0000:[INFO] [{SharedBackupDelete}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:10:30.342Z] INFO[2024-01-19 00:10:30] Verifying : Description : Creating source and destination cluster [2024-01-19T00:10:30.342Z] INFO[2024-01-19 00:10:30] Actual:true, Expected: true [2024-01-19T00:10:30.598Z] INFO[2024-01-19 00:10:30] Verifying : Description : Verifying backup restore [tp-restore-1705623030] is not possible for backup [tp-backup-1705622461] with user [testuser5-1705622438] [2024-01-19T00:10:30.598Z] INFO[2024-01-19 00:10:30] Actual:true, Expected: true [2024-01-19T00:10:30.598Z] 2024-01-19 00:10:30 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [a4c4106c-2fa5-494f-adb2-f36bb650cc3a] doesn't have permission to delete backup [tp-backup-1705622461] [2024-01-19T00:10:30.598Z] INFO[2024-01-19 00:10:30] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622461] with user [testuser5-1705622438] [2024-01-19T00:10:30.598Z] INFO[2024-01-19 00:10:30] Actual:true, Expected: true [2024-01-19T00:10:30.598Z] INFO[2024-01-19 00:10:30] Verifying : Description : Verifying backup restore [tp-restore-1705623030] is not possible for backup [tp-backup-1705622471] with user [testuser5-1705622438] [2024-01-19T00:10:30.598Z] INFO[2024-01-19 00:10:30] Actual:true, Expected: true [2024-01-19T00:10:30.854Z] 2024-01-19 00:10:30 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [a4c4106c-2fa5-494f-adb2-f36bb650cc3a] doesn't have permission to delete backup [tp-backup-1705622471] [2024-01-19T00:10:30.854Z] INFO[2024-01-19 00:10:30] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622471] with user [testuser5-1705622438] [2024-01-19T00:10:30.854Z] INFO[2024-01-19 00:10:30] Actual:true, Expected: true [2024-01-19T00:10:30.854Z] INFO[2024-01-19 00:10:30] Verifying : Description : Verifying backup restore [tp-restore-1705623030] is not possible for backup [tp-backup-1705622481] with user [testuser5-1705622438] [2024-01-19T00:10:30.854Z] INFO[2024-01-19 00:10:30] Actual:true, Expected: true [2024-01-19T00:10:30.854Z] 2024-01-19 00:10:30 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [a4c4106c-2fa5-494f-adb2-f36bb650cc3a] doesn't have permission to delete backup [tp-backup-1705622481] [2024-01-19T00:10:30.854Z] INFO[2024-01-19 00:10:30] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622481] with user [testuser5-1705622438] [2024-01-19T00:10:30.854Z] INFO[2024-01-19 00:10:30] Actual:true, Expected: true [2024-01-19T00:10:31.113Z] INFO[2024-01-19 00:10:30] Verifying : Description : Verifying backup restore [tp-restore-1705623030] is not possible for backup [tp-backup-1705622491] with user [testuser5-1705622438] [2024-01-19T00:10:31.113Z] INFO[2024-01-19 00:10:30] Actual:true, Expected: true [2024-01-19T00:10:31.113Z] 2024-01-19 00:10:30 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [a4c4106c-2fa5-494f-adb2-f36bb650cc3a] doesn't have permission to delete backup [tp-backup-1705622491] [2024-01-19T00:10:31.114Z] INFO[2024-01-19 00:10:30] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622491] with user [testuser5-1705622438] [2024-01-19T00:10:31.114Z] INFO[2024-01-19 00:10:30] Actual:true, Expected: true [2024-01-19T00:10:31.375Z] INFO[2024-01-19 00:10:31] Verifying : Description : Verifying backup restore [tp-restore-1705623031] is not possible for backup [tp-backup-1705622501] with user [testuser5-1705622438] [2024-01-19T00:10:31.375Z] INFO[2024-01-19 00:10:31] Actual:true, Expected: true [2024-01-19T00:10:31.375Z] 2024-01-19 00:10:31 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [a4c4106c-2fa5-494f-adb2-f36bb650cc3a] doesn't have permission to delete backup [tp-backup-1705622501] [2024-01-19T00:10:31.375Z] INFO[2024-01-19 00:10:31] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622501] with user [testuser5-1705622438] [2024-01-19T00:10:31.375Z] INFO[2024-01-19 00:10:31] Actual:true, Expected: true [2024-01-19T00:10:31.636Z] INFO[2024-01-19 00:10:31] Verifying : Description : Verifying backup restore [tp-restore-1705623031] is not possible for backup [tp-backup-1705622511] with user [testuser5-1705622438] [2024-01-19T00:10:31.636Z] INFO[2024-01-19 00:10:31] Actual:true, Expected: true [2024-01-19T00:10:31.636Z] 2024-01-19 00:10:31 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [a4c4106c-2fa5-494f-adb2-f36bb650cc3a] doesn't have permission to delete backup [tp-backup-1705622511] [2024-01-19T00:10:31.636Z] INFO[2024-01-19 00:10:31] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622511] with user [testuser5-1705622438] [2024-01-19T00:10:31.636Z] INFO[2024-01-19 00:10:31] Actual:true, Expected: true [2024-01-19T00:10:31.636Z] INFO[2024-01-19 00:10:31] Verifying : Description : Verifying backup restore [tp-restore-1705623031] is not possible for backup [tp-backup-1705622521] with user [testuser5-1705622438] [2024-01-19T00:10:31.636Z] INFO[2024-01-19 00:10:31] Actual:true, Expected: true [2024-01-19T00:10:31.894Z] 2024-01-19 00:10:31 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [a4c4106c-2fa5-494f-adb2-f36bb650cc3a] doesn't have permission to delete backup [tp-backup-1705622521] [2024-01-19T00:10:31.894Z] INFO[2024-01-19 00:10:31] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622521] with user [testuser5-1705622438] [2024-01-19T00:10:31.894Z] INFO[2024-01-19 00:10:31] Actual:true, Expected: true [2024-01-19T00:10:31.894Z] INFO[2024-01-19 00:10:31] Verifying : Description : Verifying backup restore [tp-restore-1705623031] is not possible for backup [tp-backup-1705622531] with user [testuser5-1705622438] [2024-01-19T00:10:31.894Z] INFO[2024-01-19 00:10:31] Actual:true, Expected: true [2024-01-19T00:10:31.894Z] 2024-01-19 00:10:31 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [a4c4106c-2fa5-494f-adb2-f36bb650cc3a] doesn't have permission to delete backup [tp-backup-1705622531] [2024-01-19T00:10:31.894Z] INFO[2024-01-19 00:10:31] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622531] with user [testuser5-1705622438] [2024-01-19T00:10:31.894Z] INFO[2024-01-19 00:10:31] Actual:true, Expected: true [2024-01-19T00:10:32.152Z] INFO[2024-01-19 00:10:31] Verifying : Description : Verifying backup restore [tp-restore-1705623031] is not possible for backup [tp-backup-1705622541] with user [testuser5-1705622438] [2024-01-19T00:10:32.152Z] INFO[2024-01-19 00:10:31] Actual:true, Expected: true [2024-01-19T00:10:32.152Z] 2024-01-19 00:10:31 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [a4c4106c-2fa5-494f-adb2-f36bb650cc3a] doesn't have permission to delete backup [tp-backup-1705622541] [2024-01-19T00:10:32.152Z] INFO[2024-01-19 00:10:31] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622541] with user [testuser5-1705622438] [2024-01-19T00:10:32.152Z] INFO[2024-01-19 00:10:31] Actual:true, Expected: true [2024-01-19T00:10:32.409Z] INFO[2024-01-19 00:10:32] Verifying : Description : Verifying backup restore [tp-restore-1705623032] is not possible for backup [tp-backup-1705622551] with user [testuser5-1705622438] [2024-01-19T00:10:32.409Z] INFO[2024-01-19 00:10:32] Actual:true, Expected: true [2024-01-19T00:10:32.410Z] 2024-01-19 00:10:32 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [a4c4106c-2fa5-494f-adb2-f36bb650cc3a] doesn't have permission to delete backup [tp-backup-1705622551] [2024-01-19T00:10:32.410Z] INFO[2024-01-19 00:10:32] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622551] with user [testuser5-1705622438] [2024-01-19T00:10:32.410Z] INFO[2024-01-19 00:10:32] Actual:true, Expected: true [2024-01-19T00:10:32.410Z] 2024-01-19 00:10:32 +0000:[INFO] [{SharedBackupDelete}] Registering Source and Destination clusters from user context for user -testuser2-1705622438 [2024-01-19T00:10:32.410Z] INFO[2024-01-19 00:10:32] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T00:10:32.410Z] INFO[2024-01-19 00:10:32] Actual:true, Expected: true [2024-01-19T00:10:32.410Z] 2024-01-19 00:10:32 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T00:10:32.410Z] 2024-01-19 00:10:32 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T00:10:32.410Z] 2024-01-19 00:10:32 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T00:10:32.410Z] 2024-01-19 00:10:32 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T00:10:32.410Z] 2024-01-19 00:10:32 +0000:[INFO] [{SharedBackupDelete}] Create cluster [source-cluster] in org [default] [2024-01-19T00:10:32.410Z] 2024-01-19 00:10:32 +0000:[INFO] [{SharedBackupDelete}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T00:10:32.410Z] 2024-01-19 00:10:32 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T00:10:32.410Z] 2024-01-19 00:10:32 +0000:[INFO] [{SharedBackupDelete}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:10:32.410Z] 2024-01-19 00:10:32 +0000:[INFO] [{SharedBackupDelete}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T00:10:32.410Z] 2024-01-19 00:10:32 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T00:10:32.410Z] 2024-01-19 00:10:32 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-nrkzb] for IBM [2024-01-19T00:10:32.410Z] 2024-01-19 00:10:32 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-nrkzb] for org [default] with [ibm] as provider [2024-01-19T00:10:32.410Z] 2024-01-19 00:10:32 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:32.410Z] 2024-01-19 00:10:32 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:32.410Z] 2024-01-19 00:10:32 +0000:[WARNING] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-nrkzb] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName2 LastName2 with roles: [px-backup-app.user]] [2024-01-19T00:10:32.411Z] 2024-01-19 00:10:32 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName2 LastName2 with roles: [px-backup-app.user] [2024-01-19T00:10:32.668Z] 2024-01-19 00:10:32 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-nrkzb from admin context and sharing with all the users [2024-01-19T00:10:32.668Z] 2024-01-19 00:10:32 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-nrkzb] for org [default] with [ibm] as provider [2024-01-19T00:10:32.668Z] 2024-01-19 00:10:32 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:32.668Z] 2024-01-19 00:10:32 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:32.925Z] 2024-01-19 00:10:32 +0000:[INFO] [{SharedBackupDelete}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-nrkzb] is present [2024-01-19T00:10:32.925Z] 2024-01-19 00:10:32 +0000:[INFO] [{SharedBackupDelete}] Create cluster [source-cluster] in org [default] [2024-01-19T00:10:34.290Z] 2024-01-19 00:10:33 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-pmaxe] for IBM [2024-01-19T00:10:34.290Z] 2024-01-19 00:10:33 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-pmaxe] for org [default] with [ibm] as provider [2024-01-19T00:10:34.290Z] 2024-01-19 00:10:33 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:34.290Z] 2024-01-19 00:10:33 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:34.291Z] 2024-01-19 00:10:33 +0000:[WARNING] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-pmaxe] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName2 LastName2 with roles: [px-backup-app.user]] [2024-01-19T00:10:34.291Z] 2024-01-19 00:10:33 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName2 LastName2 with roles: [px-backup-app.user] [2024-01-19T00:10:34.291Z] 2024-01-19 00:10:34 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-pmaxe from admin context and sharing with all the users [2024-01-19T00:10:34.291Z] 2024-01-19 00:10:34 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-pmaxe] for org [default] with [ibm] as provider [2024-01-19T00:10:34.291Z] 2024-01-19 00:10:34 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:34.291Z] 2024-01-19 00:10:34 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:34.547Z] 2024-01-19 00:10:34 +0000:[INFO] [{SharedBackupDelete}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-pmaxe] is present [2024-01-19T00:10:34.547Z] 2024-01-19 00:10:34 +0000:[INFO] [{SharedBackupDelete}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:10:35.480Z] INFO[2024-01-19 00:10:35] Verifying : Description : Creating source and destination cluster [2024-01-19T00:10:35.480Z] INFO[2024-01-19 00:10:35] Actual:true, Expected: true [2024-01-19T00:10:35.481Z] INFO[2024-01-19 00:10:35] Verifying : Description : Verifying backup restore [tp-restore-1705623035] is not possible for backup [tp-backup-1705622461] with user [testuser2-1705622438] [2024-01-19T00:10:35.481Z] INFO[2024-01-19 00:10:35] Actual:true, Expected: true [2024-01-19T00:10:35.481Z] 2024-01-19 00:10:35 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [c161acfd-6047-4d40-93ba-9f88c22efdf5] doesn't have permission to delete backup [tp-backup-1705622461] [2024-01-19T00:10:35.481Z] INFO[2024-01-19 00:10:35] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622461] with user [testuser2-1705622438] [2024-01-19T00:10:35.481Z] INFO[2024-01-19 00:10:35] Actual:true, Expected: true [2024-01-19T00:10:35.736Z] INFO[2024-01-19 00:10:35] Verifying : Description : Verifying backup restore [tp-restore-1705623035] is not possible for backup [tp-backup-1705622471] with user [testuser2-1705622438] [2024-01-19T00:10:35.736Z] INFO[2024-01-19 00:10:35] Actual:true, Expected: true [2024-01-19T00:10:35.737Z] 2024-01-19 00:10:35 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [c161acfd-6047-4d40-93ba-9f88c22efdf5] doesn't have permission to delete backup [tp-backup-1705622471] [2024-01-19T00:10:35.737Z] INFO[2024-01-19 00:10:35] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622471] with user [testuser2-1705622438] [2024-01-19T00:10:35.737Z] INFO[2024-01-19 00:10:35] Actual:true, Expected: true [2024-01-19T00:10:35.992Z] INFO[2024-01-19 00:10:35] Verifying : Description : Verifying backup restore [tp-restore-1705623035] is not possible for backup [tp-backup-1705622481] with user [testuser2-1705622438] [2024-01-19T00:10:35.992Z] INFO[2024-01-19 00:10:35] Actual:true, Expected: true [2024-01-19T00:10:35.992Z] 2024-01-19 00:10:35 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [c161acfd-6047-4d40-93ba-9f88c22efdf5] doesn't have permission to delete backup [tp-backup-1705622481] [2024-01-19T00:10:35.992Z] INFO[2024-01-19 00:10:35] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622481] with user [testuser2-1705622438] [2024-01-19T00:10:35.992Z] INFO[2024-01-19 00:10:35] Actual:true, Expected: true [2024-01-19T00:10:35.993Z] INFO[2024-01-19 00:10:35] Verifying : Description : Verifying backup restore [tp-restore-1705623035] is not possible for backup [tp-backup-1705622491] with user [testuser2-1705622438] [2024-01-19T00:10:35.993Z] INFO[2024-01-19 00:10:35] Actual:true, Expected: true [2024-01-19T00:10:36.248Z] 2024-01-19 00:10:35 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [c161acfd-6047-4d40-93ba-9f88c22efdf5] doesn't have permission to delete backup [tp-backup-1705622491] [2024-01-19T00:10:36.248Z] INFO[2024-01-19 00:10:35] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622491] with user [testuser2-1705622438] [2024-01-19T00:10:36.248Z] INFO[2024-01-19 00:10:35] Actual:true, Expected: true [2024-01-19T00:10:36.249Z] INFO[2024-01-19 00:10:36] Verifying : Description : Verifying backup restore [tp-restore-1705623036] is not possible for backup [tp-backup-1705622501] with user [testuser2-1705622438] [2024-01-19T00:10:36.249Z] INFO[2024-01-19 00:10:36] Actual:true, Expected: true [2024-01-19T00:10:36.249Z] 2024-01-19 00:10:36 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [c161acfd-6047-4d40-93ba-9f88c22efdf5] doesn't have permission to delete backup [tp-backup-1705622501] [2024-01-19T00:10:36.249Z] INFO[2024-01-19 00:10:36] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622501] with user [testuser2-1705622438] [2024-01-19T00:10:36.249Z] INFO[2024-01-19 00:10:36] Actual:true, Expected: true [2024-01-19T00:10:36.505Z] INFO[2024-01-19 00:10:36] Verifying : Description : Verifying backup restore [tp-restore-1705623036] is not possible for backup [tp-backup-1705622511] with user [testuser2-1705622438] [2024-01-19T00:10:36.505Z] INFO[2024-01-19 00:10:36] Actual:true, Expected: true [2024-01-19T00:10:36.505Z] 2024-01-19 00:10:36 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [c161acfd-6047-4d40-93ba-9f88c22efdf5] doesn't have permission to delete backup [tp-backup-1705622511] [2024-01-19T00:10:36.505Z] INFO[2024-01-19 00:10:36] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622511] with user [testuser2-1705622438] [2024-01-19T00:10:36.505Z] INFO[2024-01-19 00:10:36] Actual:true, Expected: true [2024-01-19T00:10:36.505Z] INFO[2024-01-19 00:10:36] Verifying : Description : Verifying backup restore [tp-restore-1705623036] is not possible for backup [tp-backup-1705622521] with user [testuser2-1705622438] [2024-01-19T00:10:36.505Z] INFO[2024-01-19 00:10:36] Actual:true, Expected: true [2024-01-19T00:10:36.762Z] 2024-01-19 00:10:36 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [c161acfd-6047-4d40-93ba-9f88c22efdf5] doesn't have permission to delete backup [tp-backup-1705622521] [2024-01-19T00:10:36.762Z] INFO[2024-01-19 00:10:36] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622521] with user [testuser2-1705622438] [2024-01-19T00:10:36.762Z] INFO[2024-01-19 00:10:36] Actual:true, Expected: true [2024-01-19T00:10:36.762Z] INFO[2024-01-19 00:10:36] Verifying : Description : Verifying backup restore [tp-restore-1705623036] is not possible for backup [tp-backup-1705622531] with user [testuser2-1705622438] [2024-01-19T00:10:36.762Z] INFO[2024-01-19 00:10:36] Actual:true, Expected: true [2024-01-19T00:10:36.762Z] 2024-01-19 00:10:36 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [c161acfd-6047-4d40-93ba-9f88c22efdf5] doesn't have permission to delete backup [tp-backup-1705622531] [2024-01-19T00:10:36.762Z] INFO[2024-01-19 00:10:36] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622531] with user [testuser2-1705622438] [2024-01-19T00:10:36.762Z] INFO[2024-01-19 00:10:36] Actual:true, Expected: true [2024-01-19T00:10:37.018Z] INFO[2024-01-19 00:10:36] Verifying : Description : Verifying backup restore [tp-restore-1705623036] is not possible for backup [tp-backup-1705622541] with user [testuser2-1705622438] [2024-01-19T00:10:37.018Z] INFO[2024-01-19 00:10:36] Actual:true, Expected: true [2024-01-19T00:10:37.018Z] 2024-01-19 00:10:36 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [c161acfd-6047-4d40-93ba-9f88c22efdf5] doesn't have permission to delete backup [tp-backup-1705622541] [2024-01-19T00:10:37.018Z] INFO[2024-01-19 00:10:36] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622541] with user [testuser2-1705622438] [2024-01-19T00:10:37.018Z] INFO[2024-01-19 00:10:36] Actual:true, Expected: true [2024-01-19T00:10:37.018Z] INFO[2024-01-19 00:10:36] Verifying : Description : Verifying backup restore [tp-restore-1705623036] is not possible for backup [tp-backup-1705622551] with user [testuser2-1705622438] [2024-01-19T00:10:37.018Z] INFO[2024-01-19 00:10:36] Actual:true, Expected: true [2024-01-19T00:10:37.274Z] 2024-01-19 00:10:37 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [c161acfd-6047-4d40-93ba-9f88c22efdf5] doesn't have permission to delete backup [tp-backup-1705622551] [2024-01-19T00:10:37.274Z] INFO[2024-01-19 00:10:37] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622551] with user [testuser2-1705622438] [2024-01-19T00:10:37.274Z] INFO[2024-01-19 00:10:37] Actual:true, Expected: true [2024-01-19T00:10:37.274Z] 2024-01-19 00:10:37 +0000:[INFO] [{SharedBackupDelete}] Registering Source and Destination clusters from user context for user -testuser6-1705622438 [2024-01-19T00:10:37.274Z] INFO[2024-01-19 00:10:37] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T00:10:37.274Z] INFO[2024-01-19 00:10:37] Actual:true, Expected: true [2024-01-19T00:10:37.274Z] 2024-01-19 00:10:37 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T00:10:37.274Z] 2024-01-19 00:10:37 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T00:10:37.274Z] 2024-01-19 00:10:37 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T00:10:37.274Z] 2024-01-19 00:10:37 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T00:10:37.275Z] 2024-01-19 00:10:37 +0000:[INFO] [{SharedBackupDelete}] Create cluster [source-cluster] in org [default] [2024-01-19T00:10:37.275Z] 2024-01-19 00:10:37 +0000:[INFO] [{SharedBackupDelete}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T00:10:37.275Z] 2024-01-19 00:10:37 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T00:10:37.275Z] 2024-01-19 00:10:37 +0000:[INFO] [{SharedBackupDelete}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:10:37.275Z] 2024-01-19 00:10:37 +0000:[INFO] [{SharedBackupDelete}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T00:10:37.275Z] 2024-01-19 00:10:37 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T00:10:37.275Z] 2024-01-19 00:10:37 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-bnkuv] for IBM [2024-01-19T00:10:37.275Z] 2024-01-19 00:10:37 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-bnkuv] for org [default] with [ibm] as provider [2024-01-19T00:10:37.275Z] 2024-01-19 00:10:37 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:37.275Z] 2024-01-19 00:10:37 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:37.275Z] 2024-01-19 00:10:37 +0000:[WARNING] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-bnkuv] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName6 LastName6 with roles: [px-backup-app.user]] [2024-01-19T00:10:37.275Z] 2024-01-19 00:10:37 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName6 LastName6 with roles: [px-backup-app.user] [2024-01-19T00:10:37.531Z] 2024-01-19 00:10:37 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-bnkuv from admin context and sharing with all the users [2024-01-19T00:10:37.531Z] 2024-01-19 00:10:37 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-bnkuv] for org [default] with [ibm] as provider [2024-01-19T00:10:37.531Z] 2024-01-19 00:10:37 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:37.531Z] 2024-01-19 00:10:37 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:37.786Z] 2024-01-19 00:10:37 +0000:[INFO] [{SharedBackupDelete}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-bnkuv] is present [2024-01-19T00:10:37.786Z] 2024-01-19 00:10:37 +0000:[INFO] [{SharedBackupDelete}] Create cluster [source-cluster] in org [default] [2024-01-19T00:10:38.714Z] 2024-01-19 00:10:38 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-wconk] for IBM [2024-01-19T00:10:38.715Z] 2024-01-19 00:10:38 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-wconk] for org [default] with [ibm] as provider [2024-01-19T00:10:38.715Z] 2024-01-19 00:10:38 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:38.715Z] 2024-01-19 00:10:38 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:38.715Z] 2024-01-19 00:10:38 +0000:[WARNING] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-wconk] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName6 LastName6 with roles: [px-backup-app.user]] [2024-01-19T00:10:38.715Z] 2024-01-19 00:10:38 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName6 LastName6 with roles: [px-backup-app.user] [2024-01-19T00:10:38.971Z] 2024-01-19 00:10:38 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-wconk from admin context and sharing with all the users [2024-01-19T00:10:38.971Z] 2024-01-19 00:10:38 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-wconk] for org [default] with [ibm] as provider [2024-01-19T00:10:38.971Z] 2024-01-19 00:10:38 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:38.971Z] 2024-01-19 00:10:38 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:38.971Z] 2024-01-19 00:10:38 +0000:[INFO] [{SharedBackupDelete}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-wconk] is present [2024-01-19T00:10:39.227Z] 2024-01-19 00:10:39 +0000:[INFO] [{SharedBackupDelete}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:10:40.154Z] INFO[2024-01-19 00:10:39] Verifying : Description : Creating source and destination cluster [2024-01-19T00:10:40.154Z] INFO[2024-01-19 00:10:39] Actual:true, Expected: true [2024-01-19T00:10:40.154Z] INFO[2024-01-19 00:10:39] Verifying : Description : Verifying backup restore [tp-restore-1705623039] is not possible for backup [tp-backup-1705622461] with user [testuser6-1705622438] [2024-01-19T00:10:40.154Z] INFO[2024-01-19 00:10:39] Actual:true, Expected: true [2024-01-19T00:10:40.154Z] 2024-01-19 00:10:39 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [88f8d071-50ff-4aad-a907-c3cd4ea9027b] doesn't have permission to delete backup [tp-backup-1705622461] [2024-01-19T00:10:40.154Z] INFO[2024-01-19 00:10:39] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622461] with user [testuser6-1705622438] [2024-01-19T00:10:40.154Z] INFO[2024-01-19 00:10:39] Actual:true, Expected: true [2024-01-19T00:10:40.410Z] INFO[2024-01-19 00:10:40] Verifying : Description : Verifying backup restore [tp-restore-1705623040] is not possible for backup [tp-backup-1705622471] with user [testuser6-1705622438] [2024-01-19T00:10:40.411Z] INFO[2024-01-19 00:10:40] Actual:true, Expected: true [2024-01-19T00:10:40.411Z] 2024-01-19 00:10:40 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [88f8d071-50ff-4aad-a907-c3cd4ea9027b] doesn't have permission to delete backup [tp-backup-1705622471] [2024-01-19T00:10:40.411Z] INFO[2024-01-19 00:10:40] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622471] with user [testuser6-1705622438] [2024-01-19T00:10:40.411Z] INFO[2024-01-19 00:10:40] Actual:true, Expected: true [2024-01-19T00:10:40.411Z] INFO[2024-01-19 00:10:40] Verifying : Description : Verifying backup restore [tp-restore-1705623040] is not possible for backup [tp-backup-1705622481] with user [testuser6-1705622438] [2024-01-19T00:10:40.411Z] INFO[2024-01-19 00:10:40] Actual:true, Expected: true [2024-01-19T00:10:40.411Z] 2024-01-19 00:10:40 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [88f8d071-50ff-4aad-a907-c3cd4ea9027b] doesn't have permission to delete backup [tp-backup-1705622481] [2024-01-19T00:10:40.411Z] INFO[2024-01-19 00:10:40] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622481] with user [testuser6-1705622438] [2024-01-19T00:10:40.411Z] INFO[2024-01-19 00:10:40] Actual:true, Expected: true [2024-01-19T00:10:40.667Z] INFO[2024-01-19 00:10:40] Verifying : Description : Verifying backup restore [tp-restore-1705623040] is not possible for backup [tp-backup-1705622491] with user [testuser6-1705622438] [2024-01-19T00:10:40.667Z] INFO[2024-01-19 00:10:40] Actual:true, Expected: true [2024-01-19T00:10:40.667Z] 2024-01-19 00:10:40 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [88f8d071-50ff-4aad-a907-c3cd4ea9027b] doesn't have permission to delete backup [tp-backup-1705622491] [2024-01-19T00:10:40.667Z] INFO[2024-01-19 00:10:40] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622491] with user [testuser6-1705622438] [2024-01-19T00:10:40.667Z] INFO[2024-01-19 00:10:40] Actual:true, Expected: true [2024-01-19T00:10:40.923Z] INFO[2024-01-19 00:10:40] Verifying : Description : Verifying backup restore [tp-restore-1705623040] is not possible for backup [tp-backup-1705622501] with user [testuser6-1705622438] [2024-01-19T00:10:40.923Z] INFO[2024-01-19 00:10:40] Actual:true, Expected: true [2024-01-19T00:10:40.923Z] 2024-01-19 00:10:40 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [88f8d071-50ff-4aad-a907-c3cd4ea9027b] doesn't have permission to delete backup [tp-backup-1705622501] [2024-01-19T00:10:40.923Z] INFO[2024-01-19 00:10:40] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622501] with user [testuser6-1705622438] [2024-01-19T00:10:40.923Z] INFO[2024-01-19 00:10:40] Actual:true, Expected: true [2024-01-19T00:10:40.923Z] INFO[2024-01-19 00:10:40] Verifying : Description : Verifying backup restore [tp-restore-1705623040] is not possible for backup [tp-backup-1705622511] with user [testuser6-1705622438] [2024-01-19T00:10:40.923Z] INFO[2024-01-19 00:10:40] Actual:true, Expected: true [2024-01-19T00:10:40.923Z] 2024-01-19 00:10:40 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [88f8d071-50ff-4aad-a907-c3cd4ea9027b] doesn't have permission to delete backup [tp-backup-1705622511] [2024-01-19T00:10:40.923Z] INFO[2024-01-19 00:10:40] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622511] with user [testuser6-1705622438] [2024-01-19T00:10:40.923Z] INFO[2024-01-19 00:10:40] Actual:true, Expected: true [2024-01-19T00:10:41.180Z] INFO[2024-01-19 00:10:41] Verifying : Description : Verifying backup restore [tp-restore-1705623040] is not possible for backup [tp-backup-1705622521] with user [testuser6-1705622438] [2024-01-19T00:10:41.180Z] INFO[2024-01-19 00:10:41] Actual:true, Expected: true [2024-01-19T00:10:41.180Z] 2024-01-19 00:10:41 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [88f8d071-50ff-4aad-a907-c3cd4ea9027b] doesn't have permission to delete backup [tp-backup-1705622521] [2024-01-19T00:10:41.180Z] INFO[2024-01-19 00:10:41] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622521] with user [testuser6-1705622438] [2024-01-19T00:10:41.180Z] INFO[2024-01-19 00:10:41] Actual:true, Expected: true [2024-01-19T00:10:41.436Z] INFO[2024-01-19 00:10:41] Verifying : Description : Verifying backup restore [tp-restore-1705623041] is not possible for backup [tp-backup-1705622531] with user [testuser6-1705622438] [2024-01-19T00:10:41.436Z] INFO[2024-01-19 00:10:41] Actual:true, Expected: true [2024-01-19T00:10:41.436Z] 2024-01-19 00:10:41 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [88f8d071-50ff-4aad-a907-c3cd4ea9027b] doesn't have permission to delete backup [tp-backup-1705622531] [2024-01-19T00:10:41.436Z] INFO[2024-01-19 00:10:41] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622531] with user [testuser6-1705622438] [2024-01-19T00:10:41.436Z] INFO[2024-01-19 00:10:41] Actual:true, Expected: true [2024-01-19T00:10:41.692Z] INFO[2024-01-19 00:10:41] Verifying : Description : Verifying backup restore [tp-restore-1705623041] is not possible for backup [tp-backup-1705622541] with user [testuser6-1705622438] [2024-01-19T00:10:41.692Z] INFO[2024-01-19 00:10:41] Actual:true, Expected: true [2024-01-19T00:10:41.692Z] 2024-01-19 00:10:41 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [88f8d071-50ff-4aad-a907-c3cd4ea9027b] doesn't have permission to delete backup [tp-backup-1705622541] [2024-01-19T00:10:41.692Z] INFO[2024-01-19 00:10:41] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622541] with user [testuser6-1705622438] [2024-01-19T00:10:41.692Z] INFO[2024-01-19 00:10:41] Actual:true, Expected: true [2024-01-19T00:10:41.692Z] INFO[2024-01-19 00:10:41] Verifying : Description : Verifying backup restore [tp-restore-1705623041] is not possible for backup [tp-backup-1705622551] with user [testuser6-1705622438] [2024-01-19T00:10:41.692Z] INFO[2024-01-19 00:10:41] Actual:true, Expected: true [2024-01-19T00:10:41.692Z] 2024-01-19 00:10:41 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [88f8d071-50ff-4aad-a907-c3cd4ea9027b] doesn't have permission to delete backup [tp-backup-1705622551] [2024-01-19T00:10:41.692Z] INFO[2024-01-19 00:10:41] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622551] with user [testuser6-1705622438] [2024-01-19T00:10:41.692Z] INFO[2024-01-19 00:10:41] Actual:true, Expected: true [2024-01-19T00:10:41.949Z] 2024-01-19 00:10:41 +0000:[INFO] [{SharedBackupDelete}] Registering Source and Destination clusters from user context for user -testuser1-1705622438 [2024-01-19T00:10:41.949Z] INFO[2024-01-19 00:10:41] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T00:10:41.949Z] INFO[2024-01-19 00:10:41] Actual:true, Expected: true [2024-01-19T00:10:41.949Z] 2024-01-19 00:10:41 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T00:10:41.949Z] 2024-01-19 00:10:41 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T00:10:41.949Z] 2024-01-19 00:10:41 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T00:10:41.950Z] 2024-01-19 00:10:41 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T00:10:41.950Z] 2024-01-19 00:10:41 +0000:[INFO] [{SharedBackupDelete}] Create cluster [source-cluster] in org [default] [2024-01-19T00:10:41.950Z] 2024-01-19 00:10:41 +0000:[INFO] [{SharedBackupDelete}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T00:10:41.950Z] 2024-01-19 00:10:41 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T00:10:41.950Z] 2024-01-19 00:10:41 +0000:[INFO] [{SharedBackupDelete}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:10:41.950Z] 2024-01-19 00:10:41 +0000:[INFO] [{SharedBackupDelete}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T00:10:41.950Z] 2024-01-19 00:10:41 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T00:10:41.950Z] 2024-01-19 00:10:41 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-hzzir] for IBM [2024-01-19T00:10:41.950Z] 2024-01-19 00:10:41 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-hzzir] for org [default] with [ibm] as provider [2024-01-19T00:10:41.950Z] 2024-01-19 00:10:41 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:41.950Z] 2024-01-19 00:10:41 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:41.950Z] 2024-01-19 00:10:41 +0000:[WARNING] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-hzzir] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user]] [2024-01-19T00:10:41.950Z] 2024-01-19 00:10:41 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user] [2024-01-19T00:10:42.207Z] 2024-01-19 00:10:42 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-hzzir from admin context and sharing with all the users [2024-01-19T00:10:42.207Z] 2024-01-19 00:10:42 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-hzzir] for org [default] with [ibm] as provider [2024-01-19T00:10:42.207Z] 2024-01-19 00:10:42 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:42.207Z] 2024-01-19 00:10:42 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:42.463Z] 2024-01-19 00:10:42 +0000:[INFO] [{SharedBackupDelete}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-hzzir] is present [2024-01-19T00:10:42.463Z] 2024-01-19 00:10:42 +0000:[INFO] [{SharedBackupDelete}] Create cluster [source-cluster] in org [default] [2024-01-19T00:10:43.415Z] 2024-01-19 00:10:43 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-unirp] for IBM [2024-01-19T00:10:43.415Z] 2024-01-19 00:10:43 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-unirp] for org [default] with [ibm] as provider [2024-01-19T00:10:43.415Z] 2024-01-19 00:10:43 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:43.415Z] 2024-01-19 00:10:43 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:43.415Z] 2024-01-19 00:10:43 +0000:[WARNING] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-unirp] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user]] [2024-01-19T00:10:43.415Z] 2024-01-19 00:10:43 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user] [2024-01-19T00:10:43.415Z] 2024-01-19 00:10:43 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-unirp from admin context and sharing with all the users [2024-01-19T00:10:43.415Z] 2024-01-19 00:10:43 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-unirp] for org [default] with [ibm] as provider [2024-01-19T00:10:43.415Z] 2024-01-19 00:10:43 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:43.415Z] 2024-01-19 00:10:43 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:43.671Z] 2024-01-19 00:10:43 +0000:[INFO] [{SharedBackupDelete}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-unirp] is present [2024-01-19T00:10:43.671Z] 2024-01-19 00:10:43 +0000:[INFO] [{SharedBackupDelete}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:10:44.598Z] INFO[2024-01-19 00:10:44] Verifying : Description : Creating source and destination cluster [2024-01-19T00:10:44.598Z] INFO[2024-01-19 00:10:44] Actual:true, Expected: true [2024-01-19T00:10:44.861Z] INFO[2024-01-19 00:10:44] Verifying : Description : Verifying backup restore [tp-restore-1705623044] is not possible for backup [tp-backup-1705622461] with user [testuser1-1705622438] [2024-01-19T00:10:44.861Z] INFO[2024-01-19 00:10:44] Actual:true, Expected: true [2024-01-19T00:10:44.861Z] 2024-01-19 00:10:44 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [e7b7bdc4-893c-434d-852f-21edd74993e1] doesn't have permission to delete backup [tp-backup-1705622461] [2024-01-19T00:10:44.861Z] INFO[2024-01-19 00:10:44] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622461] with user [testuser1-1705622438] [2024-01-19T00:10:44.861Z] INFO[2024-01-19 00:10:44] Actual:true, Expected: true [2024-01-19T00:10:44.861Z] INFO[2024-01-19 00:10:44] Verifying : Description : Verifying backup restore [tp-restore-1705623044] is not possible for backup [tp-backup-1705622471] with user [testuser1-1705622438] [2024-01-19T00:10:44.861Z] INFO[2024-01-19 00:10:44] Actual:true, Expected: true [2024-01-19T00:10:45.118Z] 2024-01-19 00:10:44 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [e7b7bdc4-893c-434d-852f-21edd74993e1] doesn't have permission to delete backup [tp-backup-1705622471] [2024-01-19T00:10:45.118Z] INFO[2024-01-19 00:10:44] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622471] with user [testuser1-1705622438] [2024-01-19T00:10:45.118Z] INFO[2024-01-19 00:10:44] Actual:true, Expected: true [2024-01-19T00:10:45.118Z] INFO[2024-01-19 00:10:44] Verifying : Description : Verifying backup restore [tp-restore-1705623044] is not possible for backup [tp-backup-1705622481] with user [testuser1-1705622438] [2024-01-19T00:10:45.118Z] INFO[2024-01-19 00:10:44] Actual:true, Expected: true [2024-01-19T00:10:45.118Z] 2024-01-19 00:10:45 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [e7b7bdc4-893c-434d-852f-21edd74993e1] doesn't have permission to delete backup [tp-backup-1705622481] [2024-01-19T00:10:45.118Z] INFO[2024-01-19 00:10:45] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622481] with user [testuser1-1705622438] [2024-01-19T00:10:45.118Z] INFO[2024-01-19 00:10:45] Actual:true, Expected: true [2024-01-19T00:10:45.376Z] INFO[2024-01-19 00:10:45] Verifying : Description : Verifying backup restore [tp-restore-1705623045] is not possible for backup [tp-backup-1705622491] with user [testuser1-1705622438] [2024-01-19T00:10:45.376Z] INFO[2024-01-19 00:10:45] Actual:true, Expected: true [2024-01-19T00:10:45.376Z] 2024-01-19 00:10:45 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [e7b7bdc4-893c-434d-852f-21edd74993e1] doesn't have permission to delete backup [tp-backup-1705622491] [2024-01-19T00:10:45.376Z] INFO[2024-01-19 00:10:45] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622491] with user [testuser1-1705622438] [2024-01-19T00:10:45.376Z] INFO[2024-01-19 00:10:45] Actual:true, Expected: true [2024-01-19T00:10:45.376Z] INFO[2024-01-19 00:10:45] Verifying : Description : Verifying backup restore [tp-restore-1705623045] is not possible for backup [tp-backup-1705622501] with user [testuser1-1705622438] [2024-01-19T00:10:45.376Z] INFO[2024-01-19 00:10:45] Actual:true, Expected: true [2024-01-19T00:10:45.634Z] 2024-01-19 00:10:45 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [e7b7bdc4-893c-434d-852f-21edd74993e1] doesn't have permission to delete backup [tp-backup-1705622501] [2024-01-19T00:10:45.634Z] INFO[2024-01-19 00:10:45] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622501] with user [testuser1-1705622438] [2024-01-19T00:10:45.634Z] INFO[2024-01-19 00:10:45] Actual:true, Expected: true [2024-01-19T00:10:45.634Z] INFO[2024-01-19 00:10:45] Verifying : Description : Verifying backup restore [tp-restore-1705623045] is not possible for backup [tp-backup-1705622511] with user [testuser1-1705622438] [2024-01-19T00:10:45.634Z] INFO[2024-01-19 00:10:45] Actual:true, Expected: true [2024-01-19T00:10:45.634Z] 2024-01-19 00:10:45 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [e7b7bdc4-893c-434d-852f-21edd74993e1] doesn't have permission to delete backup [tp-backup-1705622511] [2024-01-19T00:10:45.634Z] INFO[2024-01-19 00:10:45] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622511] with user [testuser1-1705622438] [2024-01-19T00:10:45.634Z] INFO[2024-01-19 00:10:45] Actual:true, Expected: true [2024-01-19T00:10:45.893Z] INFO[2024-01-19 00:10:45] Verifying : Description : Verifying backup restore [tp-restore-1705623045] is not possible for backup [tp-backup-1705622521] with user [testuser1-1705622438] [2024-01-19T00:10:45.893Z] INFO[2024-01-19 00:10:45] Actual:true, Expected: true [2024-01-19T00:10:45.893Z] 2024-01-19 00:10:45 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [e7b7bdc4-893c-434d-852f-21edd74993e1] doesn't have permission to delete backup [tp-backup-1705622521] [2024-01-19T00:10:45.893Z] INFO[2024-01-19 00:10:45] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622521] with user [testuser1-1705622438] [2024-01-19T00:10:45.893Z] INFO[2024-01-19 00:10:45] Actual:true, Expected: true [2024-01-19T00:10:46.149Z] INFO[2024-01-19 00:10:45] Verifying : Description : Verifying backup restore [tp-restore-1705623045] is not possible for backup [tp-backup-1705622531] with user [testuser1-1705622438] [2024-01-19T00:10:46.149Z] INFO[2024-01-19 00:10:45] Actual:true, Expected: true [2024-01-19T00:10:46.149Z] 2024-01-19 00:10:45 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [e7b7bdc4-893c-434d-852f-21edd74993e1] doesn't have permission to delete backup [tp-backup-1705622531] [2024-01-19T00:10:46.149Z] INFO[2024-01-19 00:10:45] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622531] with user [testuser1-1705622438] [2024-01-19T00:10:46.149Z] INFO[2024-01-19 00:10:45] Actual:true, Expected: true [2024-01-19T00:10:46.149Z] INFO[2024-01-19 00:10:46] Verifying : Description : Verifying backup restore [tp-restore-1705623046] is not possible for backup [tp-backup-1705622541] with user [testuser1-1705622438] [2024-01-19T00:10:46.149Z] INFO[2024-01-19 00:10:46] Actual:true, Expected: true [2024-01-19T00:10:46.405Z] 2024-01-19 00:10:46 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [e7b7bdc4-893c-434d-852f-21edd74993e1] doesn't have permission to delete backup [tp-backup-1705622541] [2024-01-19T00:10:46.405Z] INFO[2024-01-19 00:10:46] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622541] with user [testuser1-1705622438] [2024-01-19T00:10:46.405Z] INFO[2024-01-19 00:10:46] Actual:true, Expected: true [2024-01-19T00:10:46.405Z] INFO[2024-01-19 00:10:46] Verifying : Description : Verifying backup restore [tp-restore-1705623046] is not possible for backup [tp-backup-1705622551] with user [testuser1-1705622438] [2024-01-19T00:10:46.405Z] INFO[2024-01-19 00:10:46] Actual:true, Expected: true [2024-01-19T00:10:46.405Z] 2024-01-19 00:10:46 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [e7b7bdc4-893c-434d-852f-21edd74993e1] doesn't have permission to delete backup [tp-backup-1705622551] [2024-01-19T00:10:46.405Z] INFO[2024-01-19 00:10:46] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622551] with user [testuser1-1705622438] [2024-01-19T00:10:46.405Z] INFO[2024-01-19 00:10:46] Actual:true, Expected: true [2024-01-19T00:10:46.662Z] 2024-01-19 00:10:46 +0000:[INFO] [{SharedBackupDelete}] Registering Source and Destination clusters from user context for user -testuser7-1705622438 [2024-01-19T00:10:46.662Z] INFO[2024-01-19 00:10:46] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T00:10:46.662Z] INFO[2024-01-19 00:10:46] Actual:true, Expected: true [2024-01-19T00:10:46.662Z] 2024-01-19 00:10:46 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T00:10:46.662Z] 2024-01-19 00:10:46 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T00:10:46.662Z] 2024-01-19 00:10:46 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T00:10:46.662Z] 2024-01-19 00:10:46 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T00:10:46.662Z] 2024-01-19 00:10:46 +0000:[INFO] [{SharedBackupDelete}] Create cluster [source-cluster] in org [default] [2024-01-19T00:10:46.662Z] 2024-01-19 00:10:46 +0000:[INFO] [{SharedBackupDelete}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T00:10:46.662Z] 2024-01-19 00:10:46 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T00:10:46.662Z] 2024-01-19 00:10:46 +0000:[INFO] [{SharedBackupDelete}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:10:46.662Z] 2024-01-19 00:10:46 +0000:[INFO] [{SharedBackupDelete}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T00:10:46.662Z] 2024-01-19 00:10:46 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T00:10:46.662Z] 2024-01-19 00:10:46 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-qjrgi] for IBM [2024-01-19T00:10:46.662Z] 2024-01-19 00:10:46 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-qjrgi] for org [default] with [ibm] as provider [2024-01-19T00:10:46.663Z] 2024-01-19 00:10:46 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:46.663Z] 2024-01-19 00:10:46 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:46.663Z] 2024-01-19 00:10:46 +0000:[WARNING] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-qjrgi] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName7 LastName7 with roles: [px-backup-app.user]] [2024-01-19T00:10:46.663Z] 2024-01-19 00:10:46 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName7 LastName7 with roles: [px-backup-app.user] [2024-01-19T00:10:46.920Z] 2024-01-19 00:10:46 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-qjrgi from admin context and sharing with all the users [2024-01-19T00:10:46.920Z] 2024-01-19 00:10:46 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-qjrgi] for org [default] with [ibm] as provider [2024-01-19T00:10:46.920Z] 2024-01-19 00:10:46 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:46.920Z] 2024-01-19 00:10:46 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:47.178Z] 2024-01-19 00:10:46 +0000:[INFO] [{SharedBackupDelete}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-qjrgi] is present [2024-01-19T00:10:47.178Z] 2024-01-19 00:10:47 +0000:[INFO] [{SharedBackupDelete}] Create cluster [source-cluster] in org [default] [2024-01-19T00:10:48.105Z] 2024-01-19 00:10:47 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-kynmv] for IBM [2024-01-19T00:10:48.105Z] 2024-01-19 00:10:47 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-kynmv] for org [default] with [ibm] as provider [2024-01-19T00:10:48.105Z] 2024-01-19 00:10:47 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:48.105Z] 2024-01-19 00:10:47 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:48.105Z] 2024-01-19 00:10:47 +0000:[WARNING] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-kynmv] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName7 LastName7 with roles: [px-backup-app.user]] [2024-01-19T00:10:48.105Z] 2024-01-19 00:10:47 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName7 LastName7 with roles: [px-backup-app.user] [2024-01-19T00:10:48.365Z] 2024-01-19 00:10:48 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-kynmv from admin context and sharing with all the users [2024-01-19T00:10:48.365Z] 2024-01-19 00:10:48 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-kynmv] for org [default] with [ibm] as provider [2024-01-19T00:10:48.365Z] 2024-01-19 00:10:48 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:48.366Z] 2024-01-19 00:10:48 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:48.366Z] 2024-01-19 00:10:48 +0000:[INFO] [{SharedBackupDelete}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-kynmv] is present [2024-01-19T00:10:48.622Z] 2024-01-19 00:10:48 +0000:[INFO] [{SharedBackupDelete}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:10:49.551Z] INFO[2024-01-19 00:10:49] Verifying : Description : Creating source and destination cluster [2024-01-19T00:10:49.551Z] INFO[2024-01-19 00:10:49] Actual:true, Expected: true [2024-01-19T00:10:49.551Z] INFO[2024-01-19 00:10:49] Verifying : Description : Verifying backup restore [tp-restore-1705623049] is not possible for backup [tp-backup-1705622461] with user [testuser7-1705622438] [2024-01-19T00:10:49.551Z] INFO[2024-01-19 00:10:49] Actual:true, Expected: true [2024-01-19T00:10:49.551Z] 2024-01-19 00:10:49 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [1a84069f-7604-48f6-8a17-90dfb3e1769f] doesn't have permission to delete backup [tp-backup-1705622461] [2024-01-19T00:10:49.551Z] INFO[2024-01-19 00:10:49] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622461] with user [testuser7-1705622438] [2024-01-19T00:10:49.551Z] INFO[2024-01-19 00:10:49] Actual:true, Expected: true [2024-01-19T00:10:49.807Z] INFO[2024-01-19 00:10:49] Verifying : Description : Verifying backup restore [tp-restore-1705623049] is not possible for backup [tp-backup-1705622471] with user [testuser7-1705622438] [2024-01-19T00:10:49.807Z] INFO[2024-01-19 00:10:49] Actual:true, Expected: true [2024-01-19T00:10:49.807Z] 2024-01-19 00:10:49 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [1a84069f-7604-48f6-8a17-90dfb3e1769f] doesn't have permission to delete backup [tp-backup-1705622471] [2024-01-19T00:10:49.807Z] INFO[2024-01-19 00:10:49] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622471] with user [testuser7-1705622438] [2024-01-19T00:10:49.807Z] INFO[2024-01-19 00:10:49] Actual:true, Expected: true [2024-01-19T00:10:50.062Z] INFO[2024-01-19 00:10:49] Verifying : Description : Verifying backup restore [tp-restore-1705623049] is not possible for backup [tp-backup-1705622481] with user [testuser7-1705622438] [2024-01-19T00:10:50.062Z] INFO[2024-01-19 00:10:49] Actual:true, Expected: true [2024-01-19T00:10:50.062Z] 2024-01-19 00:10:49 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [1a84069f-7604-48f6-8a17-90dfb3e1769f] doesn't have permission to delete backup [tp-backup-1705622481] [2024-01-19T00:10:50.062Z] INFO[2024-01-19 00:10:49] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622481] with user [testuser7-1705622438] [2024-01-19T00:10:50.063Z] INFO[2024-01-19 00:10:49] Actual:true, Expected: true [2024-01-19T00:10:50.063Z] INFO[2024-01-19 00:10:50] Verifying : Description : Verifying backup restore [tp-restore-1705623049] is not possible for backup [tp-backup-1705622491] with user [testuser7-1705622438] [2024-01-19T00:10:50.063Z] INFO[2024-01-19 00:10:50] Actual:true, Expected: true [2024-01-19T00:10:50.319Z] 2024-01-19 00:10:50 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [1a84069f-7604-48f6-8a17-90dfb3e1769f] doesn't have permission to delete backup [tp-backup-1705622491] [2024-01-19T00:10:50.319Z] INFO[2024-01-19 00:10:50] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622491] with user [testuser7-1705622438] [2024-01-19T00:10:50.319Z] INFO[2024-01-19 00:10:50] Actual:true, Expected: true [2024-01-19T00:10:50.319Z] INFO[2024-01-19 00:10:50] Verifying : Description : Verifying backup restore [tp-restore-1705623050] is not possible for backup [tp-backup-1705622501] with user [testuser7-1705622438] [2024-01-19T00:10:50.319Z] INFO[2024-01-19 00:10:50] Actual:true, Expected: true [2024-01-19T00:10:50.319Z] 2024-01-19 00:10:50 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [1a84069f-7604-48f6-8a17-90dfb3e1769f] doesn't have permission to delete backup [tp-backup-1705622501] [2024-01-19T00:10:50.319Z] INFO[2024-01-19 00:10:50] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622501] with user [testuser7-1705622438] [2024-01-19T00:10:50.319Z] INFO[2024-01-19 00:10:50] Actual:true, Expected: true [2024-01-19T00:10:50.591Z] INFO[2024-01-19 00:10:50] Verifying : Description : Verifying backup restore [tp-restore-1705623050] is not possible for backup [tp-backup-1705622511] with user [testuser7-1705622438] [2024-01-19T00:10:50.591Z] INFO[2024-01-19 00:10:50] Actual:true, Expected: true [2024-01-19T00:10:50.591Z] 2024-01-19 00:10:50 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [1a84069f-7604-48f6-8a17-90dfb3e1769f] doesn't have permission to delete backup [tp-backup-1705622511] [2024-01-19T00:10:50.591Z] INFO[2024-01-19 00:10:50] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622511] with user [testuser7-1705622438] [2024-01-19T00:10:50.591Z] INFO[2024-01-19 00:10:50] Actual:true, Expected: true [2024-01-19T00:10:50.847Z] INFO[2024-01-19 00:10:50] Verifying : Description : Verifying backup restore [tp-restore-1705623050] is not possible for backup [tp-backup-1705622521] with user [testuser7-1705622438] [2024-01-19T00:10:50.847Z] INFO[2024-01-19 00:10:50] Actual:true, Expected: true [2024-01-19T00:10:50.847Z] 2024-01-19 00:10:50 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [1a84069f-7604-48f6-8a17-90dfb3e1769f] doesn't have permission to delete backup [tp-backup-1705622521] [2024-01-19T00:10:50.847Z] INFO[2024-01-19 00:10:50] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622521] with user [testuser7-1705622438] [2024-01-19T00:10:50.847Z] INFO[2024-01-19 00:10:50] Actual:true, Expected: true [2024-01-19T00:10:50.847Z] INFO[2024-01-19 00:10:50] Verifying : Description : Verifying backup restore [tp-restore-1705623050] is not possible for backup [tp-backup-1705622531] with user [testuser7-1705622438] [2024-01-19T00:10:50.847Z] INFO[2024-01-19 00:10:50] Actual:true, Expected: true [2024-01-19T00:10:50.847Z] 2024-01-19 00:10:50 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [1a84069f-7604-48f6-8a17-90dfb3e1769f] doesn't have permission to delete backup [tp-backup-1705622531] [2024-01-19T00:10:50.847Z] INFO[2024-01-19 00:10:50] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622531] with user [testuser7-1705622438] [2024-01-19T00:10:50.847Z] INFO[2024-01-19 00:10:50] Actual:true, Expected: true [2024-01-19T00:10:51.105Z] INFO[2024-01-19 00:10:50] Verifying : Description : Verifying backup restore [tp-restore-1705623050] is not possible for backup [tp-backup-1705622541] with user [testuser7-1705622438] [2024-01-19T00:10:51.105Z] INFO[2024-01-19 00:10:50] Actual:true, Expected: true [2024-01-19T00:10:51.105Z] 2024-01-19 00:10:51 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [1a84069f-7604-48f6-8a17-90dfb3e1769f] doesn't have permission to delete backup [tp-backup-1705622541] [2024-01-19T00:10:51.105Z] INFO[2024-01-19 00:10:51] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622541] with user [testuser7-1705622438] [2024-01-19T00:10:51.105Z] INFO[2024-01-19 00:10:51] Actual:true, Expected: true [2024-01-19T00:10:51.366Z] INFO[2024-01-19 00:10:51] Verifying : Description : Verifying backup restore [tp-restore-1705623051] is not possible for backup [tp-backup-1705622551] with user [testuser7-1705622438] [2024-01-19T00:10:51.366Z] INFO[2024-01-19 00:10:51] Actual:true, Expected: true [2024-01-19T00:10:51.366Z] 2024-01-19 00:10:51 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [1a84069f-7604-48f6-8a17-90dfb3e1769f] doesn't have permission to delete backup [tp-backup-1705622551] [2024-01-19T00:10:51.366Z] INFO[2024-01-19 00:10:51] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622551] with user [testuser7-1705622438] [2024-01-19T00:10:51.366Z] INFO[2024-01-19 00:10:51] Actual:true, Expected: true [2024-01-19T00:10:51.366Z] 2024-01-19 00:10:51 +0000:[INFO] [{SharedBackupDelete}] Registering Source and Destination clusters from user context for user -testuser8-1705622438 [2024-01-19T00:10:51.366Z] INFO[2024-01-19 00:10:51] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T00:10:51.366Z] INFO[2024-01-19 00:10:51] Actual:true, Expected: true [2024-01-19T00:10:51.366Z] 2024-01-19 00:10:51 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T00:10:51.629Z] 2024-01-19 00:10:51 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T00:10:51.629Z] 2024-01-19 00:10:51 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T00:10:51.629Z] 2024-01-19 00:10:51 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T00:10:51.629Z] 2024-01-19 00:10:51 +0000:[INFO] [{SharedBackupDelete}] Create cluster [source-cluster] in org [default] [2024-01-19T00:10:51.629Z] 2024-01-19 00:10:51 +0000:[INFO] [{SharedBackupDelete}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T00:10:51.629Z] 2024-01-19 00:10:51 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T00:10:51.629Z] 2024-01-19 00:10:51 +0000:[INFO] [{SharedBackupDelete}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:10:51.629Z] 2024-01-19 00:10:51 +0000:[INFO] [{SharedBackupDelete}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T00:10:51.629Z] 2024-01-19 00:10:51 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T00:10:51.629Z] 2024-01-19 00:10:51 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-jttmr] for IBM [2024-01-19T00:10:51.629Z] 2024-01-19 00:10:51 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-jttmr] for org [default] with [ibm] as provider [2024-01-19T00:10:51.630Z] 2024-01-19 00:10:51 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:51.630Z] 2024-01-19 00:10:51 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:51.630Z] 2024-01-19 00:10:51 +0000:[WARNING] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-jttmr] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName8 LastName8 with roles: [px-backup-app.user]] [2024-01-19T00:10:51.630Z] 2024-01-19 00:10:51 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName8 LastName8 with roles: [px-backup-app.user] [2024-01-19T00:10:51.886Z] 2024-01-19 00:10:51 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-jttmr from admin context and sharing with all the users [2024-01-19T00:10:51.886Z] 2024-01-19 00:10:51 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-jttmr] for org [default] with [ibm] as provider [2024-01-19T00:10:51.886Z] 2024-01-19 00:10:51 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:51.886Z] 2024-01-19 00:10:51 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:51.886Z] 2024-01-19 00:10:51 +0000:[INFO] [{SharedBackupDelete}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-jttmr] is present [2024-01-19T00:10:52.151Z] 2024-01-19 00:10:51 +0000:[INFO] [{SharedBackupDelete}] Create cluster [source-cluster] in org [default] [2024-01-19T00:10:52.718Z] 2024-01-19 00:10:52 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-mnxik] for IBM [2024-01-19T00:10:52.718Z] 2024-01-19 00:10:52 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-mnxik] for org [default] with [ibm] as provider [2024-01-19T00:10:52.718Z] 2024-01-19 00:10:52 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:52.718Z] 2024-01-19 00:10:52 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:52.976Z] 2024-01-19 00:10:52 +0000:[WARNING] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-mnxik] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName8 LastName8 with roles: [px-backup-app.user]] [2024-01-19T00:10:52.976Z] 2024-01-19 00:10:52 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName8 LastName8 with roles: [px-backup-app.user] [2024-01-19T00:10:53.237Z] 2024-01-19 00:10:52 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-mnxik from admin context and sharing with all the users [2024-01-19T00:10:53.237Z] 2024-01-19 00:10:52 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-mnxik] for org [default] with [ibm] as provider [2024-01-19T00:10:53.237Z] 2024-01-19 00:10:52 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:53.237Z] 2024-01-19 00:10:52 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:53.237Z] 2024-01-19 00:10:53 +0000:[INFO] [{SharedBackupDelete}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-mnxik] is present [2024-01-19T00:10:53.492Z] 2024-01-19 00:10:53 +0000:[INFO] [{SharedBackupDelete}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:10:54.058Z] INFO[2024-01-19 00:10:54] Verifying : Description : Creating source and destination cluster [2024-01-19T00:10:54.058Z] INFO[2024-01-19 00:10:54] Actual:true, Expected: true [2024-01-19T00:10:54.314Z] INFO[2024-01-19 00:10:54] Verifying : Description : Verifying backup restore [tp-restore-1705623054] is not possible for backup [tp-backup-1705622461] with user [testuser8-1705622438] [2024-01-19T00:10:54.314Z] INFO[2024-01-19 00:10:54] Actual:true, Expected: true [2024-01-19T00:10:54.314Z] 2024-01-19 00:10:54 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [d76ecda7-a2f7-4a32-96c7-0986ec23ef07] doesn't have permission to delete backup [tp-backup-1705622461] [2024-01-19T00:10:54.314Z] INFO[2024-01-19 00:10:54] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622461] with user [testuser8-1705622438] [2024-01-19T00:10:54.314Z] INFO[2024-01-19 00:10:54] Actual:true, Expected: true [2024-01-19T00:10:54.579Z] INFO[2024-01-19 00:10:54] Verifying : Description : Verifying backup restore [tp-restore-1705623054] is not possible for backup [tp-backup-1705622471] with user [testuser8-1705622438] [2024-01-19T00:10:54.579Z] INFO[2024-01-19 00:10:54] Actual:true, Expected: true [2024-01-19T00:10:54.579Z] 2024-01-19 00:10:54 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [d76ecda7-a2f7-4a32-96c7-0986ec23ef07] doesn't have permission to delete backup [tp-backup-1705622471] [2024-01-19T00:10:54.579Z] INFO[2024-01-19 00:10:54] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622471] with user [testuser8-1705622438] [2024-01-19T00:10:54.579Z] INFO[2024-01-19 00:10:54] Actual:true, Expected: true [2024-01-19T00:10:54.579Z] INFO[2024-01-19 00:10:54] Verifying : Description : Verifying backup restore [tp-restore-1705623054] is not possible for backup [tp-backup-1705622481] with user [testuser8-1705622438] [2024-01-19T00:10:54.579Z] INFO[2024-01-19 00:10:54] Actual:true, Expected: true [2024-01-19T00:10:54.579Z] 2024-01-19 00:10:54 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [d76ecda7-a2f7-4a32-96c7-0986ec23ef07] doesn't have permission to delete backup [tp-backup-1705622481] [2024-01-19T00:10:54.579Z] INFO[2024-01-19 00:10:54] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622481] with user [testuser8-1705622438] [2024-01-19T00:10:54.579Z] INFO[2024-01-19 00:10:54] Actual:true, Expected: true [2024-01-19T00:10:54.836Z] INFO[2024-01-19 00:10:54] Verifying : Description : Verifying backup restore [tp-restore-1705623054] is not possible for backup [tp-backup-1705622491] with user [testuser8-1705622438] [2024-01-19T00:10:54.836Z] INFO[2024-01-19 00:10:54] Actual:true, Expected: true [2024-01-19T00:10:54.836Z] 2024-01-19 00:10:54 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [d76ecda7-a2f7-4a32-96c7-0986ec23ef07] doesn't have permission to delete backup [tp-backup-1705622491] [2024-01-19T00:10:54.836Z] INFO[2024-01-19 00:10:54] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622491] with user [testuser8-1705622438] [2024-01-19T00:10:54.836Z] INFO[2024-01-19 00:10:54] Actual:true, Expected: true [2024-01-19T00:10:55.092Z] INFO[2024-01-19 00:10:54] Verifying : Description : Verifying backup restore [tp-restore-1705623054] is not possible for backup [tp-backup-1705622501] with user [testuser8-1705622438] [2024-01-19T00:10:55.092Z] INFO[2024-01-19 00:10:54] Actual:true, Expected: true [2024-01-19T00:10:55.092Z] 2024-01-19 00:10:54 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [d76ecda7-a2f7-4a32-96c7-0986ec23ef07] doesn't have permission to delete backup [tp-backup-1705622501] [2024-01-19T00:10:55.092Z] INFO[2024-01-19 00:10:54] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622501] with user [testuser8-1705622438] [2024-01-19T00:10:55.092Z] INFO[2024-01-19 00:10:54] Actual:true, Expected: true [2024-01-19T00:10:55.092Z] INFO[2024-01-19 00:10:55] Verifying : Description : Verifying backup restore [tp-restore-1705623054] is not possible for backup [tp-backup-1705622511] with user [testuser8-1705622438] [2024-01-19T00:10:55.092Z] INFO[2024-01-19 00:10:55] Actual:true, Expected: true [2024-01-19T00:10:55.349Z] 2024-01-19 00:10:55 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [d76ecda7-a2f7-4a32-96c7-0986ec23ef07] doesn't have permission to delete backup [tp-backup-1705622511] [2024-01-19T00:10:55.349Z] INFO[2024-01-19 00:10:55] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622511] with user [testuser8-1705622438] [2024-01-19T00:10:55.349Z] INFO[2024-01-19 00:10:55] Actual:true, Expected: true [2024-01-19T00:10:55.349Z] INFO[2024-01-19 00:10:55] Verifying : Description : Verifying backup restore [tp-restore-1705623055] is not possible for backup [tp-backup-1705622521] with user [testuser8-1705622438] [2024-01-19T00:10:55.349Z] INFO[2024-01-19 00:10:55] Actual:true, Expected: true [2024-01-19T00:10:55.349Z] 2024-01-19 00:10:55 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [d76ecda7-a2f7-4a32-96c7-0986ec23ef07] doesn't have permission to delete backup [tp-backup-1705622521] [2024-01-19T00:10:55.349Z] INFO[2024-01-19 00:10:55] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622521] with user [testuser8-1705622438] [2024-01-19T00:10:55.349Z] INFO[2024-01-19 00:10:55] Actual:true, Expected: true [2024-01-19T00:10:55.605Z] INFO[2024-01-19 00:10:55] Verifying : Description : Verifying backup restore [tp-restore-1705623055] is not possible for backup [tp-backup-1705622531] with user [testuser8-1705622438] [2024-01-19T00:10:55.605Z] INFO[2024-01-19 00:10:55] Actual:true, Expected: true [2024-01-19T00:10:55.605Z] 2024-01-19 00:10:55 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [d76ecda7-a2f7-4a32-96c7-0986ec23ef07] doesn't have permission to delete backup [tp-backup-1705622531] [2024-01-19T00:10:55.605Z] INFO[2024-01-19 00:10:55] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622531] with user [testuser8-1705622438] [2024-01-19T00:10:55.605Z] INFO[2024-01-19 00:10:55] Actual:true, Expected: true [2024-01-19T00:10:55.605Z] INFO[2024-01-19 00:10:55] Verifying : Description : Verifying backup restore [tp-restore-1705623055] is not possible for backup [tp-backup-1705622541] with user [testuser8-1705622438] [2024-01-19T00:10:55.605Z] INFO[2024-01-19 00:10:55] Actual:true, Expected: true [2024-01-19T00:10:55.862Z] 2024-01-19 00:10:55 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [d76ecda7-a2f7-4a32-96c7-0986ec23ef07] doesn't have permission to delete backup [tp-backup-1705622541] [2024-01-19T00:10:55.862Z] INFO[2024-01-19 00:10:55] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622541] with user [testuser8-1705622438] [2024-01-19T00:10:55.862Z] INFO[2024-01-19 00:10:55] Actual:true, Expected: true [2024-01-19T00:10:55.863Z] INFO[2024-01-19 00:10:55] Verifying : Description : Verifying backup restore [tp-restore-1705623055] is not possible for backup [tp-backup-1705622551] with user [testuser8-1705622438] [2024-01-19T00:10:55.863Z] INFO[2024-01-19 00:10:55] Actual:true, Expected: true [2024-01-19T00:10:56.118Z] 2024-01-19 00:10:55 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [d76ecda7-a2f7-4a32-96c7-0986ec23ef07] doesn't have permission to delete backup [tp-backup-1705622551] [2024-01-19T00:10:56.118Z] INFO[2024-01-19 00:10:55] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622551] with user [testuser8-1705622438] [2024-01-19T00:10:56.118Z] INFO[2024-01-19 00:10:55] Actual:true, Expected: true [2024-01-19T00:10:56.118Z] 2024-01-19 00:10:55 +0000:[INFO] [{SharedBackupDelete}] Registering Source and Destination clusters from user context for user -testuser9-1705622438 [2024-01-19T00:10:56.118Z] INFO[2024-01-19 00:10:55] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T00:10:56.118Z] INFO[2024-01-19 00:10:55] Actual:true, Expected: true [2024-01-19T00:10:56.119Z] 2024-01-19 00:10:55 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T00:10:56.119Z] 2024-01-19 00:10:55 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T00:10:56.119Z] 2024-01-19 00:10:55 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T00:10:56.119Z] 2024-01-19 00:10:55 +0000:[INFO] [{SharedBackupDelete}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T00:10:56.119Z] 2024-01-19 00:10:55 +0000:[INFO] [{SharedBackupDelete}] Create cluster [source-cluster] in org [default] [2024-01-19T00:10:56.119Z] 2024-01-19 00:10:55 +0000:[INFO] [{SharedBackupDelete}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T00:10:56.119Z] 2024-01-19 00:10:55 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T00:10:56.119Z] 2024-01-19 00:10:55 +0000:[INFO] [{SharedBackupDelete}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:10:56.119Z] 2024-01-19 00:10:55 +0000:[INFO] [{SharedBackupDelete}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T00:10:56.119Z] 2024-01-19 00:10:55 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T00:10:56.119Z] 2024-01-19 00:10:55 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-uokzd] for IBM [2024-01-19T00:10:56.119Z] 2024-01-19 00:10:55 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-uokzd] for org [default] with [ibm] as provider [2024-01-19T00:10:56.119Z] 2024-01-19 00:10:55 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:56.119Z] 2024-01-19 00:10:55 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:56.119Z] 2024-01-19 00:10:56 +0000:[WARNING] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-uokzd] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName9 LastName9 with roles: [px-backup-app.user]] [2024-01-19T00:10:56.119Z] 2024-01-19 00:10:56 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName9 LastName9 with roles: [px-backup-app.user] [2024-01-19T00:10:56.379Z] 2024-01-19 00:10:56 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-uokzd from admin context and sharing with all the users [2024-01-19T00:10:56.379Z] 2024-01-19 00:10:56 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-uokzd] for org [default] with [ibm] as provider [2024-01-19T00:10:56.379Z] 2024-01-19 00:10:56 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:56.379Z] 2024-01-19 00:10:56 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:56.641Z] 2024-01-19 00:10:56 +0000:[INFO] [{SharedBackupDelete}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-uokzd] is present [2024-01-19T00:10:56.641Z] 2024-01-19 00:10:56 +0000:[INFO] [{SharedBackupDelete}] Create cluster [source-cluster] in org [default] [2024-01-19T00:10:57.578Z] 2024-01-19 00:10:57 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-wrmiu] for IBM [2024-01-19T00:10:57.578Z] 2024-01-19 00:10:57 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-wrmiu] for org [default] with [ibm] as provider [2024-01-19T00:10:57.578Z] 2024-01-19 00:10:57 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:57.578Z] 2024-01-19 00:10:57 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:57.578Z] 2024-01-19 00:10:57 +0000:[WARNING] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-wrmiu] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName9 LastName9 with roles: [px-backup-app.user]] [2024-01-19T00:10:57.578Z] 2024-01-19 00:10:57 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName9 LastName9 with roles: [px-backup-app.user] [2024-01-19T00:10:57.578Z] 2024-01-19 00:10:57 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-wrmiu from admin context and sharing with all the users [2024-01-19T00:10:57.578Z] 2024-01-19 00:10:57 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-wrmiu] for org [default] with [ibm] as provider [2024-01-19T00:10:57.578Z] 2024-01-19 00:10:57 +0000:[INFO] [{SharedBackupDelete}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:10:57.578Z] 2024-01-19 00:10:57 +0000:[INFO] [{SharedBackupDelete}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:10:57.833Z] 2024-01-19 00:10:57 +0000:[INFO] [{SharedBackupDelete}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-wrmiu] is present [2024-01-19T00:10:57.833Z] 2024-01-19 00:10:57 +0000:[INFO] [{SharedBackupDelete}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:10:58.760Z] INFO[2024-01-19 00:10:58] Verifying : Description : Creating source and destination cluster [2024-01-19T00:10:58.760Z] INFO[2024-01-19 00:10:58] Actual:true, Expected: true [2024-01-19T00:10:59.015Z] INFO[2024-01-19 00:10:58] Verifying : Description : Verifying backup restore [tp-restore-1705623058] is not possible for backup [tp-backup-1705622461] with user [testuser9-1705622438] [2024-01-19T00:10:59.015Z] INFO[2024-01-19 00:10:58] Actual:true, Expected: true [2024-01-19T00:10:59.016Z] 2024-01-19 00:10:58 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [e253b80a-8c31-4ad2-9a21-702b0c386ec1] doesn't have permission to delete backup [tp-backup-1705622461] [2024-01-19T00:10:59.016Z] INFO[2024-01-19 00:10:58] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622461] with user [testuser9-1705622438] [2024-01-19T00:10:59.016Z] INFO[2024-01-19 00:10:58] Actual:true, Expected: true [2024-01-19T00:10:59.016Z] INFO[2024-01-19 00:10:58] Verifying : Description : Verifying backup restore [tp-restore-1705623058] is not possible for backup [tp-backup-1705622471] with user [testuser9-1705622438] [2024-01-19T00:10:59.016Z] INFO[2024-01-19 00:10:58] Actual:true, Expected: true [2024-01-19T00:10:59.016Z] 2024-01-19 00:10:58 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [e253b80a-8c31-4ad2-9a21-702b0c386ec1] doesn't have permission to delete backup [tp-backup-1705622471] [2024-01-19T00:10:59.016Z] INFO[2024-01-19 00:10:58] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622471] with user [testuser9-1705622438] [2024-01-19T00:10:59.016Z] INFO[2024-01-19 00:10:58] Actual:true, Expected: true [2024-01-19T00:10:59.272Z] INFO[2024-01-19 00:10:59] Verifying : Description : Verifying backup restore [tp-restore-1705623059] is not possible for backup [tp-backup-1705622481] with user [testuser9-1705622438] [2024-01-19T00:10:59.272Z] INFO[2024-01-19 00:10:59] Actual:true, Expected: true [2024-01-19T00:10:59.272Z] 2024-01-19 00:10:59 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [e253b80a-8c31-4ad2-9a21-702b0c386ec1] doesn't have permission to delete backup [tp-backup-1705622481] [2024-01-19T00:10:59.272Z] INFO[2024-01-19 00:10:59] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622481] with user [testuser9-1705622438] [2024-01-19T00:10:59.272Z] INFO[2024-01-19 00:10:59] Actual:true, Expected: true [2024-01-19T00:10:59.528Z] INFO[2024-01-19 00:10:59] Verifying : Description : Verifying backup restore [tp-restore-1705623059] is not possible for backup [tp-backup-1705622491] with user [testuser9-1705622438] [2024-01-19T00:10:59.528Z] INFO[2024-01-19 00:10:59] Actual:true, Expected: true [2024-01-19T00:10:59.528Z] 2024-01-19 00:10:59 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [e253b80a-8c31-4ad2-9a21-702b0c386ec1] doesn't have permission to delete backup [tp-backup-1705622491] [2024-01-19T00:10:59.528Z] INFO[2024-01-19 00:10:59] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622491] with user [testuser9-1705622438] [2024-01-19T00:10:59.528Z] INFO[2024-01-19 00:10:59] Actual:true, Expected: true [2024-01-19T00:10:59.528Z] INFO[2024-01-19 00:10:59] Verifying : Description : Verifying backup restore [tp-restore-1705623059] is not possible for backup [tp-backup-1705622501] with user [testuser9-1705622438] [2024-01-19T00:10:59.528Z] INFO[2024-01-19 00:10:59] Actual:true, Expected: true [2024-01-19T00:10:59.529Z] 2024-01-19 00:10:59 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [e253b80a-8c31-4ad2-9a21-702b0c386ec1] doesn't have permission to delete backup [tp-backup-1705622501] [2024-01-19T00:10:59.529Z] INFO[2024-01-19 00:10:59] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622501] with user [testuser9-1705622438] [2024-01-19T00:10:59.529Z] INFO[2024-01-19 00:10:59] Actual:true, Expected: true [2024-01-19T00:10:59.784Z] INFO[2024-01-19 00:10:59] Verifying : Description : Verifying backup restore [tp-restore-1705623059] is not possible for backup [tp-backup-1705622511] with user [testuser9-1705622438] [2024-01-19T00:10:59.784Z] INFO[2024-01-19 00:10:59] Actual:true, Expected: true [2024-01-19T00:10:59.784Z] 2024-01-19 00:10:59 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [e253b80a-8c31-4ad2-9a21-702b0c386ec1] doesn't have permission to delete backup [tp-backup-1705622511] [2024-01-19T00:10:59.784Z] INFO[2024-01-19 00:10:59] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622511] with user [testuser9-1705622438] [2024-01-19T00:10:59.784Z] INFO[2024-01-19 00:10:59] Actual:true, Expected: true [2024-01-19T00:11:00.040Z] INFO[2024-01-19 00:10:59] Verifying : Description : Verifying backup restore [tp-restore-1705623059] is not possible for backup [tp-backup-1705622521] with user [testuser9-1705622438] [2024-01-19T00:11:00.040Z] INFO[2024-01-19 00:10:59] Actual:true, Expected: true [2024-01-19T00:11:00.040Z] 2024-01-19 00:10:59 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [e253b80a-8c31-4ad2-9a21-702b0c386ec1] doesn't have permission to delete backup [tp-backup-1705622521] [2024-01-19T00:11:00.040Z] INFO[2024-01-19 00:10:59] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622521] with user [testuser9-1705622438] [2024-01-19T00:11:00.040Z] INFO[2024-01-19 00:10:59] Actual:true, Expected: true [2024-01-19T00:11:00.040Z] INFO[2024-01-19 00:10:59] Verifying : Description : Verifying backup restore [tp-restore-1705623059] is not possible for backup [tp-backup-1705622531] with user [testuser9-1705622438] [2024-01-19T00:11:00.040Z] INFO[2024-01-19 00:10:59] Actual:true, Expected: true [2024-01-19T00:11:00.040Z] 2024-01-19 00:11:00 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [e253b80a-8c31-4ad2-9a21-702b0c386ec1] doesn't have permission to delete backup [tp-backup-1705622531] [2024-01-19T00:11:00.040Z] INFO[2024-01-19 00:11:00] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622531] with user [testuser9-1705622438] [2024-01-19T00:11:00.040Z] INFO[2024-01-19 00:11:00] Actual:true, Expected: true [2024-01-19T00:11:00.295Z] INFO[2024-01-19 00:11:00] Verifying : Description : Verifying backup restore [tp-restore-1705623060] is not possible for backup [tp-backup-1705622541] with user [testuser9-1705622438] [2024-01-19T00:11:00.295Z] INFO[2024-01-19 00:11:00] Actual:true, Expected: true [2024-01-19T00:11:00.295Z] 2024-01-19 00:11:00 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [e253b80a-8c31-4ad2-9a21-702b0c386ec1] doesn't have permission to delete backup [tp-backup-1705622541] [2024-01-19T00:11:00.295Z] INFO[2024-01-19 00:11:00] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622541] with user [testuser9-1705622438] [2024-01-19T00:11:00.295Z] INFO[2024-01-19 00:11:00] Actual:true, Expected: true [2024-01-19T00:11:00.551Z] INFO[2024-01-19 00:11:00] Verifying : Description : Verifying backup restore [tp-restore-1705623060] is not possible for backup [tp-backup-1705622551] with user [testuser9-1705622438] [2024-01-19T00:11:00.551Z] INFO[2024-01-19 00:11:00] Actual:true, Expected: true [2024-01-19T00:11:00.551Z] 2024-01-19 00:11:00 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.6:#2163] - Error message - rpc error: code = Unknown desc = user [e253b80a-8c31-4ad2-9a21-702b0c386ec1] doesn't have permission to delete backup [tp-backup-1705622551] [2024-01-19T00:11:00.551Z] INFO[2024-01-19 00:11:00] Verifying : Description : Verifying backup deletion is not possible for backup [tp-backup-1705622551] with user [testuser9-1705622438] [2024-01-19T00:11:00.551Z] INFO[2024-01-19 00:11:00] Actual:true, Expected: true [2024-01-19T00:11:00.551Z] 2024-01-19 00:11:00 +0000:[INFO] [{SharedBackupDelete}] Delete the backups and validate [2024-01-19T00:11:01.478Z] 2024/01/19 00:11:01 DoRetryWithTimeout - Error: {backup [tp-backup-1705622461] is in [Deleting] state}, Next try in [30s], timeout [1h0m0s] [2024-01-19T00:11:01.478Z] 2024/01/19 00:11:01 DoRetryWithTimeout - Error: {backup [tp-backup-1705622531] is in [Deleting] state}, Next try in [30s], timeout [1h0m0s] [2024-01-19T00:11:01.478Z] 2024/01/19 00:11:01 DoRetryWithTimeout - Error: {backup [tp-backup-1705622471] is in [Deleting] state}, Next try in [30s], timeout [1h0m0s] [2024-01-19T00:11:01.478Z] 2024/01/19 00:11:01 DoRetryWithTimeout - Error: {backup [tp-backup-1705622511] is in [Deleting] state}, Next try in [30s], timeout [1h0m0s] [2024-01-19T00:11:01.479Z] 2024/01/19 00:11:01 DoRetryWithTimeout - Error: {backup [tp-backup-1705622521] is in [Deleting] state}, Next try in [30s], timeout [1h0m0s] [2024-01-19T00:11:01.479Z] 2024/01/19 00:11:01 DoRetryWithTimeout - Error: {backup [tp-backup-1705622501] is in [Deleting] state}, Next try in [30s], timeout [1h0m0s] [2024-01-19T00:11:01.479Z] 2024/01/19 00:11:01 DoRetryWithTimeout - Error: {backup [tp-backup-1705622491] is in [Deleting] state}, Next try in [30s], timeout [1h0m0s] [2024-01-19T00:11:01.479Z] 2024/01/19 00:11:01 DoRetryWithTimeout - Error: {backup [tp-backup-1705622541] is in [Deleting] state}, Next try in [30s], timeout [1h0m0s] [2024-01-19T00:11:01.479Z] 2024/01/19 00:11:01 DoRetryWithTimeout - Error: {backup [tp-backup-1705622481] is in [Deleting] state}, Next try in [30s], timeout [1h0m0s] [2024-01-19T00:11:01.479Z] 2024/01/19 00:11:01 DoRetryWithTimeout - Error: {backup [tp-backup-1705622551] is in [Deleting] state}, Next try in [30s], timeout [1h0m0s] [2024-01-19T00:11:33.509Z] 2024-01-19 00:11:31 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.7:#2193] - Validating user testuser3-1705622438 has access to no backups [2024-01-19T00:11:33.509Z] INFO[2024-01-19 00:11:31] Verifying : Description : Validating that user [testuser3-1705622438] has access to no backups [2024-01-19T00:11:33.509Z] INFO[2024-01-19 00:11:31] Actual:0, Expected: 0 [2024-01-19T00:11:33.509Z] 2024-01-19 00:11:31 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.7:#2193] - Validating user testuser10-1705622438 has access to no backups [2024-01-19T00:11:33.509Z] INFO[2024-01-19 00:11:31] Verifying : Description : Validating that user [testuser10-1705622438] has access to no backups [2024-01-19T00:11:33.509Z] INFO[2024-01-19 00:11:31] Actual:0, Expected: 0 [2024-01-19T00:11:33.509Z] 2024-01-19 00:11:31 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.7:#2193] - Validating user testuser4-1705622438 has access to no backups [2024-01-19T00:11:33.509Z] INFO[2024-01-19 00:11:31] Verifying : Description : Validating that user [testuser4-1705622438] has access to no backups [2024-01-19T00:11:33.509Z] INFO[2024-01-19 00:11:31] Actual:0, Expected: 0 [2024-01-19T00:11:33.509Z] 2024-01-19 00:11:31 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.7:#2193] - Validating user testuser5-1705622438 has access to no backups [2024-01-19T00:11:33.509Z] INFO[2024-01-19 00:11:31] Verifying : Description : Validating that user [testuser5-1705622438] has access to no backups [2024-01-19T00:11:33.509Z] INFO[2024-01-19 00:11:31] Actual:0, Expected: 0 [2024-01-19T00:11:33.509Z] 2024-01-19 00:11:31 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.7:#2193] - Validating user testuser2-1705622438 has access to no backups [2024-01-19T00:11:33.509Z] INFO[2024-01-19 00:11:31] Verifying : Description : Validating that user [testuser2-1705622438] has access to no backups [2024-01-19T00:11:33.509Z] INFO[2024-01-19 00:11:31] Actual:0, Expected: 0 [2024-01-19T00:11:33.509Z] 2024-01-19 00:11:31 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.7:#2193] - Validating user testuser6-1705622438 has access to no backups [2024-01-19T00:11:33.509Z] INFO[2024-01-19 00:11:31] Verifying : Description : Validating that user [testuser6-1705622438] has access to no backups [2024-01-19T00:11:33.509Z] INFO[2024-01-19 00:11:31] Actual:0, Expected: 0 [2024-01-19T00:11:33.509Z] 2024-01-19 00:11:31 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.7:#2193] - Validating user testuser1-1705622438 has access to no backups [2024-01-19T00:11:33.510Z] INFO[2024-01-19 00:11:32] Verifying : Description : Validating that user [testuser1-1705622438] has access to no backups [2024-01-19T00:11:33.510Z] INFO[2024-01-19 00:11:32] Actual:0, Expected: 0 [2024-01-19T00:11:33.510Z] 2024-01-19 00:11:32 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.7:#2193] - Validating user testuser7-1705622438 has access to no backups [2024-01-19T00:11:33.510Z] INFO[2024-01-19 00:11:32] Verifying : Description : Validating that user [testuser7-1705622438] has access to no backups [2024-01-19T00:11:33.510Z] INFO[2024-01-19 00:11:32] Actual:0, Expected: 0 [2024-01-19T00:11:33.510Z] 2024-01-19 00:11:32 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.7:#2193] - Validating user testuser8-1705622438 has access to no backups [2024-01-19T00:11:33.510Z] INFO[2024-01-19 00:11:32] Verifying : Description : Validating that user [testuser8-1705622438] has access to no backups [2024-01-19T00:11:33.510Z] INFO[2024-01-19 00:11:32] Actual:0, Expected: 0 [2024-01-19T00:11:33.510Z] 2024-01-19 00:11:32 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.2.7:#2193] - Validating user testuser9-1705622438 has access to no backups [2024-01-19T00:11:33.510Z] INFO[2024-01-19 00:11:33] Verifying : Description : Validating that user [testuser9-1705622438] has access to no backups [2024-01-19T00:11:33.510Z] INFO[2024-01-19 00:11:33] Actual:0, Expected: 0 [2024-01-19T00:11:33.510Z] 2024-01-19 00:11:33 +0000:[INFO] [{SharedBackupDelete}] Deleting the deployed apps after the testcase [2024-01-19T00:11:33.510Z] 2024-01-19 00:11:33 +0000:[INFO] [{SharedBackupDelete}] Destroying apps [2024-01-19T00:11:33.510Z] 2024-01-19 00:11:33 +0000:[INFO] [{SharedBackupDelete}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T00:11:33.767Z] 2024-01-19 00:11:33 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T00:11:33.767Z] 2024-01-19 00:11:33 +0000:[INFO] [{SharedBackupDelete}] start destroying mysql-ibm app [2024-01-19T00:11:34.024Z] 2024-01-19 00:11:33 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T00:11:34.024Z] 2024-01-19 00:11:33 +0000:[INFO] [{SharedBackupDelete}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T00:11:34.024Z] 2024-01-19 00:11:33 +0000:[WARNING] [{SharedBackupDelete}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T00:11:34.024Z] INFO[2024-01-19 00:11:33] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T00:11:34.024Z] INFO[2024-01-19 00:11:33] Actual:true, Expected: true [2024-01-19T00:11:34.024Z] 2024-01-19 00:11:33 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.3:#2206] - Deleting registered clusters for non-admin context [2024-01-19T00:11:34.024Z] 2024-01-19 00:11:33 +0000:[INFO] [{SharedBackupDelete}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T00:11:34.024Z] INFO[2024-01-19 00:11:33] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T00:11:34.024Z] INFO[2024-01-19 00:11:33] Actual:true, Expected: true [2024-01-19T00:11:34.586Z] INFO[2024-01-19 00:11:34] Verifying : Description : Deleting cluster source-cluster [2024-01-19T00:11:34.586Z] INFO[2024-01-19 00:11:34] Actual:true, Expected: true [2024-01-19T00:11:35.513Z] INFO[2024-01-19 00:11:35] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-gsmhp] [2024-01-19T00:11:35.513Z] INFO[2024-01-19 00:11:35] Actual:true, Expected: true [2024-01-19T00:11:35.769Z] INFO[2024-01-19 00:11:35] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T00:11:35.769Z] INFO[2024-01-19 00:11:35] Actual:true, Expected: true [2024-01-19T00:11:37.135Z] INFO[2024-01-19 00:11:36] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-vizom] [2024-01-19T00:11:37.135Z] INFO[2024-01-19 00:11:36] Actual:true, Expected: true [2024-01-19T00:11:37.135Z] 2024-01-19 00:11:36 +0000:[INFO] [{SharedBackupDelete}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T00:11:37.135Z] INFO[2024-01-19 00:11:36] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T00:11:37.135Z] INFO[2024-01-19 00:11:36] Actual:true, Expected: true [2024-01-19T00:11:37.135Z] INFO[2024-01-19 00:11:37] Verifying : Description : Deleting cluster source-cluster [2024-01-19T00:11:37.135Z] INFO[2024-01-19 00:11:37] Actual:true, Expected: true [2024-01-19T00:11:38.500Z] INFO[2024-01-19 00:11:38] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-wfyfe] [2024-01-19T00:11:38.500Z] INFO[2024-01-19 00:11:38] Actual:true, Expected: true [2024-01-19T00:11:38.756Z] INFO[2024-01-19 00:11:38] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T00:11:38.756Z] INFO[2024-01-19 00:11:38] Actual:true, Expected: true [2024-01-19T00:11:39.683Z] INFO[2024-01-19 00:11:39] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-lfzhw] [2024-01-19T00:11:39.684Z] INFO[2024-01-19 00:11:39] Actual:true, Expected: true [2024-01-19T00:11:39.684Z] 2024-01-19 00:11:39 +0000:[INFO] [{SharedBackupDelete}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T00:11:39.684Z] INFO[2024-01-19 00:11:39] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T00:11:39.684Z] INFO[2024-01-19 00:11:39] Actual:true, Expected: true [2024-01-19T00:11:39.940Z] INFO[2024-01-19 00:11:39] Verifying : Description : Deleting cluster source-cluster [2024-01-19T00:11:39.940Z] INFO[2024-01-19 00:11:39] Actual:true, Expected: true [2024-01-19T00:11:41.305Z] INFO[2024-01-19 00:11:40] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-mtvko] [2024-01-19T00:11:41.305Z] INFO[2024-01-19 00:11:40] Actual:true, Expected: true [2024-01-19T00:11:41.560Z] INFO[2024-01-19 00:11:41] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T00:11:41.560Z] INFO[2024-01-19 00:11:41] Actual:true, Expected: true [2024-01-19T00:11:42.487Z] INFO[2024-01-19 00:11:42] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-dggka] [2024-01-19T00:11:42.487Z] INFO[2024-01-19 00:11:42] Actual:true, Expected: true [2024-01-19T00:11:42.487Z] 2024-01-19 00:11:42 +0000:[INFO] [{SharedBackupDelete}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T00:11:42.487Z] INFO[2024-01-19 00:11:42] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T00:11:42.487Z] INFO[2024-01-19 00:11:42] Actual:true, Expected: true [2024-01-19T00:11:42.743Z] INFO[2024-01-19 00:11:42] Verifying : Description : Deleting cluster source-cluster [2024-01-19T00:11:42.743Z] INFO[2024-01-19 00:11:42] Actual:true, Expected: true [2024-01-19T00:11:44.117Z] INFO[2024-01-19 00:11:43] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-pksdz] [2024-01-19T00:11:44.117Z] INFO[2024-01-19 00:11:43] Actual:true, Expected: true [2024-01-19T00:11:44.117Z] INFO[2024-01-19 00:11:44] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T00:11:44.117Z] INFO[2024-01-19 00:11:44] Actual:true, Expected: true [2024-01-19T00:11:45.482Z] INFO[2024-01-19 00:11:45] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-qtxon] [2024-01-19T00:11:45.482Z] INFO[2024-01-19 00:11:45] Actual:true, Expected: true [2024-01-19T00:11:45.482Z] 2024-01-19 00:11:45 +0000:[INFO] [{SharedBackupDelete}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T00:11:45.482Z] INFO[2024-01-19 00:11:45] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T00:11:45.482Z] INFO[2024-01-19 00:11:45] Actual:true, Expected: true [2024-01-19T00:11:45.482Z] INFO[2024-01-19 00:11:45] Verifying : Description : Deleting cluster source-cluster [2024-01-19T00:11:45.482Z] INFO[2024-01-19 00:11:45] Actual:true, Expected: true [2024-01-19T00:11:46.847Z] INFO[2024-01-19 00:11:46] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-nrkzb] [2024-01-19T00:11:46.847Z] INFO[2024-01-19 00:11:46] Actual:true, Expected: true [2024-01-19T00:11:47.104Z] INFO[2024-01-19 00:11:46] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T00:11:47.104Z] INFO[2024-01-19 00:11:46] Actual:true, Expected: true [2024-01-19T00:11:48.032Z] INFO[2024-01-19 00:11:47] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-pmaxe] [2024-01-19T00:11:48.032Z] INFO[2024-01-19 00:11:47] Actual:true, Expected: true [2024-01-19T00:11:48.032Z] 2024-01-19 00:11:47 +0000:[INFO] [{SharedBackupDelete}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T00:11:48.032Z] INFO[2024-01-19 00:11:47] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T00:11:48.032Z] INFO[2024-01-19 00:11:47] Actual:true, Expected: true [2024-01-19T00:11:48.288Z] INFO[2024-01-19 00:11:48] Verifying : Description : Deleting cluster source-cluster [2024-01-19T00:11:48.288Z] INFO[2024-01-19 00:11:48] Actual:true, Expected: true [2024-01-19T00:11:49.653Z] INFO[2024-01-19 00:11:49] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-bnkuv] [2024-01-19T00:11:49.653Z] INFO[2024-01-19 00:11:49] Actual:true, Expected: true [2024-01-19T00:11:49.653Z] INFO[2024-01-19 00:11:49] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T00:11:49.653Z] INFO[2024-01-19 00:11:49] Actual:true, Expected: true [2024-01-19T00:11:51.016Z] INFO[2024-01-19 00:11:50] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-wconk] [2024-01-19T00:11:51.016Z] INFO[2024-01-19 00:11:50] Actual:true, Expected: true [2024-01-19T00:11:51.016Z] 2024-01-19 00:11:50 +0000:[INFO] [{SharedBackupDelete}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T00:11:51.016Z] INFO[2024-01-19 00:11:50] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T00:11:51.016Z] INFO[2024-01-19 00:11:50] Actual:true, Expected: true [2024-01-19T00:11:51.272Z] INFO[2024-01-19 00:11:51] Verifying : Description : Deleting cluster source-cluster [2024-01-19T00:11:51.272Z] INFO[2024-01-19 00:11:51] Actual:true, Expected: true [2024-01-19T00:11:52.201Z] INFO[2024-01-19 00:11:52] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-hzzir] [2024-01-19T00:11:52.201Z] INFO[2024-01-19 00:11:52] Actual:true, Expected: true [2024-01-19T00:11:52.457Z] INFO[2024-01-19 00:11:52] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T00:11:52.457Z] INFO[2024-01-19 00:11:52] Actual:true, Expected: true [2024-01-19T00:11:53.854Z] INFO[2024-01-19 00:11:53] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-unirp] [2024-01-19T00:11:53.854Z] INFO[2024-01-19 00:11:53] Actual:true, Expected: true [2024-01-19T00:11:53.854Z] 2024-01-19 00:11:53 +0000:[INFO] [{SharedBackupDelete}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T00:11:53.854Z] INFO[2024-01-19 00:11:53] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T00:11:53.854Z] INFO[2024-01-19 00:11:53] Actual:true, Expected: true [2024-01-19T00:11:53.854Z] INFO[2024-01-19 00:11:53] Verifying : Description : Deleting cluster source-cluster [2024-01-19T00:11:53.854Z] INFO[2024-01-19 00:11:53] Actual:true, Expected: true [2024-01-19T00:11:55.217Z] INFO[2024-01-19 00:11:54] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-qjrgi] [2024-01-19T00:11:55.217Z] INFO[2024-01-19 00:11:54] Actual:true, Expected: true [2024-01-19T00:11:55.217Z] INFO[2024-01-19 00:11:55] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T00:11:55.217Z] INFO[2024-01-19 00:11:55] Actual:true, Expected: true [2024-01-19T00:11:56.583Z] INFO[2024-01-19 00:11:56] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-kynmv] [2024-01-19T00:11:56.583Z] INFO[2024-01-19 00:11:56] Actual:true, Expected: true [2024-01-19T00:11:56.583Z] 2024-01-19 00:11:56 +0000:[INFO] [{SharedBackupDelete}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T00:11:56.583Z] INFO[2024-01-19 00:11:56] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T00:11:56.583Z] INFO[2024-01-19 00:11:56] Actual:true, Expected: true [2024-01-19T00:11:56.839Z] INFO[2024-01-19 00:11:56] Verifying : Description : Deleting cluster source-cluster [2024-01-19T00:11:56.839Z] INFO[2024-01-19 00:11:56] Actual:true, Expected: true [2024-01-19T00:11:57.769Z] INFO[2024-01-19 00:11:57] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-jttmr] [2024-01-19T00:11:57.769Z] INFO[2024-01-19 00:11:57] Actual:true, Expected: true [2024-01-19T00:11:58.025Z] INFO[2024-01-19 00:11:57] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T00:11:58.025Z] INFO[2024-01-19 00:11:57] Actual:true, Expected: true [2024-01-19T00:11:59.388Z] INFO[2024-01-19 00:11:59] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-mnxik] [2024-01-19T00:11:59.388Z] INFO[2024-01-19 00:11:59] Actual:true, Expected: true [2024-01-19T00:11:59.388Z] 2024-01-19 00:11:59 +0000:[INFO] [{SharedBackupDelete}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T00:11:59.388Z] INFO[2024-01-19 00:11:59] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T00:11:59.388Z] INFO[2024-01-19 00:11:59] Actual:true, Expected: true [2024-01-19T00:11:59.388Z] INFO[2024-01-19 00:11:59] Verifying : Description : Deleting cluster source-cluster [2024-01-19T00:11:59.388Z] INFO[2024-01-19 00:11:59] Actual:true, Expected: true [2024-01-19T00:12:00.752Z] INFO[2024-01-19 00:12:00] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-uokzd] [2024-01-19T00:12:00.752Z] INFO[2024-01-19 00:12:00] Actual:true, Expected: true [2024-01-19T00:12:01.009Z] INFO[2024-01-19 00:12:00] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T00:12:01.009Z] INFO[2024-01-19 00:12:00] Actual:true, Expected: true [2024-01-19T00:12:02.377Z] INFO[2024-01-19 00:12:01] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-wrmiu] [2024-01-19T00:12:02.377Z] INFO[2024-01-19 00:12:01] Actual:true, Expected: true [2024-01-19T00:12:02.377Z] 2024-01-19 00:12:01 +0000:[INFO] [{SharedBackupDelete}] [backup.glob..func87.3:#2212] - Cleaning up users [2024-01-19T00:12:03.305Z] I0119 00:12:02.981218 14 request.go:682] Waited for 1.050438285s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/services [2024-01-19T00:12:13.246Z] 2024-01-19 00:12:12 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser9-1705622438 - e253b80a-8c31-4ad2-9a21-702b0c386ec1 [2024-01-19T00:12:13.246Z] 2024-01-19 00:12:12 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2-1705622438 - c161acfd-6047-4d40-93ba-9f88c22efdf5 [2024-01-19T00:12:13.246Z] 2024-01-19 00:12:12 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser3-1705622438 - cdc67b42-871e-4ca1-8d95-9ba7dea58b08 [2024-01-19T00:12:13.247Z] 2024-01-19 00:12:12 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser6-1705622438 - 88f8d071-50ff-4aad-a907-c3cd4ea9027b [2024-01-19T00:12:13.247Z] 2024-01-19 00:12:12 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser10-1705622438 - a57151ec-69ca-4f7c-ad7b-7048c5513968 [2024-01-19T00:12:13.247Z] 2024-01-19 00:12:13 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705622438 - e7b7bdc4-893c-434d-852f-21edd74993e1 [2024-01-19T00:12:13.247Z] I0119 00:12:13.181251 14 request.go:682] Waited for 1.902413699s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-19T00:12:13.247Z] 2024-01-19 00:12:13 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser4-1705622438 - 95ebe0ab-2852-448c-a013-75d46766a6eb [2024-01-19T00:12:13.503Z] 2024-01-19 00:12:13 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser7-1705622438 - 1a84069f-7604-48f6-8a17-90dfb3e1769f [2024-01-19T00:12:13.759Z] 2024-01-19 00:12:13 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser8-1705622438 - d76ecda7-a2f7-4a32-96c7-0986ec23ef07 [2024-01-19T00:12:14.014Z] 2024-01-19 00:12:13 +0000:[INFO] [{SharedBackupDelete}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser5-1705622438 - a4c4106c-2fa5-494f-adb2-f36bb650cc3a [2024-01-19T00:12:16.529Z] 2024-01-19 00:12:16 +0000:[INFO] [{SharedBackupDelete}] [backup.DeleteUser:#755] - Response for user [testuser9-1705622438] deletion - [2024-01-19T00:12:16.529Z] 2024-01-19 00:12:16 +0000:[INFO] [{SharedBackupDelete}] [backup.DeleteUser:#759] - Deleted User - testuser9-1705622438 [2024-01-19T00:12:16.529Z] 2024-01-19 00:12:16 +0000:[INFO] [{SharedBackupDelete}] [backup.DeleteUser:#755] - Response for user [testuser2-1705622438] deletion - [2024-01-19T00:12:16.529Z] 2024-01-19 00:12:16 +0000:[INFO] [{SharedBackupDelete}] [backup.DeleteUser:#759] - Deleted User - testuser2-1705622438 [2024-01-19T00:12:16.785Z] 2024-01-19 00:12:16 +0000:[INFO] [{SharedBackupDelete}] [backup.DeleteUser:#755] - Response for user [testuser3-1705622438] deletion - [2024-01-19T00:12:16.785Z] 2024-01-19 00:12:16 +0000:[INFO] [{SharedBackupDelete}] [backup.DeleteUser:#759] - Deleted User - testuser3-1705622438 [2024-01-19T00:12:16.785Z] 2024-01-19 00:12:16 +0000:[INFO] [{SharedBackupDelete}] [backup.DeleteUser:#755] - Response for user [testuser6-1705622438] deletion - [2024-01-19T00:12:16.785Z] 2024-01-19 00:12:16 +0000:[INFO] [{SharedBackupDelete}] [backup.DeleteUser:#759] - Deleted User - testuser6-1705622438 [2024-01-19T00:12:17.041Z] 2024-01-19 00:12:16 +0000:[INFO] [{SharedBackupDelete}] [backup.DeleteUser:#755] - Response for user [testuser10-1705622438] deletion - [2024-01-19T00:12:17.041Z] 2024-01-19 00:12:16 +0000:[INFO] [{SharedBackupDelete}] [backup.DeleteUser:#759] - Deleted User - testuser10-1705622438 [2024-01-19T00:12:17.297Z] 2024-01-19 00:12:17 +0000:[INFO] [{SharedBackupDelete}] [backup.DeleteUser:#755] - Response for user [testuser1-1705622438] deletion - [2024-01-19T00:12:17.297Z] 2024-01-19 00:12:17 +0000:[INFO] [{SharedBackupDelete}] [backup.DeleteUser:#759] - Deleted User - testuser1-1705622438 [2024-01-19T00:12:17.552Z] 2024-01-19 00:12:17 +0000:[INFO] [{SharedBackupDelete}] [backup.DeleteUser:#755] - Response for user [testuser4-1705622438] deletion - [2024-01-19T00:12:17.552Z] 2024-01-19 00:12:17 +0000:[INFO] [{SharedBackupDelete}] [backup.DeleteUser:#759] - Deleted User - testuser4-1705622438 [2024-01-19T00:12:17.552Z] 2024-01-19 00:12:17 +0000:[INFO] [{SharedBackupDelete}] [backup.DeleteUser:#755] - Response for user [testuser7-1705622438] deletion - [2024-01-19T00:12:17.552Z] 2024-01-19 00:12:17 +0000:[INFO] [{SharedBackupDelete}] [backup.DeleteUser:#759] - Deleted User - testuser7-1705622438 [2024-01-19T00:12:17.807Z] 2024-01-19 00:12:17 +0000:[INFO] [{SharedBackupDelete}] [backup.DeleteUser:#755] - Response for user [testuser8-1705622438] deletion - [2024-01-19T00:12:17.807Z] 2024-01-19 00:12:17 +0000:[INFO] [{SharedBackupDelete}] [backup.DeleteUser:#759] - Deleted User - testuser8-1705622438 [2024-01-19T00:12:18.062Z] 2024-01-19 00:12:17 +0000:[INFO] [{SharedBackupDelete}] [backup.DeleteUser:#755] - Response for user [testuser5-1705622438] deletion - [2024-01-19T00:12:18.062Z] 2024-01-19 00:12:17 +0000:[INFO] [{SharedBackupDelete}] [backup.DeleteUser:#759] - Deleted User - testuser5-1705622438 [2024-01-19T00:12:19.430Z] 2024-01-19 00:12:19 +0000:[INFO] [{SharedBackupDelete}] Cleaning backup locations in map [map[e4afce94-3012-4119-b893-5e6fd4811d8d:autogenerated-backup-location-1705622446]], cloud credential [autogenerated-cred-1705622446], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T00:12:19.430Z] INFO[2024-01-19 00:12:19] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-1705622446] [2024-01-19T00:12:19.430Z] INFO[2024-01-19 00:12:19] Actual:true, Expected: true [2024-01-19T00:12:19.430Z] 2024-01-19 00:12:19 +0000:[INFO] [{SharedBackupDelete}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705622446] is present [2024-01-19T00:12:19.689Z] 2024/01/19 00:12:19 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705622446 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T00:12:51.713Z] 2024-01-19 00:12:49 +0000:[INFO] [{SharedBackupDelete}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 aws-global-aws-px-backup-system-test-4cac76dc-bl] [2024-01-19T00:12:51.714Z] INFO[2024-01-19 00:12:49] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-1705622446 [2024-01-19T00:12:51.714Z] INFO[2024-01-19 00:12:49] Actual:true, Expected: true [2024-01-19T00:12:51.714Z] 2024-01-19 00:12:49 +0000:[INFO] [{SharedBackupDelete}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-1705622446] is present [2024-01-19T00:12:51.714Z] INFO[2024-01-19 00:12:49] Verifying : Description : Verifying if cloud cred [autogenerated-cred-1705622446] is present [2024-01-19T00:12:51.714Z] INFO[2024-01-19 00:12:49] Actual:true, Expected: true [2024-01-19T00:12:51.714Z] INFO[2024-01-19 00:12:49] Verifying : Description : Verifying deletion of cloud cred [autogenerated-cred-1705622446] [2024-01-19T00:12:51.714Z] INFO[2024-01-19 00:12:49] Actual:true, Expected: true [2024-01-19T00:12:51.714Z] INFO[2024-01-19 00:12:49] Verifying : Description : Deleting cloud cred autogenerated-cred-1705622446 [2024-01-19T00:12:51.714Z] INFO[2024-01-19 00:12:49] Actual:true, Expected: true [2024-01-19T00:12:51.714Z] INFO[2024-01-19 00:12:49] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T00:12:51.714Z] INFO[2024-01-19 00:12:49] Actual:true, Expected: true [2024-01-19T00:12:51.714Z] INFO[2024-01-19 00:12:49] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T00:12:51.714Z] ERRO[2024-01-19 00:12:49] Actual:rpc error: code = FailedPrecondition desc = cluster [destination-cluster] in use by backup [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-19-001118], cannot be deleted, Expected:  [2024-01-19T00:12:51.714Z] INFO[2024-01-19 00:12:50] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-zbbvt] [2024-01-19T00:12:51.714Z] ERRO[2024-01-19 00:12:50] Actual:rpc error: code = Unknown desc = cluster [destination-cluster] has reference to cloud credential [ibm-destination-config-cloud-cred-zbbvt:a271ea5b-dc9d-40f1-a82a-651539c7e486], Expected:  [2024-01-19T00:12:51.714Z] INFO[2024-01-19 00:12:50] Verifying : Description : Deleting cluster source-cluster [2024-01-19T00:12:51.714Z] INFO[2024-01-19 00:12:50] Actual:true, Expected: true [2024-01-19T00:12:51.714Z] INFO[2024-01-19 00:12:50] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-pntez] [2024-01-19T00:12:51.714Z] INFO[2024-01-19 00:12:50] Actual:true, Expected: true [2024-01-19T00:12:51.714Z] INFO[2024-01-19 00:12:50] --------Test End------ [2024-01-19T00:12:51.714Z] INFO[2024-01-19 00:12:50] #Test: [2024-01-19T00:12:51.714Z] INFO[2024-01-19 00:12:50] #Description: [2024-01-19T00:12:51.714Z] INFO[2024-01-19 00:12:50] ------------------------ [2024-01-19T00:12:51.714Z] 2024-01-19 00:12:50 +0000:[INFO] [{SharedBackupDelete}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-82946-01-18-23h59m07s] [2024-01-19T00:12:51.714Z] 2024-01-19 00:12:51 +0000:[INFO] [{SharedBackupDelete}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T00:12:51.714Z] 2024-01-19 00:12:51 +0000:[INFO] [{SharedBackupDelete}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T00:12:51.714Z] 2024-01-19 00:12:51 +0000:[INFO] [{SharedBackupDelete}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-82946-01-18-23h59m07s] [2024-01-19T00:12:51.714Z] 2024-01-19 00:12:51 +0000:[INFO] [{SharedBackupDelete}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {SharedBackupDelete} Share the backups and delete [2024-01-19T00:12:51.714Z] 2024-01-19 00:12:51 +0000:[INFO] [{SharedBackupDelete}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T00:12:51.714Z] 2024/01/19 00:12:51 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:12:56.966Z] 2024/01/19 00:12:56 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:13:02.207Z] 2024/01/19 00:13:01 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:13:07.449Z] 2024/01/19 00:13:06 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:13:11.614Z] 2024/01/19 00:13:11 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:13:16.855Z] 2024/01/19 00:13:16 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:13:22.099Z] 2024/01/19 00:13:21 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:13:27.347Z] 2024/01/19 00:13:26 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:13:31.515Z] 2024/01/19 00:13:31 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.64.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.8 10.240.64.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T00:13:31.515Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T00:13:31.515Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T00:13:31.516Z] {dial tcp 10.240.64.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T00:13:36.764Z] 2024-01-19 00:13:36 +0000:[WARNING] [{SharedBackupDelete}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.64.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.8 10.240.64.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T00:13:36.764Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T00:13:36.764Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T00:13:36.764Z] {dial tcp 10.240.64.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T00:13:36.764Z] 2024-01-19 00:13:36 +0000:[ERROR] [{SharedBackupDelete}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.64.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.8 10.240.64.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T00:13:36.764Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T00:13:36.764Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T00:13:36.764Z] {dial tcp 10.240.64.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T00:13:36.764Z] 2024-01-19 00:13:36 +0000:[INFO] [{SharedBackupDelete}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T00:13:36.764Z] 2024-01-19 00:13:36 +0000:[INFO] [{SharedBackupDelete}] Switching context to [/tmp/source-config] [2024-01-19T00:13:36.764Z] 2024-01-19 00:13:36 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T00:13:36.764Z] 2024-01-19 00:13:36 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T00:13:36.764Z] 2024-01-19 00:13:36 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T00:13:36.764Z] 2024-01-19 00:13:36 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T00:13:36.764Z] 2024-01-19 00:13:36 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T00:13:37.029Z] 2024-01-19 00:13:36 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T00:13:37.591Z] 2024-01-19 00:13:37 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T00:13:37.846Z] 2024-01-19 00:13:37 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T00:13:38.408Z] 2024-01-19 00:13:38 +0000:[INFO] [{SharedBackupDelete}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T00:13:38.408Z] 2024/01/19 00:13:38 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:13:48.347Z] 2024/01/19 00:13:48 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:14:00.516Z] 2024/01/19 00:13:58 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:14:08.600Z] 2024/01/19 00:14:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:14:18.541Z] 2024/01/19 00:14:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:14:28.485Z] 2024/01/19 00:14:28 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:14:38.540Z] INFO[2024-01-19 00:14:37] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T00:14:38.540Z] INFO[2024-01-19 00:14:37] sched stats: tick interval: 31237 < 1.2s (100%) [2024-01-19T00:14:38.540Z] INFO[2024-01-19 00:14:37] sched stats: tick process durations: 31236 < 500ms (100%) [2024-01-19T00:14:38.540Z] 2024/01/19 00:14:38 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:14:50.722Z] 2024/01/19 00:14:48 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:14:58.804Z] 2024/01/19 00:14:58 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:15:08.752Z] 2024/01/19 00:15:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:15:18.695Z] 2024/01/19 00:15:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:15:28.645Z] 2024/01/19 00:15:28 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:15:40.840Z] 2024/01/19 00:15:38 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:15:48.924Z] 2024/01/19 00:15:48 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:15:58.864Z] 2024/01/19 00:15:58 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:16:08.823Z] 2024/01/19 00:16:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:16:20.992Z] 2024/01/19 00:16:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:16:29.076Z] 2024/01/19 00:16:28 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:16:39.020Z] 2024/01/19 00:16:38 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:16:48.961Z] 2024/01/19 00:16:48 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:17:01.162Z] 2024/01/19 00:16:58 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:17:09.324Z] 2024/01/19 00:17:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:17:19.269Z] 2024/01/19 00:17:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:17:29.208Z] 2024/01/19 00:17:29 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:17:39.156Z] 2024/01/19 00:17:39 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:17:51.327Z] 2024/01/19 00:17:49 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:17:59.413Z] 2024/01/19 00:17:59 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:18:09.349Z] 2024/01/19 00:18:09 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:18:19.295Z] 2024/01/19 00:18:19 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:18:31.467Z] 2024/01/19 00:18:29 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:18:39.546Z] 2024-01-19 00:18:39 +0000:[ERROR] [{SharedBackupDelete}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.546Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.546Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.546Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.546Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.546Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.546Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.546Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.546Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.546Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.546Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.546Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.546Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.547Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.547Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.547Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.547Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.547Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.547Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.547Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.547Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.547Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.547Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.547Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.547Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.547Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.547Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.547Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.547Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.547Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T00:18:39.547Z] [2024-01-19T00:18:39.547Z] ------------------------------ [2024-01-19T00:18:39.547Z] • Failure [1172.220 seconds] [2024-01-19T00:18:39.547Z] {SharedBackupDelete} [2024-01-19T00:18:39.547Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:2024 [2024-01-19T00:18:39.547Z] Share the backups and delete [] [2024-01-19T00:18:39.547Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:2058 [2024-01-19T00:18:39.547Z] [2024-01-19T00:18:39.547Z] Unexpected error: [2024-01-19T00:18:39.547Z] <*errors.errorString | 0xc000e64a80>: { [2024-01-19T00:18:39.548Z] s: "failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{No... [2024-01-19T00:18:39.548Z] [2024-01-19T00:18:39.548Z] Gomega truncated this representation as it exceeds 'format.MaxLength'. [2024-01-19T00:18:39.548Z] Consider having the object provide a custom 'GomegaStringer' representation [2024-01-19T00:18:39.548Z] or adjust the parameters in Gomega's 'format' package. [2024-01-19T00:18:39.548Z] [2024-01-19T00:18:39.548Z] Learn more here: https://onsi.github.io/gomega/#adjusting-output [2024-01-19T00:18:39.548Z] [2024-01-19T00:18:39.549Z] failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.549Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.549Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.549Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.549Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.549Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.549Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.549Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.549Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.549Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.549Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.549Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.549Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.549Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.549Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.549Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.549Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.549Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.549Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.549Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.549Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.550Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.550Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.550Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.550Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.550Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.550Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.550Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.550Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:18:39.550Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T00:18:39.550Z] occurred [2024-01-19T00:18:39.550Z] [2024-01-19T00:18:39.550Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T00:18:39.550Z] [2024-01-19T00:18:39.550Z] Full Stack Trace [2024-01-19T00:18:39.550Z] github.com/portworx/torpedo/pkg/log.FailOnError({0x88a5060, 0xc000e64a80}, {0x7f74bfd?, 0x6cc0320?}, {0x0?, 0x6cc0320?, 0xc00179c350?}) [2024-01-19T00:18:39.550Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 +0x290 [2024-01-19T00:18:39.550Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest.func1() [2024-01-19T00:18:39.550Z] /go/src/github.com/portworx/torpedo/tests/common.go:7283 +0x32 [2024-01-19T00:18:39.550Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest({0xc0003aa740, 0x1, 0x1}) [2024-01-19T00:18:39.550Z] /go/src/github.com/portworx/torpedo/tests/common.go:7299 +0x778 [2024-01-19T00:18:39.550Z] github.com/portworx/torpedo/tests/backup.glob..func87.3() [2024-01-19T00:18:39.550Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:2226 +0x247 [2024-01-19T00:18:39.550Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T00:18:39.550Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T00:18:39.550Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T00:18:39.550Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T00:18:39.550Z] created by testing.(*T).Run [2024-01-19T00:18:39.550Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T00:18:39.550Z] ------------------------------ [2024-01-19T00:18:39.550Z] STEP: schedule applications [2024-01-19T00:18:39.550Z] STEP: Validate applications [2024-01-19T00:18:39.550Z] STEP: validate applications [2024-01-19T00:18:39.550Z] STEP: validate mysql-ibm app's volumes [2024-01-19T00:18:39.550Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T00:18:39.550Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T00:18:39.550Z] STEP: get mysql-ibm app's volume: pvc-ffab43c6-a0ca-498c-9340-b90e440e70fe inspected by the volume driver [2024-01-19T00:18:39.550Z] STEP: wait for mysql-ibm app to start running [2024-01-19T00:18:39.550Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T00:18:39.550Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T00:18:39.550Z] STEP: Validate Px pod restart count [2024-01-19T00:18:39.550Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T00:18:39.550Z] STEP: Create Users [2024-01-19T00:18:39.550Z] STEP: Adding Credentials and Registering Backup Location [2024-01-19T00:18:39.550Z] STEP: Register source and destination cluster for backup [2024-01-19T00:18:39.550Z] STEP: Taking backup of applications [2024-01-19T00:18:39.550Z] STEP: Share backup with multiple users [2024-01-19T00:18:39.550Z] STEP: Delete the backups and validate [2024-01-19T00:18:39.550Z] STEP: destroy apps [2024-01-19T00:18:39.550Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T00:18:39.550Z] STEP: start destroying mysql-ibm app [2024-01-19T00:18:39.550Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T00:18:39.550Z] 2024-01-19 00:18:39 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] Log Dir: /testresults//LicensingCountWithNodeLabelledBeforeClusterAddition.log [2024-01-19T00:18:39.551Z] INFO[2024-01-19 00:18:39] --------Test Start------ [2024-01-19T00:18:39.551Z] INFO[2024-01-19 00:18:39] #Test: LicensingCountWithNodeLabelledBeforeClusterAddition [2024-01-19T00:18:39.551Z] INFO[2024-01-19 00:18:39] #Description: Applies label portworx.io/nobackup=true before adding of application cluster to backup and verifies the license count [2024-01-19T00:18:39.551Z] INFO[2024-01-19 00:18:39] ------------------------ [2024-01-19T00:18:39.551Z] 2024-01-19 00:18:39 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] Deploy applications needed for taking backup [2024-01-19T00:18:39.551Z] 2024-01-19 00:18:39 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T00:18:39.551Z] 2024-01-19 00:18:39 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T00:18:39.551Z] 2024-01-19 00:18:39 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T00:18:39.551Z] 2024-01-19 00:18:39 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T00:18:39.551Z] 2024-01-19 00:18:39 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T00:18:39.551Z] 2024-01-19 00:18:39 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T00:18:39.552Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T00:18:39.552Z] 2024-01-19 00:18:39 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T00:18:39.552Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T00:18:39.807Z] 2024-01-19 00:18:39 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T00:18:40.063Z] INFO[2024-01-19 00:18:39] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T00:18:40.063Z] ERRO[2024-01-19 00:18:39] Actual:rpc error: code = FailedPrecondition desc = cluster [destination-cluster] in use by backup [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-19-001118], cannot be deleted, Expected:  [2024-01-19T00:18:40.063Z] 2024-01-19 00:18:39 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] Destroying apps [2024-01-19T00:18:40.063Z] 2024-01-19 00:18:39 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T00:18:40.318Z] 2024-01-19 00:18:40 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T00:18:40.318Z] 2024-01-19 00:18:40 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] start destroying mysql-ibm app [2024-01-19T00:18:40.318Z] 2024-01-19 00:18:40 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T00:18:40.319Z] 2024-01-19 00:18:40 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T00:18:40.319Z] 2024-01-19 00:18:40 +0000:[WARNING] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T00:18:40.319Z] INFO[2024-01-19 00:18:40] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T00:18:40.319Z] INFO[2024-01-19 00:18:40] Actual:true, Expected: true [2024-01-19T00:18:40.319Z] 2024-01-19 00:18:40 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T00:18:40.319Z] 2024-01-19 00:18:40 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T00:18:40.319Z] INFO[2024-01-19 00:18:40] Verifying : Description : Switching context to destination cluster [2024-01-19T00:18:40.319Z] INFO[2024-01-19 00:18:40] Actual:true, Expected: true [2024-01-19T00:18:40.319Z] 2024-01-19 00:18:40 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] Removing label portworx.io/nobackup=true from all worker nodes on destination cluster if present [2024-01-19T00:18:40.319Z] 2024-01-19 00:18:40 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] Switching cluster context back to source cluster [2024-01-19T00:18:40.319Z] 2024-01-19 00:18:40 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T00:18:40.319Z] 2024-01-19 00:18:40 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] Switching context to [/tmp/source-config] [2024-01-19T00:18:40.319Z] 2024-01-19 00:18:40 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T00:18:40.319Z] 2024-01-19 00:18:40 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T00:18:40.574Z] 2024-01-19 00:18:40 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T00:18:40.574Z] 2024-01-19 00:18:40 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T00:18:40.574Z] 2024-01-19 00:18:40 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T00:18:40.829Z] 2024-01-19 00:18:40 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T00:18:41.085Z] 2024-01-19 00:18:40 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T00:18:41.645Z] 2024-01-19 00:18:41 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T00:18:41.901Z] 2024-01-19 00:18:41 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T00:18:42.156Z] 2024/01/19 00:18:41 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:18:52.098Z] 2024/01/19 00:18:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:19:04.261Z] 2024/01/19 00:19:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:19:12.343Z] 2024/01/19 00:19:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:19:22.284Z] 2024/01/19 00:19:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:19:32.230Z] 2024/01/19 00:19:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:19:44.400Z] 2024/01/19 00:19:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:19:52.499Z] 2024/01/19 00:19:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:20:02.443Z] 2024/01/19 00:20:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:20:12.395Z] 2024/01/19 00:20:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:20:22.332Z] 2024/01/19 00:20:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:20:34.500Z] 2024/01/19 00:20:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:20:42.585Z] 2024/01/19 00:20:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:20:52.532Z] 2024/01/19 00:20:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:21:02.505Z] 2024/01/19 00:21:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:21:12.476Z] 2024/01/19 00:21:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:21:24.673Z] 2024/01/19 00:21:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:21:33.740Z] 2024/01/19 00:21:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:21:43.684Z] 2024/01/19 00:21:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:21:53.632Z] 2024/01/19 00:21:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:22:03.572Z] 2024/01/19 00:22:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:22:13.524Z] 2024/01/19 00:22:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:22:23.474Z] 2024/01/19 00:22:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:22:33.414Z] 2024/01/19 00:22:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:22:43.390Z] 2024/01/19 00:22:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:22:53.333Z] 2024/01/19 00:22:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:23:03.280Z] 2024/01/19 00:23:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:23:13.231Z] 2024/01/19 00:23:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:23:23.172Z] 2024/01/19 00:23:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:23:33.123Z] 2024/01/19 00:23:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:23:43.069Z] INFO[2024-01-19 00:23:42] Verifying : Description : Switching context to source cluster [2024-01-19T00:23:43.069Z] ERRO[2024-01-19 00:23:42] failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.069Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.069Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.069Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.069Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.069Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.069Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.070Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.070Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.070Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.070Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.070Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.070Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.070Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.070Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.070Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.070Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.070Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.070Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.070Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.070Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.070Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.070Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.070Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.070Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.070Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.070Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.071Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.071Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:23:43.071Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T00:23:43.071Z] 2024-01-19 00:23:42 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] Removing label portworx.io/nobackup=true from all worker nodes on source cluster if present [2024-01-19T00:23:43.328Z] 2024-01-19 00:23:43 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] Deleting the restore taken [2024-01-19T00:23:43.328Z] INFO[2024-01-19 00:23:43] Verifying : Description : Getting the backup driver [2024-01-19T00:23:43.328Z] INFO[2024-01-19 00:23:43] Actual:true, Expected: true [2024-01-19T00:23:43.328Z] INFO[2024-01-19 00:23:43] Verifying : Description : Deleting the restore taken: [2024-01-19T00:23:43.328Z] ERRO[2024-01-19 00:23:43] restore with name '' not found for org 'default' [2024-01-19T00:23:43.328Z] 2024-01-19 00:23:43 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T00:23:43.328Z] INFO[2024-01-19 00:23:43] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T00:23:43.328Z] INFO[2024-01-19 00:23:43] Actual:true, Expected: true [2024-01-19T00:23:43.584Z] INFO[2024-01-19 00:23:43] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T00:23:43.584Z] ERRO[2024-01-19 00:23:43] Actual:rpc error: code = FailedPrecondition desc = cluster [destination-cluster] in use by backupschedule [tp-backup-schedule-1705605709], cannot be deleted, Expected:  [2024-01-19T00:23:44.146Z] INFO[2024-01-19 00:23:43] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-zbbvt] [2024-01-19T00:23:44.146Z] ERRO[2024-01-19 00:23:43] Actual:rpc error: code = Unknown desc = cluster [destination-cluster] has reference to cloud credential [ibm-destination-config-cloud-cred-zbbvt:a271ea5b-dc9d-40f1-a82a-651539c7e486], Expected:  [2024-01-19T00:23:44.146Z] INFO[2024-01-19 00:23:43] --------Test End------ [2024-01-19T00:23:44.146Z] INFO[2024-01-19 00:23:43] #Test: [2024-01-19T00:23:44.146Z] INFO[2024-01-19 00:23:43] #Description: [2024-01-19T00:23:44.147Z] INFO[2024-01-19 00:23:43] ------------------------ [2024-01-19T00:23:44.147Z] 2024-01-19 00:23:43 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [tests.EndPxBackupTorpedoTest:#7261] - >>>> FAILED TEST: {LicensingCountWithNodeLabelledBeforeClusterAddition} Label the application cluster nodes before adding to backup and verify the license count [2024-01-19T00:23:44.147Z] 2024-01-19 00:23:43 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-82954-01-19-00h18m39s] [2024-01-19T00:23:44.147Z] 2024-01-19 00:23:44 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T00:23:44.147Z] 2024-01-19 00:23:44 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T00:23:44.708Z] 2024-01-19 00:23:44 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-82954-01-19-00h18m39s] [2024-01-19T00:23:44.708Z] 2024-01-19 00:23:44 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {LicensingCountWithNodeLabelledBeforeClusterAddition} Label the application cluster nodes before adding to backup and verify the license count [2024-01-19T00:23:44.708Z] 2024-01-19 00:23:44 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T00:23:44.708Z] 2024/01/19 00:23:44 DoRetryWithTimeout - Error: {dial tcp 10.240.128.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:23:49.959Z] 2024/01/19 00:23:49 DoRetryWithTimeout - Error: {dial tcp 10.240.128.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:23:55.205Z] 2024/01/19 00:23:54 DoRetryWithTimeout - Error: {dial tcp 10.240.128.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:24:00.450Z] 2024/01/19 00:23:59 DoRetryWithTimeout - Error: {dial tcp 10.240.128.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:24:04.617Z] 2024/01/19 00:24:04 DoRetryWithTimeout - Error: {dial tcp 10.240.128.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:24:09.867Z] 2024/01/19 00:24:09 DoRetryWithTimeout - Error: {dial tcp 10.240.128.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:24:15.115Z] 2024/01/19 00:24:14 DoRetryWithTimeout - Error: {dial tcp 10.240.128.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:24:20.360Z] 2024/01/19 00:24:19 DoRetryWithTimeout - Error: {dial tcp 10.240.128.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:24:24.525Z] 2024/01/19 00:24:24 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.128.6. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.6 10.240.128.6]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T00:24:24.525Z] {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T00:24:24.525Z] {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T00:24:24.525Z] {dial tcp 10.240.128.6:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T00:24:29.772Z] 2024-01-19 00:24:29 +0000:[WARNING] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.6. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.6 10.240.128.6]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T00:24:29.772Z] {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T00:24:29.772Z] {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T00:24:29.772Z] {dial tcp 10.240.128.6:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T00:24:29.772Z] 2024-01-19 00:24:29 +0000:[ERROR] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.6. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.6 10.240.128.6]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T00:24:29.772Z] {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T00:24:29.772Z] {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T00:24:29.772Z] {dial tcp 10.240.128.6:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T00:24:29.772Z] 2024-01-19 00:24:29 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T00:24:29.772Z] 2024-01-19 00:24:29 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] Switching context to [/tmp/source-config] [2024-01-19T00:24:29.772Z] 2024-01-19 00:24:29 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T00:24:29.772Z] 2024-01-19 00:24:29 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T00:24:29.772Z] 2024-01-19 00:24:29 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T00:24:30.028Z] 2024-01-19 00:24:29 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T00:24:30.028Z] 2024-01-19 00:24:29 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T00:24:30.028Z] 2024-01-19 00:24:29 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T00:24:30.589Z] 2024-01-19 00:24:30 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T00:24:30.845Z] 2024-01-19 00:24:30 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T00:24:31.406Z] 2024-01-19 00:24:31 +0000:[INFO] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T00:24:31.406Z] 2024/01/19 00:24:31 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:24:39.545Z] INFO[2024-01-19 00:24:38] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T00:24:39.545Z] INFO[2024-01-19 00:24:38] sched stats: tick interval: 31838 < 1.2s (100%) [2024-01-19T00:24:39.545Z] INFO[2024-01-19 00:24:38] sched stats: tick process durations: 31837 < 500ms (100%) [2024-01-19T00:24:41.436Z] 2024/01/19 00:24:41 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:24:53.610Z] 2024/01/19 00:24:51 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:25:01.702Z] 2024/01/19 00:25:01 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:25:11.639Z] 2024/01/19 00:25:11 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:25:21.583Z] 2024/01/19 00:25:21 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:25:33.754Z] 2024/01/19 00:25:31 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:25:41.836Z] 2024/01/19 00:25:41 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:25:51.824Z] 2024/01/19 00:25:51 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:26:01.767Z] 2024/01/19 00:26:01 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:26:11.705Z] 2024/01/19 00:26:11 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:26:23.877Z] 2024/01/19 00:26:21 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:26:31.965Z] 2024/01/19 00:26:31 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:26:41.911Z] 2024/01/19 00:26:41 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:26:51.853Z] 2024/01/19 00:26:51 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:27:04.023Z] 2024/01/19 00:27:01 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:27:12.105Z] 2024/01/19 00:27:11 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:27:22.066Z] 2024/01/19 00:27:21 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:27:32.034Z] 2024/01/19 00:27:31 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:27:42.026Z] 2024/01/19 00:27:41 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:27:51.978Z] 2024/01/19 00:27:51 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:28:04.152Z] 2024/01/19 00:28:01 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:28:12.239Z] 2024/01/19 00:28:11 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:28:22.187Z] 2024/01/19 00:28:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:28:32.146Z] 2024/01/19 00:28:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:28:42.134Z] 2024/01/19 00:28:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:28:54.315Z] 2024/01/19 00:28:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:29:02.423Z] 2024/01/19 00:29:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:29:12.365Z] 2024/01/19 00:29:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:29:22.311Z] 2024/01/19 00:29:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:29:32.275Z] 2024-01-19 00:29:32 +0000:[ERROR] [{LicensingCountWithNodeLabelledBeforeClusterAddition}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.275Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.276Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.276Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.276Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.276Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.276Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.276Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.276Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.276Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.276Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.276Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.276Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.276Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.276Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.276Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.276Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.276Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.276Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.276Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.276Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.276Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.276Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.276Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.276Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.276Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.277Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.277Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.277Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:29:32.277Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T00:29:32.277Z] • Failure [652.934 seconds] [2024-01-19T00:29:32.277Z] {LicensingCountWithNodeLabelledBeforeClusterAddition} [2024-01-19T00:29:32.277Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_license_test.go:134 [2024-01-19T00:29:32.277Z] Label the application cluster nodes before adding to backup and verify the license count [It] [2024-01-19T00:29:32.277Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_license_test.go:167 [2024-01-19T00:29:32.277Z] [2024-01-19T00:29:32.277Z] Unexpected error: [2024-01-19T00:29:32.277Z] <*errors.errorString | 0xc00117abb0>: { [2024-01-19T00:29:32.277Z] s: "error: rpc error: code = FailedPrecondition desc = cluster [destination-cluster] in use by backup [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-19-001118], cannot be deleted, Description: Deleting cluster destination-cluster", [2024-01-19T00:29:32.277Z] } [2024-01-19T00:29:32.277Z] error: rpc error: code = FailedPrecondition desc = cluster [destination-cluster] in use by backup [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-19-001118], cannot be deleted, Description: Deleting cluster destination-cluster [2024-01-19T00:29:32.277Z] occurred [2024-01-19T00:29:32.277Z] [2024-01-19T00:29:32.277Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-19T00:29:32.277Z] [2024-01-19T00:29:32.277Z] Full Stack Trace [2024-01-19T00:29:32.277Z] github.com/portworx/torpedo/pkg/aetosutil.(*Dashboard).VerifyFatal(0x7ed11b1?, {0x72ffd80?, 0xc00048cfb8}, {0x0?, 0x0}, {0xc001e637a0, 0x24}) [2024-01-19T00:29:32.277Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 +0x143 [2024-01-19T00:29:32.277Z] github.com/portworx/torpedo/tests/backup.glob..func12.2.1() [2024-01-19T00:29:32.277Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_license_test.go:178 +0xf3 [2024-01-19T00:29:32.277Z] github.com/portworx/torpedo/tests/backup.glob..func12.2() [2024-01-19T00:29:32.277Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_license_test.go:170 +0x1c9 [2024-01-19T00:29:32.277Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T00:29:32.277Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T00:29:32.277Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T00:29:32.277Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T00:29:32.277Z] created by testing.(*T).Run [2024-01-19T00:29:32.277Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T00:29:32.277Z] ------------------------------ [2024-01-19T00:29:32.277Z] STEP: schedule applications [2024-01-19T00:29:32.277Z] STEP: Removing any cluster if present [2024-01-19T00:29:32.277Z] STEP: destroy apps [2024-01-19T00:29:32.277Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T00:29:32.277Z] STEP: start destroying mysql-ibm app [2024-01-19T00:29:32.277Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T00:29:32.277Z] SS2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Log Dir: /testresults//NamespaceLabelledBackupSharedWithDifferentAccessMode.log [2024-01-19T00:29:32.277Z] INFO[2024-01-19 00:29:32] --------Test Start------ [2024-01-19T00:29:32.277Z] INFO[2024-01-19 00:29:32] #Test: NamespaceLabelledBackupSharedWithDifferentAccessMode [2024-01-19T00:29:32.277Z] INFO[2024-01-19 00:29:32] #Description: Take namespace labelled backup and share with users having different access mode [2024-01-19T00:29:32.277Z] INFO[2024-01-19 00:29:32] ------------------------ [2024-01-19T00:29:32.277Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.glob..func18.1:#47] - Deploy applications needed for backup [2024-01-19T00:29:32.277Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T00:29:32.278Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T00:29:32.278Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T00:29:32.534Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T00:29:32.534Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T00:29:32.534Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T00:29:32.535Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T00:29:32.535Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T00:29:32.535Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T00:29:32.535Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T00:29:32.535Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T00:29:32.535Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T00:29:32.535Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T00:29:32.791Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T00:29:32.791Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T00:29:32.791Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T00:29:32.792Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T00:29:32.792Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T00:29:32.792Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T00:29:32.792Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T00:29:32.792Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T00:29:32.793Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T00:29:32.793Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T00:29:33.048Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T00:29:33.048Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T00:29:33.048Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T00:29:33.049Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T00:29:33.049Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T00:29:33.049Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T00:29:33.050Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T00:29:33.050Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T00:29:33.050Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T00:29:33.050Z] 2024-01-19 00:29:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T00:29:33.306Z] 2024-01-19 00:29:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T00:29:33.306Z] 2024-01-19 00:29:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T00:29:33.306Z] 2024-01-19 00:29:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T00:29:33.307Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T00:29:33.307Z] 2024-01-19 00:29:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T00:29:33.308Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T00:29:33.308Z] 2024-01-19 00:29:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T00:29:33.308Z] 2024-01-19 00:29:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T00:29:33.308Z] 2024-01-19 00:29:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T00:29:33.308Z] 2024-01-19 00:29:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T00:29:33.564Z] 2024-01-19 00:29:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T00:29:33.564Z] 2024-01-19 00:29:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T00:29:33.564Z] 2024-01-19 00:29:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T00:29:33.565Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T00:29:33.565Z] 2024-01-19 00:29:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T00:29:33.565Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T00:29:33.565Z] 2024-01-19 00:29:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T00:29:33.565Z] 2024-01-19 00:29:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T00:29:33.565Z] 2024-01-19 00:29:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T00:29:33.565Z] 2024-01-19 00:29:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T00:29:33.824Z] 2024-01-19 00:29:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T00:29:33.824Z] 2024-01-19 00:29:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T00:29:34.080Z] 2024-01-19 00:29:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T00:29:34.081Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T00:29:34.081Z] 2024-01-19 00:29:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T00:29:34.082Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T00:29:34.082Z] 2024-01-19 00:29:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T00:29:34.082Z] 2024-01-19 00:29:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] List of all namespaces deployed are [mysql-ibm-pxb-0-85040-01-19-00h29m32s mysql-ibm-pxb-1-85040-01-19-00h29m32s mysql-ibm-pxb-2-85040-01-19-00h29m32s mysql-ibm-pxb-3-85040-01-19-00h29m32s mysql-ibm-pxb-4-85040-01-19-00h29m32s mysql-ibm-pxb-5-85040-01-19-00h29m32s] [2024-01-19T00:29:34.082Z] 2024-01-19 00:29:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.glob..func18.2.1:#65] - Validate applications [2024-01-19T00:29:34.082Z] 2024-01-19 00:29:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validate applications [2024-01-19T00:29:34.082Z] 2024-01-19 00:29:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating mysql-ibm app [2024-01-19T00:29:34.082Z] 2024-01-19 00:29:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating mysql-ibm app's volumes [2024-01-19T00:29:34.082Z] 2024-01-19 00:29:34 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:29:34.338Z] 2024/01/19 00:29:34 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T00:29:46.505Z] 2024/01/19 00:29:44 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T00:29:54.599Z] 2024-01-19 00:29:54 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-85040-01-19-00h29m32s [2024-01-19T00:29:54.599Z] 2024-01-19 00:29:54 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T00:29:54.599Z] 2024-01-19 00:29:54 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T00:29:54.599Z] 2024-01-19 00:29:54 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] wait for mysql-ibm app to start running [2024-01-19T00:29:54.600Z] 2024-01-19 00:29:54 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T00:29:54.600Z] 2024/01/19 00:29:54 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:29:54.600Z] pod name:mysql-7d766cd65c-68mlp namespace:mysql-ibm-pxb-0-85040-01-19-00h29m32s running:false ready:false node:10.240.64.27 [2024-01-19T00:29:54.600Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:30:06.772Z] 2024/01/19 00:30:04 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:30:06.772Z] pod name:mysql-7d766cd65c-68mlp namespace:mysql-ibm-pxb-0-85040-01-19-00h29m32s running:false ready:false node:10.240.64.27 [2024-01-19T00:30:06.772Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:30:16.721Z] 2024/01/19 00:30:14 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:30:16.721Z] pod name:mysql-7d766cd65c-68mlp namespace:mysql-ibm-pxb-0-85040-01-19-00h29m32s running:true ready:false node:10.240.64.27 [2024-01-19T00:30:16.721Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:30:24.802Z] 2024/01/19 00:30:24 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:30:24.803Z] pod name:mysql-7d766cd65c-68mlp namespace:mysql-ibm-pxb-0-85040-01-19-00h29m32s running:true ready:false node:10.240.64.27 [2024-01-19T00:30:24.803Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:30:36.971Z] 2024/01/19 00:30:34 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:30:36.971Z] pod name:mysql-7d766cd65c-68mlp namespace:mysql-ibm-pxb-0-85040-01-19-00h29m32s running:true ready:false node:10.240.64.27 [2024-01-19T00:30:36.971Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:30:46.231Z] 2024/01/19 00:30:44 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:30:46.231Z] pod name:mysql-7d766cd65c-68mlp namespace:mysql-ibm-pxb-0-85040-01-19-00h29m32s running:true ready:false node:10.240.64.27 [2024-01-19T00:30:46.231Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:30:56.172Z] 2024/01/19 00:30:55 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:30:56.172Z] pod name:mysql-7d766cd65c-68mlp namespace:mysql-ibm-pxb-0-85040-01-19-00h29m32s running:true ready:false node:10.240.64.27 [2024-01-19T00:30:56.172Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:31:06.107Z] 2024/01/19 00:31:05 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:31:06.107Z] pod name:mysql-7d766cd65c-68mlp namespace:mysql-ibm-pxb-0-85040-01-19-00h29m32s running:true ready:false node:10.240.64.27 [2024-01-19T00:31:06.107Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:31:16.047Z] 2024/01/19 00:31:15 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:31:16.047Z] pod name:mysql-7d766cd65c-68mlp namespace:mysql-ibm-pxb-0-85040-01-19-00h29m32s running:true ready:false node:10.240.64.27 [2024-01-19T00:31:16.047Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:31:26.065Z] 2024-01-19 00:31:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T00:31:26.065Z] 2024-01-19 00:31:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T00:31:26.065Z] 2024-01-19 00:31:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] validate if mysql-ibm app's volumes are setup [2024-01-19T00:31:26.065Z] 2024-01-19 00:31:25 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:31:26.065Z] 2024-01-19 00:31:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:31:26.065Z] 2024-01-19 00:31:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T00:31:26.065Z] 2024-01-19 00:31:25 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T00:31:26.065Z] 2024-01-19 00:31:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating mysql-ibm app [2024-01-19T00:31:26.065Z] 2024-01-19 00:31:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating mysql-ibm app's volumes [2024-01-19T00:31:26.066Z] 2024-01-19 00:31:25 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:31:26.066Z] 2024-01-19 00:31:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:31:26.066Z] 2024-01-19 00:31:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-1-85040-01-19-00h29m32s [2024-01-19T00:31:26.066Z] 2024-01-19 00:31:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T00:31:26.066Z] 2024-01-19 00:31:25 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T00:31:26.066Z] 2024-01-19 00:31:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] wait for mysql-ibm app to start running [2024-01-19T00:31:26.066Z] 2024-01-19 00:31:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T00:31:26.066Z] 2024-01-19 00:31:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T00:31:26.629Z] 2024-01-19 00:31:26 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T00:31:27.558Z] 2024-01-19 00:31:27 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] validate if mysql-ibm app's volumes are setup [2024-01-19T00:31:27.558Z] 2024-01-19 00:31:27 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:31:27.558Z] 2024-01-19 00:31:27 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:31:27.559Z] 2024-01-19 00:31:27 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T00:31:27.559Z] 2024-01-19 00:31:27 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T00:31:27.559Z] 2024-01-19 00:31:27 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating mysql-ibm app [2024-01-19T00:31:27.559Z] 2024-01-19 00:31:27 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating mysql-ibm app's volumes [2024-01-19T00:31:27.559Z] 2024-01-19 00:31:27 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:31:27.559Z] 2024-01-19 00:31:27 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:31:27.559Z] 2024-01-19 00:31:27 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-2-85040-01-19-00h29m32s [2024-01-19T00:31:27.559Z] 2024-01-19 00:31:27 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T00:31:27.820Z] 2024-01-19 00:31:27 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T00:31:27.820Z] 2024-01-19 00:31:27 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] wait for mysql-ibm app to start running [2024-01-19T00:31:27.820Z] 2024-01-19 00:31:27 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T00:31:28.085Z] 2024-01-19 00:31:27 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T00:31:28.646Z] 2024-01-19 00:31:28 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T00:31:29.588Z] 2024-01-19 00:31:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] validate if mysql-ibm app's volumes are setup [2024-01-19T00:31:29.588Z] 2024-01-19 00:31:29 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:31:29.588Z] 2024-01-19 00:31:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:31:29.588Z] 2024-01-19 00:31:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T00:31:29.588Z] 2024-01-19 00:31:29 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T00:31:29.588Z] 2024-01-19 00:31:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating mysql-ibm app [2024-01-19T00:31:29.588Z] 2024-01-19 00:31:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating mysql-ibm app's volumes [2024-01-19T00:31:29.588Z] 2024-01-19 00:31:29 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:31:29.589Z] 2024-01-19 00:31:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:31:29.589Z] 2024-01-19 00:31:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-3-85040-01-19-00h29m32s [2024-01-19T00:31:29.589Z] 2024-01-19 00:31:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T00:31:29.589Z] 2024-01-19 00:31:29 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T00:31:29.589Z] 2024-01-19 00:31:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] wait for mysql-ibm app to start running [2024-01-19T00:31:29.844Z] 2024-01-19 00:31:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T00:31:30.100Z] 2024-01-19 00:31:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T00:31:30.661Z] 2024-01-19 00:31:30 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T00:31:31.592Z] 2024-01-19 00:31:31 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] validate if mysql-ibm app's volumes are setup [2024-01-19T00:31:31.592Z] 2024-01-19 00:31:31 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:31:31.592Z] 2024-01-19 00:31:31 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:31:31.592Z] 2024-01-19 00:31:31 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T00:31:31.592Z] 2024-01-19 00:31:31 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T00:31:31.592Z] 2024-01-19 00:31:31 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating mysql-ibm app [2024-01-19T00:31:31.593Z] 2024-01-19 00:31:31 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating mysql-ibm app's volumes [2024-01-19T00:31:31.593Z] 2024-01-19 00:31:31 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:31:31.593Z] 2024-01-19 00:31:31 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:31:31.593Z] 2024-01-19 00:31:31 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-4-85040-01-19-00h29m32s [2024-01-19T00:31:31.593Z] 2024-01-19 00:31:31 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T00:31:31.593Z] 2024-01-19 00:31:31 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T00:31:31.593Z] 2024-01-19 00:31:31 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] wait for mysql-ibm app to start running [2024-01-19T00:31:31.593Z] 2024-01-19 00:31:31 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T00:31:32.154Z] 2024-01-19 00:31:31 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T00:31:32.715Z] 2024-01-19 00:31:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T00:31:33.280Z] 2024-01-19 00:31:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] validate if mysql-ibm app's volumes are setup [2024-01-19T00:31:33.537Z] 2024-01-19 00:31:33 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:31:33.537Z] 2024-01-19 00:31:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:31:33.537Z] 2024-01-19 00:31:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T00:31:33.537Z] 2024-01-19 00:31:33 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T00:31:33.537Z] 2024-01-19 00:31:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating mysql-ibm app [2024-01-19T00:31:33.537Z] 2024-01-19 00:31:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating mysql-ibm app's volumes [2024-01-19T00:31:33.537Z] 2024-01-19 00:31:33 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:31:33.537Z] 2024-01-19 00:31:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:31:33.537Z] 2024-01-19 00:31:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-5-85040-01-19-00h29m32s [2024-01-19T00:31:33.537Z] 2024-01-19 00:31:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T00:31:33.793Z] 2024-01-19 00:31:33 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T00:31:33.793Z] 2024-01-19 00:31:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] wait for mysql-ibm app to start running [2024-01-19T00:31:33.793Z] 2024-01-19 00:31:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T00:31:34.049Z] 2024-01-19 00:31:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T00:31:34.614Z] 2024-01-19 00:31:34 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T00:31:35.541Z] 2024-01-19 00:31:35 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] validate if mysql-ibm app's volumes are setup [2024-01-19T00:31:35.541Z] 2024-01-19 00:31:35 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:31:35.541Z] 2024-01-19 00:31:35 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:31:35.541Z] 2024-01-19 00:31:35 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T00:31:35.541Z] 2024-01-19 00:31:35 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T00:31:35.541Z] 2024-01-19 00:31:35 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Generating multiple labels [2024-01-19T00:31:35.542Z] 2024-01-19 00:31:35 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Generated labels are map[0-6aff95ad-3aab-4d8a-8a87-7200386facb5:6aff95ad-3aab-4d8a-8a87-7200386facb5 1-6aff95ad-3aab-4d8a-8a87-7200386facb5:6aff95ad-3aab-4d8a-8a87-7200386facb5 2-6aff95ad-3aab-4d8a-8a87-7200386facb5:6aff95ad-3aab-4d8a-8a87-7200386facb5] [2024-01-19T00:31:35.542Z] 2024-01-19 00:31:35 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Adding labels to only few namespaces [2024-01-19T00:31:35.542Z] 2024-01-19 00:31:35 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] List of labels applied namespaces are [mysql-ibm-pxb-0-85040-01-19-00h29m32s mysql-ibm-pxb-1-85040-01-19-00h29m32s mysql-ibm-pxb-2-85040-01-19-00h29m32s] [2024-01-19T00:31:35.542Z] 2024-01-19 00:31:35 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Creating 3 users [2024-01-19T00:31:35.542Z] 2024-01-19 00:31:35 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Creating 3 users [2024-01-19T00:31:38.111Z] 2024-01-19 00:31:37 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.AddUser:#728] - Response for user [testuser3-1705624295] creation - [2024-01-19T00:31:38.111Z] 2024-01-19 00:31:37 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.AddUser:#733] - User [testuser3-1705624295] created [2024-01-19T00:31:38.111Z] INFO[2024-01-19 00:31:37] Verifying : Description : Creating user - testuser3-1705624295 [2024-01-19T00:31:38.111Z] INFO[2024-01-19 00:31:37] Actual:true, Expected: true [2024-01-19T00:31:38.111Z] 2024-01-19 00:31:37 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.AddUser:#728] - Response for user [testuser2-1705624295] creation - [2024-01-19T00:31:38.111Z] 2024-01-19 00:31:37 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.AddUser:#733] - User [testuser2-1705624295] created [2024-01-19T00:31:38.111Z] INFO[2024-01-19 00:31:37] Verifying : Description : Creating user - testuser2-1705624295 [2024-01-19T00:31:38.111Z] INFO[2024-01-19 00:31:37] Actual:true, Expected: true [2024-01-19T00:31:38.111Z] 2024-01-19 00:31:37 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.AddUser:#728] - Response for user [testuser1-1705624295] creation - [2024-01-19T00:31:38.111Z] 2024-01-19 00:31:37 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.AddUser:#733] - User [testuser1-1705624295] created [2024-01-19T00:31:38.111Z] INFO[2024-01-19 00:31:37] Verifying : Description : Creating user - testuser1-1705624295 [2024-01-19T00:31:38.111Z] INFO[2024-01-19 00:31:37] Actual:true, Expected: true [2024-01-19T00:31:38.111Z] 2024-01-19 00:31:37 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.glob..func18.2.4:#86] - Created 3 users and users list is [testuser3-1705624295 testuser2-1705624295 testuser1-1705624295] [2024-01-19T00:31:38.111Z] 2024-01-19 00:31:37 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Creating backup location and cloud setting [2024-01-19T00:31:39.477Z] 2024-01-19 00:31:39 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [cred-aws-1705624299] for org [default] with [aws] as provider [2024-01-19T00:31:39.477Z] 2024-01-19 00:31:39 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T00:31:39.477Z] 2024-01-19 00:31:39 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cred-aws-1705624299] is present [2024-01-19T00:31:43.644Z] 2024-01-19 00:31:43 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Register source and destination cluster for backup [2024-01-19T00:31:43.902Z] INFO[2024-01-19 00:31:43] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T00:31:43.902Z] INFO[2024-01-19 00:31:43] Actual:true, Expected: true [2024-01-19T00:31:43.902Z] 2024-01-19 00:31:43 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T00:31:43.902Z] 2024-01-19 00:31:43 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T00:31:43.902Z] 2024-01-19 00:31:43 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T00:31:43.902Z] 2024-01-19 00:31:43 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T00:31:43.902Z] 2024-01-19 00:31:43 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Create cluster [source-cluster] in org [default] [2024-01-19T00:31:43.902Z] 2024-01-19 00:31:43 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T00:31:43.902Z] 2024-01-19 00:31:43 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T00:31:43.902Z] 2024-01-19 00:31:43 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:31:43.902Z] 2024-01-19 00:31:43 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T00:31:43.902Z] 2024-01-19 00:31:43 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T00:31:43.903Z] 2024-01-19 00:31:43 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-xmque] for IBM [2024-01-19T00:31:43.903Z] 2024-01-19 00:31:43 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-xmque] for org [default] with [ibm] as provider [2024-01-19T00:31:43.903Z] 2024-01-19 00:31:43 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:31:43.903Z] 2024-01-19 00:31:43 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:31:44.159Z] 2024-01-19 00:31:43 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-xmque] is present [2024-01-19T00:31:44.159Z] 2024-01-19 00:31:43 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Create cluster [source-cluster] in org [default] [2024-01-19T00:31:44.721Z] 2024-01-19 00:31:44 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-gcbql] for IBM [2024-01-19T00:31:44.721Z] 2024-01-19 00:31:44 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-gcbql] for org [default] with [ibm] as provider [2024-01-19T00:31:44.721Z] 2024-01-19 00:31:44 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:31:44.721Z] 2024-01-19 00:31:44 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:31:44.721Z] 2024-01-19 00:31:44 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-gcbql] is present [2024-01-19T00:31:44.721Z] 2024-01-19 00:31:44 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:31:44.977Z] INFO[2024-01-19 00:31:44] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T00:31:44.977Z] INFO[2024-01-19 00:31:44] Actual:Online, Expected: Online [2024-01-19T00:31:44.977Z] INFO[2024-01-19 00:31:44] Verifying : Description : Verifying if [destination-cluster] cluster is online [2024-01-19T00:31:44.977Z] INFO[2024-01-19 00:31:44] Actual:Online, Expected: Online [2024-01-19T00:31:44.977Z] 2024-01-19 00:31:44 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Taking namespace labelled backup for each user [2024-01-19T00:31:45.233Z] 2024-01-19 00:31:44 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T00:31:45.233Z] 2024-01-19 00:31:44 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T00:31:45.233Z] 2024-01-19 00:31:44 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T00:31:45.233Z] 2024-01-19 00:31:44 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Volume snapshot classes found in the cluster - [2024-01-19T00:31:45.233Z] 2024-01-19 00:31:44 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] ibmc-vpcblock-snapshot [2024-01-19T00:31:45.233Z] 2024-01-19 00:31:44 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T00:31:45.233Z] 2024-01-19 00:31:44 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Volume snapshot classes found in the cluster - [2024-01-19T00:31:45.233Z] 2024-01-19 00:31:44 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] ibmc-vpcblock-snapshot [2024-01-19T00:31:45.233Z] 2024-01-19 00:31:44 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T00:31:45.233Z] 2024-01-19 00:31:44 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Volume snapshot classes found in the cluster - [2024-01-19T00:31:45.233Z] 2024-01-19 00:31:44 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] ibmc-vpcblock-snapshot [2024-01-19T00:31:45.233Z] 2024-01-19 00:31:44 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T00:31:45.795Z] 2024/01/19 00:31:45 DoRetryWithTimeout - Error: {backup status for [tp-backup-testuser1-1705624295] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T00:31:46.356Z] 2024/01/19 00:31:46 DoRetryWithTimeout - Error: {backup status for [tp-backup-testuser2-1705624295] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T00:31:46.356Z] 2024/01/19 00:31:46 DoRetryWithTimeout - Error: {backup status for [tp-backup-testuser3-1705624295] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T00:32:18.382Z] 2024/01/19 00:32:15 DoRetryWithTimeout - Error: {backup status for [tp-backup-testuser1-1705624295] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:32:18.382Z] 2024/01/19 00:32:16 DoRetryWithTimeout - Error: {backup status for [tp-backup-testuser2-1705624295] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:32:18.382Z] 2024/01/19 00:32:16 DoRetryWithTimeout - Error: {backup status for [tp-backup-testuser3-1705624295] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:32:50.413Z] 2024/01/19 00:32:45 DoRetryWithTimeout - Error: {backup status for [tp-backup-testuser1-1705624295] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:32:50.413Z] 2024/01/19 00:32:46 DoRetryWithTimeout - Error: {backup status for [tp-backup-testuser2-1705624295] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:32:50.414Z] 2024/01/19 00:32:46 DoRetryWithTimeout - Error: {backup status for [tp-backup-testuser3-1705624295] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:33:16.935Z] 2024/01/19 00:33:15 DoRetryWithTimeout - Error: {backup status for [tp-backup-testuser1-1705624295] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:33:16.935Z] 2024/01/19 00:33:16 DoRetryWithTimeout - Error: {backup status for [tp-backup-testuser2-1705624295] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:33:16.935Z] 2024/01/19 00:33:16 DoRetryWithTimeout - Error: {backup status for [tp-backup-testuser3-1705624295] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:33:48.996Z] 2024/01/19 00:33:45 DoRetryWithTimeout - Error: {backup status for [tp-backup-testuser1-1705624295] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:33:48.996Z] 2024/01/19 00:33:46 DoRetryWithTimeout - Error: {backup status for [tp-backup-testuser2-1705624295] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:33:48.997Z] 2024/01/19 00:33:46 DoRetryWithTimeout - Error: {backup status for [tp-backup-testuser3-1705624295] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:34:21.059Z] 2024/01/19 00:34:15 DoRetryWithTimeout - Error: {backup status for [tp-backup-testuser1-1705624295] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:34:21.059Z] 2024-01-19 00:34:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.CreateBackupWithNamespaceLabel:#3030] - Successfully created backup [tp-backup-testuser2-1705624295] with namespace label [0-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5,1-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5,2-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5] [2024-01-19T00:34:21.059Z] 2024-01-19 00:34:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating backup [tp-backup-testuser2-1705624295] in org [default] [2024-01-19T00:34:21.059Z] 2024-01-19 00:34:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-testuser2-1705624295] [2024-01-19T00:34:21.059Z] 2024/01/19 00:34:16 DoRetryWithTimeout - Error: {backup status for [tp-backup-testuser3-1705624295] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:34:21.060Z] 2024-01-19 00:34:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-85040-01-19-00h29m32s] in backup [tp-backup-testuser2-1705624295] [2024-01-19T00:34:21.060Z] 2024-01-19 00:34:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-85040-01-19-00h29m32s] in backup [tp-backup-testuser2-1705624295] [2024-01-19T00:34:21.060Z] 2024-01-19 00:34:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-85040-01-19-00h29m32s] [2024-01-19T00:34:21.060Z] 2024-01-19 00:34:16 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:34:21.060Z] 2024-01-19 00:34:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:34:21.060Z] 2024-01-19 00:34:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-85040-01-19-00h29m32s] are [[mysql-data]] [2024-01-19T00:34:21.060Z] 2024-01-19 00:34:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T00:34:21.060Z] 2024-01-19 00:34:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-85040-01-19-00h29m32s] in backup [tp-backup-testuser2-1705624295] [2024-01-19T00:34:21.060Z] 2024-01-19 00:34:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-85040-01-19-00h29m32s] in backup [tp-backup-testuser2-1705624295] [2024-01-19T00:34:21.060Z] 2024-01-19 00:34:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-85040-01-19-00h29m32s] [2024-01-19T00:34:21.060Z] 2024-01-19 00:34:16 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:34:21.060Z] 2024-01-19 00:34:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:34:21.060Z] 2024-01-19 00:34:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-85040-01-19-00h29m32s] are [[mysql-data]] [2024-01-19T00:34:21.061Z] 2024-01-19 00:34:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T00:34:21.061Z] 2024-01-19 00:34:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-85040-01-19-00h29m32s] in backup [tp-backup-testuser2-1705624295] [2024-01-19T00:34:21.061Z] 2024-01-19 00:34:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-85040-01-19-00h29m32s] in backup [tp-backup-testuser2-1705624295] [2024-01-19T00:34:21.061Z] 2024-01-19 00:34:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-2-85040-01-19-00h29m32s] [2024-01-19T00:34:21.061Z] 2024-01-19 00:34:16 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:34:21.061Z] 2024-01-19 00:34:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:34:21.061Z] 2024-01-19 00:34:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-2-85040-01-19-00h29m32s] are [[mysql-data]] [2024-01-19T00:34:21.061Z] 2024-01-19 00:34:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T00:34:21.061Z] 2024-01-19 00:34:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating CR cleanup [2024-01-19T00:34:21.061Z] 2024-01-19 00:34:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T00:34:21.061Z] 2024-01-19 00:34:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-testuser1-1705624295-06073c5 tp-backup-testuser3-1705624295-cd61102]] [2024-01-19T00:34:21.061Z] INFO[2024-01-19 00:34:16] Verifying : Description : Creation and Validation of namespace labelled backup [tp-backup-testuser2-1705624295] of namespaces (scheduled contexts) [[mysql-ibm-pxb-0-85040-01-19-00h29m32s mysql-ibm-pxb-1-85040-01-19-00h29m32s mysql-ibm-pxb-2-85040-01-19-00h29m32s]] [2024-01-19T00:34:21.061Z] INFO[2024-01-19 00:34:16] Actual:true, Expected: true [2024-01-19T00:34:39.132Z] INFO[2024-01-19 00:34:38] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T00:34:39.132Z] INFO[2024-01-19 00:34:38] sched stats: tick interval: 32438 < 1.2s (100%) [2024-01-19T00:34:39.132Z] INFO[2024-01-19 00:34:38] sched stats: tick process durations: 32437 < 500ms (100%) [2024-01-19T00:34:47.242Z] 2024/01/19 00:34:45 DoRetryWithTimeout - Error: {backup status for [tp-backup-testuser1-1705624295] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:34:47.242Z] 2024/01/19 00:34:46 DoRetryWithTimeout - Error: {backup status for [tp-backup-testuser3-1705624295] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:35:19.285Z] 2024-01-19 00:35:15 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.CreateBackupWithNamespaceLabel:#3030] - Successfully created backup [tp-backup-testuser1-1705624295] with namespace label [0-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5,1-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5,2-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5] [2024-01-19T00:35:19.285Z] 2024-01-19 00:35:15 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating backup [tp-backup-testuser1-1705624295] in org [default] [2024-01-19T00:35:19.285Z] 2024-01-19 00:35:15 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-testuser1-1705624295] [2024-01-19T00:35:19.285Z] 2024-01-19 00:35:15 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-85040-01-19-00h29m32s] in backup [tp-backup-testuser1-1705624295] [2024-01-19T00:35:19.285Z] 2024-01-19 00:35:15 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-85040-01-19-00h29m32s] in backup [tp-backup-testuser1-1705624295] [2024-01-19T00:35:19.285Z] 2024-01-19 00:35:15 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-85040-01-19-00h29m32s] [2024-01-19T00:35:19.285Z] 2024-01-19 00:35:15 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:35:19.285Z] 2024-01-19 00:35:15 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:35:19.285Z] 2024-01-19 00:35:15 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-85040-01-19-00h29m32s] are [[mysql-data]] [2024-01-19T00:35:19.285Z] 2024-01-19 00:35:15 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T00:35:19.285Z] 2024-01-19 00:35:15 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-85040-01-19-00h29m32s] in backup [tp-backup-testuser1-1705624295] [2024-01-19T00:35:19.285Z] 2024-01-19 00:35:15 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-85040-01-19-00h29m32s] in backup [tp-backup-testuser1-1705624295] [2024-01-19T00:35:19.285Z] 2024-01-19 00:35:15 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-85040-01-19-00h29m32s] [2024-01-19T00:35:19.285Z] 2024-01-19 00:35:16 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:35:19.285Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:35:19.286Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-85040-01-19-00h29m32s] are [[mysql-data]] [2024-01-19T00:35:19.286Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T00:35:19.286Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-85040-01-19-00h29m32s] in backup [tp-backup-testuser1-1705624295] [2024-01-19T00:35:19.286Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-85040-01-19-00h29m32s] in backup [tp-backup-testuser1-1705624295] [2024-01-19T00:35:19.286Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-2-85040-01-19-00h29m32s] [2024-01-19T00:35:19.286Z] 2024-01-19 00:35:16 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:35:19.286Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:35:19.286Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-2-85040-01-19-00h29m32s] are [[mysql-data]] [2024-01-19T00:35:19.286Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T00:35:19.286Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating CR cleanup [2024-01-19T00:35:19.286Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T00:35:19.286Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133]] [2024-01-19T00:35:19.286Z] INFO[2024-01-19 00:35:16] Verifying : Description : Creation and Validation of namespace labelled backup [tp-backup-testuser1-1705624295] of namespaces (scheduled contexts) [[mysql-ibm-pxb-0-85040-01-19-00h29m32s mysql-ibm-pxb-1-85040-01-19-00h29m32s mysql-ibm-pxb-2-85040-01-19-00h29m32s]] [2024-01-19T00:35:19.286Z] INFO[2024-01-19 00:35:16] Actual:true, Expected: true [2024-01-19T00:35:19.286Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.CreateBackupWithNamespaceLabel:#3030] - Successfully created backup [tp-backup-testuser3-1705624295] with namespace label [1-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5,2-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5,0-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5] [2024-01-19T00:35:19.286Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating backup [tp-backup-testuser3-1705624295] in org [default] [2024-01-19T00:35:19.286Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-testuser3-1705624295] [2024-01-19T00:35:19.286Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-85040-01-19-00h29m32s] in backup [tp-backup-testuser3-1705624295] [2024-01-19T00:35:19.287Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-85040-01-19-00h29m32s] in backup [tp-backup-testuser3-1705624295] [2024-01-19T00:35:19.287Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-85040-01-19-00h29m32s] [2024-01-19T00:35:19.287Z] 2024-01-19 00:35:16 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:35:19.287Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:35:19.287Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-85040-01-19-00h29m32s] are [[mysql-data]] [2024-01-19T00:35:19.287Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T00:35:19.287Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-85040-01-19-00h29m32s] in backup [tp-backup-testuser3-1705624295] [2024-01-19T00:35:19.287Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-85040-01-19-00h29m32s] in backup [tp-backup-testuser3-1705624295] [2024-01-19T00:35:19.287Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-85040-01-19-00h29m32s] [2024-01-19T00:35:19.287Z] 2024-01-19 00:35:16 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:35:19.287Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:35:19.287Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-85040-01-19-00h29m32s] are [[mysql-data]] [2024-01-19T00:35:19.287Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T00:35:19.287Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-85040-01-19-00h29m32s] in backup [tp-backup-testuser3-1705624295] [2024-01-19T00:35:19.287Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-85040-01-19-00h29m32s] in backup [tp-backup-testuser3-1705624295] [2024-01-19T00:35:19.287Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-2-85040-01-19-00h29m32s] [2024-01-19T00:35:19.287Z] 2024-01-19 00:35:16 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:35:19.287Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:35:19.287Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-2-85040-01-19-00h29m32s] are [[mysql-data]] [2024-01-19T00:35:19.288Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T00:35:19.288Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating CR cleanup [2024-01-19T00:35:19.288Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T00:35:19.288Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133]] [2024-01-19T00:35:19.288Z] INFO[2024-01-19 00:35:16] Verifying : Description : Creation and Validation of namespace labelled backup [tp-backup-testuser3-1705624295] of namespaces (scheduled contexts) [[mysql-ibm-pxb-0-85040-01-19-00h29m32s mysql-ibm-pxb-1-85040-01-19-00h29m32s mysql-ibm-pxb-2-85040-01-19-00h29m32s]] [2024-01-19T00:35:19.288Z] INFO[2024-01-19 00:35:16] Actual:true, Expected: true [2024-01-19T00:35:19.288Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] List of namespace labelled backups- [tp-backup-testuser3-1705624295 tp-backup-testuser2-1705624295 tp-backup-testuser1-1705624295] [2024-01-19T00:35:19.288Z] 2024-01-19 00:35:16 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Verifying that correct namespaces are backed up for the namespace labelled backup taken and correct labels are applied to the backup [2024-01-19T00:35:19.288Z] 2024-01-19 00:35:18 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup tp-backup-testuser3-1705624295 [2024-01-19T00:35:19.288Z] 2024-01-19 00:35:18 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [mysql-ibm-pxb-0-85040-01-19-00h29m32s mysql-ibm-pxb-1-85040-01-19-00h29m32s mysql-ibm-pxb-2-85040-01-19-00h29m32s] [2024-01-19T00:35:19.288Z] 2024-01-19 00:35:18 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are 1-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5,2-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5,0-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5 [2024-01-19T00:35:19.288Z] INFO[2024-01-19 00:35:18] Verifying : Description : Verifying if only the labeled namespaces: [[mysql-ibm-pxb-0-85040-01-19-00h29m32s mysql-ibm-pxb-1-85040-01-19-00h29m32s mysql-ibm-pxb-2-85040-01-19-00h29m32s]] are backed up and correct labels [0-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5,1-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5,2-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5] are applied to backup [tp-backup-testuser3-1705624295] [2024-01-19T00:35:19.288Z] INFO[2024-01-19 00:35:18] Actual:true, Expected: true [2024-01-19T00:35:19.288Z] 2024-01-19 00:35:18 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup tp-backup-testuser2-1705624295 [2024-01-19T00:35:19.288Z] 2024-01-19 00:35:18 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [mysql-ibm-pxb-0-85040-01-19-00h29m32s mysql-ibm-pxb-1-85040-01-19-00h29m32s mysql-ibm-pxb-2-85040-01-19-00h29m32s] [2024-01-19T00:35:19.288Z] 2024-01-19 00:35:18 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are 0-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5,1-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5,2-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5 [2024-01-19T00:35:19.289Z] INFO[2024-01-19 00:35:18] Verifying : Description : Verifying if only the labeled namespaces: [[mysql-ibm-pxb-0-85040-01-19-00h29m32s mysql-ibm-pxb-1-85040-01-19-00h29m32s mysql-ibm-pxb-2-85040-01-19-00h29m32s]] are backed up and correct labels [0-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5,1-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5,2-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5] are applied to backup [tp-backup-testuser2-1705624295] [2024-01-19T00:35:19.289Z] INFO[2024-01-19 00:35:18] Actual:true, Expected: true [2024-01-19T00:35:19.289Z] 2024-01-19 00:35:18 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup tp-backup-testuser1-1705624295 [2024-01-19T00:35:19.289Z] 2024-01-19 00:35:18 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [mysql-ibm-pxb-0-85040-01-19-00h29m32s mysql-ibm-pxb-1-85040-01-19-00h29m32s mysql-ibm-pxb-2-85040-01-19-00h29m32s] [2024-01-19T00:35:19.289Z] 2024-01-19 00:35:18 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are 0-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5,1-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5,2-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5 [2024-01-19T00:35:19.289Z] INFO[2024-01-19 00:35:18] Verifying : Description : Verifying if only the labeled namespaces: [[mysql-ibm-pxb-0-85040-01-19-00h29m32s mysql-ibm-pxb-1-85040-01-19-00h29m32s mysql-ibm-pxb-2-85040-01-19-00h29m32s]] are backed up and correct labels [0-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5,1-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5,2-6aff95ad-3aab-4d8a-8a87-7200386facb5=6aff95ad-3aab-4d8a-8a87-7200386facb5] are applied to backup [tp-backup-testuser1-1705624295] [2024-01-19T00:35:19.289Z] INFO[2024-01-19 00:35:18] Actual:true, Expected: true [2024-01-19T00:35:19.289Z] 2024-01-19 00:35:18 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Share backup with users with different access level [2024-01-19T00:35:19.853Z] 2024-01-19 00:35:19 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Sharing backups with users with different access level [2024-01-19T00:35:19.853Z] 2024-01-19 00:35:19 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ShareBackup:#715] - Backup UID for tp-backup-testuser3-1705624295 - cd611020-472c-4f1a-98e1-5392a3c2df0c [2024-01-19T00:35:20.781Z] 2024-01-19 00:35:20 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser3-1705624295 - 9ed58ac1-4544-44f9-aae4-abe5ca573145 [2024-01-19T00:35:21.038Z] 2024-01-19 00:35:20 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ShareBackup:#715] - Backup UID for tp-backup-testuser2-1705624295 - 083ebee4-7609-46a7-a4bd-45e5ec5ff693 [2024-01-19T00:35:21.966Z] 2024-01-19 00:35:21 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2-1705624295 - e93e77ff-3a3d-4417-83b2-30f0acdf3ed5 [2024-01-19T00:35:22.227Z] 2024-01-19 00:35:22 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ShareBackup:#715] - Backup UID for tp-backup-testuser1-1705624295 - 06073c58-a402-4817-9bd6-a21e9f022aeb [2024-01-19T00:35:23.156Z] 2024-01-19 00:35:22 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705624295 - 83ca30da-aa80-45fc-a95c-d3eebd44cf4e [2024-01-19T00:35:23.412Z] 2024-01-19 00:35:23 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validate if the users with different access level can restore/delete backup as per their access mode [2024-01-19T00:35:24.777Z] INFO[2024-01-19 00:35:24] Verifying : Description : Fetching px-central-admin ctx [2024-01-19T00:35:24.777Z] INFO[2024-01-19 00:35:24] Actual:true, Expected: true [2024-01-19T00:35:25.339Z] INFO[2024-01-19 00:35:25] Verifying : Description : Fetching testuser3-1705624295 user ctx [2024-01-19T00:35:25.339Z] INFO[2024-01-19 00:35:25] Actual:true, Expected: true [2024-01-19T00:35:25.339Z] 2024-01-19 00:35:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Registering Source and Destination clusters from user context [2024-01-19T00:35:25.339Z] INFO[2024-01-19 00:35:25] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T00:35:25.339Z] INFO[2024-01-19 00:35:25] Actual:true, Expected: true [2024-01-19T00:35:25.339Z] 2024-01-19 00:35:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T00:35:25.339Z] 2024-01-19 00:35:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T00:35:25.339Z] 2024-01-19 00:35:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T00:35:25.339Z] 2024-01-19 00:35:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T00:35:25.339Z] 2024-01-19 00:35:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Create cluster [source-cluster] in org [default] [2024-01-19T00:35:25.339Z] 2024-01-19 00:35:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T00:35:25.340Z] 2024-01-19 00:35:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T00:35:25.340Z] 2024-01-19 00:35:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:35:25.340Z] 2024-01-19 00:35:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T00:35:25.340Z] 2024-01-19 00:35:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T00:35:25.340Z] 2024-01-19 00:35:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-avylz] for IBM [2024-01-19T00:35:25.340Z] 2024-01-19 00:35:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-avylz] for org [default] with [ibm] as provider [2024-01-19T00:35:25.340Z] 2024-01-19 00:35:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:35:25.340Z] 2024-01-19 00:35:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:35:25.340Z] 2024-01-19 00:35:25 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-avylz] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName3 LastName3 with roles: [px-backup-app.user]] [2024-01-19T00:35:25.340Z] 2024-01-19 00:35:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName3 LastName3 with roles: [px-backup-app.user] [2024-01-19T00:35:26.705Z] 2024-01-19 00:35:26 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-avylz from admin context and sharing with all the users [2024-01-19T00:35:26.706Z] 2024-01-19 00:35:26 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-avylz] for org [default] with [ibm] as provider [2024-01-19T00:35:26.706Z] 2024-01-19 00:35:26 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:35:26.706Z] 2024-01-19 00:35:26 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:35:26.961Z] 2024-01-19 00:35:26 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-avylz] is present [2024-01-19T00:35:26.961Z] 2024-01-19 00:35:26 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Create cluster [source-cluster] in org [default] [2024-01-19T00:35:27.888Z] 2024-01-19 00:35:27 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-qzoph] for IBM [2024-01-19T00:35:27.889Z] 2024-01-19 00:35:27 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-qzoph] for org [default] with [ibm] as provider [2024-01-19T00:35:27.889Z] 2024-01-19 00:35:27 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:35:27.889Z] 2024-01-19 00:35:27 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:35:27.889Z] 2024-01-19 00:35:27 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-qzoph] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName3 LastName3 with roles: [px-backup-app.user]] [2024-01-19T00:35:27.889Z] 2024-01-19 00:35:27 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName3 LastName3 with roles: [px-backup-app.user] [2024-01-19T00:35:28.145Z] 2024-01-19 00:35:28 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-qzoph from admin context and sharing with all the users [2024-01-19T00:35:28.145Z] 2024-01-19 00:35:28 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-qzoph] for org [default] with [ibm] as provider [2024-01-19T00:35:28.145Z] 2024-01-19 00:35:28 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:35:28.145Z] 2024-01-19 00:35:28 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:35:28.401Z] 2024-01-19 00:35:28 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-qzoph] is present [2024-01-19T00:35:28.401Z] 2024-01-19 00:35:28 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:35:28.965Z] INFO[2024-01-19 00:35:28] Verifying : Description : Creating source and destination cluster [2024-01-19T00:35:28.965Z] INFO[2024-01-19 00:35:28] Actual:true, Expected: true [2024-01-19T00:35:28.965Z] 2024-01-19 00:35:28 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating if user [testuser3-1705624295] with access [ViewOnlyAccess] can restore and delete backup tp-backup-testuser3-1705624295 or not [2024-01-19T00:35:28.965Z] 2024-01-19 00:35:28 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateSharedBackupWithUsers:#1405] - The expected error returned is rpc error: code = NotFound desc = failed to retrieve backup location [autogenerated-backup-location-1705624299]: object not found [2024-01-19T00:35:28.965Z] INFO[2024-01-19 00:35:28] Verifying : Description : Verifying backup restore is not possible [2024-01-19T00:35:28.965Z] INFO[2024-01-19 00:35:28] Actual:true, Expected: true [2024-01-19T00:35:29.221Z] INFO[2024-01-19 00:35:28] Verifying : Description : Getting backup UID for- tp-backup-testuser3-1705624295 [2024-01-19T00:35:29.221Z] INFO[2024-01-19 00:35:28] Actual:true, Expected: true [2024-01-19T00:35:29.221Z] 2024-01-19 00:35:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateSharedBackupWithUsers:#1412] - The expected error returned is rpc error: code = Unknown desc = user [9ed58ac1-4544-44f9-aae4-abe5ca573145] doesn't have permission to delete backup [tp-backup-testuser3-1705624295] [2024-01-19T00:35:29.221Z] INFO[2024-01-19 00:35:29] Verifying : Description : Verifying backup deletion is not possible [2024-01-19T00:35:29.221Z] INFO[2024-01-19 00:35:29] Actual:true, Expected: true [2024-01-19T00:35:29.221Z] 2024-01-19 00:35:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.glob..func18.2.10:#162] - Adding the restores testuser2-1705624295-tp-restore-1705624529 which will be created while validating RestoreAccess access [2024-01-19T00:35:29.477Z] INFO[2024-01-19 00:35:29] Verifying : Description : Fetching px-central-admin ctx [2024-01-19T00:35:29.477Z] INFO[2024-01-19 00:35:29] Actual:true, Expected: true [2024-01-19T00:35:30.039Z] INFO[2024-01-19 00:35:29] Verifying : Description : Fetching testuser2-1705624295 user ctx [2024-01-19T00:35:30.039Z] INFO[2024-01-19 00:35:29] Actual:true, Expected: true [2024-01-19T00:35:30.039Z] 2024-01-19 00:35:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Registering Source and Destination clusters from user context [2024-01-19T00:35:30.039Z] INFO[2024-01-19 00:35:29] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T00:35:30.039Z] INFO[2024-01-19 00:35:29] Actual:true, Expected: true [2024-01-19T00:35:30.039Z] 2024-01-19 00:35:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T00:35:30.039Z] 2024-01-19 00:35:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T00:35:30.039Z] 2024-01-19 00:35:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T00:35:30.039Z] 2024-01-19 00:35:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T00:35:30.039Z] 2024-01-19 00:35:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Create cluster [source-cluster] in org [default] [2024-01-19T00:35:30.039Z] 2024-01-19 00:35:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T00:35:30.039Z] 2024-01-19 00:35:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T00:35:30.039Z] 2024-01-19 00:35:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:35:30.040Z] 2024-01-19 00:35:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T00:35:30.040Z] 2024-01-19 00:35:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T00:35:30.040Z] 2024-01-19 00:35:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-cegyx] for IBM [2024-01-19T00:35:30.040Z] 2024-01-19 00:35:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-cegyx] for org [default] with [ibm] as provider [2024-01-19T00:35:30.040Z] 2024-01-19 00:35:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:35:30.040Z] 2024-01-19 00:35:29 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:35:30.040Z] 2024-01-19 00:35:30 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-cegyx] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName2 LastName2 with roles: [px-backup-app.user]] [2024-01-19T00:35:30.040Z] 2024-01-19 00:35:30 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName2 LastName2 with roles: [px-backup-app.user] [2024-01-19T00:35:31.426Z] 2024-01-19 00:35:31 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-cegyx from admin context and sharing with all the users [2024-01-19T00:35:31.427Z] 2024-01-19 00:35:31 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-cegyx] for org [default] with [ibm] as provider [2024-01-19T00:35:31.427Z] 2024-01-19 00:35:31 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:35:31.427Z] 2024-01-19 00:35:31 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:35:31.683Z] 2024-01-19 00:35:31 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-cegyx] is present [2024-01-19T00:35:31.683Z] 2024-01-19 00:35:31 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Create cluster [source-cluster] in org [default] [2024-01-19T00:35:32.245Z] 2024-01-19 00:35:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-jgdyr] for IBM [2024-01-19T00:35:32.246Z] 2024-01-19 00:35:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-jgdyr] for org [default] with [ibm] as provider [2024-01-19T00:35:32.246Z] 2024-01-19 00:35:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:35:32.246Z] 2024-01-19 00:35:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:35:32.246Z] 2024-01-19 00:35:32 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-jgdyr] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName2 LastName2 with roles: [px-backup-app.user]] [2024-01-19T00:35:32.246Z] 2024-01-19 00:35:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName2 LastName2 with roles: [px-backup-app.user] [2024-01-19T00:35:32.811Z] 2024-01-19 00:35:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-jgdyr from admin context and sharing with all the users [2024-01-19T00:35:32.812Z] 2024-01-19 00:35:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-jgdyr] for org [default] with [ibm] as provider [2024-01-19T00:35:32.812Z] 2024-01-19 00:35:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:35:32.812Z] 2024-01-19 00:35:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:35:33.067Z] 2024-01-19 00:35:32 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-jgdyr] is present [2024-01-19T00:35:33.323Z] 2024-01-19 00:35:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:35:33.886Z] INFO[2024-01-19 00:35:33] Verifying : Description : Creating source and destination cluster [2024-01-19T00:35:33.886Z] INFO[2024-01-19 00:35:33] Actual:true, Expected: true [2024-01-19T00:35:33.886Z] 2024-01-19 00:35:33 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating if user [testuser2-1705624295] with access [RestoreAccess] can restore and delete backup tp-backup-testuser2-1705624295 or not [2024-01-19T00:35:34.142Z] 2024/01/19 00:35:34 DoRetryWithTimeout - Error: {restore status for [testuser2-1705624295-tp-restore-1705624529] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T00:36:06.161Z] 2024/01/19 00:36:04 DoRetryWithTimeout - Error: {restore status for [testuser2-1705624295-tp-restore-1705624529] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:36:38.192Z] 2024-01-19 00:36:34 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.CreateRestore:#936] - Restore [testuser2-1705624295-tp-restore-1705624529] created successfully [2024-01-19T00:36:38.192Z] INFO[2024-01-19 00:36:34] Verifying : Description : Verifying that restore is possible [2024-01-19T00:36:38.192Z] INFO[2024-01-19 00:36:34] Actual:true, Expected: true [2024-01-19T00:36:38.192Z] INFO[2024-01-19 00:36:34] Verifying : Description : Getting backup UID for- tp-backup-testuser2-1705624295 [2024-01-19T00:36:38.193Z] INFO[2024-01-19 00:36:34] Actual:true, Expected: true [2024-01-19T00:36:38.193Z] 2024-01-19 00:36:34 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.ValidateSharedBackupWithUsers:#1424] - The expected error returned is rpc error: code = Unknown desc = user [e93e77ff-3a3d-4417-83b2-30f0acdf3ed5] doesn't have permission to delete backup [tp-backup-testuser2-1705624295] [2024-01-19T00:36:38.193Z] INFO[2024-01-19 00:36:34] Verifying : Description : Verifying backup deletion is not possible [2024-01-19T00:36:38.193Z] INFO[2024-01-19 00:36:34] Actual:true, Expected: true [2024-01-19T00:36:38.193Z] 2024-01-19 00:36:34 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.glob..func18.2.10:#162] - Adding the restores testuser1-1705624295-tp-restore-1705624594 which will be created while validating FullAccess access [2024-01-19T00:36:38.193Z] INFO[2024-01-19 00:36:34] Verifying : Description : Fetching px-central-admin ctx [2024-01-19T00:36:38.193Z] INFO[2024-01-19 00:36:34] Actual:true, Expected: true [2024-01-19T00:36:38.193Z] INFO[2024-01-19 00:36:34] Verifying : Description : Fetching testuser1-1705624295 user ctx [2024-01-19T00:36:38.193Z] INFO[2024-01-19 00:36:34] Actual:true, Expected: true [2024-01-19T00:36:38.193Z] 2024-01-19 00:36:34 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Registering Source and Destination clusters from user context [2024-01-19T00:36:38.193Z] INFO[2024-01-19 00:36:34] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T00:36:38.193Z] INFO[2024-01-19 00:36:34] Actual:true, Expected: true [2024-01-19T00:36:38.193Z] 2024-01-19 00:36:34 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T00:36:38.193Z] 2024-01-19 00:36:34 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T00:36:38.193Z] 2024-01-19 00:36:34 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T00:36:38.193Z] 2024-01-19 00:36:34 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T00:36:38.193Z] 2024-01-19 00:36:34 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Create cluster [source-cluster] in org [default] [2024-01-19T00:36:38.193Z] 2024-01-19 00:36:34 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T00:36:38.193Z] 2024-01-19 00:36:34 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T00:36:38.193Z] 2024-01-19 00:36:34 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:36:38.193Z] 2024-01-19 00:36:34 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T00:36:38.193Z] 2024-01-19 00:36:34 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T00:36:38.193Z] 2024-01-19 00:36:34 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-qczvf] for IBM [2024-01-19T00:36:38.193Z] 2024-01-19 00:36:34 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-qczvf] for org [default] with [ibm] as provider [2024-01-19T00:36:38.193Z] 2024-01-19 00:36:34 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:36:38.194Z] 2024-01-19 00:36:34 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:36:38.194Z] 2024-01-19 00:36:34 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-qczvf] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user]] [2024-01-19T00:36:38.194Z] 2024-01-19 00:36:34 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user] [2024-01-19T00:36:38.194Z] 2024-01-19 00:36:35 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-qczvf from admin context and sharing with all the users [2024-01-19T00:36:38.194Z] 2024-01-19 00:36:35 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-qczvf] for org [default] with [ibm] as provider [2024-01-19T00:36:38.194Z] 2024-01-19 00:36:35 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:36:38.194Z] 2024-01-19 00:36:35 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:36:38.194Z] 2024-01-19 00:36:35 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-qczvf] is present [2024-01-19T00:36:38.194Z] 2024-01-19 00:36:35 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Create cluster [source-cluster] in org [default] [2024-01-19T00:36:38.194Z] 2024-01-19 00:36:36 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-lzaum] for IBM [2024-01-19T00:36:38.194Z] 2024-01-19 00:36:36 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-lzaum] for org [default] with [ibm] as provider [2024-01-19T00:36:38.194Z] 2024-01-19 00:36:36 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:36:38.194Z] 2024-01-19 00:36:36 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:36:38.194Z] 2024-01-19 00:36:36 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-lzaum] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user]] [2024-01-19T00:36:38.194Z] 2024-01-19 00:36:36 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user] [2024-01-19T00:36:38.195Z] 2024-01-19 00:36:36 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-lzaum from admin context and sharing with all the users [2024-01-19T00:36:38.195Z] 2024-01-19 00:36:36 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-lzaum] for org [default] with [ibm] as provider [2024-01-19T00:36:38.195Z] 2024-01-19 00:36:36 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:36:38.195Z] 2024-01-19 00:36:36 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:36:38.195Z] 2024-01-19 00:36:37 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-lzaum] is present [2024-01-19T00:36:38.195Z] 2024-01-19 00:36:37 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:36:38.195Z] INFO[2024-01-19 00:36:38] Verifying : Description : Creating source and destination cluster [2024-01-19T00:36:38.195Z] INFO[2024-01-19 00:36:38] Actual:true, Expected: true [2024-01-19T00:36:38.195Z] 2024-01-19 00:36:38 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Validating if user [testuser1-1705624295] with access [FullAccess] can restore and delete backup tp-backup-testuser1-1705624295 or not [2024-01-19T00:36:38.757Z] 2024/01/19 00:36:38 DoRetryWithTimeout - Error: {restore status for [testuser1-1705624295-tp-restore-1705624594] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T00:37:10.824Z] 2024-01-19 00:37:08 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.CreateRestore:#936] - Restore [testuser1-1705624295-tp-restore-1705624594] created successfully [2024-01-19T00:37:10.825Z] INFO[2024-01-19 00:37:08] Verifying : Description : Verifying that restore is possible [2024-01-19T00:37:10.825Z] INFO[2024-01-19 00:37:08] Actual:true, Expected: true [2024-01-19T00:37:10.825Z] INFO[2024-01-19 00:37:08] Verifying : Description : Getting backup UID for- tp-backup-testuser1-1705624295 [2024-01-19T00:37:10.825Z] INFO[2024-01-19 00:37:08] Actual:true, Expected: true [2024-01-19T00:37:10.825Z] INFO[2024-01-19 00:37:08] Verifying : Description : Verifying that delete backup is possible [2024-01-19T00:37:10.825Z] INFO[2024-01-19 00:37:08] Actual:true, Expected: true [2024-01-19T00:37:10.825Z] 2024-01-19 00:37:09 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Deleting labels from namespaces - [mysql-ibm-pxb-0-85040-01-19-00h29m32s mysql-ibm-pxb-1-85040-01-19-00h29m32s mysql-ibm-pxb-2-85040-01-19-00h29m32s] [2024-01-19T00:37:10.835Z] INFO[2024-01-19 00:37:09] Verifying : Description : Deleting labels [map[0-6aff95ad-3aab-4d8a-8a87-7200386facb5:6aff95ad-3aab-4d8a-8a87-7200386facb5 1-6aff95ad-3aab-4d8a-8a87-7200386facb5:6aff95ad-3aab-4d8a-8a87-7200386facb5 2-6aff95ad-3aab-4d8a-8a87-7200386facb5:6aff95ad-3aab-4d8a-8a87-7200386facb5]] to namespaces [[mysql-ibm-pxb-0-85040-01-19-00h29m32s mysql-ibm-pxb-1-85040-01-19-00h29m32s mysql-ibm-pxb-2-85040-01-19-00h29m32s]] [2024-01-19T00:37:10.835Z] INFO[2024-01-19 00:37:09] Actual:true, Expected: true [2024-01-19T00:37:10.835Z] 2024-01-19 00:37:09 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Destroying apps [2024-01-19T00:37:10.835Z] 2024-01-19 00:37:09 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T00:37:10.835Z] 2024-01-19 00:37:09 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T00:37:10.835Z] 2024-01-19 00:37:09 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] start destroying mysql-ibm app [2024-01-19T00:37:10.835Z] 2024-01-19 00:37:10 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T00:37:10.835Z] 2024-01-19 00:37:10 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T00:37:10.835Z] 2024-01-19 00:37:10 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T00:37:10.835Z] INFO[2024-01-19 00:37:10] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T00:37:10.835Z] INFO[2024-01-19 00:37:10] Actual:true, Expected: true [2024-01-19T00:37:10.835Z] 2024-01-19 00:37:10 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T00:37:10.835Z] 2024-01-19 00:37:10 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T00:37:10.835Z] 2024-01-19 00:37:10 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] start destroying mysql-ibm app [2024-01-19T00:37:10.835Z] 2024-01-19 00:37:10 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T00:37:10.835Z] 2024-01-19 00:37:10 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T00:37:10.836Z] 2024-01-19 00:37:10 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T00:37:10.836Z] INFO[2024-01-19 00:37:10] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T00:37:10.836Z] INFO[2024-01-19 00:37:10] Actual:true, Expected: true [2024-01-19T00:37:10.836Z] 2024-01-19 00:37:10 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T00:37:11.092Z] 2024-01-19 00:37:11 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T00:37:11.092Z] 2024-01-19 00:37:11 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] start destroying mysql-ibm app [2024-01-19T00:37:11.348Z] 2024-01-19 00:37:11 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T00:37:11.603Z] 2024-01-19 00:37:11 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T00:37:11.603Z] 2024-01-19 00:37:11 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T00:37:11.603Z] INFO[2024-01-19 00:37:11] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T00:37:11.603Z] INFO[2024-01-19 00:37:11] Actual:true, Expected: true [2024-01-19T00:37:11.603Z] 2024-01-19 00:37:11 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T00:37:11.859Z] 2024-01-19 00:37:11 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T00:37:11.859Z] 2024-01-19 00:37:11 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] start destroying mysql-ibm app [2024-01-19T00:37:12.115Z] 2024-01-19 00:37:11 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T00:37:12.115Z] 2024-01-19 00:37:11 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T00:37:12.115Z] 2024-01-19 00:37:11 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T00:37:12.115Z] INFO[2024-01-19 00:37:11] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T00:37:12.116Z] INFO[2024-01-19 00:37:11] Actual:true, Expected: true [2024-01-19T00:37:12.116Z] 2024-01-19 00:37:11 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T00:37:12.384Z] 2024-01-19 00:37:12 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T00:37:12.384Z] 2024-01-19 00:37:12 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] start destroying mysql-ibm app [2024-01-19T00:37:12.641Z] 2024-01-19 00:37:12 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T00:37:12.641Z] 2024-01-19 00:37:12 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T00:37:12.642Z] 2024-01-19 00:37:12 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T00:37:12.642Z] INFO[2024-01-19 00:37:12] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T00:37:12.642Z] INFO[2024-01-19 00:37:12] Actual:true, Expected: true [2024-01-19T00:37:12.642Z] 2024-01-19 00:37:12 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T00:37:13.203Z] 2024-01-19 00:37:12 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T00:37:13.203Z] 2024-01-19 00:37:12 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] start destroying mysql-ibm app [2024-01-19T00:37:13.203Z] 2024-01-19 00:37:13 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T00:37:13.203Z] 2024-01-19 00:37:13 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T00:37:13.204Z] 2024-01-19 00:37:13 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T00:37:13.204Z] INFO[2024-01-19 00:37:13] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T00:37:13.204Z] INFO[2024-01-19 00:37:13] Actual:true, Expected: true [2024-01-19T00:37:13.204Z] 2024-01-19 00:37:13 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.glob..func18.3:#180] - Generating user context [2024-01-19T00:37:14.569Z] 2024-01-19 00:37:14 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.glob..func18.3:#186] - Deleting restore created by users [2024-01-19T00:37:14.569Z] INFO[2024-01-19 00:37:14] Verifying : Description : Getting the backup driver [2024-01-19T00:37:14.569Z] INFO[2024-01-19 00:37:14] Actual:true, Expected: true [2024-01-19T00:37:14.569Z] INFO[2024-01-19 00:37:14] Verifying : Description : Deleting Restore testuser2-1705624295-tp-restore-1705624529 [2024-01-19T00:37:14.569Z] INFO[2024-01-19 00:37:14] Actual:true, Expected: true [2024-01-19T00:37:14.569Z] INFO[2024-01-19 00:37:14] Verifying : Description : Getting the backup driver [2024-01-19T00:37:14.569Z] INFO[2024-01-19 00:37:14] Actual:true, Expected: true [2024-01-19T00:37:14.569Z] INFO[2024-01-19 00:37:14] Verifying : Description : Deleting Restore testuser1-1705624295-tp-restore-1705624594 [2024-01-19T00:37:14.569Z] INFO[2024-01-19 00:37:14] Actual:true, Expected: true [2024-01-19T00:37:14.569Z] 2024-01-19 00:37:14 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.glob..func18.3:#191] - Deleting registered clusters for non-admin context [2024-01-19T00:37:14.569Z] 2024-01-19 00:37:14 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T00:37:14.569Z] INFO[2024-01-19 00:37:14] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T00:37:14.569Z] INFO[2024-01-19 00:37:14] Actual:true, Expected: true [2024-01-19T00:37:14.825Z] INFO[2024-01-19 00:37:14] Verifying : Description : Deleting cluster source-cluster [2024-01-19T00:37:14.825Z] INFO[2024-01-19 00:37:14] Actual:true, Expected: true [2024-01-19T00:37:16.192Z] INFO[2024-01-19 00:37:15] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-avylz] [2024-01-19T00:37:16.192Z] INFO[2024-01-19 00:37:15] Actual:true, Expected: true [2024-01-19T00:37:16.192Z] INFO[2024-01-19 00:37:16] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T00:37:16.192Z] INFO[2024-01-19 00:37:16] Actual:true, Expected: true [2024-01-19T00:37:17.557Z] INFO[2024-01-19 00:37:17] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-qzoph] [2024-01-19T00:37:17.557Z] INFO[2024-01-19 00:37:17] Actual:true, Expected: true [2024-01-19T00:37:17.557Z] 2024-01-19 00:37:17 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T00:37:17.558Z] INFO[2024-01-19 00:37:17] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T00:37:17.558Z] INFO[2024-01-19 00:37:17] Actual:true, Expected: true [2024-01-19T00:37:17.558Z] INFO[2024-01-19 00:37:17] Verifying : Description : Deleting cluster source-cluster [2024-01-19T00:37:17.558Z] INFO[2024-01-19 00:37:17] Actual:true, Expected: true [2024-01-19T00:37:18.922Z] INFO[2024-01-19 00:37:18] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-cegyx] [2024-01-19T00:37:18.923Z] INFO[2024-01-19 00:37:18] Actual:true, Expected: true [2024-01-19T00:37:18.923Z] INFO[2024-01-19 00:37:18] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T00:37:18.923Z] INFO[2024-01-19 00:37:18] Actual:true, Expected: true [2024-01-19T00:37:20.288Z] INFO[2024-01-19 00:37:19] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-jgdyr] [2024-01-19T00:37:20.288Z] INFO[2024-01-19 00:37:19] Actual:true, Expected: true [2024-01-19T00:37:20.289Z] 2024-01-19 00:37:19 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T00:37:20.289Z] INFO[2024-01-19 00:37:19] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T00:37:20.289Z] INFO[2024-01-19 00:37:19] Actual:true, Expected: true [2024-01-19T00:37:20.289Z] INFO[2024-01-19 00:37:20] Verifying : Description : Deleting cluster source-cluster [2024-01-19T00:37:20.289Z] INFO[2024-01-19 00:37:20] Actual:true, Expected: true [2024-01-19T00:37:21.655Z] INFO[2024-01-19 00:37:21] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-qczvf] [2024-01-19T00:37:21.655Z] INFO[2024-01-19 00:37:21] Actual:true, Expected: true [2024-01-19T00:37:21.912Z] INFO[2024-01-19 00:37:21] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T00:37:21.912Z] INFO[2024-01-19 00:37:21] Actual:true, Expected: true [2024-01-19T00:37:22.842Z] INFO[2024-01-19 00:37:22] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-lzaum] [2024-01-19T00:37:22.842Z] INFO[2024-01-19 00:37:22] Actual:true, Expected: true [2024-01-19T00:37:22.842Z] 2024-01-19 00:37:22 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.glob..func18.3:#195] - Cleaning up users [2024-01-19T00:37:26.111Z] 2024-01-19 00:37:25 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705624295 - 83ca30da-aa80-45fc-a95c-d3eebd44cf4e [2024-01-19T00:37:26.111Z] 2024-01-19 00:37:26 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser3-1705624295 - 9ed58ac1-4544-44f9-aae4-abe5ca573145 [2024-01-19T00:37:26.378Z] 2024-01-19 00:37:26 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2-1705624295 - e93e77ff-3a3d-4417-83b2-30f0acdf3ed5 [2024-01-19T00:37:27.315Z] 2024-01-19 00:37:27 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.DeleteUser:#755] - Response for user [testuser1-1705624295] deletion - [2024-01-19T00:37:27.315Z] 2024-01-19 00:37:27 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.DeleteUser:#759] - Deleted User - testuser1-1705624295 [2024-01-19T00:37:27.315Z] INFO[2024-01-19 00:37:27] Verifying : Description : Deleting user testuser1-1705624295 [2024-01-19T00:37:27.315Z] INFO[2024-01-19 00:37:27] Actual:true, Expected: true [2024-01-19T00:37:27.573Z] 2024-01-19 00:37:27 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.DeleteUser:#755] - Response for user [testuser3-1705624295] deletion - [2024-01-19T00:37:27.573Z] 2024-01-19 00:37:27 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.DeleteUser:#759] - Deleted User - testuser3-1705624295 [2024-01-19T00:37:27.573Z] INFO[2024-01-19 00:37:27] Verifying : Description : Deleting user testuser3-1705624295 [2024-01-19T00:37:27.573Z] INFO[2024-01-19 00:37:27] Actual:true, Expected: true [2024-01-19T00:37:27.829Z] 2024-01-19 00:37:27 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.DeleteUser:#755] - Response for user [testuser2-1705624295] deletion - [2024-01-19T00:37:27.829Z] 2024-01-19 00:37:27 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.DeleteUser:#759] - Deleted User - testuser2-1705624295 [2024-01-19T00:37:27.829Z] INFO[2024-01-19 00:37:27] Verifying : Description : Deleting user testuser2-1705624295 [2024-01-19T00:37:27.829Z] INFO[2024-01-19 00:37:27] Actual:true, Expected: true [2024-01-19T00:37:27.829Z] 2024-01-19 00:37:27 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Cleaning backup locations in map [map[bff63c2c-ee79-49a8-88da-dc888d0dda83:autogenerated-backup-location-1705624299]], cloud credential [cred-aws-1705624299], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T00:37:27.829Z] INFO[2024-01-19 00:37:27] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-1705624299] [2024-01-19T00:37:27.829Z] INFO[2024-01-19 00:37:27] Actual:true, Expected: true [2024-01-19T00:37:27.829Z] 2024-01-19 00:37:27 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705624299] is present [2024-01-19T00:37:28.087Z] 2024/01/19 00:37:27 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705624299 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T00:38:00.106Z] 2024-01-19 00:37:57 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705624299] is present [2024-01-19T00:38:00.106Z] 2024/01/19 00:37:57 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705624299 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T00:38:32.145Z] 2024-01-19 00:38:27 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705624299] is present [2024-01-19T00:38:32.145Z] 2024/01/19 00:38:28 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705624299 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T00:38:58.644Z] 2024-01-19 00:38:58 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 aws-global-aws-px-backup-system-test-4cac76dc-bl] [2024-01-19T00:38:58.644Z] INFO[2024-01-19 00:38:58] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-1705624299 [2024-01-19T00:38:58.644Z] INFO[2024-01-19 00:38:58] Actual:true, Expected: true [2024-01-19T00:38:58.644Z] 2024-01-19 00:38:58 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cred-aws-1705624299] is present [2024-01-19T00:38:58.644Z] INFO[2024-01-19 00:38:58] Verifying : Description : Verifying if cloud cred [cred-aws-1705624299] is present [2024-01-19T00:38:58.644Z] INFO[2024-01-19 00:38:58] Actual:true, Expected: true [2024-01-19T00:38:58.644Z] INFO[2024-01-19 00:38:58] Verifying : Description : Verifying deletion of cloud cred [cred-aws-1705624299] [2024-01-19T00:38:58.644Z] INFO[2024-01-19 00:38:58] Actual:true, Expected: true [2024-01-19T00:38:58.644Z] INFO[2024-01-19 00:38:58] Verifying : Description : Deleting cloud cred cred-aws-1705624299 [2024-01-19T00:38:58.644Z] INFO[2024-01-19 00:38:58] Actual:true, Expected: true [2024-01-19T00:38:58.644Z] INFO[2024-01-19 00:38:58] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T00:38:58.644Z] INFO[2024-01-19 00:38:58] Actual:true, Expected: true [2024-01-19T00:38:58.644Z] INFO[2024-01-19 00:38:58] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T00:38:58.644Z] ERRO[2024-01-19 00:38:58] Actual:rpc error: code = FailedPrecondition desc = cluster [destination-cluster] in use by backupschedule [tp-backup-schedule-1705605709], cannot be deleted, Expected:  [2024-01-19T00:38:58.901Z] INFO[2024-01-19 00:38:58] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-zbbvt] [2024-01-19T00:38:58.901Z] ERRO[2024-01-19 00:38:58] Actual:rpc error: code = Unknown desc = cluster [destination-cluster] has reference to cloud credential [ibm-destination-config-cloud-cred-zbbvt:a271ea5b-dc9d-40f1-a82a-651539c7e486], Expected:  [2024-01-19T00:38:59.161Z] INFO[2024-01-19 00:38:58] Verifying : Description : Deleting cluster source-cluster [2024-01-19T00:38:59.161Z] INFO[2024-01-19 00:38:58] Actual:true, Expected: true [2024-01-19T00:38:59.419Z] INFO[2024-01-19 00:38:59] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-xmque] [2024-01-19T00:38:59.419Z] INFO[2024-01-19 00:38:59] Actual:true, Expected: true [2024-01-19T00:38:59.419Z] INFO[2024-01-19 00:38:59] --------Test End------ [2024-01-19T00:38:59.419Z] INFO[2024-01-19 00:38:59] #Test: [2024-01-19T00:38:59.419Z] INFO[2024-01-19 00:38:59] #Description: [2024-01-19T00:38:59.419Z] INFO[2024-01-19 00:38:59] ------------------------ [2024-01-19T00:38:59.675Z] 2024-01-19 00:38:59 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-85040-01-19-00h29m32s] [2024-01-19T00:38:59.930Z] 2024-01-19 00:38:59 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-1-85040-01-19-00h29m32s] [2024-01-19T00:38:59.930Z] 2024-01-19 00:38:59 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-2-85040-01-19-00h29m32s] [2024-01-19T00:39:00.186Z] 2024-01-19 00:39:00 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-3-85040-01-19-00h29m32s] [2024-01-19T00:39:00.441Z] 2024-01-19 00:39:00 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-4-85040-01-19-00h29m32s] [2024-01-19T00:39:00.697Z] 2024-01-19 00:39:00 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-5-85040-01-19-00h29m32s] [2024-01-19T00:39:00.697Z] 2024-01-19 00:39:00 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T00:39:00.697Z] 2024-01-19 00:39:00 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T00:39:01.258Z] 2024-01-19 00:39:00 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-85040-01-19-00h29m32s] [2024-01-19T00:39:01.258Z] 2024-01-19 00:39:01 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-1-85040-01-19-00h29m32s] [2024-01-19T00:39:01.513Z] 2024-01-19 00:39:01 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-2-85040-01-19-00h29m32s] [2024-01-19T00:39:01.513Z] 2024-01-19 00:39:01 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-3-85040-01-19-00h29m32s] [2024-01-19T00:39:01.769Z] 2024-01-19 00:39:01 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-4-85040-01-19-00h29m32s] [2024-01-19T00:39:02.025Z] 2024-01-19 00:39:01 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-5-85040-01-19-00h29m32s] [2024-01-19T00:39:02.282Z] 2024-01-19 00:39:02 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {NamespaceLabelledBackupSharedWithDifferentAccessMode} Take namespace labelled backup and share with users having different access mode [2024-01-19T00:39:02.282Z] 2024-01-19 00:39:02 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T00:39:02.282Z] 2024/01/19 00:39:02 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:39:07.528Z] 2024/01/19 00:39:07 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:39:12.776Z] 2024/01/19 00:39:12 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:39:18.018Z] 2024/01/19 00:39:17 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:39:22.211Z] 2024/01/19 00:39:22 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:39:27.460Z] 2024/01/19 00:39:27 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:39:32.707Z] 2024/01/19 00:39:32 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:39:37.954Z] 2024/01/19 00:39:37 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:39:42.121Z] 2024/01/19 00:39:42 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.0.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.7 10.240.0.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T00:39:42.121Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T00:39:42.121Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T00:39:42.121Z] {dial tcp 10.240.0.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T00:39:47.364Z] 2024-01-19 00:39:47 +0000:[WARNING] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.7 10.240.0.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T00:39:47.364Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T00:39:47.364Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T00:39:47.364Z] {dial tcp 10.240.0.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T00:39:47.364Z] 2024-01-19 00:39:47 +0000:[ERROR] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.7 10.240.0.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T00:39:47.364Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T00:39:47.364Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T00:39:47.364Z] {dial tcp 10.240.0.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T00:39:47.364Z] 2024-01-19 00:39:47 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T00:39:47.364Z] 2024-01-19 00:39:47 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] Switching context to [/tmp/source-config] [2024-01-19T00:39:47.364Z] 2024-01-19 00:39:47 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T00:39:47.365Z] 2024-01-19 00:39:47 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T00:39:47.365Z] 2024-01-19 00:39:47 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T00:39:47.365Z] 2024-01-19 00:39:47 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T00:39:47.365Z] 2024-01-19 00:39:47 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T00:39:47.621Z] 2024-01-19 00:39:47 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T00:39:48.182Z] 2024-01-19 00:39:47 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T00:39:48.438Z] 2024-01-19 00:39:48 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T00:39:48.999Z] 2024-01-19 00:39:48 +0000:[INFO] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T00:39:48.999Z] 2024/01/19 00:39:48 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:40:01.165Z] 2024/01/19 00:39:58 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:40:09.251Z] 2024/01/19 00:40:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:40:19.191Z] 2024/01/19 00:40:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:40:29.134Z] 2024/01/19 00:40:29 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:40:41.303Z] 2024/01/19 00:40:39 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:40:49.412Z] 2024/01/19 00:40:49 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:40:59.354Z] 2024/01/19 00:40:59 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:41:09.295Z] 2024/01/19 00:41:09 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:41:19.238Z] 2024/01/19 00:41:19 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:41:31.411Z] 2024/01/19 00:41:29 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:41:39.506Z] 2024/01/19 00:41:39 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:41:49.452Z] 2024/01/19 00:41:49 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:41:59.402Z] 2024/01/19 00:41:59 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:42:09.352Z] 2024/01/19 00:42:09 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:42:21.557Z] 2024/01/19 00:42:19 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:42:29.643Z] 2024/01/19 00:42:29 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:42:39.586Z] 2024/01/19 00:42:39 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:42:49.531Z] 2024/01/19 00:42:49 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:42:59.488Z] 2024/01/19 00:42:59 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:43:11.658Z] 2024/01/19 00:43:09 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:43:19.744Z] 2024/01/19 00:43:19 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:43:29.685Z] 2024/01/19 00:43:29 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:43:39.627Z] 2024/01/19 00:43:39 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:43:51.796Z] 2024/01/19 00:43:49 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:43:59.888Z] 2024/01/19 00:43:59 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:44:09.834Z] 2024/01/19 00:44:09 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:44:19.780Z] 2024/01/19 00:44:19 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:44:29.738Z] 2024/01/19 00:44:29 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:44:39.691Z] INFO[2024-01-19 00:44:39] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T00:44:39.691Z] INFO[2024-01-19 00:44:39] sched stats: tick interval: 33039 < 1.2s (100%) [2024-01-19T00:44:39.691Z] INFO[2024-01-19 00:44:39] sched stats: tick process durations: 33038 < 500ms (100%) [2024-01-19T00:44:39.948Z] 2024/01/19 00:44:39 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:44:49.884Z] 2024-01-19 00:44:49 +0000:[ERROR] [{NamespaceLabelledBackupSharedWithDifferentAccessMode}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.884Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.884Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.884Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.884Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.884Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.884Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.884Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.884Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.885Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.885Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.885Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.885Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.885Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.885Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.885Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.885Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.885Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.885Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.885Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.885Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.885Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.885Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.885Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.885Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.885Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.885Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.885Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.885Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.885Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T00:44:49.885Z] [2024-01-19T00:44:49.885Z] ------------------------------ [2024-01-19T00:44:49.885Z] • Failure [917.521 seconds] [2024-01-19T00:44:49.885Z] {NamespaceLabelledBackupSharedWithDifferentAccessMode} [2024-01-19T00:44:49.885Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_namespace_labelled_test.go:22 [2024-01-19T00:44:49.885Z] Take namespace labelled backup and share with users having different access mode [] [2024-01-19T00:44:49.886Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_namespace_labelled_test.go:62 [2024-01-19T00:44:49.886Z] [2024-01-19T00:44:49.886Z] Unexpected error: [2024-01-19T00:44:49.886Z] <*errors.errorString | 0xc00133bd20>: { [2024-01-19T00:44:49.887Z] s: "failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{No... [2024-01-19T00:44:49.887Z] [2024-01-19T00:44:49.887Z] Gomega truncated this representation as it exceeds 'format.MaxLength'. [2024-01-19T00:44:49.887Z] Consider having the object provide a custom 'GomegaStringer' representation [2024-01-19T00:44:49.887Z] or adjust the parameters in Gomega's 'format' package. [2024-01-19T00:44:49.887Z] [2024-01-19T00:44:49.887Z] Learn more here: https://onsi.github.io/gomega/#adjusting-output [2024-01-19T00:44:49.887Z] [2024-01-19T00:44:49.887Z] failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.887Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.887Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.887Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.887Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.887Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.887Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.887Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.887Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.887Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.887Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.887Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.887Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.887Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.887Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.888Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.888Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.888Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.888Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.888Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.888Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.888Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.888Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.888Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.888Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.888Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.888Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.888Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.888Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:44:49.888Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T00:44:49.888Z] occurred [2024-01-19T00:44:49.888Z] [2024-01-19T00:44:49.888Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T00:44:49.888Z] [2024-01-19T00:44:49.888Z] Full Stack Trace [2024-01-19T00:44:49.888Z] github.com/portworx/torpedo/pkg/log.FailOnError({0x88a5060, 0xc00133bd20}, {0x7f74bfd?, 0x6cc0320?}, {0x0?, 0x6cc0320?, 0xc00020dda0?}) [2024-01-19T00:44:49.888Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 +0x290 [2024-01-19T00:44:49.888Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest.func1() [2024-01-19T00:44:49.888Z] /go/src/github.com/portworx/torpedo/tests/common.go:7283 +0x32 [2024-01-19T00:44:49.888Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest({0xc000d60380, 0x6, 0x8}) [2024-01-19T00:44:49.888Z] /go/src/github.com/portworx/torpedo/tests/common.go:7299 +0x778 [2024-01-19T00:44:49.888Z] github.com/portworx/torpedo/tests/backup.glob..func18.3() [2024-01-19T00:44:49.888Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_namespace_labelled_test.go:206 +0x54c [2024-01-19T00:44:49.888Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T00:44:49.888Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T00:44:49.888Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T00:44:49.888Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T00:44:49.888Z] created by testing.(*T).Run [2024-01-19T00:44:49.888Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T00:44:49.888Z] ------------------------------ [2024-01-19T00:44:49.888Z] STEP: schedule applications [2024-01-19T00:44:49.888Z] STEP: schedule applications [2024-01-19T00:44:49.888Z] STEP: schedule applications [2024-01-19T00:44:49.888Z] STEP: schedule applications [2024-01-19T00:44:49.888Z] STEP: schedule applications [2024-01-19T00:44:49.888Z] STEP: schedule applications [2024-01-19T00:44:49.889Z] STEP: Validate applications [2024-01-19T00:44:49.889Z] STEP: validate applications [2024-01-19T00:44:49.889Z] STEP: validate mysql-ibm app's volumes [2024-01-19T00:44:49.889Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T00:44:49.889Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T00:44:49.889Z] STEP: get mysql-ibm app's volume: pvc-dadf9890-e77b-4798-9726-51d949287c09 inspected by the volume driver [2024-01-19T00:44:49.889Z] STEP: wait for mysql-ibm app to start running [2024-01-19T00:44:49.889Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T00:44:49.889Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T00:44:49.889Z] STEP: Validate Px pod restart count [2024-01-19T00:44:49.889Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T00:44:49.889Z] STEP: validate mysql-ibm app's volumes [2024-01-19T00:44:49.889Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T00:44:49.889Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T00:44:49.889Z] STEP: get mysql-ibm app's volume: pvc-149a1cf8-c076-46c1-b7b7-3301f9441ba2 inspected by the volume driver [2024-01-19T00:44:49.889Z] STEP: wait for mysql-ibm app to start running [2024-01-19T00:44:49.889Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T00:44:49.889Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T00:44:49.889Z] STEP: Validate Px pod restart count [2024-01-19T00:44:49.889Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T00:44:49.889Z] STEP: validate mysql-ibm app's volumes [2024-01-19T00:44:49.889Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T00:44:49.889Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T00:44:49.889Z] STEP: get mysql-ibm app's volume: pvc-ef03831f-f197-43dc-a7dc-b033cfa4f0d0 inspected by the volume driver [2024-01-19T00:44:49.889Z] STEP: wait for mysql-ibm app to start running [2024-01-19T00:44:49.889Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T00:44:49.889Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T00:44:49.889Z] STEP: Validate Px pod restart count [2024-01-19T00:44:49.889Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T00:44:49.889Z] STEP: validate mysql-ibm app's volumes [2024-01-19T00:44:49.889Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T00:44:49.889Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T00:44:49.889Z] STEP: get mysql-ibm app's volume: pvc-e32e0c76-c576-479d-945f-eb001dda1ab0 inspected by the volume driver [2024-01-19T00:44:49.889Z] STEP: wait for mysql-ibm app to start running [2024-01-19T00:44:49.889Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T00:44:49.889Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T00:44:49.889Z] STEP: Validate Px pod restart count [2024-01-19T00:44:49.889Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T00:44:49.889Z] STEP: validate mysql-ibm app's volumes [2024-01-19T00:44:49.889Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T00:44:49.889Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T00:44:49.889Z] STEP: get mysql-ibm app's volume: pvc-f81cae6b-1f96-4642-a30e-ca10fe1d869f inspected by the volume driver [2024-01-19T00:44:49.889Z] STEP: wait for mysql-ibm app to start running [2024-01-19T00:44:49.889Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T00:44:49.889Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T00:44:49.889Z] STEP: Validate Px pod restart count [2024-01-19T00:44:49.889Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T00:44:49.889Z] STEP: validate mysql-ibm app's volumes [2024-01-19T00:44:49.889Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T00:44:49.889Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T00:44:49.889Z] STEP: get mysql-ibm app's volume: pvc-c5842f53-4323-4801-bc30-cdcc2a6ec0a8 inspected by the volume driver [2024-01-19T00:44:49.889Z] STEP: wait for mysql-ibm app to start running [2024-01-19T00:44:49.889Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T00:44:49.889Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T00:44:49.889Z] STEP: Validate Px pod restart count [2024-01-19T00:44:49.889Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T00:44:49.889Z] STEP: Generating multiple labels [2024-01-19T00:44:49.889Z] STEP: Adding labels to only few namespaces [2024-01-19T00:44:49.889Z] STEP: Create multiple users [2024-01-19T00:44:49.889Z] STEP: Creating backup location and cloud setting [2024-01-19T00:44:49.890Z] STEP: Register source and destination cluster for backup [2024-01-19T00:44:49.890Z] STEP: Taking namespace labelled backup for each user [2024-01-19T00:44:49.890Z] STEP: Verifying that correct namespaces are backed up for the namespace labelled backup taken and correct labels are applied to the backup [2024-01-19T00:44:49.890Z] STEP: Share backup with users with different access level [2024-01-19T00:44:49.890Z] STEP: Validate if the users with different access level can restore/delete backup as per their access mode [2024-01-19T00:44:49.890Z] STEP: destroy apps [2024-01-19T00:44:49.890Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T00:44:49.890Z] STEP: start destroying mysql-ibm app [2024-01-19T00:44:49.890Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T00:44:49.890Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T00:44:49.890Z] STEP: start destroying mysql-ibm app [2024-01-19T00:44:49.890Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T00:44:49.890Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T00:44:49.890Z] STEP: start destroying mysql-ibm app [2024-01-19T00:44:49.890Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T00:44:49.890Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T00:44:49.890Z] STEP: start destroying mysql-ibm app [2024-01-19T00:44:49.890Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T00:44:49.890Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T00:44:49.890Z] STEP: start destroying mysql-ibm app [2024-01-19T00:44:49.890Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T00:44:49.890Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T00:44:49.890Z] STEP: start destroying mysql-ibm app [2024-01-19T00:44:49.890Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T00:44:49.890Z] S2024-01-19 00:44:49 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Log Dir: /testresults//CancelAllRunningRestoreJobs.log [2024-01-19T00:44:49.890Z] INFO[2024-01-19 00:44:49] --------Test Start------ [2024-01-19T00:44:49.890Z] INFO[2024-01-19 00:44:49] #Test: CancelAllRunningRestoreJobs [2024-01-19T00:44:49.890Z] INFO[2024-01-19 00:44:49] #Description: Cancel all the running restore jobs while restores are in progress [2024-01-19T00:44:49.890Z] INFO[2024-01-19 00:44:49] ------------------------ [2024-01-19T00:44:49.890Z] 2024-01-19 00:44:49 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Deploying applications required for the testcase [2024-01-19T00:44:49.890Z] 2024-01-19 00:44:49 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T00:44:49.890Z] 2024-01-19 00:44:49 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T00:44:49.890Z] 2024-01-19 00:44:49 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T00:44:49.890Z] 2024-01-19 00:44:49 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T00:44:49.890Z] 2024-01-19 00:44:49 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T00:44:50.146Z] 2024-01-19 00:44:49 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T00:44:50.146Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T00:44:50.146Z] 2024-01-19 00:44:49 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T00:44:50.147Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T00:44:50.147Z] 2024-01-19 00:44:49 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T00:44:50.147Z] 2024-01-19 00:44:49 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Validating the deployed applications [2024-01-19T00:44:50.147Z] 2024-01-19 00:44:49 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Validate applications [2024-01-19T00:44:50.147Z] 2024-01-19 00:44:49 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Validating mysql-ibm app [2024-01-19T00:44:50.147Z] 2024-01-19 00:44:49 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Validating mysql-ibm app's volumes [2024-01-19T00:44:50.147Z] 2024-01-19 00:44:49 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:44:50.147Z] 2024/01/19 00:44:50 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T00:45:00.090Z] 2024/01/19 00:45:00 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T00:45:12.286Z] 2024-01-19 00:45:10 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-58058-01-19-00h44m49s [2024-01-19T00:45:12.286Z] 2024-01-19 00:45:10 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T00:45:12.287Z] 2024-01-19 00:45:10 +0000:[WARNING] [{CancelAllRunningRestoreJobs}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T00:45:12.287Z] 2024-01-19 00:45:10 +0000:[INFO] [{CancelAllRunningRestoreJobs}] wait for mysql-ibm app to start running [2024-01-19T00:45:12.287Z] 2024-01-19 00:45:10 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T00:45:12.287Z] 2024/01/19 00:45:10 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:45:12.287Z] pod name:mysql-7d766cd65c-cbsx4 namespace:mysql-ibm-pxb-0-58058-01-19-00h44m49s running:false ready:false node:10.240.64.8 [2024-01-19T00:45:12.287Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:45:22.240Z] 2024/01/19 00:45:20 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:45:22.240Z] pod name:mysql-7d766cd65c-cbsx4 namespace:mysql-ibm-pxb-0-58058-01-19-00h44m49s running:true ready:false node:10.240.64.8 [2024-01-19T00:45:22.240Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:45:32.180Z] 2024/01/19 00:45:30 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:45:32.180Z] pod name:mysql-7d766cd65c-cbsx4 namespace:mysql-ibm-pxb-0-58058-01-19-00h44m49s running:true ready:false node:10.240.64.8 [2024-01-19T00:45:32.180Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:45:42.145Z] 2024/01/19 00:45:40 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:45:42.146Z] pod name:mysql-7d766cd65c-cbsx4 namespace:mysql-ibm-pxb-0-58058-01-19-00h44m49s running:true ready:false node:10.240.64.8 [2024-01-19T00:45:42.146Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:45:52.090Z] 2024/01/19 00:45:50 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:45:52.090Z] pod name:mysql-7d766cd65c-cbsx4 namespace:mysql-ibm-pxb-0-58058-01-19-00h44m49s running:true ready:false node:10.240.64.8 [2024-01-19T00:45:52.090Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:46:02.037Z] 2024/01/19 00:46:00 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:46:02.037Z] pod name:mysql-7d766cd65c-cbsx4 namespace:mysql-ibm-pxb-0-58058-01-19-00h44m49s running:true ready:false node:10.240.64.8 [2024-01-19T00:46:02.037Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:46:11.984Z] 2024/01/19 00:46:10 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:46:11.985Z] pod name:mysql-7d766cd65c-cbsx4 namespace:mysql-ibm-pxb-0-58058-01-19-00h44m49s running:true ready:false node:10.240.64.8 [2024-01-19T00:46:11.985Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:46:21.933Z] 2024/01/19 00:46:21 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:46:21.933Z] pod name:mysql-7d766cd65c-cbsx4 namespace:mysql-ibm-pxb-0-58058-01-19-00h44m49s running:true ready:false node:10.240.64.8 [2024-01-19T00:46:21.933Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:46:31.879Z] 2024-01-19 00:46:31 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T00:46:31.879Z] 2024-01-19 00:46:31 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T00:46:31.879Z] 2024-01-19 00:46:31 +0000:[INFO] [{CancelAllRunningRestoreJobs}] validate if mysql-ibm app's volumes are setup [2024-01-19T00:46:31.879Z] 2024-01-19 00:46:31 +0000:[WARNING] [{CancelAllRunningRestoreJobs}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:46:31.879Z] 2024-01-19 00:46:31 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:46:31.879Z] 2024-01-19 00:46:31 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T00:46:31.879Z] 2024-01-19 00:46:31 +0000:[WARNING] [{CancelAllRunningRestoreJobs}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T00:46:31.879Z] 2024-01-19 00:46:31 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Adding cloud account and backup location [2024-01-19T00:46:31.879Z] 2024-01-19 00:46:31 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [cloudcred-aws-1705625191] for org [default] with [aws] as provider [2024-01-19T00:46:31.879Z] 2024-01-19 00:46:31 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T00:46:32.136Z] 2024-01-19 00:46:31 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cloudcred-aws-1705625191] is present [2024-01-19T00:46:32.136Z] INFO[2024-01-19 00:46:31] Verifying : Description : Verifying creation of cloud account named [cloudcred-aws-1705625191] for org [default] with [aws] as provider [2024-01-19T00:46:32.136Z] INFO[2024-01-19 00:46:31] Actual:true, Expected: true [2024-01-19T00:46:35.417Z] INFO[2024-01-19 00:46:34] Verifying : Description : Creating backup location aws-global-aws-px-backup-system-test-4cac76dc-1705625191-bl [2024-01-19T00:46:35.417Z] INFO[2024-01-19 00:46:34] Actual:true, Expected: true [2024-01-19T00:46:35.417Z] 2024-01-19 00:46:34 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Registering source and destination clusters for backup [2024-01-19T00:46:35.417Z] INFO[2024-01-19 00:46:35] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T00:46:35.417Z] INFO[2024-01-19 00:46:35] Actual:true, Expected: true [2024-01-19T00:46:35.417Z] 2024-01-19 00:46:35 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T00:46:35.417Z] 2024-01-19 00:46:35 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T00:46:35.417Z] 2024-01-19 00:46:35 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T00:46:35.417Z] 2024-01-19 00:46:35 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T00:46:35.417Z] 2024-01-19 00:46:35 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Create cluster [source-cluster] in org [default] [2024-01-19T00:46:35.417Z] 2024-01-19 00:46:35 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T00:46:35.417Z] 2024-01-19 00:46:35 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T00:46:35.417Z] 2024-01-19 00:46:35 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:46:35.417Z] 2024-01-19 00:46:35 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T00:46:35.417Z] 2024-01-19 00:46:35 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T00:46:35.417Z] 2024-01-19 00:46:35 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-xfizb] for IBM [2024-01-19T00:46:35.417Z] 2024-01-19 00:46:35 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-xfizb] for org [default] with [ibm] as provider [2024-01-19T00:46:35.417Z] 2024-01-19 00:46:35 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:46:35.418Z] 2024-01-19 00:46:35 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:46:35.418Z] 2024-01-19 00:46:35 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-xfizb] is present [2024-01-19T00:46:35.418Z] 2024-01-19 00:46:35 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Create cluster [source-cluster] in org [default] [2024-01-19T00:46:35.979Z] 2024-01-19 00:46:35 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-myzyc] for IBM [2024-01-19T00:46:35.979Z] 2024-01-19 00:46:35 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-myzyc] for org [default] with [ibm] as provider [2024-01-19T00:46:35.979Z] 2024-01-19 00:46:35 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:46:35.979Z] 2024-01-19 00:46:35 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:46:36.235Z] 2024-01-19 00:46:35 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-myzyc] is present [2024-01-19T00:46:36.235Z] 2024-01-19 00:46:35 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:46:36.235Z] INFO[2024-01-19 00:46:36] Verifying : Description : Creating source cluster source-cluster and destination cluster destination-cluster [2024-01-19T00:46:36.235Z] INFO[2024-01-19 00:46:36] Actual:true, Expected: true [2024-01-19T00:46:36.235Z] INFO[2024-01-19 00:46:36] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T00:46:36.235Z] INFO[2024-01-19 00:46:36] Actual:Online, Expected: Online [2024-01-19T00:46:36.235Z] INFO[2024-01-19 00:46:36] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T00:46:36.235Z] INFO[2024-01-19 00:46:36] Actual:true, Expected: true [2024-01-19T00:46:36.235Z] INFO[2024-01-19 00:46:36] Verifying : Description : Verifying if [destination-cluster] cluster is online [2024-01-19T00:46:36.235Z] INFO[2024-01-19 00:46:36] Actual:Online, Expected: Online [2024-01-19T00:46:36.235Z] 2024-01-19 00:46:36 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Taking backup of applications [2024-01-19T00:46:48.405Z] 2024-01-19 00:46:46 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T00:46:48.405Z] 2024-01-19 00:46:46 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Volume snapshot classes found in the cluster - [2024-01-19T00:46:48.405Z] 2024-01-19 00:46:46 +0000:[INFO] [{CancelAllRunningRestoreJobs}] ibmc-vpcblock-snapshot [2024-01-19T00:46:48.405Z] 2024-01-19 00:46:46 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T00:46:48.405Z] 2024/01/19 00:46:46 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-0-1705625206] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T00:46:56.487Z] 2024-01-19 00:46:56 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T00:46:56.487Z] 2024-01-19 00:46:56 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Volume snapshot classes found in the cluster - [2024-01-19T00:46:56.487Z] 2024-01-19 00:46:56 +0000:[INFO] [{CancelAllRunningRestoreJobs}] ibmc-vpcblock-snapshot [2024-01-19T00:46:56.487Z] 2024-01-19 00:46:56 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T00:46:59.026Z] 2024/01/19 00:46:57 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T00:47:07.114Z] 2024-01-19 00:47:06 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T00:47:07.114Z] 2024-01-19 00:47:06 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Volume snapshot classes found in the cluster - [2024-01-19T00:47:07.114Z] 2024-01-19 00:47:06 +0000:[INFO] [{CancelAllRunningRestoreJobs}] ibmc-vpcblock-snapshot [2024-01-19T00:47:07.114Z] 2024-01-19 00:47:06 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T00:47:07.372Z] 2024/01/19 00:47:07 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-2-1705625226] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T00:47:17.310Z] 2024-01-19 00:47:16 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T00:47:17.310Z] 2024-01-19 00:47:16 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Volume snapshot classes found in the cluster - [2024-01-19T00:47:17.310Z] 2024-01-19 00:47:16 +0000:[INFO] [{CancelAllRunningRestoreJobs}] ibmc-vpcblock-snapshot [2024-01-19T00:47:17.310Z] 2024-01-19 00:47:16 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T00:47:17.310Z] 2024/01/19 00:47:16 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-0-1705625206] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:47:17.310Z] 2024/01/19 00:47:17 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-3-1705625236] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T00:47:27.251Z] 2024/01/19 00:47:27 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:47:39.418Z] 2024/01/19 00:47:37 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-2-1705625226] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:47:47.514Z] 2024/01/19 00:47:46 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-0-1705625206] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:47:47.514Z] 2024/01/19 00:47:47 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-3-1705625236] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:47:57.458Z] 2024/01/19 00:47:57 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:48:07.404Z] 2024/01/19 00:48:07 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-2-1705625226] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:48:17.346Z] 2024/01/19 00:48:16 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-0-1705625206] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:48:17.603Z] 2024/01/19 00:48:17 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-3-1705625236] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:48:27.543Z] 2024/01/19 00:48:27 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:48:37.491Z] 2024/01/19 00:48:37 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-2-1705625226] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:48:47.434Z] 2024/01/19 00:48:46 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-0-1705625206] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:48:47.434Z] 2024/01/19 00:48:47 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-3-1705625236] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:48:57.397Z] 2024/01/19 00:48:57 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:49:09.622Z] 2024/01/19 00:49:07 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-2-1705625226] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:49:17.715Z] 2024/01/19 00:49:16 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-0-1705625206] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:49:17.715Z] 2024/01/19 00:49:17 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-3-1705625236] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:49:27.658Z] 2024/01/19 00:49:27 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:49:37.596Z] 2024/01/19 00:49:37 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-2-1705625226] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T00:49:47.540Z] 2024-01-19 00:49:46 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.CreateBackup:#313] - Backup [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-0-1705625206] created successfully [2024-01-19T00:49:47.540Z] 2024-01-19 00:49:46 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Validating backup [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-0-1705625206] in org [default] [2024-01-19T00:49:47.540Z] 2024-01-19 00:49:46 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-0-1705625206] [2024-01-19T00:49:47.540Z] 2024-01-19 00:49:46 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58058-01-19-00h44m49s] in backup [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-0-1705625206] [2024-01-19T00:49:47.540Z] 2024-01-19 00:49:46 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58058-01-19-00h44m49s] in backup [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-0-1705625206] [2024-01-19T00:49:47.540Z] 2024-01-19 00:49:46 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58058-01-19-00h44m49s] [2024-01-19T00:49:47.540Z] 2024-01-19 00:49:47 +0000:[WARNING] [{CancelAllRunningRestoreJobs}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:49:47.540Z] 2024-01-19 00:49:47 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:49:47.541Z] 2024-01-19 00:49:47 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-58058-01-19-00h44m49s] are [[mysql-data]] [2024-01-19T00:49:47.541Z] 2024-01-19 00:49:47 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T00:49:47.541Z] 2024-01-19 00:49:47 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Validating CR cleanup [2024-01-19T00:49:47.541Z] 2024-01-19 00:49:47 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T00:49:47.541Z] 2024-01-19 00:49:47 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.CreateBackup:#313] - Backup [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-3-1705625236] created successfully [2024-01-19T00:49:47.541Z] 2024-01-19 00:49:47 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Validating backup [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-3-1705625236] in org [default] [2024-01-19T00:49:47.541Z] 2024-01-19 00:49:47 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-3-1705625236] [2024-01-19T00:49:47.541Z] 2024-01-19 00:49:47 +0000:[INFO] [{CancelAllRunningRestoreJobs}] All CRs in [mysql-ibm-pxb-0-58058-01-19-00h44m49s] are [[tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-3-1705625236-0b6cfa4]] [2024-01-19T00:49:47.541Z] INFO[2024-01-19 00:49:47] Verifying : Description : Creation and Validation of backup [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-0-1705625206] of namespace (scheduled Context) [mysql-ibm-pxb-0-58058-01-19-00h44m49s] [2024-01-19T00:49:47.541Z] INFO[2024-01-19 00:49:47] Actual:true, Expected: true [2024-01-19T00:49:47.541Z] 2024-01-19 00:49:47 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58058-01-19-00h44m49s] in backup [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-3-1705625236] [2024-01-19T00:49:47.541Z] 2024-01-19 00:49:47 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58058-01-19-00h44m49s] in backup [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-3-1705625236] [2024-01-19T00:49:47.541Z] 2024-01-19 00:49:47 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58058-01-19-00h44m49s] [2024-01-19T00:49:47.541Z] 2024-01-19 00:49:47 +0000:[WARNING] [{CancelAllRunningRestoreJobs}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:49:47.541Z] 2024-01-19 00:49:47 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:49:47.541Z] 2024-01-19 00:49:47 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-58058-01-19-00h44m49s] are [[mysql-data]] [2024-01-19T00:49:47.541Z] 2024-01-19 00:49:47 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T00:49:47.541Z] 2024-01-19 00:49:47 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Validating CR cleanup [2024-01-19T00:49:47.797Z] 2024-01-19 00:49:47 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T00:49:47.797Z] 2024-01-19 00:49:47 +0000:[INFO] [{CancelAllRunningRestoreJobs}] All CRs in [mysql-ibm-pxb-0-58058-01-19-00h44m49s] are [[]] [2024-01-19T00:49:47.797Z] INFO[2024-01-19 00:49:47] Verifying : Description : Creation and Validation of backup [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-3-1705625236] of namespace (scheduled Context) [mysql-ibm-pxb-0-58058-01-19-00h44m49s] [2024-01-19T00:49:47.797Z] INFO[2024-01-19 00:49:47] Actual:true, Expected: true [2024-01-19T00:49:57.739Z] 2024-01-19 00:49:57 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.CreateBackup:#313] - Backup [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216] created successfully [2024-01-19T00:49:57.739Z] 2024-01-19 00:49:57 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Validating backup [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216] in org [default] [2024-01-19T00:49:57.739Z] 2024-01-19 00:49:57 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216] [2024-01-19T00:49:57.739Z] 2024-01-19 00:49:57 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58058-01-19-00h44m49s] in backup [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216] [2024-01-19T00:49:57.739Z] 2024-01-19 00:49:57 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58058-01-19-00h44m49s] in backup [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216] [2024-01-19T00:49:57.739Z] 2024-01-19 00:49:57 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58058-01-19-00h44m49s] [2024-01-19T00:49:57.739Z] 2024-01-19 00:49:57 +0000:[WARNING] [{CancelAllRunningRestoreJobs}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:49:57.739Z] 2024-01-19 00:49:57 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:49:57.739Z] 2024-01-19 00:49:57 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-58058-01-19-00h44m49s] are [[mysql-data]] [2024-01-19T00:49:57.739Z] 2024-01-19 00:49:57 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T00:49:57.739Z] 2024-01-19 00:49:57 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Validating CR cleanup [2024-01-19T00:49:57.739Z] 2024-01-19 00:49:57 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T00:49:57.739Z] 2024-01-19 00:49:57 +0000:[INFO] [{CancelAllRunningRestoreJobs}] All CRs in [mysql-ibm-pxb-0-58058-01-19-00h44m49s] are [[]] [2024-01-19T00:49:57.739Z] INFO[2024-01-19 00:49:57] Verifying : Description : Creation and Validation of backup [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216] of namespace (scheduled Context) [mysql-ibm-pxb-0-58058-01-19-00h44m49s] [2024-01-19T00:49:57.739Z] INFO[2024-01-19 00:49:57] Actual:true, Expected: true [2024-01-19T00:50:07.691Z] 2024-01-19 00:50:07 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.CreateBackup:#313] - Backup [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-2-1705625226] created successfully [2024-01-19T00:50:07.691Z] 2024-01-19 00:50:07 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Validating backup [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-2-1705625226] in org [default] [2024-01-19T00:50:07.691Z] 2024-01-19 00:50:07 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-2-1705625226] [2024-01-19T00:50:07.691Z] 2024-01-19 00:50:07 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58058-01-19-00h44m49s] in backup [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-2-1705625226] [2024-01-19T00:50:07.691Z] 2024-01-19 00:50:07 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58058-01-19-00h44m49s] in backup [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-2-1705625226] [2024-01-19T00:50:07.691Z] 2024-01-19 00:50:07 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58058-01-19-00h44m49s] [2024-01-19T00:50:07.691Z] 2024-01-19 00:50:07 +0000:[WARNING] [{CancelAllRunningRestoreJobs}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:50:07.691Z] 2024-01-19 00:50:07 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:50:07.691Z] 2024-01-19 00:50:07 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-58058-01-19-00h44m49s] are [[mysql-data]] [2024-01-19T00:50:07.691Z] 2024-01-19 00:50:07 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T00:50:07.691Z] 2024-01-19 00:50:07 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Validating CR cleanup [2024-01-19T00:50:07.691Z] 2024-01-19 00:50:07 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T00:50:07.691Z] 2024-01-19 00:50:07 +0000:[INFO] [{CancelAllRunningRestoreJobs}] All CRs in [mysql-ibm-pxb-0-58058-01-19-00h44m49s] are [[]] [2024-01-19T00:50:07.691Z] INFO[2024-01-19 00:50:07] Verifying : Description : Creation and Validation of backup [tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-2-1705625226] of namespace (scheduled Context) [mysql-ibm-pxb-0-58058-01-19-00h44m49s] [2024-01-19T00:50:07.691Z] INFO[2024-01-19 00:50:07] Actual:true, Expected: true [2024-01-19T00:50:07.692Z] 2024-01-19 00:50:07 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.glob..func54.2.4:#1684] - The list of backups taken are: [] [2024-01-19T00:50:07.692Z] 2024-01-19 00:50:07 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Restoring the backed up applications [2024-01-19T00:50:08.261Z] 2024-01-19 00:50:07 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.CreateRestoreWithoutCheck:#1089] - Getting the UID of the backup needed to be restored [2024-01-19T00:50:08.261Z] 2024-01-19 00:50:07 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.CreateRestoreWithoutCheck:#1089] - Getting the UID of the backup needed to be restored [2024-01-19T00:50:08.261Z] 2024-01-19 00:50:07 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.CreateRestoreWithoutCheck:#1089] - Getting the UID of the backup needed to be restored [2024-01-19T00:50:08.261Z] 2024-01-19 00:50:07 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.CreateRestoreWithoutCheck:#1089] - Getting the UID of the backup needed to be restored [2024-01-19T00:50:08.261Z] 2024-01-19 00:50:07 +0000:[DEBUG] [{CancelAllRunningRestoreJobs}] [backup.CreateRestoreWithoutCheck:#1093] - Enumerate backup response - [2024-01-19T00:50:08.289Z] backups: last_update_time: create_time_in_sec:1705625236 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"74f0578a-05e7-47bf-a46b-457eddceb57b" > backup_path:"mysql-ibm-pxb-0-58058-01-19-00h44m49s/tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-3-1705625236-0b6cfa4/bf3b9a58-b909-4d9f-ba1a-3ea46b0a4572" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-3-1705625236-0b6cfa4" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"bf3b9a58-b909-4d9f-ba1a-3ea46b0a4572" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58058-01-19-00h44m49s" > > backups: last_update_time: create_time_in_sec:1705625226 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"74f0578a-05e7-47bf-a46b-457eddceb57b" > backup_path:"mysql-ibm-pxb-0-58058-01-19-00h44m49s/tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-2-1705625226-d8e782d/1746207f-bb79-4b31-b390-69f7de468dea" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-2-1705625226-d8e782d" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"1746207f-bb79-4b31-b390-69f7de468dea" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58058-01-19-00h44m49s" > > backups: last_update_time: create_time_in_sec:1705625216 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"74f0578a-05e7-47bf-a46b-457eddceb57b" > backup_path:"mysql-ibm-pxb-0-58058-01-19-00h44m49s/tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216-8b4bf56/fdc9d117-f9a9-4949-ad4e-195bf6c9a294" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216-8b4bf56" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"fdc9d117-f9a9-4949-ad4e-195bf6c9a294" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58058-01-19-00h44m49s" > > backups: last_update_time: create_time_in_sec:1705625206 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"74f0578a-05e7-47bf-a46b-457eddceb57b" > backup_path:"mysql-ibm-pxb-0-58058-01-19-00h44m49s/tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-0-1705625206-5a82112/7ac42ad8-7a19-43b8-8585-88ce78f51e4f" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-0-1705625206-5a82112" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"7ac42ad8-7a19-43b8-8585-88ce78f51e4f" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58058-01-19-00h44m49s" > > backups: last_update_time: create_time_in_sec:1705624891 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"d682bac3-b7dc-4a64-8619-522c22901cd2" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c1cfe9bb-8463-4533-9cb7-8178e6c82fde" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"2a0050cf-04d5-45ee-b2af-095697a62abd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"12f69145-19e7-46eb-99da-150260083318" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"7e62d614-582d-43b4-a49e-6de1f54109a0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"46a11c8e-8722-4252-a239-74c9d52fcfd0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"2fd15fb5-96d5-4efe-ba8d-963b018043a0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c2f8e10-3c5f-477a-8a5a-5918addbbbdc" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"72c57770-1e12-4da4-91b7-967895866f13" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bfbe9aa1-d48e-4284-b4ad-7c5d202e7250" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"ddde003a-4407-4f78-bbe9-1457892e6a8b" > stage:Applications backup_schedule: cr_name:"tp-backup-schedule-1705605709-41dbde6-interval-2024-01-19-004131" resource_count:1070 backup_location_ref: cloud_credential_ref: cr_uid:"d4a648d6-e3a3-4303-8f5b-d7bda503d30f" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705623985 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"d682bac3-b7dc-4a64-8619-522c22901cd2" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c1cfe9bb-8463-4533-9cb7-8178e6c82fde" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"12f69145-19e7-46eb-99da-150260083318" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"46a11c8e-8722-4252-a239-74c9d52fcfd0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c2f8e10-3c5f-477a-8a5a-5918addbbbdc" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"72c57770-1e12-4da4-91b7-967895866f13" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bfbe9aa1-d48e-4284-b4ad-7c5d202e7250" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"ddde003a-4407-4f78-bbe9-1457892e6a8b" > backup_path:"kube-system/tp-backup-schedule-1705605709-41dbde6-interval-2024-01-19-002624/ee69de0b-1dbf-47ae-afc4-bd892dea2ce0" stage:Final backup_schedule: cr_name:"tp-backup-schedule-1705605709-41dbde6-interval-2024-01-19-002624" total_size:107374182400 resource_count:1054 backup_location_ref: cloud_credential_ref: cr_uid:"ee69de0b-1dbf-47ae-afc4-bd892dea2ce0" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705623078 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"d682bac3-b7dc-4a64-8619-522c22901cd2" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c1cfe9bb-8463-4533-9cb7-8178e6c82fde" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"12f69145-19e7-46eb-99da-150260083318" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"46a11c8e-8722-4252-a239-74c9d52fcfd0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c2f8e10-3c5f-477a-8a5a-5918addbbbdc" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"72c57770-1e12-4da4-91b7-967895866f13" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bfbe9aa1-d48e-4284-b4ad-7c5d202e7250" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"ddde003a-4407-4f78-bbe9-1457892e6a8b" > backup_path:"kube-system/tp-backup-schedule-1705605709-41dbde6-interval-2024-01-19-001118/ccb6ad20-bf7d-4274-9b54-0e9e5b64ecc2" stage:Final backup_schedule: cr_name:"tp-backup-schedule-1705605709-41dbde6-interval-2024-01-19-001118" total_size:107374182400 resource_count:1054 backup_location_ref: cloud_credential_ref: cr_uid:"ccb6ad20-bf7d-4274-9b54-0e9e5b64ecc2" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705622172 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"d682bac3-b7dc-4a64-8619-522c22901cd2" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c1cfe9bb-8463-4533-9cb7-8178e6c82fde" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"12f69145-19e7-46eb-99da-150260083318" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"46a11c8e-8722-4252-a239-74c9d52fcfd0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c2f8e10-3c5f-477a-8a5a-5918addbbbdc" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"72c57770-1e12-4da4-91b7-967895866f13" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bfbe9aa1-d48e-4284-b4ad-7c5d202e7250" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"ddde003a-4407-4f78-bbe9-1457892e6a8b" > backup_path:"kube-system/tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-235611/5212b02a-5de1-462d-9c39-9d336c1b219e" stage:Final backup_schedule: cr_name:"tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-235611" total_size:107374182400 resource_count:1054 backup_location_ref: cloud_credential_ref: cr_uid:"5212b02a-5de1-462d-9c39-9d336c1b219e" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705621265 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c1cfe9bb-8463-4533-9cb7-8178e6c82fde" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"12f69145-19e7-46eb-99da-150260083318" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"46a11c8e-8722-4252-a239-74c9d52fcfd0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c2f8e10-3c5f-477a-8a5a-5918addbbbdc" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"72c57770-1e12-4da4-91b7-967895866f13" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bfbe9aa1-d48e-4284-b4ad-7c5d202e7250" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"ddde003a-4407-4f78-bbe9-1457892e6a8b" > backup_path:"kube-system/tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-234104/77012505-473e-44f9-ba96-d3bfc34afdc9" stage:Final backup_schedule: cr_name:"tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-234104" total_size:96636764160 resource_count:1050 backup_location_ref: cloud_credential_ref: cr_uid:"77012505-473e-44f9-ba96-d3bfc34afdc9" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705620358 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c1cfe9bb-8463-4533-9cb7-8178e6c82fde" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"12f69145-19e7-46eb-99da-150260083318" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"46a11c8e-8722-4252-a239-74c9d52fcfd0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c2f8e10-3c5f-477a-8a5a-5918addbbbdc" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"72c57770-1e12-4da4-91b7-967895866f13" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bfbe9aa1-d48e-4284-b4ad-7c5d202e7250" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"ddde003a-4407-4f78-bbe9-1457892e6a8b" > backup_path:"kube-system/tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-232558/18e7a184-138c-42c4-b475-4b4a8fc7ec23" stage:Final backup_schedule: cr_name:"tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-232558" total_size:96636764160 resource_count:1050 backup_location_ref: cloud_credential_ref: cr_uid:"18e7a184-138c-42c4-b475-4b4a8fc7ec23" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705608695 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"3533cf38-8bcc-45fc-bee6-b8bb4d41c70f" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"d7d7cb90-2508-46c8-807c-098ae6519ec2" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"257b09ac-f456-49a3-b554-d1a6b955a67a" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"beacc472-7dad-4629-9976-00fb76efdb0c" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"6a12597a-9251-44de-8b14-d45388db6303" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"3b2558ec-36dc-473d-bcd4-5a8ef6073298" > backup_path:"kube-system/tp-backup-1705608575-5486a15/df9dc151-38f3-4d61-8e59-b8435003bcdc" stage:Final cr_name:"tp-backup-1705608575-5486a15" total_size:64424509440 resource_count:30 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"df9dc151-38f3-4d61-8e59-b8435003bcdc" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705581755 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"95eda04e-e6b8-45ef-ad93-86b879759a29" > backup_path:"mysql-ibm-pxb-1-58051-01-18-12h40m37s/tp-backup-mysql-ibm-pxb-1-58051-01-18-12h40m37s-mkmce-ed02fc4/fe3a32bd-dcc4-4413-8892-441f4633ab25" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-1-58051-01-18-12h40m37s-mkmce-ed02fc4" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"fe3a32bd-dcc4-4413-8892-441f4633ab25" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-1-58051-01-18-12h40m37s" > > backups: last_update_time: create_time_in_sec:1705581755 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"1f21aca3-0b40-4991-8207-7fc618bb624d" > backup_path:"mysql-ibm-pxb-3-58051-01-18-12h40m37s/tp-backup-mysql-ibm-pxb-3-58051-01-18-12h40m37s-otfyc-9ab1f7b/947706f8-1325-407e-a9bf-13e160054d3f" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-3-58051-01-18-12h40m37s-otfyc-9ab1f7b" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"947706f8-1325-407e-a9bf-13e160054d3f" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-3-58051-01-18-12h40m37s" > > backups: last_update_time: create_time_in_sec:1705581755 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"1374e8d7-6696-4f76-99e3-39ffd108582a" > backup_path:"mysql-ibm-pxb-2-58051-01-18-12h40m37s/tp-backup-mysql-ibm-pxb-2-58051-01-18-12h40m37s-hhxop-2106116/7b6f9711-36e2-4cb0-9cb3-a40515399d07" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-2-58051-01-18-12h40m37s-hhxop-2106116" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"7b6f9711-36e2-4cb0-9cb3-a40515399d07" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-2-58051-01-18-12h40m37s" > > backups: last_update_time: create_time_in_sec:1705581755 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"640dd5b5-65a6-4113-98fa-311bd2423b07" > backup_path:"mysql-ibm-pxb-4-58051-01-18-12h40m37s/tp-backup-mysql-ibm-pxb-4-58051-01-18-12h40m37s-ltnvx-230d407/130fd826-2bee-4842-b4de-6cfde09f56bb" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-4-58051-01-18-12h40m37s-ltnvx-230d407" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"130fd826-2bee-4842-b4de-6cfde09f56bb" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-4-58051-01-18-12h40m37s" > > backups: last_update_time: create_time_in_sec:1705581755 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c16981a5-7cb4-45f2-96d3-48daba527664" > backup_path:"mysql-ibm-pxb-0-58051-01-18-12h40m37s/tp-backup-mysql-ibm-pxb-0-58051-01-18-12h40m37s-gllfn-3e3f1a0/f37ce93a-d3c4-49a0-8dd9-a0fa2d5e4f0e" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58051-01-18-12h40m37s-gllfn-3e3f1a0" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"f37ce93a-d3c4-49a0-8dd9-a0fa2d5e4f0e" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58051-01-18-12h40m37s" > > backups: last_update_time: create_time_in_sec:1705577915 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"314c2ef4-b6b9-41bd-997a-c1c64541d4c3" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"f68d9981-3579-4196-89cd-b3c72121c24e" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"a1171d90-aa6f-400e-8bfd-f1df926bd4c8" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > backup_path:"kube-system/tp-backup-all-1705577915-b24d80f/9bd7f0db-8d79-405e-ad15-67d682eb3468" stage:Final cr_name:"tp-backup-all-1705577915-b24d80f" total_size:53687091200 resource_count:1160 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"9bd7f0db-8d79-405e-ad15-67d682eb3468" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > total_count:17 [2024-01-19T00:50:08.289Z] 2024-01-19 00:50:08 +0000:[DEBUG] [{CancelAllRunningRestoreJobs}] [backup.CreateRestoreWithoutCheck:#1093] - Enumerate backup response - [2024-01-19T00:50:08.315Z] backups: last_update_time: create_time_in_sec:1705625236 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"74f0578a-05e7-47bf-a46b-457eddceb57b" > backup_path:"mysql-ibm-pxb-0-58058-01-19-00h44m49s/tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-3-1705625236-0b6cfa4/bf3b9a58-b909-4d9f-ba1a-3ea46b0a4572" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-3-1705625236-0b6cfa4" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"bf3b9a58-b909-4d9f-ba1a-3ea46b0a4572" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58058-01-19-00h44m49s" > > backups: last_update_time: create_time_in_sec:1705625226 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"74f0578a-05e7-47bf-a46b-457eddceb57b" > backup_path:"mysql-ibm-pxb-0-58058-01-19-00h44m49s/tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-2-1705625226-d8e782d/1746207f-bb79-4b31-b390-69f7de468dea" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-2-1705625226-d8e782d" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"1746207f-bb79-4b31-b390-69f7de468dea" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58058-01-19-00h44m49s" > > backups: last_update_time: create_time_in_sec:1705625216 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"74f0578a-05e7-47bf-a46b-457eddceb57b" > backup_path:"mysql-ibm-pxb-0-58058-01-19-00h44m49s/tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216-8b4bf56/fdc9d117-f9a9-4949-ad4e-195bf6c9a294" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216-8b4bf56" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"fdc9d117-f9a9-4949-ad4e-195bf6c9a294" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58058-01-19-00h44m49s" > > backups: last_update_time: create_time_in_sec:1705625206 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"74f0578a-05e7-47bf-a46b-457eddceb57b" > backup_path:"mysql-ibm-pxb-0-58058-01-19-00h44m49s/tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-0-1705625206-5a82112/7ac42ad8-7a19-43b8-8585-88ce78f51e4f" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-0-1705625206-5a82112" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"7ac42ad8-7a19-43b8-8585-88ce78f51e4f" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58058-01-19-00h44m49s" > > backups: last_update_time: create_time_in_sec:1705624891 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"d682bac3-b7dc-4a64-8619-522c22901cd2" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c1cfe9bb-8463-4533-9cb7-8178e6c82fde" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"2a0050cf-04d5-45ee-b2af-095697a62abd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"12f69145-19e7-46eb-99da-150260083318" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"7e62d614-582d-43b4-a49e-6de1f54109a0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"46a11c8e-8722-4252-a239-74c9d52fcfd0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"2fd15fb5-96d5-4efe-ba8d-963b018043a0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c2f8e10-3c5f-477a-8a5a-5918addbbbdc" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"72c57770-1e12-4da4-91b7-967895866f13" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bfbe9aa1-d48e-4284-b4ad-7c5d202e7250" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"ddde003a-4407-4f78-bbe9-1457892e6a8b" > stage:Applications backup_schedule: cr_name:"tp-backup-schedule-1705605709-41dbde6-interval-2024-01-19-004131" resource_count:1070 backup_location_ref: cloud_credential_ref: cr_uid:"d4a648d6-e3a3-4303-8f5b-d7bda503d30f" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705623985 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"d682bac3-b7dc-4a64-8619-522c22901cd2" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c1cfe9bb-8463-4533-9cb7-8178e6c82fde" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"12f69145-19e7-46eb-99da-150260083318" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"46a11c8e-8722-4252-a239-74c9d52fcfd0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c2f8e10-3c5f-477a-8a5a-5918addbbbdc" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"72c57770-1e12-4da4-91b7-967895866f13" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bfbe9aa1-d48e-4284-b4ad-7c5d202e7250" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"ddde003a-4407-4f78-bbe9-1457892e6a8b" > backup_path:"kube-system/tp-backup-schedule-1705605709-41dbde6-interval-2024-01-19-002624/ee69de0b-1dbf-47ae-afc4-bd892dea2ce0" stage:Final backup_schedule: cr_name:"tp-backup-schedule-1705605709-41dbde6-interval-2024-01-19-002624" total_size:107374182400 resource_count:1054 backup_location_ref: cloud_credential_ref: cr_uid:"ee69de0b-1dbf-47ae-afc4-bd892dea2ce0" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705623078 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"d682bac3-b7dc-4a64-8619-522c22901cd2" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c1cfe9bb-8463-4533-9cb7-8178e6c82fde" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"12f69145-19e7-46eb-99da-150260083318" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"46a11c8e-8722-4252-a239-74c9d52fcfd0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c2f8e10-3c5f-477a-8a5a-5918addbbbdc" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"72c57770-1e12-4da4-91b7-967895866f13" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bfbe9aa1-d48e-4284-b4ad-7c5d202e7250" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"ddde003a-4407-4f78-bbe9-1457892e6a8b" > backup_path:"kube-system/tp-backup-schedule-1705605709-41dbde6-interval-2024-01-19-001118/ccb6ad20-bf7d-4274-9b54-0e9e5b64ecc2" stage:Final backup_schedule: cr_name:"tp-backup-schedule-1705605709-41dbde6-interval-2024-01-19-001118" total_size:107374182400 resource_count:1054 backup_location_ref: cloud_credential_ref: cr_uid:"ccb6ad20-bf7d-4274-9b54-0e9e5b64ecc2" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705622172 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"d682bac3-b7dc-4a64-8619-522c22901cd2" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c1cfe9bb-8463-4533-9cb7-8178e6c82fde" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"12f69145-19e7-46eb-99da-150260083318" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"46a11c8e-8722-4252-a239-74c9d52fcfd0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c2f8e10-3c5f-477a-8a5a-5918addbbbdc" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"72c57770-1e12-4da4-91b7-967895866f13" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bfbe9aa1-d48e-4284-b4ad-7c5d202e7250" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"ddde003a-4407-4f78-bbe9-1457892e6a8b" > backup_path:"kube-system/tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-235611/5212b02a-5de1-462d-9c39-9d336c1b219e" stage:Final backup_schedule: cr_name:"tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-235611" total_size:107374182400 resource_count:1054 backup_location_ref: cloud_credential_ref: cr_uid:"5212b02a-5de1-462d-9c39-9d336c1b219e" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705621265 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c1cfe9bb-8463-4533-9cb7-8178e6c82fde" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"12f69145-19e7-46eb-99da-150260083318" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"46a11c8e-8722-4252-a239-74c9d52fcfd0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c2f8e10-3c5f-477a-8a5a-5918addbbbdc" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"72c57770-1e12-4da4-91b7-967895866f13" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bfbe9aa1-d48e-4284-b4ad-7c5d202e7250" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"ddde003a-4407-4f78-bbe9-1457892e6a8b" > backup_path:"kube-system/tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-234104/77012505-473e-44f9-ba96-d3bfc34afdc9" stage:Final backup_schedule: cr_name:"tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-234104" total_size:96636764160 resource_count:1050 backup_location_ref: cloud_credential_ref: cr_uid:"77012505-473e-44f9-ba96-d3bfc34afdc9" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705620358 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c1cfe9bb-8463-4533-9cb7-8178e6c82fde" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"12f69145-19e7-46eb-99da-150260083318" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"46a11c8e-8722-4252-a239-74c9d52fcfd0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c2f8e10-3c5f-477a-8a5a-5918addbbbdc" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"72c57770-1e12-4da4-91b7-967895866f13" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bfbe9aa1-d48e-4284-b4ad-7c5d202e7250" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"ddde003a-4407-4f78-bbe9-1457892e6a8b" > backup_path:"kube-system/tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-232558/18e7a184-138c-42c4-b475-4b4a8fc7ec23" stage:Final backup_schedule: cr_name:"tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-232558" total_size:96636764160 resource_count:1050 backup_location_ref: cloud_credential_ref: cr_uid:"18e7a184-138c-42c4-b475-4b4a8fc7ec23" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705608695 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"3533cf38-8bcc-45fc-bee6-b8bb4d41c70f" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"d7d7cb90-2508-46c8-807c-098ae6519ec2" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"257b09ac-f456-49a3-b554-d1a6b955a67a" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"beacc472-7dad-4629-9976-00fb76efdb0c" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"6a12597a-9251-44de-8b14-d45388db6303" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"3b2558ec-36dc-473d-bcd4-5a8ef6073298" > backup_path:"kube-system/tp-backup-1705608575-5486a15/df9dc151-38f3-4d61-8e59-b8435003bcdc" stage:Final cr_name:"tp-backup-1705608575-5486a15" total_size:64424509440 resource_count:30 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"df9dc151-38f3-4d61-8e59-b8435003bcdc" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705581755 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"95eda04e-e6b8-45ef-ad93-86b879759a29" > backup_path:"mysql-ibm-pxb-1-58051-01-18-12h40m37s/tp-backup-mysql-ibm-pxb-1-58051-01-18-12h40m37s-mkmce-ed02fc4/fe3a32bd-dcc4-4413-8892-441f4633ab25" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-1-58051-01-18-12h40m37s-mkmce-ed02fc4" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"fe3a32bd-dcc4-4413-8892-441f4633ab25" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-1-58051-01-18-12h40m37s" > > backups: last_update_time: create_time_in_sec:1705581755 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"1f21aca3-0b40-4991-8207-7fc618bb624d" > backup_path:"mysql-ibm-pxb-3-58051-01-18-12h40m37s/tp-backup-mysql-ibm-pxb-3-58051-01-18-12h40m37s-otfyc-9ab1f7b/947706f8-1325-407e-a9bf-13e160054d3f" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-3-58051-01-18-12h40m37s-otfyc-9ab1f7b" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"947706f8-1325-407e-a9bf-13e160054d3f" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-3-58051-01-18-12h40m37s" > > backups: last_update_time: create_time_in_sec:1705581755 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"1374e8d7-6696-4f76-99e3-39ffd108582a" > backup_path:"mysql-ibm-pxb-2-58051-01-18-12h40m37s/tp-backup-mysql-ibm-pxb-2-58051-01-18-12h40m37s-hhxop-2106116/7b6f9711-36e2-4cb0-9cb3-a40515399d07" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-2-58051-01-18-12h40m37s-hhxop-2106116" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"7b6f9711-36e2-4cb0-9cb3-a40515399d07" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-2-58051-01-18-12h40m37s" > > backups: last_update_time: create_time_in_sec:1705581755 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"640dd5b5-65a6-4113-98fa-311bd2423b07" > backup_path:"mysql-ibm-pxb-4-58051-01-18-12h40m37s/tp-backup-mysql-ibm-pxb-4-58051-01-18-12h40m37s-ltnvx-230d407/130fd826-2bee-4842-b4de-6cfde09f56bb" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-4-58051-01-18-12h40m37s-ltnvx-230d407" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"130fd826-2bee-4842-b4de-6cfde09f56bb" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-4-58051-01-18-12h40m37s" > > backups: last_update_time: create_time_in_sec:1705581755 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c16981a5-7cb4-45f2-96d3-48daba527664" > backup_path:"mysql-ibm-pxb-0-58051-01-18-12h40m37s/tp-backup-mysql-ibm-pxb-0-58051-01-18-12h40m37s-gllfn-3e3f1a0/f37ce93a-d3c4-49a0-8dd9-a0fa2d5e4f0e" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58051-01-18-12h40m37s-gllfn-3e3f1a0" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"f37ce93a-d3c4-49a0-8dd9-a0fa2d5e4f0e" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58051-01-18-12h40m37s" > > backups: last_update_time: create_time_in_sec:1705577915 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"314c2ef4-b6b9-41bd-997a-c1c64541d4c3" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"f68d9981-3579-4196-89cd-b3c72121c24e" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"a1171d90-aa6f-400e-8bfd-f1df926bd4c8" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > backup_path:"kube-system/tp-backup-all-1705577915-b24d80f/9bd7f0db-8d79-405e-ad15-67d682eb3468" stage:Final cr_name:"tp-backup-all-1705577915-b24d80f" total_size:53687091200 resource_count:1160 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"9bd7f0db-8d79-405e-ad15-67d682eb3468" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > total_count:17 [2024-01-19T00:50:08.315Z] 2024-01-19 00:50:08 +0000:[DEBUG] [{CancelAllRunningRestoreJobs}] [backup.CreateRestoreWithoutCheck:#1093] - Enumerate backup response - [2024-01-19T00:50:08.341Z] backups: last_update_time: create_time_in_sec:1705625236 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"74f0578a-05e7-47bf-a46b-457eddceb57b" > backup_path:"mysql-ibm-pxb-0-58058-01-19-00h44m49s/tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-3-1705625236-0b6cfa4/bf3b9a58-b909-4d9f-ba1a-3ea46b0a4572" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-3-1705625236-0b6cfa4" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"bf3b9a58-b909-4d9f-ba1a-3ea46b0a4572" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58058-01-19-00h44m49s" > > backups: last_update_time: create_time_in_sec:1705625226 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"74f0578a-05e7-47bf-a46b-457eddceb57b" > backup_path:"mysql-ibm-pxb-0-58058-01-19-00h44m49s/tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-2-1705625226-d8e782d/1746207f-bb79-4b31-b390-69f7de468dea" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-2-1705625226-d8e782d" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"1746207f-bb79-4b31-b390-69f7de468dea" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58058-01-19-00h44m49s" > > backups: last_update_time: create_time_in_sec:1705625216 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"74f0578a-05e7-47bf-a46b-457eddceb57b" > backup_path:"mysql-ibm-pxb-0-58058-01-19-00h44m49s/tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216-8b4bf56/fdc9d117-f9a9-4949-ad4e-195bf6c9a294" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216-8b4bf56" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"fdc9d117-f9a9-4949-ad4e-195bf6c9a294" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58058-01-19-00h44m49s" > > backups: last_update_time: create_time_in_sec:1705625206 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"74f0578a-05e7-47bf-a46b-457eddceb57b" > backup_path:"mysql-ibm-pxb-0-58058-01-19-00h44m49s/tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-0-1705625206-5a82112/7ac42ad8-7a19-43b8-8585-88ce78f51e4f" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-0-1705625206-5a82112" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"7ac42ad8-7a19-43b8-8585-88ce78f51e4f" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58058-01-19-00h44m49s" > > backups: last_update_time: create_time_in_sec:1705624891 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"d682bac3-b7dc-4a64-8619-522c22901cd2" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c1cfe9bb-8463-4533-9cb7-8178e6c82fde" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"2a0050cf-04d5-45ee-b2af-095697a62abd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"12f69145-19e7-46eb-99da-150260083318" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"7e62d614-582d-43b4-a49e-6de1f54109a0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"46a11c8e-8722-4252-a239-74c9d52fcfd0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"2fd15fb5-96d5-4efe-ba8d-963b018043a0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c2f8e10-3c5f-477a-8a5a-5918addbbbdc" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"72c57770-1e12-4da4-91b7-967895866f13" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bfbe9aa1-d48e-4284-b4ad-7c5d202e7250" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"ddde003a-4407-4f78-bbe9-1457892e6a8b" > stage:Applications backup_schedule: cr_name:"tp-backup-schedule-1705605709-41dbde6-interval-2024-01-19-004131" resource_count:1070 backup_location_ref: cloud_credential_ref: cr_uid:"d4a648d6-e3a3-4303-8f5b-d7bda503d30f" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705623985 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"d682bac3-b7dc-4a64-8619-522c22901cd2" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c1cfe9bb-8463-4533-9cb7-8178e6c82fde" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"12f69145-19e7-46eb-99da-150260083318" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"46a11c8e-8722-4252-a239-74c9d52fcfd0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c2f8e10-3c5f-477a-8a5a-5918addbbbdc" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"72c57770-1e12-4da4-91b7-967895866f13" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bfbe9aa1-d48e-4284-b4ad-7c5d202e7250" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"ddde003a-4407-4f78-bbe9-1457892e6a8b" > backup_path:"kube-system/tp-backup-schedule-1705605709-41dbde6-interval-2024-01-19-002624/ee69de0b-1dbf-47ae-afc4-bd892dea2ce0" stage:Final backup_schedule: cr_name:"tp-backup-schedule-1705605709-41dbde6-interval-2024-01-19-002624" total_size:107374182400 resource_count:1054 backup_location_ref: cloud_credential_ref: cr_uid:"ee69de0b-1dbf-47ae-afc4-bd892dea2ce0" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705623078 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"d682bac3-b7dc-4a64-8619-522c22901cd2" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c1cfe9bb-8463-4533-9cb7-8178e6c82fde" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"12f69145-19e7-46eb-99da-150260083318" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"46a11c8e-8722-4252-a239-74c9d52fcfd0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c2f8e10-3c5f-477a-8a5a-5918addbbbdc" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"72c57770-1e12-4da4-91b7-967895866f13" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bfbe9aa1-d48e-4284-b4ad-7c5d202e7250" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"ddde003a-4407-4f78-bbe9-1457892e6a8b" > backup_path:"kube-system/tp-backup-schedule-1705605709-41dbde6-interval-2024-01-19-001118/ccb6ad20-bf7d-4274-9b54-0e9e5b64ecc2" stage:Final backup_schedule: cr_name:"tp-backup-schedule-1705605709-41dbde6-interval-2024-01-19-001118" total_size:107374182400 resource_count:1054 backup_location_ref: cloud_credential_ref: cr_uid:"ccb6ad20-bf7d-4274-9b54-0e9e5b64ecc2" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705622172 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"d682bac3-b7dc-4a64-8619-522c22901cd2" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c1cfe9bb-8463-4533-9cb7-8178e6c82fde" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"12f69145-19e7-46eb-99da-150260083318" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"46a11c8e-8722-4252-a239-74c9d52fcfd0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c2f8e10-3c5f-477a-8a5a-5918addbbbdc" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"72c57770-1e12-4da4-91b7-967895866f13" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bfbe9aa1-d48e-4284-b4ad-7c5d202e7250" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"ddde003a-4407-4f78-bbe9-1457892e6a8b" > backup_path:"kube-system/tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-235611/5212b02a-5de1-462d-9c39-9d336c1b219e" stage:Final backup_schedule: cr_name:"tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-235611" total_size:107374182400 resource_count:1054 backup_location_ref: cloud_credential_ref: cr_uid:"5212b02a-5de1-462d-9c39-9d336c1b219e" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705621265 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c1cfe9bb-8463-4533-9cb7-8178e6c82fde" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"12f69145-19e7-46eb-99da-150260083318" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"46a11c8e-8722-4252-a239-74c9d52fcfd0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c2f8e10-3c5f-477a-8a5a-5918addbbbdc" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"72c57770-1e12-4da4-91b7-967895866f13" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bfbe9aa1-d48e-4284-b4ad-7c5d202e7250" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"ddde003a-4407-4f78-bbe9-1457892e6a8b" > backup_path:"kube-system/tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-234104/77012505-473e-44f9-ba96-d3bfc34afdc9" stage:Final backup_schedule: cr_name:"tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-234104" total_size:96636764160 resource_count:1050 backup_location_ref: cloud_credential_ref: cr_uid:"77012505-473e-44f9-ba96-d3bfc34afdc9" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705620358 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c1cfe9bb-8463-4533-9cb7-8178e6c82fde" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"12f69145-19e7-46eb-99da-150260083318" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"46a11c8e-8722-4252-a239-74c9d52fcfd0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c2f8e10-3c5f-477a-8a5a-5918addbbbdc" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"72c57770-1e12-4da4-91b7-967895866f13" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bfbe9aa1-d48e-4284-b4ad-7c5d202e7250" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"ddde003a-4407-4f78-bbe9-1457892e6a8b" > backup_path:"kube-system/tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-232558/18e7a184-138c-42c4-b475-4b4a8fc7ec23" stage:Final backup_schedule: cr_name:"tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-232558" total_size:96636764160 resource_count:1050 backup_location_ref: cloud_credential_ref: cr_uid:"18e7a184-138c-42c4-b475-4b4a8fc7ec23" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705608695 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"3533cf38-8bcc-45fc-bee6-b8bb4d41c70f" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"d7d7cb90-2508-46c8-807c-098ae6519ec2" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"257b09ac-f456-49a3-b554-d1a6b955a67a" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"beacc472-7dad-4629-9976-00fb76efdb0c" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"6a12597a-9251-44de-8b14-d45388db6303" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"3b2558ec-36dc-473d-bcd4-5a8ef6073298" > backup_path:"kube-system/tp-backup-1705608575-5486a15/df9dc151-38f3-4d61-8e59-b8435003bcdc" stage:Final cr_name:"tp-backup-1705608575-5486a15" total_size:64424509440 resource_count:30 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"df9dc151-38f3-4d61-8e59-b8435003bcdc" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705581755 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"95eda04e-e6b8-45ef-ad93-86b879759a29" > backup_path:"mysql-ibm-pxb-1-58051-01-18-12h40m37s/tp-backup-mysql-ibm-pxb-1-58051-01-18-12h40m37s-mkmce-ed02fc4/fe3a32bd-dcc4-4413-8892-441f4633ab25" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-1-58051-01-18-12h40m37s-mkmce-ed02fc4" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"fe3a32bd-dcc4-4413-8892-441f4633ab25" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-1-58051-01-18-12h40m37s" > > backups: last_update_time: create_time_in_sec:1705581755 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"1f21aca3-0b40-4991-8207-7fc618bb624d" > backup_path:"mysql-ibm-pxb-3-58051-01-18-12h40m37s/tp-backup-mysql-ibm-pxb-3-58051-01-18-12h40m37s-otfyc-9ab1f7b/947706f8-1325-407e-a9bf-13e160054d3f" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-3-58051-01-18-12h40m37s-otfyc-9ab1f7b" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"947706f8-1325-407e-a9bf-13e160054d3f" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-3-58051-01-18-12h40m37s" > > backups: last_update_time: create_time_in_sec:1705581755 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"1374e8d7-6696-4f76-99e3-39ffd108582a" > backup_path:"mysql-ibm-pxb-2-58051-01-18-12h40m37s/tp-backup-mysql-ibm-pxb-2-58051-01-18-12h40m37s-hhxop-2106116/7b6f9711-36e2-4cb0-9cb3-a40515399d07" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-2-58051-01-18-12h40m37s-hhxop-2106116" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"7b6f9711-36e2-4cb0-9cb3-a40515399d07" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-2-58051-01-18-12h40m37s" > > backups: last_update_time: create_time_in_sec:1705581755 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"640dd5b5-65a6-4113-98fa-311bd2423b07" > backup_path:"mysql-ibm-pxb-4-58051-01-18-12h40m37s/tp-backup-mysql-ibm-pxb-4-58051-01-18-12h40m37s-ltnvx-230d407/130fd826-2bee-4842-b4de-6cfde09f56bb" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-4-58051-01-18-12h40m37s-ltnvx-230d407" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"130fd826-2bee-4842-b4de-6cfde09f56bb" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-4-58051-01-18-12h40m37s" > > backups: last_update_time: create_time_in_sec:1705581755 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c16981a5-7cb4-45f2-96d3-48daba527664" > backup_path:"mysql-ibm-pxb-0-58051-01-18-12h40m37s/tp-backup-mysql-ibm-pxb-0-58051-01-18-12h40m37s-gllfn-3e3f1a0/f37ce93a-d3c4-49a0-8dd9-a0fa2d5e4f0e" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58051-01-18-12h40m37s-gllfn-3e3f1a0" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"f37ce93a-d3c4-49a0-8dd9-a0fa2d5e4f0e" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58051-01-18-12h40m37s" > > backups: last_update_time: create_time_in_sec:1705577915 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"314c2ef4-b6b9-41bd-997a-c1c64541d4c3" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"f68d9981-3579-4196-89cd-b3c72121c24e" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"a1171d90-aa6f-400e-8bfd-f1df926bd4c8" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > backup_path:"kube-system/tp-backup-all-1705577915-b24d80f/9bd7f0db-8d79-405e-ad15-67d682eb3468" stage:Final cr_name:"tp-backup-all-1705577915-b24d80f" total_size:53687091200 resource_count:1160 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"9bd7f0db-8d79-405e-ad15-67d682eb3468" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > total_count:17 [2024-01-19T00:50:08.341Z] 2024-01-19 00:50:08 +0000:[DEBUG] [{CancelAllRunningRestoreJobs}] [backup.CreateRestoreWithoutCheck:#1093] - Enumerate backup response - [2024-01-19T00:50:08.368Z] backups: last_update_time: create_time_in_sec:1705625236 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"74f0578a-05e7-47bf-a46b-457eddceb57b" > backup_path:"mysql-ibm-pxb-0-58058-01-19-00h44m49s/tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-3-1705625236-0b6cfa4/bf3b9a58-b909-4d9f-ba1a-3ea46b0a4572" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-3-1705625236-0b6cfa4" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"bf3b9a58-b909-4d9f-ba1a-3ea46b0a4572" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58058-01-19-00h44m49s" > > backups: last_update_time: create_time_in_sec:1705625226 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"74f0578a-05e7-47bf-a46b-457eddceb57b" > backup_path:"mysql-ibm-pxb-0-58058-01-19-00h44m49s/tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-2-1705625226-d8e782d/1746207f-bb79-4b31-b390-69f7de468dea" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-2-1705625226-d8e782d" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"1746207f-bb79-4b31-b390-69f7de468dea" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58058-01-19-00h44m49s" > > backups: last_update_time: create_time_in_sec:1705625216 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"74f0578a-05e7-47bf-a46b-457eddceb57b" > backup_path:"mysql-ibm-pxb-0-58058-01-19-00h44m49s/tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216-8b4bf56/fdc9d117-f9a9-4949-ad4e-195bf6c9a294" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216-8b4bf56" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"fdc9d117-f9a9-4949-ad4e-195bf6c9a294" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58058-01-19-00h44m49s" > > backups: last_update_time: create_time_in_sec:1705625206 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"74f0578a-05e7-47bf-a46b-457eddceb57b" > backup_path:"mysql-ibm-pxb-0-58058-01-19-00h44m49s/tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-0-1705625206-5a82112/7ac42ad8-7a19-43b8-8585-88ce78f51e4f" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-0-1705625206-5a82112" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"7ac42ad8-7a19-43b8-8585-88ce78f51e4f" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58058-01-19-00h44m49s" > > backups: last_update_time: create_time_in_sec:1705624891 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"d682bac3-b7dc-4a64-8619-522c22901cd2" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c1cfe9bb-8463-4533-9cb7-8178e6c82fde" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"2a0050cf-04d5-45ee-b2af-095697a62abd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"12f69145-19e7-46eb-99da-150260083318" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"7e62d614-582d-43b4-a49e-6de1f54109a0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"46a11c8e-8722-4252-a239-74c9d52fcfd0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"2fd15fb5-96d5-4efe-ba8d-963b018043a0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c2f8e10-3c5f-477a-8a5a-5918addbbbdc" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"72c57770-1e12-4da4-91b7-967895866f13" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bfbe9aa1-d48e-4284-b4ad-7c5d202e7250" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"ddde003a-4407-4f78-bbe9-1457892e6a8b" > stage:Applications backup_schedule: cr_name:"tp-backup-schedule-1705605709-41dbde6-interval-2024-01-19-004131" resource_count:1070 backup_location_ref: cloud_credential_ref: cr_uid:"d4a648d6-e3a3-4303-8f5b-d7bda503d30f" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705623985 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"d682bac3-b7dc-4a64-8619-522c22901cd2" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c1cfe9bb-8463-4533-9cb7-8178e6c82fde" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"12f69145-19e7-46eb-99da-150260083318" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"46a11c8e-8722-4252-a239-74c9d52fcfd0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c2f8e10-3c5f-477a-8a5a-5918addbbbdc" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"72c57770-1e12-4da4-91b7-967895866f13" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bfbe9aa1-d48e-4284-b4ad-7c5d202e7250" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"ddde003a-4407-4f78-bbe9-1457892e6a8b" > backup_path:"kube-system/tp-backup-schedule-1705605709-41dbde6-interval-2024-01-19-002624/ee69de0b-1dbf-47ae-afc4-bd892dea2ce0" stage:Final backup_schedule: cr_name:"tp-backup-schedule-1705605709-41dbde6-interval-2024-01-19-002624" total_size:107374182400 resource_count:1054 backup_location_ref: cloud_credential_ref: cr_uid:"ee69de0b-1dbf-47ae-afc4-bd892dea2ce0" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705623078 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"d682bac3-b7dc-4a64-8619-522c22901cd2" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c1cfe9bb-8463-4533-9cb7-8178e6c82fde" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"12f69145-19e7-46eb-99da-150260083318" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"46a11c8e-8722-4252-a239-74c9d52fcfd0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c2f8e10-3c5f-477a-8a5a-5918addbbbdc" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"72c57770-1e12-4da4-91b7-967895866f13" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bfbe9aa1-d48e-4284-b4ad-7c5d202e7250" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"ddde003a-4407-4f78-bbe9-1457892e6a8b" > backup_path:"kube-system/tp-backup-schedule-1705605709-41dbde6-interval-2024-01-19-001118/ccb6ad20-bf7d-4274-9b54-0e9e5b64ecc2" stage:Final backup_schedule: cr_name:"tp-backup-schedule-1705605709-41dbde6-interval-2024-01-19-001118" total_size:107374182400 resource_count:1054 backup_location_ref: cloud_credential_ref: cr_uid:"ccb6ad20-bf7d-4274-9b54-0e9e5b64ecc2" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705622172 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"d682bac3-b7dc-4a64-8619-522c22901cd2" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c1cfe9bb-8463-4533-9cb7-8178e6c82fde" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"12f69145-19e7-46eb-99da-150260083318" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"46a11c8e-8722-4252-a239-74c9d52fcfd0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c2f8e10-3c5f-477a-8a5a-5918addbbbdc" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"72c57770-1e12-4da4-91b7-967895866f13" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bfbe9aa1-d48e-4284-b4ad-7c5d202e7250" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"ddde003a-4407-4f78-bbe9-1457892e6a8b" > backup_path:"kube-system/tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-235611/5212b02a-5de1-462d-9c39-9d336c1b219e" stage:Final backup_schedule: cr_name:"tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-235611" total_size:107374182400 resource_count:1054 backup_location_ref: cloud_credential_ref: cr_uid:"5212b02a-5de1-462d-9c39-9d336c1b219e" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705621265 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c1cfe9bb-8463-4533-9cb7-8178e6c82fde" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"12f69145-19e7-46eb-99da-150260083318" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"46a11c8e-8722-4252-a239-74c9d52fcfd0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c2f8e10-3c5f-477a-8a5a-5918addbbbdc" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"72c57770-1e12-4da4-91b7-967895866f13" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bfbe9aa1-d48e-4284-b4ad-7c5d202e7250" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"ddde003a-4407-4f78-bbe9-1457892e6a8b" > backup_path:"kube-system/tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-234104/77012505-473e-44f9-ba96-d3bfc34afdc9" stage:Final backup_schedule: cr_name:"tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-234104" total_size:96636764160 resource_count:1050 backup_location_ref: cloud_credential_ref: cr_uid:"77012505-473e-44f9-ba96-d3bfc34afdc9" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705620358 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c1cfe9bb-8463-4533-9cb7-8178e6c82fde" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"12f69145-19e7-46eb-99da-150260083318" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"46a11c8e-8722-4252-a239-74c9d52fcfd0" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c2f8e10-3c5f-477a-8a5a-5918addbbbdc" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"72c57770-1e12-4da4-91b7-967895866f13" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bfbe9aa1-d48e-4284-b4ad-7c5d202e7250" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"ddde003a-4407-4f78-bbe9-1457892e6a8b" > backup_path:"kube-system/tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-232558/18e7a184-138c-42c4-b475-4b4a8fc7ec23" stage:Final backup_schedule: cr_name:"tp-backup-schedule-1705605709-41dbde6-interval-2024-01-18-232558" total_size:96636764160 resource_count:1050 backup_location_ref: cloud_credential_ref: cr_uid:"18e7a184-138c-42c4-b475-4b4a8fc7ec23" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705608695 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"3533cf38-8bcc-45fc-bee6-b8bb4d41c70f" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"d7d7cb90-2508-46c8-807c-098ae6519ec2" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"257b09ac-f456-49a3-b554-d1a6b955a67a" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"beacc472-7dad-4629-9976-00fb76efdb0c" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"6a12597a-9251-44de-8b14-d45388db6303" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"3b2558ec-36dc-473d-bcd4-5a8ef6073298" > backup_path:"kube-system/tp-backup-1705608575-5486a15/df9dc151-38f3-4d61-8e59-b8435003bcdc" stage:Final cr_name:"tp-backup-1705608575-5486a15" total_size:64424509440 resource_count:30 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"df9dc151-38f3-4d61-8e59-b8435003bcdc" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705581755 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"95eda04e-e6b8-45ef-ad93-86b879759a29" > backup_path:"mysql-ibm-pxb-1-58051-01-18-12h40m37s/tp-backup-mysql-ibm-pxb-1-58051-01-18-12h40m37s-mkmce-ed02fc4/fe3a32bd-dcc4-4413-8892-441f4633ab25" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-1-58051-01-18-12h40m37s-mkmce-ed02fc4" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"fe3a32bd-dcc4-4413-8892-441f4633ab25" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-1-58051-01-18-12h40m37s" > > backups: last_update_time: create_time_in_sec:1705581755 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"1f21aca3-0b40-4991-8207-7fc618bb624d" > backup_path:"mysql-ibm-pxb-3-58051-01-18-12h40m37s/tp-backup-mysql-ibm-pxb-3-58051-01-18-12h40m37s-otfyc-9ab1f7b/947706f8-1325-407e-a9bf-13e160054d3f" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-3-58051-01-18-12h40m37s-otfyc-9ab1f7b" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"947706f8-1325-407e-a9bf-13e160054d3f" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-3-58051-01-18-12h40m37s" > > backups: last_update_time: create_time_in_sec:1705581755 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"1374e8d7-6696-4f76-99e3-39ffd108582a" > backup_path:"mysql-ibm-pxb-2-58051-01-18-12h40m37s/tp-backup-mysql-ibm-pxb-2-58051-01-18-12h40m37s-hhxop-2106116/7b6f9711-36e2-4cb0-9cb3-a40515399d07" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-2-58051-01-18-12h40m37s-hhxop-2106116" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"7b6f9711-36e2-4cb0-9cb3-a40515399d07" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-2-58051-01-18-12h40m37s" > > backups: last_update_time: create_time_in_sec:1705581755 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"640dd5b5-65a6-4113-98fa-311bd2423b07" > backup_path:"mysql-ibm-pxb-4-58051-01-18-12h40m37s/tp-backup-mysql-ibm-pxb-4-58051-01-18-12h40m37s-ltnvx-230d407/130fd826-2bee-4842-b4de-6cfde09f56bb" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-4-58051-01-18-12h40m37s-ltnvx-230d407" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"130fd826-2bee-4842-b4de-6cfde09f56bb" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-4-58051-01-18-12h40m37s" > > backups: last_update_time: create_time_in_sec:1705581755 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c16981a5-7cb4-45f2-96d3-48daba527664" > backup_path:"mysql-ibm-pxb-0-58051-01-18-12h40m37s/tp-backup-mysql-ibm-pxb-0-58051-01-18-12h40m37s-gllfn-3e3f1a0/f37ce93a-d3c4-49a0-8dd9-a0fa2d5e4f0e" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58051-01-18-12h40m37s-gllfn-3e3f1a0" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"f37ce93a-d3c4-49a0-8dd9-a0fa2d5e4f0e" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58051-01-18-12h40m37s" > > backups: last_update_time: create_time_in_sec:1705577915 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"314c2ef4-b6b9-41bd-997a-c1c64541d4c3" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"f68d9981-3579-4196-89cd-b3c72121c24e" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"a1171d90-aa6f-400e-8bfd-f1df926bd4c8" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"5c153ee7-1f23-4eaa-b2c8-79dc9b949781" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"8eb78f4b-c86d-4a06-bae2-b7b2b92078fd" > backup_path:"kube-system/tp-backup-all-1705577915-b24d80f/9bd7f0db-8d79-405e-ad15-67d682eb3468" stage:Final cr_name:"tp-backup-all-1705577915-b24d80f" total_size:53687091200 resource_count:1160 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"9bd7f0db-8d79-405e-ad15-67d682eb3468" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > total_count:17 [2024-01-19T00:50:08.930Z] INFO[2024-01-19 00:50:08] Verifying : Description : Restoring the backup tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-3-1705625236 with name tp-restore-tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-3-1705625236 [2024-01-19T00:50:08.930Z] INFO[2024-01-19 00:50:08] Actual:true, Expected: true [2024-01-19T00:50:08.931Z] INFO[2024-01-19 00:50:08] Verifying : Description : Restoring the backup tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-2-1705625226 with name tp-restore-tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-2-1705625226 [2024-01-19T00:50:08.931Z] INFO[2024-01-19 00:50:08] Actual:true, Expected: true [2024-01-19T00:50:08.931Z] INFO[2024-01-19 00:50:08] Verifying : Description : Restoring the backup tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-0-1705625206 with name tp-restore-tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-0-1705625206 [2024-01-19T00:50:08.931Z] INFO[2024-01-19 00:50:08] Actual:true, Expected: true [2024-01-19T00:50:10.300Z] INFO[2024-01-19 00:50:10] Verifying : Description : Restoring the backup tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216 with name tp-restore-tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216 [2024-01-19T00:50:10.300Z] INFO[2024-01-19 00:50:10] Actual:true, Expected: true [2024-01-19T00:50:10.300Z] 2024-01-19 00:50:10 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.glob..func54.2.5:#1708] - The list of restores taken are: [tp-restore-tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-0-1705625206 tp-restore-tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216 tp-restore-tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-2-1705625226 tp-restore-tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-3-1705625236] [2024-01-19T00:50:10.301Z] 2024-01-19 00:50:10 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Cancelling the ongoing restores [2024-01-19T00:50:10.557Z] 2024/01/19 00:50:10 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-0-1705625206] expected was [InProgress] but got [Pending]}, Next try in [1s], timeout [40m0s] [2024-01-19T00:50:10.557Z] INFO[2024-01-19 00:50:10] Verifying : Description : Verfiying restore tp-restore-tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-2-1705625226 is in progress [2024-01-19T00:50:10.557Z] INFO[2024-01-19 00:50:10] Actual:true, Expected: true [2024-01-19T00:50:10.557Z] INFO[2024-01-19 00:50:10] Verifying : Description : Getting the backup driver [2024-01-19T00:50:10.558Z] INFO[2024-01-19 00:50:10] Actual:true, Expected: true [2024-01-19T00:50:10.558Z] 2024/01/19 00:50:10 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-3-1705625236] expected was [InProgress] but got [Pending]}, Next try in [1s], timeout [40m0s] [2024-01-19T00:50:10.558Z] 2024/01/19 00:50:10 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216] expected was [InProgress] but got [Pending]}, Next try in [1s], timeout [40m0s] [2024-01-19T00:50:11.487Z] INFO[2024-01-19 00:50:11] Verifying : Description : Deleting restore tp-restore-tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-2-1705625226 while restore is in progress [2024-01-19T00:50:11.487Z] INFO[2024-01-19 00:50:11] Actual:true, Expected: true [2024-01-19T00:50:11.487Z] INFO[2024-01-19 00:50:11] Verifying : Description : Verfiying restore tp-restore-tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-0-1705625206 is in progress [2024-01-19T00:50:11.487Z] INFO[2024-01-19 00:50:11] Actual:true, Expected: true [2024-01-19T00:50:11.487Z] INFO[2024-01-19 00:50:11] Verifying : Description : Getting the backup driver [2024-01-19T00:50:11.487Z] INFO[2024-01-19 00:50:11] Actual:true, Expected: true [2024-01-19T00:50:11.487Z] INFO[2024-01-19 00:50:11] Verifying : Description : Verfiying restore tp-restore-tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-3-1705625236 is in progress [2024-01-19T00:50:11.487Z] INFO[2024-01-19 00:50:11] Actual:true, Expected: true [2024-01-19T00:50:11.752Z] INFO[2024-01-19 00:50:11] Verifying : Description : Getting the backup driver [2024-01-19T00:50:11.752Z] INFO[2024-01-19 00:50:11] Actual:true, Expected: true [2024-01-19T00:50:11.752Z] 2024/01/19 00:50:11 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216] expected was [InProgress] but got [Pending]}, Next try in [1s], timeout [40m0s] [2024-01-19T00:50:12.008Z] INFO[2024-01-19 00:50:11] Verifying : Description : Deleting restore tp-restore-tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-0-1705625206 while restore is in progress [2024-01-19T00:50:12.008Z] INFO[2024-01-19 00:50:11] Actual:true, Expected: true [2024-01-19T00:50:12.943Z] 2024/01/19 00:50:12 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216] expected was [InProgress] but got [Pending]}, Next try in [1s], timeout [40m0s] [2024-01-19T00:50:12.943Z] INFO[2024-01-19 00:50:12] Verifying : Description : Deleting restore tp-restore-tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-3-1705625236 while restore is in progress [2024-01-19T00:50:12.943Z] INFO[2024-01-19 00:50:12] Actual:true, Expected: true [2024-01-19T00:50:13.882Z] INFO[2024-01-19 00:50:13] Verifying : Description : Verfiying restore tp-restore-tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216 is in progress [2024-01-19T00:50:13.882Z] INFO[2024-01-19 00:50:13] Actual:true, Expected: true [2024-01-19T00:50:13.882Z] INFO[2024-01-19 00:50:13] Verifying : Description : Getting the backup driver [2024-01-19T00:50:13.882Z] INFO[2024-01-19 00:50:13] Actual:true, Expected: true [2024-01-19T00:50:14.138Z] INFO[2024-01-19 00:50:14] Verifying : Description : Deleting restore tp-restore-tp-backup-mysql-ibm-pxb-0-58058-01-19-00h44m49s-1-1705625216 while restore is in progress [2024-01-19T00:50:14.138Z] INFO[2024-01-19 00:50:14] Actual:true, Expected: true [2024-01-19T00:50:14.138Z] 2024-01-19 00:50:14 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Verifying if all the restore creation is cancelled [2024-01-19T00:50:14.699Z] 2024-01-19 00:50:14 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.glob..func54.2.7:#1754] - The list of restore after restore cancellation is [tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724 empty-namespace-restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-cuonk tp-restore-1705608575 test-restore-kjiy test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-18-12h40m37s-rbnmv test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-18-12h40m37s-vfatb test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-18-12h40m37s-sgdlv test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-18-12h40m37s-uatmq test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-18-12h40m37s-necbd] [2024-01-19T00:50:15.260Z] INFO[2024-01-19 00:50:14] Verifying : Description : Verifying restore jobs cancellation while restores is in progress [2024-01-19T00:50:15.260Z] INFO[2024-01-19 00:50:14] Actual:true, Expected: true [2024-01-19T00:50:15.260Z] 2024-01-19 00:50:14 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.glob..func54.2.7:#1776] - All the restores created by this testcase are deleted after restore cancellation [2024-01-19T00:50:16.631Z] 2024-01-19 00:50:16 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.glob..func54.3:#1783] - Deleting the deployed applications [2024-01-19T00:50:16.631Z] 2024-01-19 00:50:16 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Destroying apps [2024-01-19T00:50:16.631Z] 2024-01-19 00:50:16 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T00:50:16.887Z] 2024-01-19 00:50:16 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T00:50:16.887Z] 2024-01-19 00:50:16 +0000:[INFO] [{CancelAllRunningRestoreJobs}] start destroying mysql-ibm app [2024-01-19T00:50:17.142Z] 2024-01-19 00:50:16 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T00:50:17.142Z] 2024-01-19 00:50:17 +0000:[INFO] [{CancelAllRunningRestoreJobs}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T00:50:17.142Z] 2024-01-19 00:50:17 +0000:[WARNING] [{CancelAllRunningRestoreJobs}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T00:50:17.142Z] INFO[2024-01-19 00:50:17] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T00:50:17.142Z] INFO[2024-01-19 00:50:17] Actual:true, Expected: true [2024-01-19T00:50:17.142Z] 2024-01-19 00:50:17 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Deleting the remaining restores in case of failure [2024-01-19T00:50:18.509Z] INFO[2024-01-19 00:50:18] Verifying : Description : Getting the backup driver [2024-01-19T00:50:18.509Z] INFO[2024-01-19 00:50:18] Actual:true, Expected: true [2024-01-19T00:50:18.509Z] INFO[2024-01-19 00:50:18] Verifying : Description : Deleting user restore tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb [2024-01-19T00:50:18.509Z] INFO[2024-01-19 00:50:18] Actual:true, Expected: true [2024-01-19T00:50:18.509Z] INFO[2024-01-19 00:50:18] Verifying : Description : Getting the backup driver [2024-01-19T00:50:18.509Z] INFO[2024-01-19 00:50:18] Actual:true, Expected: true [2024-01-19T00:50:18.509Z] INFO[2024-01-19 00:50:18] Verifying : Description : Deleting user restore tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb [2024-01-19T00:50:18.509Z] INFO[2024-01-19 00:50:18] Actual:true, Expected: true [2024-01-19T00:50:18.509Z] INFO[2024-01-19 00:50:18] Verifying : Description : Getting the backup driver [2024-01-19T00:50:18.509Z] INFO[2024-01-19 00:50:18] Actual:true, Expected: true [2024-01-19T00:50:18.768Z] INFO[2024-01-19 00:50:18] Verifying : Description : Deleting user restore tp-restore-tp-backup-mysql-ibm-pxb-0-87563-01-18-23h40m54s-0-cegb [2024-01-19T00:50:18.768Z] INFO[2024-01-19 00:50:18] Actual:true, Expected: true [2024-01-19T00:50:18.768Z] INFO[2024-01-19 00:50:18] Verifying : Description : Getting the backup driver [2024-01-19T00:50:18.768Z] INFO[2024-01-19 00:50:18] Actual:true, Expected: true [2024-01-19T00:50:18.768Z] INFO[2024-01-19 00:50:18] Verifying : Description : Deleting user restore tp-restore-tp-backup-schedule-1705592040-e11b1ba-interval-2024-01-18-231724 [2024-01-19T00:50:18.768Z] INFO[2024-01-19 00:50:18] Actual:true, Expected: true [2024-01-19T00:50:18.768Z] INFO[2024-01-19 00:50:18] Verifying : Description : Getting the backup driver [2024-01-19T00:50:18.768Z] INFO[2024-01-19 00:50:18] Actual:true, Expected: true [2024-01-19T00:50:18.768Z] INFO[2024-01-19 00:50:18] Verifying : Description : Deleting user restore empty-namespace-restore-tp-backup-schedule-sluzc-300f06e-interval-2024-01-18-220129-cuonk [2024-01-19T00:50:18.768Z] INFO[2024-01-19 00:50:18] Actual:true, Expected: true [2024-01-19T00:50:18.768Z] INFO[2024-01-19 00:50:18] Verifying : Description : Getting the backup driver [2024-01-19T00:50:18.768Z] INFO[2024-01-19 00:50:18] Actual:true, Expected: true [2024-01-19T00:50:18.768Z] INFO[2024-01-19 00:50:18] Verifying : Description : Deleting user restore tp-restore-1705608575 [2024-01-19T00:50:18.768Z] INFO[2024-01-19 00:50:18] Actual:true, Expected: true [2024-01-19T00:50:18.768Z] INFO[2024-01-19 00:50:18] Verifying : Description : Getting the backup driver [2024-01-19T00:50:18.768Z] INFO[2024-01-19 00:50:18] Actual:true, Expected: true [2024-01-19T00:50:18.768Z] INFO[2024-01-19 00:50:18] Verifying : Description : Deleting user restore test-restore-kjiy [2024-01-19T00:50:18.768Z] INFO[2024-01-19 00:50:18] Actual:true, Expected: true [2024-01-19T00:50:18.768Z] INFO[2024-01-19 00:50:18] Verifying : Description : Getting the backup driver [2024-01-19T00:50:18.768Z] INFO[2024-01-19 00:50:18] Actual:true, Expected: true [2024-01-19T00:50:19.026Z] INFO[2024-01-19 00:50:18] Verifying : Description : Deleting user restore test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-18-12h40m37s-rbnmv [2024-01-19T00:50:19.026Z] INFO[2024-01-19 00:50:18] Actual:true, Expected: true [2024-01-19T00:50:19.026Z] INFO[2024-01-19 00:50:18] Verifying : Description : Getting the backup driver [2024-01-19T00:50:19.026Z] INFO[2024-01-19 00:50:18] Actual:true, Expected: true [2024-01-19T00:50:19.026Z] INFO[2024-01-19 00:50:18] Verifying : Description : Deleting user restore test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-18-12h40m37s-vfatb [2024-01-19T00:50:19.026Z] INFO[2024-01-19 00:50:18] Actual:true, Expected: true [2024-01-19T00:50:19.026Z] INFO[2024-01-19 00:50:18] Verifying : Description : Getting the backup driver [2024-01-19T00:50:19.026Z] INFO[2024-01-19 00:50:18] Actual:true, Expected: true [2024-01-19T00:50:19.026Z] INFO[2024-01-19 00:50:18] Verifying : Description : Deleting user restore test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-18-12h40m37s-sgdlv [2024-01-19T00:50:19.026Z] INFO[2024-01-19 00:50:18] Actual:true, Expected: true [2024-01-19T00:50:19.026Z] INFO[2024-01-19 00:50:18] Verifying : Description : Getting the backup driver [2024-01-19T00:50:19.026Z] INFO[2024-01-19 00:50:18] Actual:true, Expected: true [2024-01-19T00:50:19.026Z] INFO[2024-01-19 00:50:18] Verifying : Description : Deleting user restore test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-18-12h40m37s-uatmq [2024-01-19T00:50:19.026Z] INFO[2024-01-19 00:50:18] Actual:true, Expected: true [2024-01-19T00:50:19.026Z] INFO[2024-01-19 00:50:18] Verifying : Description : Getting the backup driver [2024-01-19T00:50:19.026Z] INFO[2024-01-19 00:50:18] Actual:true, Expected: true [2024-01-19T00:50:19.026Z] INFO[2024-01-19 00:50:18] Verifying : Description : Deleting user restore test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-18-12h40m37s-necbd [2024-01-19T00:50:19.026Z] INFO[2024-01-19 00:50:18] Actual:true, Expected: true [2024-01-19T00:50:19.026Z] 2024-01-19 00:50:18 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Cleaning backup locations in map [map[1227b815-abf8-4759-a32d-1dbf0da0a1c3:aws-global-aws-px-backup-system-test-4cac76dc-1705625191-bl]], cloud credential [cloudcred-aws-1705625191], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T00:50:19.281Z] INFO[2024-01-19 00:50:19] Verifying : Description : Verifying deletion of backup location [aws-global-aws-px-backup-system-test-4cac76dc-1705625191-bl] [2024-01-19T00:50:19.281Z] INFO[2024-01-19 00:50:19] Actual:true, Expected: true [2024-01-19T00:50:19.282Z] 2024-01-19 00:50:19 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.IsBackupLocationPresent:#2442] - Backup location [aws-global-aws-px-backup-system-test-4cac76dc-1705625191-bl] is present [2024-01-19T00:50:19.282Z] 2024/01/19 00:50:19 DoRetryWithTimeout - Error: {backup location aws-global-aws-px-backup-system-test-4cac76dc-1705625191-bl is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T00:50:51.383Z] 2024-01-19 00:50:49 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.IsBackupLocationPresent:#2442] - Backup location [aws-global-aws-px-backup-system-test-4cac76dc-1705625191-bl] is present [2024-01-19T00:50:51.383Z] 2024/01/19 00:50:49 DoRetryWithTimeout - Error: {backup location aws-global-aws-px-backup-system-test-4cac76dc-1705625191-bl is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T00:51:23.407Z] 2024-01-19 00:51:19 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.IsBackupLocationPresent:#2442] - Backup location [aws-global-aws-px-backup-system-test-4cac76dc-1705625191-bl] is present [2024-01-19T00:51:23.407Z] 2024/01/19 00:51:19 DoRetryWithTimeout - Error: {backup location aws-global-aws-px-backup-system-test-4cac76dc-1705625191-bl is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T00:51:49.903Z] 2024-01-19 00:51:49 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 aws-global-aws-px-backup-system-test-4cac76dc-bl] [2024-01-19T00:51:49.903Z] INFO[2024-01-19 00:51:49] Verifying : Description : Verifying backup location deletion status aws-global-aws-px-backup-system-test-4cac76dc-1705625191-bl [2024-01-19T00:51:49.903Z] INFO[2024-01-19 00:51:49] Actual:true, Expected: true [2024-01-19T00:51:49.903Z] 2024-01-19 00:51:49 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cloudcred-aws-1705625191] is present [2024-01-19T00:51:49.903Z] INFO[2024-01-19 00:51:49] Verifying : Description : Verifying if cloud cred [cloudcred-aws-1705625191] is present [2024-01-19T00:51:49.904Z] INFO[2024-01-19 00:51:49] Actual:true, Expected: true [2024-01-19T00:51:49.904Z] INFO[2024-01-19 00:51:49] Verifying : Description : Verifying deletion of cloud cred [cloudcred-aws-1705625191] [2024-01-19T00:51:49.904Z] INFO[2024-01-19 00:51:49] Actual:true, Expected: true [2024-01-19T00:51:49.904Z] INFO[2024-01-19 00:51:49] Verifying : Description : Deleting cloud cred cloudcred-aws-1705625191 [2024-01-19T00:51:49.904Z] INFO[2024-01-19 00:51:49] Actual:true, Expected: true [2024-01-19T00:51:49.904Z] INFO[2024-01-19 00:51:49] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T00:51:49.904Z] INFO[2024-01-19 00:51:49] Actual:true, Expected: true [2024-01-19T00:51:49.904Z] INFO[2024-01-19 00:51:49] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T00:51:49.904Z] ERRO[2024-01-19 00:51:49] Actual:rpc error: code = FailedPrecondition desc = cluster [destination-cluster] in use by backup [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-19-004131], cannot be deleted, Expected:  [2024-01-19T00:51:50.163Z] INFO[2024-01-19 00:51:50] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-zbbvt] [2024-01-19T00:51:50.163Z] ERRO[2024-01-19 00:51:50] Actual:rpc error: code = Unknown desc = cluster [destination-cluster] has reference to cloud credential [ibm-destination-config-cloud-cred-zbbvt:a271ea5b-dc9d-40f1-a82a-651539c7e486], Expected:  [2024-01-19T00:51:50.418Z] INFO[2024-01-19 00:51:50] Verifying : Description : Deleting cluster source-cluster [2024-01-19T00:51:50.418Z] INFO[2024-01-19 00:51:50] Actual:true, Expected: true [2024-01-19T00:51:50.979Z] INFO[2024-01-19 00:51:50] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-xfizb] [2024-01-19T00:51:50.979Z] INFO[2024-01-19 00:51:50] Actual:true, Expected: true [2024-01-19T00:51:50.979Z] INFO[2024-01-19 00:51:50] --------Test End------ [2024-01-19T00:51:50.979Z] INFO[2024-01-19 00:51:50] #Test: [2024-01-19T00:51:50.979Z] INFO[2024-01-19 00:51:50] #Description: [2024-01-19T00:51:50.979Z] INFO[2024-01-19 00:51:50] ------------------------ [2024-01-19T00:51:50.979Z] 2024-01-19 00:51:50 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-58058-01-19-00h44m49s] [2024-01-19T00:51:51.235Z] 2024-01-19 00:51:50 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T00:51:51.235Z] 2024-01-19 00:51:50 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T00:51:51.491Z] 2024-01-19 00:51:51 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-58058-01-19-00h44m49s] [2024-01-19T00:51:51.491Z] 2024-01-19 00:51:51 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {CancelAllRunningRestoreJobs} Cancel All Running Restore Jobs and validate [2024-01-19T00:51:51.492Z] 2024-01-19 00:51:51 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T00:51:51.492Z] 2024/01/19 00:51:51 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:51:56.734Z] 2024/01/19 00:51:56 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:52:01.974Z] 2024/01/19 00:52:01 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:52:07.217Z] 2024/01/19 00:52:06 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:52:12.460Z] 2024/01/19 00:52:11 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:52:16.628Z] 2024/01/19 00:52:16 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:52:21.883Z] 2024/01/19 00:52:21 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:52:27.128Z] 2024/01/19 00:52:26 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T00:52:32.372Z] 2024/01/19 00:52:31 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.128.20. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.20 10.240.128.20]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T00:52:32.372Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T00:52:32.372Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T00:52:32.372Z] {dial tcp 10.240.128.20:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T00:52:36.544Z] 2024-01-19 00:52:36 +0000:[WARNING] [{CancelAllRunningRestoreJobs}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.20. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.20 10.240.128.20]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T00:52:36.544Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T00:52:36.544Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T00:52:36.544Z] {dial tcp 10.240.128.20:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T00:52:36.544Z] 2024-01-19 00:52:36 +0000:[ERROR] [{CancelAllRunningRestoreJobs}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.20. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.20 10.240.128.20]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T00:52:36.544Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T00:52:36.544Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T00:52:36.544Z] {dial tcp 10.240.128.20:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T00:52:36.544Z] 2024-01-19 00:52:36 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T00:52:36.544Z] 2024-01-19 00:52:36 +0000:[INFO] [{CancelAllRunningRestoreJobs}] Switching context to [/tmp/source-config] [2024-01-19T00:52:36.544Z] 2024-01-19 00:52:36 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T00:52:36.803Z] 2024-01-19 00:52:36 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T00:52:36.803Z] 2024-01-19 00:52:36 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T00:52:36.803Z] 2024-01-19 00:52:36 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T00:52:36.803Z] 2024-01-19 00:52:36 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T00:52:37.059Z] 2024-01-19 00:52:36 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T00:52:37.315Z] 2024-01-19 00:52:37 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T00:52:37.876Z] 2024-01-19 00:52:37 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T00:52:38.132Z] 2024-01-19 00:52:37 +0000:[INFO] [{CancelAllRunningRestoreJobs}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T00:52:38.388Z] 2024/01/19 00:52:38 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:52:48.327Z] 2024/01/19 00:52:48 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:52:58.285Z] 2024/01/19 00:52:58 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:53:10.455Z] 2024/01/19 00:53:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:53:18.538Z] 2024/01/19 00:53:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:53:28.480Z] 2024/01/19 00:53:28 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:53:40.651Z] 2024/01/19 00:53:38 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:53:48.746Z] 2024/01/19 00:53:48 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:53:58.690Z] 2024/01/19 00:53:58 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:54:08.631Z] 2024/01/19 00:54:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:54:20.825Z] 2024/01/19 00:54:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:54:28.908Z] 2024/01/19 00:54:28 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:54:38.859Z] 2024/01/19 00:54:38 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:54:40.746Z] INFO[2024-01-19 00:54:40] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T00:54:40.746Z] INFO[2024-01-19 00:54:40] sched stats: tick interval: 33640 < 1.2s (100%) [2024-01-19T00:54:40.746Z] INFO[2024-01-19 00:54:40] sched stats: tick process durations: 33639 < 500ms (100%) [2024-01-19T00:54:48.830Z] 2024/01/19 00:54:48 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:54:58.773Z] 2024/01/19 00:54:58 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:55:10.934Z] 2024/01/19 00:55:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:55:19.014Z] 2024/01/19 00:55:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:55:28.952Z] 2024/01/19 00:55:28 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:55:38.909Z] 2024/01/19 00:55:38 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:55:49.994Z] 2024/01/19 00:55:48 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:55:59.938Z] 2024/01/19 00:55:58 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:56:09.878Z] 2024/01/19 00:56:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:56:19.816Z] 2024/01/19 00:56:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:56:29.754Z] 2024/01/19 00:56:28 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:56:39.697Z] 2024/01/19 00:56:38 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:56:49.640Z] 2024/01/19 00:56:49 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:56:59.725Z] 2024/01/19 00:56:59 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:57:09.670Z] 2024/01/19 00:57:09 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:57:19.620Z] 2024/01/19 00:57:19 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:57:29.556Z] 2024/01/19 00:57:29 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T00:57:39.497Z] 2024-01-19 00:57:39 +0000:[ERROR] [{CancelAllRunningRestoreJobs}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.497Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.497Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.497Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.497Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.497Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.497Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.497Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.497Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.497Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.497Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.497Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.498Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.498Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.498Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.498Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.498Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.498Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.498Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.498Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.498Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.498Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.498Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.498Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.498Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.498Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.498Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.499Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.499Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.499Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T00:57:39.499Z] [2024-01-19T00:57:39.499Z] ------------------------------ [2024-01-19T00:57:39.499Z] • Failure [769.421 seconds] [2024-01-19T00:57:39.499Z] {CancelAllRunningRestoreJobs} [2024-01-19T00:57:39.499Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_resiliency_test.go:1587 [2024-01-19T00:57:39.499Z] Cancel All Running Restore Jobs and validate [] [2024-01-19T00:57:39.499Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_resiliency_test.go:1622 [2024-01-19T00:57:39.499Z] [2024-01-19T00:57:39.499Z] Unexpected error: [2024-01-19T00:57:39.499Z] <*errors.errorString | 0xc001356880>: { [2024-01-19T00:57:39.500Z] s: "failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{No... [2024-01-19T00:57:39.500Z] [2024-01-19T00:57:39.500Z] Gomega truncated this representation as it exceeds 'format.MaxLength'. [2024-01-19T00:57:39.500Z] Consider having the object provide a custom 'GomegaStringer' representation [2024-01-19T00:57:39.501Z] or adjust the parameters in Gomega's 'format' package. [2024-01-19T00:57:39.501Z] [2024-01-19T00:57:39.501Z] Learn more here: https://onsi.github.io/gomega/#adjusting-output [2024-01-19T00:57:39.501Z] [2024-01-19T00:57:39.501Z] failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.501Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.501Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.501Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.501Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.501Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.501Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.501Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.501Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.501Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.501Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.501Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.501Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.501Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.502Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.502Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.502Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.502Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.502Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.502Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.502Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.502Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.502Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.502Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.502Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.502Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.502Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.502Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.502Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T00:57:39.502Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T00:57:39.502Z] occurred [2024-01-19T00:57:39.502Z] [2024-01-19T00:57:39.502Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T00:57:39.502Z] [2024-01-19T00:57:39.502Z] Full Stack Trace [2024-01-19T00:57:39.503Z] github.com/portworx/torpedo/pkg/log.FailOnError({0x88a5060, 0xc001356880}, {0x7f74bfd?, 0x6cc0320?}, {0x0?, 0x6cc0320?, 0xc0019b8090?}) [2024-01-19T00:57:39.503Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 +0x290 [2024-01-19T00:57:39.503Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest.func1() [2024-01-19T00:57:39.503Z] /go/src/github.com/portworx/torpedo/tests/common.go:7283 +0x32 [2024-01-19T00:57:39.503Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest({0xc001088228, 0x1, 0x1}) [2024-01-19T00:57:39.503Z] /go/src/github.com/portworx/torpedo/tests/common.go:7299 +0x778 [2024-01-19T00:57:39.503Z] github.com/portworx/torpedo/tests/backup.glob..func54.3() [2024-01-19T00:57:39.503Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_resiliency_test.go:1794 +0x1ca [2024-01-19T00:57:39.503Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T00:57:39.503Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T00:57:39.503Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T00:57:39.503Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T00:57:39.503Z] created by testing.(*T).Run [2024-01-19T00:57:39.503Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T00:57:39.503Z] ------------------------------ [2024-01-19T00:57:39.503Z] STEP: schedule applications [2024-01-19T00:57:39.503Z] STEP: Validating the deployed applications [2024-01-19T00:57:39.503Z] STEP: validate applications [2024-01-19T00:57:39.503Z] STEP: validate mysql-ibm app's volumes [2024-01-19T00:57:39.503Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T00:57:39.503Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T00:57:39.503Z] STEP: get mysql-ibm app's volume: pvc-74f0578a-05e7-47bf-a46b-457eddceb57b inspected by the volume driver [2024-01-19T00:57:39.503Z] STEP: wait for mysql-ibm app to start running [2024-01-19T00:57:39.503Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T00:57:39.503Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T00:57:39.503Z] STEP: Validate Px pod restart count [2024-01-19T00:57:39.503Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T00:57:39.503Z] STEP: Adding cloud account and backup location [2024-01-19T00:57:39.503Z] STEP: Registering source and destination clusters for backup [2024-01-19T00:57:39.503Z] STEP: Taking backup of applications [2024-01-19T00:57:39.503Z] STEP: Restoring the backed up applications [2024-01-19T00:57:39.503Z] STEP: Cancelling the ongoing restores [2024-01-19T00:57:39.503Z] STEP: Verifying if all the restore creation is cancelled [2024-01-19T00:57:39.503Z] STEP: destroy apps [2024-01-19T00:57:39.503Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T00:57:39.503Z] STEP: start destroying mysql-ibm app [2024-01-19T00:57:39.503Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T00:57:39.503Z] SSSS2024-01-19 00:57:39 +0000:[INFO] [{VerifyRBACForAppAdmin}] Log Dir: /testresults//VerifyRBACForAppAdmin.log [2024-01-19T00:57:39.503Z] INFO[2024-01-19 00:57:39] --------Test Start------ [2024-01-19T00:57:39.503Z] INFO[2024-01-19 00:57:39] #Test: VerifyRBACForAppAdmin [2024-01-19T00:57:39.503Z] INFO[2024-01-19 00:57:39] #Description: Validates the RBAC operation for app-admin user. [2024-01-19T00:57:39.503Z] INFO[2024-01-19 00:57:39] ------------------------ [2024-01-19T00:57:39.503Z] 2024-01-19 00:57:39 +0000:[INFO] [{VerifyRBACForAppAdmin}] scheduling applications [2024-01-19T00:57:39.504Z] 2024-01-19 00:57:39 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T00:57:39.504Z] 2024-01-19 00:57:39 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T00:57:39.504Z] 2024-01-19 00:57:39 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T00:57:39.504Z] 2024-01-19 00:57:39 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T00:57:39.504Z] 2024-01-19 00:57:39 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T00:57:39.504Z] 2024-01-19 00:57:39 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T00:57:39.505Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T00:57:39.505Z] 2024-01-19 00:57:39 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T00:57:39.506Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T00:57:39.506Z] 2024-01-19 00:57:39 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T00:57:39.506Z] 2024-01-19 00:57:39 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating applications [2024-01-19T00:57:39.506Z] 2024-01-19 00:57:39 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validate applications [2024-01-19T00:57:39.506Z] 2024-01-19 00:57:39 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating mysql-ibm app [2024-01-19T00:57:39.506Z] 2024-01-19 00:57:39 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating mysql-ibm app's volumes [2024-01-19T00:57:39.506Z] 2024-01-19 00:57:39 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:57:39.766Z] 2024/01/19 00:57:39 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T00:57:49.711Z] 2024/01/19 00:57:49 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T00:57:59.688Z] 2024-01-19 00:57:59 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-87888-01-19-00h57m39s [2024-01-19T00:57:59.688Z] 2024-01-19 00:57:59 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T00:57:59.944Z] 2024-01-19 00:57:59 +0000:[WARNING] [{VerifyRBACForAppAdmin}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T00:57:59.944Z] 2024-01-19 00:57:59 +0000:[INFO] [{VerifyRBACForAppAdmin}] wait for mysql-ibm app to start running [2024-01-19T00:57:59.944Z] 2024-01-19 00:57:59 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T00:57:59.944Z] 2024/01/19 00:57:59 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:57:59.944Z] pod name:mysql-7d766cd65c-tqs2l namespace:mysql-ibm-pxb-0-87888-01-19-00h57m39s running:false ready:false node:10.240.64.8 [2024-01-19T00:57:59.944Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:58:12.128Z] 2024/01/19 00:58:09 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:58:12.129Z] pod name:mysql-7d766cd65c-tqs2l namespace:mysql-ibm-pxb-0-87888-01-19-00h57m39s running:true ready:false node:10.240.64.8 [2024-01-19T00:58:12.129Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:58:20.218Z] 2024/01/19 00:58:19 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:58:20.218Z] pod name:mysql-7d766cd65c-tqs2l namespace:mysql-ibm-pxb-0-87888-01-19-00h57m39s running:true ready:false node:10.240.64.8 [2024-01-19T00:58:20.218Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:58:30.160Z] 2024/01/19 00:58:30 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:58:30.160Z] pod name:mysql-7d766cd65c-tqs2l namespace:mysql-ibm-pxb-0-87888-01-19-00h57m39s running:true ready:false node:10.240.64.8 [2024-01-19T00:58:30.160Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:58:42.384Z] 2024/01/19 00:58:40 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:58:42.384Z] pod name:mysql-7d766cd65c-tqs2l namespace:mysql-ibm-pxb-0-87888-01-19-00h57m39s running:true ready:false node:10.240.64.8 [2024-01-19T00:58:42.384Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:58:50.470Z] 2024/01/19 00:58:50 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:58:50.470Z] pod name:mysql-7d766cd65c-tqs2l namespace:mysql-ibm-pxb-0-87888-01-19-00h57m39s running:true ready:false node:10.240.64.8 [2024-01-19T00:58:50.470Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:59:02.640Z] 2024/01/19 00:59:00 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T00:59:02.640Z] pod name:mysql-7d766cd65c-tqs2l namespace:mysql-ibm-pxb-0-87888-01-19-00h57m39s running:true ready:false node:10.240.64.8 [2024-01-19T00:59:02.640Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T00:59:10.722Z] 2024-01-19 00:59:10 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T00:59:10.722Z] 2024-01-19 00:59:10 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T00:59:10.978Z] 2024-01-19 00:59:10 +0000:[INFO] [{VerifyRBACForAppAdmin}] validate if mysql-ibm app's volumes are setup [2024-01-19T00:59:10.978Z] 2024-01-19 00:59:10 +0000:[WARNING] [{VerifyRBACForAppAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T00:59:10.978Z] 2024-01-19 00:59:10 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T00:59:10.978Z] 2024-01-19 00:59:10 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T00:59:10.978Z] 2024-01-19 00:59:10 +0000:[WARNING] [{VerifyRBACForAppAdmin}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T00:59:10.978Z] 2024-01-19 00:59:10 +0000:[INFO] [{VerifyRBACForAppAdmin}] Adding labels to namespaces [2024-01-19T00:59:10.978Z] INFO[2024-01-19 00:59:10] Verifying : Description : Adding labels [map[0-f8cd8dc4-98ab-48f1-b5e4-477a5a43c57d:f8cd8dc4-98ab-48f1-b5e4-477a5a43c57d 1-f8cd8dc4-98ab-48f1-b5e4-477a5a43c57d:f8cd8dc4-98ab-48f1-b5e4-477a5a43c57d]] to namespaces [[mysql-ibm-pxb-0-87888-01-19-00h57m39s]] [2024-01-19T00:59:10.978Z] INFO[2024-01-19 00:59:10] Actual:true, Expected: true [2024-01-19T00:59:10.978Z] 2024-01-19 00:59:10 +0000:[INFO] [{VerifyRBACForAppAdmin}] Generating namespace label string from label map for namespaces [2024-01-19T00:59:10.978Z] 2024-01-19 00:59:10 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.glob..func45.2.3:#1170] - label string for namespaces 0-f8cd8dc4-98ab-48f1-b5e4-477a5a43c57d=f8cd8dc4-98ab-48f1-b5e4-477a5a43c57d,1-f8cd8dc4-98ab-48f1-b5e4-477a5a43c57d=f8cd8dc4-98ab-48f1-b5e4-477a5a43c57d [2024-01-19T00:59:10.978Z] 2024-01-19 00:59:10 +0000:[INFO] [{VerifyRBACForAppAdmin}] Creating a user with px-backup-app.admin role [2024-01-19T00:59:10.978Z] 2024-01-19 00:59:10 +0000:[INFO] [{VerifyRBACForAppAdmin}] Creating 1 users [2024-01-19T00:59:11.538Z] 2024-01-19 00:59:11 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.AddUser:#728] - Response for user [testuser1-1705625950] creation - [2024-01-19T00:59:11.538Z] 2024-01-19 00:59:11 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.AddUser:#733] - User [testuser1-1705625950] created [2024-01-19T00:59:11.538Z] INFO[2024-01-19 00:59:11] Verifying : Description : Creating user - testuser1-1705625950 [2024-01-19T00:59:11.538Z] INFO[2024-01-19 00:59:11] Actual:true, Expected: true [2024-01-19T00:59:11.538Z] 2024-01-19 00:59:11 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705625950 - bf03dd75-8285-4424-8ca9-3bc5ace8efbf [2024-01-19T00:59:13.448Z] INFO[2024-01-19 00:59:13] Verifying : Description : failed to add role px-backup-app.admin to the user testuser1-1705625950 [2024-01-19T00:59:13.448Z] INFO[2024-01-19 00:59:13] Actual:true, Expected: true [2024-01-19T00:59:13.448Z] 2024-01-19 00:59:13 +0000:[INFO] [{VerifyRBACForAppAdmin}] Verify App-Admin User doesnt have permission to create cloud credential [2024-01-19T00:59:13.703Z] 2024-01-19 00:59:13 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-poerz] for org [default] with [aws] as provider [2024-01-19T00:59:13.704Z] 2024-01-19 00:59:13 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T00:59:13.704Z] 2024-01-19 00:59:13 +0000:[WARNING] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [autogenerated-cred-poerz] in org [default] with [aws] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user px-backup-app.admin]] [2024-01-19T00:59:13.704Z] INFO[2024-01-19 00:59:13] Verifying : Description : Verifying creation failure of cloud credential autogenerated-cred-poerz using provider aws for the user [2024-01-19T00:59:13.704Z] INFO[2024-01-19 00:59:13] Actual:true, Expected: true [2024-01-19T00:59:13.704Z] 2024-01-19 00:59:13 +0000:[INFO] [{VerifyRBACForAppAdmin}] Verify App-Admin User doesnt have permission to create role [2024-01-19T00:59:15.590Z] INFO[2024-01-19 00:59:15] Verifying : Description : Verifying app-user doesnt have permission for creating role [custom-user-role-cvqu] [2024-01-19T00:59:15.590Z] INFO[2024-01-19 00:59:15] Actual:true, Expected: true [2024-01-19T00:59:15.590Z] 2024-01-19 00:59:15 +0000:[INFO] [{VerifyRBACForAppAdmin}] Adding Credentials and BackupLocation from px-admin user and making it public [2024-01-19T00:59:16.954Z] 2024-01-19 00:59:16 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-mjwsl] for org [default] with [aws] as provider [2024-01-19T00:59:16.954Z] 2024-01-19 00:59:16 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T00:59:17.210Z] 2024-01-19 00:59:16 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-mjwsl] is present [2024-01-19T00:59:17.210Z] INFO[2024-01-19 00:59:17] Verifying : Description : Verifying public ownership update for cloud credential autogenerated-cred-mjwsl [2024-01-19T00:59:17.210Z] INFO[2024-01-19 00:59:17] Actual:true, Expected: true [2024-01-19T00:59:19.726Z] INFO[2024-01-19 00:59:19] Verifying : Description : Verifying public ownership update for backup location autogenerated-backup-location-tabml [2024-01-19T00:59:19.726Z] INFO[2024-01-19 00:59:19] Actual:true, Expected: true [2024-01-19T00:59:19.726Z] 2024-01-19 00:59:19 +0000:[INFO] [{VerifyRBACForAppAdmin}] Verify App-Admin User can create backup location with cloud cred shared from px-admin [2024-01-19T00:59:22.241Z] INFO[2024-01-19 00:59:21] Verifying : Description : Failed to add backup location autogenerated-backup-location-dctzc using provider aws for px-admin user [2024-01-19T00:59:22.241Z] INFO[2024-01-19 00:59:21] Actual:true, Expected: true [2024-01-19T00:59:22.241Z] 2024-01-19 00:59:21 +0000:[INFO] [{VerifyRBACForAppAdmin}] Verify App-Admin User can't share backup location created with shared cloud cred from px-admin [2024-01-19T00:59:22.241Z] INFO[2024-01-19 00:59:22] Verifying : Description : Verifying ownership update failure for backup location autogenerated-backup-location-dctzc [2024-01-19T00:59:22.241Z] INFO[2024-01-19 00:59:22] Actual:true, Expected: true [2024-01-19T00:59:22.241Z] 2024-01-19 00:59:22 +0000:[INFO] [{VerifyRBACForAppAdmin}] Verify App-Admin User has permission to create a schedule policy [2024-01-19T00:59:22.241Z] INFO[2024-01-19 00:59:22] Verifying : Description : Verifying creation of periodic schedule policy of interval [15] minutes named [periodic-czjsw] [2024-01-19T00:59:22.241Z] INFO[2024-01-19 00:59:22] Actual:true, Expected: true [2024-01-19T00:59:22.241Z] 2024-01-19 00:59:22 +0000:[INFO] [{VerifyRBACForAppAdmin}] Verify App-Admin User has permission to create pre and post exec rules for applications [2024-01-19T00:59:22.497Z] 2024-01-19 00:59:22 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.CreateRuleForBackupWithMultipleApplications:#4309] - Validate pre-rules for backup [2024-01-19T00:59:22.497Z] 2024-01-19 00:59:22 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.CreateRuleForBackupWithMultipleApplications:#4322] - Validate post-rules for backup [2024-01-19T00:59:22.752Z] INFO[2024-01-19 00:59:22] Verifying : Description : Verifying creation of pre and post exec rules for applications from px-admin [2024-01-19T00:59:22.752Z] INFO[2024-01-19 00:59:22] Actual:true, Expected: true [2024-01-19T00:59:22.752Z] 2024-01-19 00:59:22 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.glob..func45.2.11:#1271] - Pre backup rule [pre-rule-ueyge] uid: [3a45b24d-adce-49f3-9dd0-649e4aa68894] [2024-01-19T00:59:22.752Z] 2024-01-19 00:59:22 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.glob..func45.2.11:#1276] - Post backup rule [post-rule-bulps] uid: [407cea14-94a1-44b5-b72d-9b72bee9382f] [2024-01-19T00:59:22.752Z] 2024-01-19 00:59:22 +0000:[INFO] [{VerifyRBACForAppAdmin}] Verify px-admin group user can list RBAC resources created by app-Admin User [2024-01-19T00:59:23.009Z] 2024-01-19 00:59:22 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.getAllBackupLocations:#1603] - The backup locations and their UID are map[4418c3bb-e07a-4cdc-83e7-b25be38d1e4c:aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 4c14da25-9552-4acb-a4e8-a02da0fa1688:aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 7f811de6-4dac-4a67-9cb4-f60356572f9a:autogenerated-backup-location-dctzc bd1936a0-e91f-4f7c-b9b3-22720421495b:autogenerated-backup-location-tabml cd190df6-e916-41da-b969-ae70c2b830cb:autogenerated-backup-location-omphu e32820d4-7435-4cf7-a3c7-d857b223c325:aws-global-aws-px-backup-system-test-4cac76dc-bl] [2024-01-19T00:59:23.264Z] 2024-01-19 00:59:22 +0000:[INFO] [{VerifyRBACForAppAdmin}] Create Users with Different types of roles [2024-01-19T00:59:23.264Z] 2024-01-19 00:59:22 +0000:[INFO] [{VerifyRBACForAppAdmin}] Creating 1 users [2024-01-19T00:59:23.520Z] 2024-01-19 00:59:23 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.AddUser:#728] - Response for user [testuser1-1705625962] creation - [2024-01-19T00:59:23.520Z] 2024-01-19 00:59:23 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.AddUser:#733] - User [testuser1-1705625962] created [2024-01-19T00:59:23.520Z] INFO[2024-01-19 00:59:23] Verifying : Description : Creating user - testuser1-1705625962 [2024-01-19T00:59:23.520Z] INFO[2024-01-19 00:59:23] Actual:true, Expected: true [2024-01-19T00:59:24.081Z] 2024-01-19 00:59:24 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705625962 - e80e9f17-1795-419e-acbf-1dc483475046 [2024-01-19T00:59:25.968Z] 2024-01-19 00:59:25 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.glob..func45.2.13:#1326] - Added role px-backup-app.admin to user testuser1-1705625962 [2024-01-19T00:59:25.968Z] 2024-01-19 00:59:25 +0000:[INFO] [{VerifyRBACForAppAdmin}] Creating 1 users [2024-01-19T00:59:26.897Z] 2024-01-19 00:59:26 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.AddUser:#728] - Response for user [testuser1-1705625965] creation - [2024-01-19T00:59:26.897Z] 2024-01-19 00:59:26 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.AddUser:#733] - User [testuser1-1705625965] created [2024-01-19T00:59:26.897Z] INFO[2024-01-19 00:59:26] Verifying : Description : Creating user - testuser1-1705625965 [2024-01-19T00:59:26.897Z] INFO[2024-01-19 00:59:26] Actual:true, Expected: true [2024-01-19T00:59:27.458Z] 2024-01-19 00:59:27 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705625965 - 2d10b1f6-0632-4166-b5e3-c90e410ef610 [2024-01-19T00:59:29.378Z] 2024-01-19 00:59:28 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.glob..func45.2.13:#1326] - Added role px-backup-infra.admin to user testuser1-1705625965 [2024-01-19T00:59:29.379Z] 2024-01-19 00:59:28 +0000:[INFO] [{VerifyRBACForAppAdmin}] Creating 1 users [2024-01-19T00:59:29.939Z] 2024-01-19 00:59:29 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.AddUser:#728] - Response for user [testuser1-1705625968] creation - [2024-01-19T00:59:29.939Z] 2024-01-19 00:59:29 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.AddUser:#733] - User [testuser1-1705625968] created [2024-01-19T00:59:29.939Z] INFO[2024-01-19 00:59:29] Verifying : Description : Creating user - testuser1-1705625968 [2024-01-19T00:59:29.939Z] INFO[2024-01-19 00:59:29] Actual:true, Expected: true [2024-01-19T00:59:30.509Z] 2024-01-19 00:59:30 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705625968 - 785699f6-b9ac-4002-acda-4fc838e1e130 [2024-01-19T00:59:32.398Z] 2024-01-19 00:59:32 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.glob..func45.2.13:#1326] - Added role px-backup-app.user to user testuser1-1705625968 [2024-01-19T00:59:32.398Z] 2024-01-19 00:59:32 +0000:[INFO] [{VerifyRBACForAppAdmin}] Verify App-Admin User has permission to share schedule-policy and rules with non-admin users [2024-01-19T00:59:32.654Z] 2024-01-19 00:59:32 +0000:[INFO] [{VerifyRBACForAppAdmin}] Update SchedulePolicy - periodic-czjsw ownership for users - [[testuser1-1705625962 testuser1-1705625965 testuser1-1705625968]] [2024-01-19T00:59:33.581Z] 2024-01-19 00:59:33 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705625962 - e80e9f17-1795-419e-acbf-1dc483475046 [2024-01-19T00:59:34.142Z] 2024-01-19 00:59:34 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705625965 - 2d10b1f6-0632-4166-b5e3-c90e410ef610 [2024-01-19T00:59:35.069Z] 2024-01-19 00:59:34 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705625968 - 785699f6-b9ac-4002-acda-4fc838e1e130 [2024-01-19T00:59:35.069Z] INFO[2024-01-19 00:59:34] Verifying : Description : Verifying updation of ownership for schedulepolicy - periodic-czjsw [2024-01-19T00:59:35.069Z] INFO[2024-01-19 00:59:34] Actual:true, Expected: true [2024-01-19T00:59:35.069Z] 2024-01-19 00:59:34 +0000:[INFO] [{VerifyRBACForAppAdmin}] Update Application Rules ownership for users - [[testuser1-1705625962 testuser1-1705625965 testuser1-1705625968]] [2024-01-19T00:59:35.999Z] 2024-01-19 00:59:35 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705625962 - e80e9f17-1795-419e-acbf-1dc483475046 [2024-01-19T00:59:36.562Z] 2024-01-19 00:59:36 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705625965 - 2d10b1f6-0632-4166-b5e3-c90e410ef610 [2024-01-19T00:59:37.489Z] 2024-01-19 00:59:37 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705625968 - 785699f6-b9ac-4002-acda-4fc838e1e130 [2024-01-19T00:59:37.489Z] INFO[2024-01-19 00:59:37] Verifying : Description : Verifying updation of ownership for pre-rule of application [2024-01-19T00:59:37.489Z] INFO[2024-01-19 00:59:37] Actual:true, Expected: true [2024-01-19T00:59:38.050Z] 2024-01-19 00:59:38 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705625962 - e80e9f17-1795-419e-acbf-1dc483475046 [2024-01-19T00:59:38.981Z] 2024-01-19 00:59:38 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705625965 - 2d10b1f6-0632-4166-b5e3-c90e410ef610 [2024-01-19T00:59:39.907Z] 2024-01-19 00:59:39 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705625968 - 785699f6-b9ac-4002-acda-4fc838e1e130 [2024-01-19T00:59:39.907Z] INFO[2024-01-19 00:59:39] Verifying : Description : Verifying updation of ownership for post-rule of application [2024-01-19T00:59:39.907Z] INFO[2024-01-19 00:59:39] Actual:true, Expected: true [2024-01-19T00:59:39.907Z] 2024-01-19 00:59:39 +0000:[INFO] [{VerifyRBACForAppAdmin}] Creating source and destination cluster from the user testuser1-1705625962 [2024-01-19T00:59:40.163Z] INFO[2024-01-19 00:59:40] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T00:59:40.163Z] INFO[2024-01-19 00:59:40] Actual:true, Expected: true [2024-01-19T00:59:40.163Z] 2024-01-19 00:59:40 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T00:59:40.419Z] 2024-01-19 00:59:40 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T00:59:40.419Z] 2024-01-19 00:59:40 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T00:59:40.419Z] 2024-01-19 00:59:40 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T00:59:40.419Z] 2024-01-19 00:59:40 +0000:[INFO] [{VerifyRBACForAppAdmin}] Create cluster [source-cluster] in org [default] [2024-01-19T00:59:40.419Z] 2024-01-19 00:59:40 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T00:59:40.419Z] 2024-01-19 00:59:40 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T00:59:40.419Z] 2024-01-19 00:59:40 +0000:[INFO] [{VerifyRBACForAppAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:59:40.419Z] 2024-01-19 00:59:40 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T00:59:40.420Z] 2024-01-19 00:59:40 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T00:59:40.420Z] 2024-01-19 00:59:40 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-lbtlt] for IBM [2024-01-19T00:59:40.420Z] 2024-01-19 00:59:40 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-lbtlt] for org [default] with [ibm] as provider [2024-01-19T00:59:40.420Z] 2024-01-19 00:59:40 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:59:40.420Z] 2024-01-19 00:59:40 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:59:40.420Z] 2024-01-19 00:59:40 +0000:[WARNING] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-lbtlt] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user px-backup-app.admin]] [2024-01-19T00:59:40.420Z] 2024-01-19 00:59:40 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user px-backup-app.admin] [2024-01-19T00:59:41.788Z] 2024-01-19 00:59:41 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-lbtlt from admin context and sharing with all the users [2024-01-19T00:59:41.788Z] 2024-01-19 00:59:41 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-lbtlt] for org [default] with [ibm] as provider [2024-01-19T00:59:41.788Z] 2024-01-19 00:59:41 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:59:41.788Z] 2024-01-19 00:59:41 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:59:41.788Z] 2024-01-19 00:59:41 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-lbtlt] is present [2024-01-19T00:59:42.045Z] 2024-01-19 00:59:41 +0000:[INFO] [{VerifyRBACForAppAdmin}] Create cluster [source-cluster] in org [default] [2024-01-19T00:59:42.609Z] 2024-01-19 00:59:42 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-tnykx] for IBM [2024-01-19T00:59:42.609Z] 2024-01-19 00:59:42 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-tnykx] for org [default] with [ibm] as provider [2024-01-19T00:59:42.609Z] 2024-01-19 00:59:42 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:59:42.609Z] 2024-01-19 00:59:42 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:59:42.609Z] 2024-01-19 00:59:42 +0000:[WARNING] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-tnykx] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user px-backup-app.admin]] [2024-01-19T00:59:42.609Z] 2024-01-19 00:59:42 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user px-backup-app.admin] [2024-01-19T00:59:43.174Z] 2024-01-19 00:59:43 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-tnykx from admin context and sharing with all the users [2024-01-19T00:59:43.175Z] 2024-01-19 00:59:43 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-tnykx] for org [default] with [ibm] as provider [2024-01-19T00:59:43.175Z] 2024-01-19 00:59:43 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:59:43.175Z] 2024-01-19 00:59:43 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:59:43.175Z] 2024-01-19 00:59:43 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-tnykx] is present [2024-01-19T00:59:43.432Z] 2024-01-19 00:59:43 +0000:[INFO] [{VerifyRBACForAppAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:59:44.002Z] INFO[2024-01-19 00:59:43] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T00:59:44.002Z] INFO[2024-01-19 00:59:43] Actual:Online, Expected: Online [2024-01-19T00:59:44.002Z] INFO[2024-01-19 00:59:43] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T00:59:44.002Z] INFO[2024-01-19 00:59:43] Actual:true, Expected: true [2024-01-19T00:59:44.002Z] INFO[2024-01-19 00:59:43] Verifying : Description : Fetching [destination-cluster] cluster uid [2024-01-19T00:59:44.002Z] INFO[2024-01-19 00:59:43] Actual:true, Expected: true [2024-01-19T00:59:44.002Z] 2024-01-19 00:59:43 +0000:[INFO] [{VerifyRBACForAppAdmin}] Creating source and destination cluster from the user testuser1-1705625965 [2024-01-19T00:59:44.002Z] INFO[2024-01-19 00:59:43] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T00:59:44.002Z] INFO[2024-01-19 00:59:43] Actual:true, Expected: true [2024-01-19T00:59:44.002Z] 2024-01-19 00:59:43 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T00:59:44.002Z] 2024-01-19 00:59:43 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T00:59:44.002Z] 2024-01-19 00:59:43 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T00:59:44.002Z] 2024-01-19 00:59:43 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T00:59:44.002Z] 2024-01-19 00:59:43 +0000:[INFO] [{VerifyRBACForAppAdmin}] Create cluster [source-cluster] in org [default] [2024-01-19T00:59:44.002Z] 2024-01-19 00:59:43 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T00:59:44.002Z] 2024-01-19 00:59:43 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T00:59:44.002Z] 2024-01-19 00:59:43 +0000:[INFO] [{VerifyRBACForAppAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:59:44.003Z] 2024-01-19 00:59:43 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T00:59:44.003Z] 2024-01-19 00:59:43 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T00:59:44.003Z] 2024-01-19 00:59:43 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-xnntd] for IBM [2024-01-19T00:59:44.003Z] 2024-01-19 00:59:43 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-xnntd] for org [default] with [ibm] as provider [2024-01-19T00:59:44.003Z] 2024-01-19 00:59:43 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:59:44.003Z] 2024-01-19 00:59:43 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:59:44.259Z] 2024-01-19 00:59:44 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-xnntd] is present [2024-01-19T00:59:44.259Z] 2024-01-19 00:59:44 +0000:[INFO] [{VerifyRBACForAppAdmin}] Create cluster [source-cluster] in org [default] [2024-01-19T00:59:44.820Z] 2024-01-19 00:59:44 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-pnouo] for IBM [2024-01-19T00:59:44.820Z] 2024-01-19 00:59:44 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-pnouo] for org [default] with [ibm] as provider [2024-01-19T00:59:44.820Z] 2024-01-19 00:59:44 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:59:44.820Z] 2024-01-19 00:59:44 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:59:44.820Z] 2024-01-19 00:59:44 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-pnouo] is present [2024-01-19T00:59:44.820Z] 2024-01-19 00:59:44 +0000:[INFO] [{VerifyRBACForAppAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:59:45.381Z] INFO[2024-01-19 00:59:45] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T00:59:45.381Z] INFO[2024-01-19 00:59:45] Actual:Online, Expected: Online [2024-01-19T00:59:45.381Z] INFO[2024-01-19 00:59:45] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T00:59:45.381Z] INFO[2024-01-19 00:59:45] Actual:true, Expected: true [2024-01-19T00:59:45.381Z] INFO[2024-01-19 00:59:45] Verifying : Description : Fetching [destination-cluster] cluster uid [2024-01-19T00:59:45.382Z] INFO[2024-01-19 00:59:45] Actual:true, Expected: true [2024-01-19T00:59:45.382Z] 2024-01-19 00:59:45 +0000:[INFO] [{VerifyRBACForAppAdmin}] Creating source and destination cluster from the user testuser1-1705625968 [2024-01-19T00:59:45.638Z] INFO[2024-01-19 00:59:45] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T00:59:45.638Z] INFO[2024-01-19 00:59:45] Actual:true, Expected: true [2024-01-19T00:59:45.638Z] 2024-01-19 00:59:45 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T00:59:45.638Z] 2024-01-19 00:59:45 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T00:59:45.638Z] 2024-01-19 00:59:45 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T00:59:45.639Z] 2024-01-19 00:59:45 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T00:59:45.639Z] 2024-01-19 00:59:45 +0000:[INFO] [{VerifyRBACForAppAdmin}] Create cluster [source-cluster] in org [default] [2024-01-19T00:59:45.639Z] 2024-01-19 00:59:45 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T00:59:45.639Z] 2024-01-19 00:59:45 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T00:59:45.639Z] 2024-01-19 00:59:45 +0000:[INFO] [{VerifyRBACForAppAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:59:45.639Z] 2024-01-19 00:59:45 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T00:59:45.639Z] 2024-01-19 00:59:45 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T00:59:45.639Z] 2024-01-19 00:59:45 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-effqh] for IBM [2024-01-19T00:59:45.639Z] 2024-01-19 00:59:45 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-effqh] for org [default] with [ibm] as provider [2024-01-19T00:59:45.639Z] 2024-01-19 00:59:45 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:59:45.640Z] 2024-01-19 00:59:45 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:59:45.641Z] 2024-01-19 00:59:45 +0000:[WARNING] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-effqh] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user]] [2024-01-19T00:59:45.641Z] 2024-01-19 00:59:45 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user] [2024-01-19T00:59:45.897Z] 2024-01-19 00:59:45 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-effqh from admin context and sharing with all the users [2024-01-19T00:59:45.897Z] 2024-01-19 00:59:45 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-effqh] for org [default] with [ibm] as provider [2024-01-19T00:59:45.897Z] 2024-01-19 00:59:45 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:59:45.897Z] 2024-01-19 00:59:45 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:59:46.154Z] 2024-01-19 00:59:45 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-effqh] is present [2024-01-19T00:59:46.154Z] 2024-01-19 00:59:46 +0000:[INFO] [{VerifyRBACForAppAdmin}] Create cluster [source-cluster] in org [default] [2024-01-19T00:59:46.715Z] 2024-01-19 00:59:46 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-bknif] for IBM [2024-01-19T00:59:46.715Z] 2024-01-19 00:59:46 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-bknif] for org [default] with [ibm] as provider [2024-01-19T00:59:46.715Z] 2024-01-19 00:59:46 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:59:46.715Z] 2024-01-19 00:59:46 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:59:46.715Z] 2024-01-19 00:59:46 +0000:[WARNING] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-bknif] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user]] [2024-01-19T00:59:46.716Z] 2024-01-19 00:59:46 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user] [2024-01-19T00:59:47.277Z] 2024-01-19 00:59:47 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-bknif from admin context and sharing with all the users [2024-01-19T00:59:47.277Z] 2024-01-19 00:59:47 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-bknif] for org [default] with [ibm] as provider [2024-01-19T00:59:47.277Z] 2024-01-19 00:59:47 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T00:59:47.277Z] 2024-01-19 00:59:47 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T00:59:47.277Z] 2024-01-19 00:59:47 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-bknif] is present [2024-01-19T00:59:47.533Z] 2024-01-19 00:59:47 +0000:[INFO] [{VerifyRBACForAppAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-19T00:59:47.788Z] INFO[2024-01-19 00:59:47] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T00:59:47.788Z] INFO[2024-01-19 00:59:47] Actual:Online, Expected: Online [2024-01-19T00:59:48.045Z] INFO[2024-01-19 00:59:47] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T00:59:48.045Z] INFO[2024-01-19 00:59:47] Actual:true, Expected: true [2024-01-19T00:59:48.045Z] INFO[2024-01-19 00:59:47] Verifying : Description : Fetching [destination-cluster] cluster uid [2024-01-19T00:59:48.045Z] INFO[2024-01-19 00:59:47] Actual:true, Expected: true [2024-01-19T00:59:48.046Z] 2024-01-19 00:59:47 +0000:[INFO] [{VerifyRBACForAppAdmin}] Taking backup of applications from the user testuser1-1705625968 [2024-01-19T00:59:48.046Z] 2024-01-19 00:59:47 +0000:[INFO] [{VerifyRBACForAppAdmin}] Taking backup of applications from the user testuser1-1705625962 [2024-01-19T00:59:48.046Z] 2024-01-19 00:59:47 +0000:[INFO] [{VerifyRBACForAppAdmin}] Taking backup of applications from the user testuser1-1705625965 [2024-01-19T00:59:48.046Z] 2024-01-19 00:59:47 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T00:59:48.046Z] 2024-01-19 00:59:47 +0000:[INFO] [{VerifyRBACForAppAdmin}] Volume snapshot classes found in the cluster - [2024-01-19T00:59:48.046Z] 2024-01-19 00:59:47 +0000:[INFO] [{VerifyRBACForAppAdmin}] ibmc-vpcblock-snapshot [2024-01-19T00:59:48.046Z] 2024-01-19 00:59:47 +0000:[INFO] [{VerifyRBACForAppAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T00:59:48.304Z] 2024-01-19 00:59:48 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T00:59:48.304Z] 2024-01-19 00:59:48 +0000:[INFO] [{VerifyRBACForAppAdmin}] Volume snapshot classes found in the cluster - [2024-01-19T00:59:48.304Z] 2024-01-19 00:59:48 +0000:[INFO] [{VerifyRBACForAppAdmin}] ibmc-vpcblock-snapshot [2024-01-19T00:59:48.304Z] 2024-01-19 00:59:48 +0000:[INFO] [{VerifyRBACForAppAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T00:59:48.560Z] 2024-01-19 00:59:48 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T00:59:48.560Z] 2024-01-19 00:59:48 +0000:[INFO] [{VerifyRBACForAppAdmin}] Volume snapshot classes found in the cluster - [2024-01-19T00:59:48.560Z] 2024-01-19 00:59:48 +0000:[INFO] [{VerifyRBACForAppAdmin}] ibmc-vpcblock-snapshot [2024-01-19T00:59:48.560Z] 2024-01-19 00:59:48 +0000:[INFO] [{VerifyRBACForAppAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T00:59:49.935Z] 2024/01/19 00:59:49 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705625968-with-rules-gvtt] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T00:59:50.861Z] 2024/01/19 00:59:50 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705625962-with-rules-dxff] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T00:59:51.117Z] 2024/01/19 00:59:50 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705625965-with-rules-mkld] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T01:00:23.146Z] 2024/01/19 01:00:19 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705625968-with-rules-gvtt] expected was [[Success]] but got [InProgress] because of [Pre-Exec rules are being executed]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:00:23.146Z] 2024/01/19 01:00:20 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705625962-with-rules-dxff] expected was [[Success]] but got [InProgress] because of [Pre-Exec rules are being executed]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:00:23.146Z] 2024/01/19 01:00:20 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705625965-with-rules-mkld] expected was [[Success]] but got [InProgress] because of [Pre-Exec rules are being executed]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:00:49.652Z] 2024/01/19 01:00:49 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705625968-with-rules-gvtt] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:00:51.017Z] 2024/01/19 01:00:50 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705625962-with-rules-dxff] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:00:51.272Z] 2024/01/19 01:00:51 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705625965-with-rules-mkld] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:01:23.327Z] 2024/01/19 01:01:19 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705625968-with-rules-gvtt] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:01:23.327Z] 2024/01/19 01:01:20 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705625962-with-rules-dxff] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:01:23.327Z] 2024/01/19 01:01:21 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705625965-with-rules-mkld] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:01:49.830Z] 2024/01/19 01:01:49 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705625968-with-rules-gvtt] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:01:50.759Z] 2024/01/19 01:01:50 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705625962-with-rules-dxff] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:01:51.323Z] 2024/01/19 01:01:51 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705625965-with-rules-mkld] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:02:23.369Z] 2024/01/19 01:02:19 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705625968-with-rules-gvtt] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:02:23.369Z] 2024/01/19 01:02:20 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705625962-with-rules-dxff] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:02:23.369Z] 2024/01/19 01:02:21 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-testuser1-1705625965-with-rules-mkld] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:02:49.877Z] 2024-01-19 01:02:49 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.CreateBackup:#313] - Backup [tp-backup-manual-single-ns-testuser1-1705625968-with-rules-gvtt] created successfully [2024-01-19T01:02:49.877Z] 2024-01-19 01:02:49 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating backup [tp-backup-manual-single-ns-testuser1-1705625968-with-rules-gvtt] in org [default] [2024-01-19T01:02:49.877Z] 2024-01-19 01:02:49 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-manual-single-ns-testuser1-1705625968-with-rules-gvtt] [2024-01-19T01:02:49.877Z] 2024-01-19 01:02:49 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87888-01-19-00h57m39s] in backup [tp-backup-manual-single-ns-testuser1-1705625968-with-rules-gvtt] [2024-01-19T01:02:49.877Z] 2024-01-19 01:02:49 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87888-01-19-00h57m39s] in backup [tp-backup-manual-single-ns-testuser1-1705625968-with-rules-gvtt] [2024-01-19T01:02:49.877Z] 2024-01-19 01:02:49 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87888-01-19-00h57m39s] [2024-01-19T01:02:49.877Z] 2024-01-19 01:02:49 +0000:[WARNING] [{VerifyRBACForAppAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T01:02:49.877Z] 2024-01-19 01:02:49 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T01:02:49.877Z] 2024-01-19 01:02:49 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[mysql-data]] [2024-01-19T01:02:49.877Z] 2024-01-19 01:02:49 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T01:02:49.877Z] 2024-01-19 01:02:49 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating CR cleanup [2024-01-19T01:02:50.138Z] 2024-01-19 01:02:49 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T01:02:50.138Z] 2024-01-19 01:02:49 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[]] [2024-01-19T01:02:50.138Z] INFO[2024-01-19 01:02:49] Verifying : Description : Verifying creation and validation of backup [tp-backup-manual-single-ns-testuser1-1705625968-with-rules-gvtt] of namespace (scheduled Context) [[mysql-ibm-pxb-0-87888-01-19-00h57m39s]] [2024-01-19T01:02:50.138Z] INFO[2024-01-19 01:02:49] Actual:true, Expected: true [2024-01-19T01:02:50.138Z] 2024-01-19 01:02:49 +0000:[INFO] [{VerifyRBACForAppAdmin}] Taking schedule backup of applications from the user testuser1-1705625968 [2024-01-19T01:02:50.138Z] 2024-01-19 01:02:50 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T01:02:50.138Z] 2024-01-19 01:02:50 +0000:[INFO] [{VerifyRBACForAppAdmin}] Volume snapshot classes found in the cluster - [2024-01-19T01:02:50.138Z] 2024-01-19 01:02:50 +0000:[INFO] [{VerifyRBACForAppAdmin}] ibmc-vpcblock-snapshot [2024-01-19T01:02:50.138Z] 2024-01-19 01:02:50 +0000:[INFO] [{VerifyRBACForAppAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T01:02:51.070Z] 2024-01-19 01:02:50 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.CreateBackup:#313] - Backup [tp-backup-manual-single-ns-testuser1-1705625962-with-rules-dxff] created successfully [2024-01-19T01:02:51.070Z] 2024-01-19 01:02:50 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating backup [tp-backup-manual-single-ns-testuser1-1705625962-with-rules-dxff] in org [default] [2024-01-19T01:02:51.070Z] 2024-01-19 01:02:50 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-manual-single-ns-testuser1-1705625962-with-rules-dxff] [2024-01-19T01:02:51.070Z] 2024-01-19 01:02:50 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87888-01-19-00h57m39s] in backup [tp-backup-manual-single-ns-testuser1-1705625962-with-rules-dxff] [2024-01-19T01:02:51.071Z] 2024-01-19 01:02:50 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87888-01-19-00h57m39s] in backup [tp-backup-manual-single-ns-testuser1-1705625962-with-rules-dxff] [2024-01-19T01:02:51.071Z] 2024-01-19 01:02:50 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87888-01-19-00h57m39s] [2024-01-19T01:02:51.071Z] 2024-01-19 01:02:50 +0000:[WARNING] [{VerifyRBACForAppAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T01:02:51.071Z] 2024-01-19 01:02:50 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T01:02:51.071Z] 2024-01-19 01:02:50 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[mysql-data]] [2024-01-19T01:02:51.071Z] 2024-01-19 01:02:50 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T01:02:51.071Z] 2024-01-19 01:02:50 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating CR cleanup [2024-01-19T01:02:51.071Z] 2024-01-19 01:02:50 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T01:02:51.071Z] 2024-01-19 01:02:50 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[]] [2024-01-19T01:02:51.071Z] INFO[2024-01-19 01:02:50] Verifying : Description : Verifying creation and validation of backup [tp-backup-manual-single-ns-testuser1-1705625962-with-rules-dxff] of namespace (scheduled Context) [[mysql-ibm-pxb-0-87888-01-19-00h57m39s]] [2024-01-19T01:02:51.071Z] INFO[2024-01-19 01:02:50] Actual:true, Expected: true [2024-01-19T01:02:51.071Z] 2024-01-19 01:02:50 +0000:[INFO] [{VerifyRBACForAppAdmin}] Taking schedule backup of applications from the user testuser1-1705625962 [2024-01-19T01:02:51.327Z] 2024-01-19 01:02:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T01:02:51.327Z] 2024-01-19 01:02:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] Volume snapshot classes found in the cluster - [2024-01-19T01:02:51.327Z] 2024-01-19 01:02:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] ibmc-vpcblock-snapshot [2024-01-19T01:02:51.327Z] 2024-01-19 01:02:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T01:02:51.327Z] 2024-01-19 01:02:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.CreateBackup:#313] - Backup [tp-backup-manual-single-ns-testuser1-1705625965-with-rules-mkld] created successfully [2024-01-19T01:02:51.327Z] 2024-01-19 01:02:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating backup [tp-backup-manual-single-ns-testuser1-1705625965-with-rules-mkld] in org [default] [2024-01-19T01:02:51.327Z] 2024-01-19 01:02:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-manual-single-ns-testuser1-1705625965-with-rules-mkld] [2024-01-19T01:02:51.327Z] 2024-01-19 01:02:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87888-01-19-00h57m39s] in backup [tp-backup-manual-single-ns-testuser1-1705625965-with-rules-mkld] [2024-01-19T01:02:51.327Z] 2024-01-19 01:02:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87888-01-19-00h57m39s] in backup [tp-backup-manual-single-ns-testuser1-1705625965-with-rules-mkld] [2024-01-19T01:02:51.327Z] 2024-01-19 01:02:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87888-01-19-00h57m39s] [2024-01-19T01:02:51.327Z] 2024-01-19 01:02:51 +0000:[WARNING] [{VerifyRBACForAppAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T01:02:51.327Z] 2024-01-19 01:02:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T01:02:51.327Z] 2024-01-19 01:02:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[mysql-data]] [2024-01-19T01:02:51.327Z] 2024-01-19 01:02:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T01:02:51.327Z] 2024-01-19 01:02:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating CR cleanup [2024-01-19T01:02:51.583Z] 2024-01-19 01:02:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T01:02:51.583Z] 2024-01-19 01:02:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] [2024-01-19T01:02:51.583Z] INFO[2024-01-19 01:02:51] Verifying : Description : Verifying creation and validation of backup [tp-backup-manual-single-ns-testuser1-1705625965-with-rules-mkld] of namespace (scheduled Context) [[mysql-ibm-pxb-0-87888-01-19-00h57m39s]] [2024-01-19T01:02:51.583Z] INFO[2024-01-19 01:02:51] Actual:true, Expected: true [2024-01-19T01:02:51.583Z] 2024-01-19 01:02:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] Taking schedule backup of applications from the user testuser1-1705625965 [2024-01-19T01:02:51.838Z] 2024-01-19 01:02:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T01:02:51.838Z] 2024-01-19 01:02:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] Volume snapshot classes found in the cluster - [2024-01-19T01:02:51.838Z] 2024-01-19 01:02:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] ibmc-vpcblock-snapshot [2024-01-19T01:02:51.838Z] 2024-01-19 01:02:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T01:03:59.484Z] 2024-01-19 01:03:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] first schedule backup for schedule name [backup-schedule-yqszc] is [backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251] [2024-01-19T01:03:59.484Z] 2024/01/19 01:03:51 DoRetryWithTimeout - Error: {backup status for [backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:03:59.484Z] 2024-01-19 01:03:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] first schedule backup for schedule name [backup-schedule-poins] is [backup-schedule-poins-ce42dc4-interval-2024-01-19-010253] [2024-01-19T01:03:59.484Z] 2024/01/19 01:03:54 DoRetryWithTimeout - Error: {backup status for [backup-schedule-poins-ce42dc4-interval-2024-01-19-010253] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:03:59.484Z] 2024-01-19 01:03:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] first schedule backup for schedule name [backup-schedule-nqnsr] is [backup-schedule-nqnsr-a303382-interval-2024-01-19-010254] [2024-01-19T01:03:59.484Z] 2024/01/19 01:03:54 DoRetryWithTimeout - Error: {backup status for [backup-schedule-nqnsr-a303382-interval-2024-01-19-010254] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:04:25.985Z] 2024/01/19 01:04:21 DoRetryWithTimeout - Error: {backup status for [backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:04:25.985Z] 2024/01/19 01:04:24 DoRetryWithTimeout - Error: {backup status for [backup-schedule-poins-ce42dc4-interval-2024-01-19-010253] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:04:25.985Z] 2024/01/19 01:04:24 DoRetryWithTimeout - Error: {backup status for [backup-schedule-nqnsr-a303382-interval-2024-01-19-010254] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:04:44.029Z] INFO[2024-01-19 01:04:41] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T01:04:44.029Z] INFO[2024-01-19 01:04:41] sched stats: tick interval: 34241 < 1.2s (100%) [2024-01-19T01:04:44.029Z] INFO[2024-01-19 01:04:41] sched stats: tick process durations: 34240 < 500ms (100%) [2024-01-19T01:04:52.120Z] 2024/01/19 01:04:51 DoRetryWithTimeout - Error: {backup status for [backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:04:54.642Z] 2024/01/19 01:04:54 DoRetryWithTimeout - Error: {backup status for [backup-schedule-poins-ce42dc4-interval-2024-01-19-010253] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:04:54.642Z] 2024/01/19 01:04:54 DoRetryWithTimeout - Error: {backup status for [backup-schedule-nqnsr-a303382-interval-2024-01-19-010254] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:05:26.723Z] 2024/01/19 01:05:21 DoRetryWithTimeout - Error: {backup status for [backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:05:26.723Z] 2024/01/19 01:05:24 DoRetryWithTimeout - Error: {backup status for [backup-schedule-poins-ce42dc4-interval-2024-01-19-010253] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:05:26.723Z] 2024/01/19 01:05:24 DoRetryWithTimeout - Error: {backup status for [backup-schedule-nqnsr-a303382-interval-2024-01-19-010254] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:05:53.298Z] 2024-01-19 01:05:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating backup [backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251] in org [default] [2024-01-19T01:05:53.298Z] 2024-01-19 01:05:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251] [2024-01-19T01:05:53.298Z] 2024-01-19 01:05:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87888-01-19-00h57m39s] in backup [backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251] [2024-01-19T01:05:53.298Z] 2024-01-19 01:05:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87888-01-19-00h57m39s] in backup [backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251] [2024-01-19T01:05:53.298Z] 2024-01-19 01:05:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87888-01-19-00h57m39s] [2024-01-19T01:05:53.298Z] 2024-01-19 01:05:51 +0000:[WARNING] [{VerifyRBACForAppAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T01:05:53.298Z] 2024-01-19 01:05:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T01:05:53.298Z] 2024-01-19 01:05:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[mysql-data]] [2024-01-19T01:05:53.298Z] 2024-01-19 01:05:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T01:05:53.298Z] 2024-01-19 01:05:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating CR cleanup [2024-01-19T01:05:53.298Z] 2024-01-19 01:05:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T01:05:53.299Z] 2024-01-19 01:05:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-nqnsr-a303382-interval-2024-01-19-010254 backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] [2024-01-19T01:05:53.299Z] 2024-01-19 01:05:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-nqnsr-a303382-interval-2024-01-19-010254 backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:05:53.299Z] 2024/01/19 01:05:51 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:05:54.663Z] 2024-01-19 01:05:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating backup [backup-schedule-poins-ce42dc4-interval-2024-01-19-010253] in org [default] [2024-01-19T01:05:54.663Z] 2024-01-19 01:05:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [backup-schedule-poins-ce42dc4-interval-2024-01-19-010253] [2024-01-19T01:05:54.663Z] 2024-01-19 01:05:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87888-01-19-00h57m39s] in backup [backup-schedule-poins-ce42dc4-interval-2024-01-19-010253] [2024-01-19T01:05:54.663Z] 2024-01-19 01:05:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87888-01-19-00h57m39s] in backup [backup-schedule-poins-ce42dc4-interval-2024-01-19-010253] [2024-01-19T01:05:54.663Z] 2024-01-19 01:05:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87888-01-19-00h57m39s] [2024-01-19T01:05:54.664Z] 2024-01-19 01:05:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating backup [backup-schedule-nqnsr-a303382-interval-2024-01-19-010254] in org [default] [2024-01-19T01:05:54.664Z] 2024-01-19 01:05:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [backup-schedule-nqnsr-a303382-interval-2024-01-19-010254] [2024-01-19T01:05:54.664Z] 2024-01-19 01:05:54 +0000:[WARNING] [{VerifyRBACForAppAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T01:05:54.664Z] 2024-01-19 01:05:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T01:05:54.664Z] 2024-01-19 01:05:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[mysql-data]] [2024-01-19T01:05:54.921Z] 2024-01-19 01:05:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T01:05:54.921Z] 2024-01-19 01:05:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating CR cleanup [2024-01-19T01:05:54.921Z] 2024-01-19 01:05:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87888-01-19-00h57m39s] in backup [backup-schedule-nqnsr-a303382-interval-2024-01-19-010254] [2024-01-19T01:05:54.921Z] 2024-01-19 01:05:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87888-01-19-00h57m39s] in backup [backup-schedule-nqnsr-a303382-interval-2024-01-19-010254] [2024-01-19T01:05:54.921Z] 2024-01-19 01:05:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87888-01-19-00h57m39s] [2024-01-19T01:05:54.921Z] 2024-01-19 01:05:54 +0000:[WARNING] [{VerifyRBACForAppAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T01:05:54.921Z] 2024-01-19 01:05:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T01:05:54.921Z] 2024-01-19 01:05:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[mysql-data]] [2024-01-19T01:05:54.921Z] 2024-01-19 01:05:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T01:05:54.921Z] 2024-01-19 01:05:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating CR cleanup [2024-01-19T01:05:54.921Z] 2024-01-19 01:05:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T01:05:54.921Z] 2024-01-19 01:05:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T01:05:54.922Z] 2024-01-19 01:05:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-nqnsr-a303382-interval-2024-01-19-010254 backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] [2024-01-19T01:05:54.922Z] 2024-01-19 01:05:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-nqnsr-a303382-interval-2024-01-19-010254 backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:05:54.922Z] 2024/01/19 01:05:54 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:05:54.922Z] 2024-01-19 01:05:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-nqnsr-a303382-interval-2024-01-19-010254 backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] [2024-01-19T01:05:54.922Z] 2024-01-19 01:05:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-nqnsr-a303382-interval-2024-01-19-010254 backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:05:54.922Z] 2024/01/19 01:05:54 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-nqnsr-a303382-interval-2024-01-19-010254]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:05:57.449Z] 2024-01-19 01:05:56 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-nqnsr-a303382-interval-2024-01-19-010254 backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] [2024-01-19T01:05:57.449Z] 2024-01-19 01:05:56 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-nqnsr-a303382-interval-2024-01-19-010254 backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:05:57.449Z] 2024/01/19 01:05:56 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:05:59.965Z] 2024-01-19 01:05:59 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-nqnsr-a303382-interval-2024-01-19-010254 backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] [2024-01-19T01:05:59.965Z] 2024-01-19 01:05:59 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-nqnsr-a303382-interval-2024-01-19-010254 backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:05:59.965Z] 2024/01/19 01:05:59 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:05:59.965Z] 2024-01-19 01:05:59 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-nqnsr-a303382-interval-2024-01-19-010254 backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] [2024-01-19T01:05:59.965Z] 2024-01-19 01:05:59 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-nqnsr-a303382-interval-2024-01-19-010254 backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:05:59.965Z] 2024/01/19 01:05:59 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-nqnsr-a303382-interval-2024-01-19-010254]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:06:02.492Z] 2024-01-19 01:06:01 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-nqnsr-a303382-interval-2024-01-19-010254 backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] [2024-01-19T01:06:02.493Z] 2024-01-19 01:06:01 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-nqnsr-a303382-interval-2024-01-19-010254 backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:06:02.493Z] 2024/01/19 01:06:01 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:06:05.021Z] 2024-01-19 01:06:04 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-nqnsr-a303382-interval-2024-01-19-010254 backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] [2024-01-19T01:06:05.021Z] 2024-01-19 01:06:04 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-nqnsr-a303382-interval-2024-01-19-010254 backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:06:05.021Z] 2024/01/19 01:06:04 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:06:05.021Z] 2024-01-19 01:06:04 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-nqnsr-a303382-interval-2024-01-19-010254 backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] [2024-01-19T01:06:05.021Z] 2024-01-19 01:06:04 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-nqnsr-a303382-interval-2024-01-19-010254 backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:06:05.021Z] 2024/01/19 01:06:04 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-nqnsr-a303382-interval-2024-01-19-010254]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:06:07.544Z] 2024-01-19 01:06:07 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-nqnsr-a303382-interval-2024-01-19-010254 backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] [2024-01-19T01:06:07.544Z] 2024-01-19 01:06:07 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-nqnsr-a303382-interval-2024-01-19-010254 backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:06:07.544Z] 2024/01/19 01:06:07 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:06:10.059Z] 2024-01-19 01:06:09 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] [2024-01-19T01:06:10.060Z] 2024-01-19 01:06:09 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:06:10.060Z] 2024/01/19 01:06:09 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:06:10.060Z] 2024-01-19 01:06:09 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] [2024-01-19T01:06:10.060Z] INFO[2024-01-19 01:06:09] Verifying : Description : Verifying creation and validation of schedule backup with schedule name [backup-schedule-nqnsr] [2024-01-19T01:06:10.060Z] INFO[2024-01-19 01:06:09] Actual:true, Expected: true [2024-01-19T01:06:10.324Z] INFO[2024-01-19 01:06:10] Verifying : Description : Suspending Backup Schedule [backup-schedule-nqnsr] for user [testuser1-1705625962] [2024-01-19T01:06:10.324Z] INFO[2024-01-19 01:06:10] Actual:true, Expected: true [2024-01-19T01:06:12.213Z] 2024-01-19 01:06:12 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] [2024-01-19T01:06:12.214Z] 2024-01-19 01:06:12 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:06:12.214Z] 2024/01/19 01:06:12 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:06:15.484Z] 2024-01-19 01:06:14 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] [2024-01-19T01:06:15.484Z] 2024-01-19 01:06:14 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:06:15.484Z] 2024/01/19 01:06:14 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:06:17.373Z] 2024-01-19 01:06:17 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] [2024-01-19T01:06:17.373Z] 2024-01-19 01:06:17 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:06:17.373Z] 2024/01/19 01:06:17 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:06:20.639Z] 2024-01-19 01:06:20 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] [2024-01-19T01:06:20.639Z] 2024-01-19 01:06:20 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253 backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:06:20.639Z] 2024/01/19 01:06:20 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:06:22.528Z] 2024-01-19 01:06:22 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]] [2024-01-19T01:06:22.528Z] INFO[2024-01-19 01:06:22] Verifying : Description : Verifying creation and validation of schedule backup with schedule name [backup-schedule-yqszc] [2024-01-19T01:06:22.528Z] INFO[2024-01-19 01:06:22] Actual:true, Expected: true [2024-01-19T01:06:22.528Z] INFO[2024-01-19 01:06:22] Verifying : Description : Suspending Backup Schedule [backup-schedule-yqszc] for user [testuser1-1705625968] [2024-01-19T01:06:22.528Z] INFO[2024-01-19 01:06:22] Actual:true, Expected: true [2024-01-19T01:06:25.797Z] 2024-01-19 01:06:25 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]] [2024-01-19T01:06:25.797Z] 2024-01-19 01:06:25 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:06:25.797Z] 2024/01/19 01:06:25 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:06:31.040Z] 2024-01-19 01:06:30 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]] [2024-01-19T01:06:31.041Z] 2024-01-19 01:06:30 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:06:31.041Z] 2024/01/19 01:06:30 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:06:35.207Z] 2024-01-19 01:06:35 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]] [2024-01-19T01:06:35.207Z] 2024-01-19 01:06:35 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:06:35.207Z] 2024/01/19 01:06:35 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:06:40.454Z] 2024-01-19 01:06:40 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]] [2024-01-19T01:06:40.454Z] 2024-01-19 01:06:40 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:06:40.454Z] 2024/01/19 01:06:40 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:06:45.698Z] 2024-01-19 01:06:45 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]] [2024-01-19T01:06:45.698Z] 2024-01-19 01:06:45 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:06:45.698Z] 2024/01/19 01:06:45 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:06:50.946Z] 2024-01-19 01:06:50 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]] [2024-01-19T01:06:50.946Z] 2024-01-19 01:06:50 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:06:50.946Z] 2024/01/19 01:06:50 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:06:56.195Z] 2024-01-19 01:06:55 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]] [2024-01-19T01:06:56.195Z] 2024-01-19 01:06:55 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:06:56.195Z] 2024/01/19 01:06:55 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:07:01.441Z] 2024-01-19 01:07:00 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]] [2024-01-19T01:07:01.441Z] 2024-01-19 01:07:00 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:07:01.441Z] 2024/01/19 01:07:00 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:07:05.607Z] 2024-01-19 01:07:05 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]] [2024-01-19T01:07:05.607Z] 2024-01-19 01:07:05 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:07:05.607Z] 2024/01/19 01:07:05 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-poins-ce42dc4-interval-2024-01-19-010253]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:07:10.877Z] 2024-01-19 01:07:10 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[]] [2024-01-19T01:07:10.877Z] INFO[2024-01-19 01:07:10] Verifying : Description : Verifying creation and validation of schedule backup with schedule name [backup-schedule-poins] [2024-01-19T01:07:10.877Z] INFO[2024-01-19 01:07:10] Actual:true, Expected: true [2024-01-19T01:07:10.877Z] INFO[2024-01-19 01:07:10] Verifying : Description : Suspending Backup Schedule [backup-schedule-poins] for user [testuser1-1705625965] [2024-01-19T01:07:10.877Z] INFO[2024-01-19 01:07:10] Actual:true, Expected: true [2024-01-19T01:07:10.877Z] 2024-01-19 01:07:10 +0000:[INFO] [{VerifyRBACForAppAdmin}] Deleting user testuser1-1705625968 backups from the user context [2024-01-19T01:07:10.877Z] 2024-01-19 01:07:10 +0000:[INFO] [{VerifyRBACForAppAdmin}] Deleting user testuser1-1705625962 backups from the user context [2024-01-19T01:07:10.877Z] 2024-01-19 01:07:10 +0000:[INFO] [{VerifyRBACForAppAdmin}] Deleting user testuser1-1705625965 backups from the user context [2024-01-19T01:07:10.877Z] 2024/01/19 01:07:10 DoRetryWithTimeout - Error: {backupObject [tp-backup-manual-single-ns-testuser1-1705625962-with-rules-dxff] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T01:07:10.877Z] 2024/01/19 01:07:10 DoRetryWithTimeout - Error: {backupObject [tp-backup-manual-single-ns-testuser1-1705625965-with-rules-mkld] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T01:07:10.877Z] 2024/01/19 01:07:10 DoRetryWithTimeout - Error: {backupObject [tp-backup-manual-single-ns-testuser1-1705625968-with-rules-gvtt] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T01:07:42.962Z] 2024/01/19 01:07:40 DoRetryWithTimeout - Error: {backupObject [backup-schedule-nqnsr-a303382-interval-2024-01-19-010254] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T01:07:42.962Z] 2024/01/19 01:07:40 DoRetryWithTimeout - Error: {backupObject [backup-schedule-poins-ce42dc4-interval-2024-01-19-010253] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T01:07:42.962Z] 2024/01/19 01:07:40 DoRetryWithTimeout - Error: {backupObject [backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T01:08:16.398Z] 2024/01/19 01:08:10 DoRetryWithTimeout - Error: {backupObject [backup-schedule-nqnsr-a303382-interval-2024-01-19-010254] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T01:08:16.399Z] 2024/01/19 01:08:10 DoRetryWithTimeout - Error: {backupObject [backup-schedule-poins-ce42dc4-interval-2024-01-19-010253] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T01:08:16.399Z] 2024/01/19 01:08:10 DoRetryWithTimeout - Error: {backupObject [backup-schedule-yqszc-f7e1f2a-interval-2024-01-19-010251] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T01:08:42.887Z] 2024-01-19 01:08:40 +0000:[INFO] [{VerifyRBACForAppAdmin}] Delete user testuser1-1705625962 backup schedule [2024-01-19T01:08:42.887Z] 2024-01-19 01:08:40 +0000:[INFO] [{VerifyRBACForAppAdmin}] Delete user testuser1-1705625965 backup schedule [2024-01-19T01:08:42.887Z] 2024-01-19 01:08:40 +0000:[INFO] [{VerifyRBACForAppAdmin}] Delete user testuser1-1705625968 backup schedule [2024-01-19T01:08:42.887Z] 2024/01/19 01:08:41 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-poins] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T01:08:42.887Z] 2024/01/19 01:08:41 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-yqszc] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T01:08:42.887Z] 2024/01/19 01:08:41 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-nqnsr] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T01:08:52.827Z] INFO[2024-01-19 01:08:51] Verifying : Description : Deleting Backup Schedule [backup-schedule-poins] for user [testuser1-1705625965] [2024-01-19T01:08:52.827Z] INFO[2024-01-19 01:08:51] Actual:true, Expected: true [2024-01-19T01:08:52.827Z] 2024-01-19 01:08:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] Deleting user testuser1-1705625965 source and destination cluster from the user context [2024-01-19T01:08:52.827Z] INFO[2024-01-19 01:08:51] Verifying : Description : Deleting Backup Schedule [backup-schedule-yqszc] for user [testuser1-1705625968] [2024-01-19T01:08:52.827Z] INFO[2024-01-19 01:08:51] Actual:true, Expected: true [2024-01-19T01:08:52.827Z] 2024-01-19 01:08:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] Deleting user testuser1-1705625968 source and destination cluster from the user context [2024-01-19T01:08:52.827Z] INFO[2024-01-19 01:08:51] Verifying : Description : Deleting Backup Schedule [backup-schedule-nqnsr] for user [testuser1-1705625962] [2024-01-19T01:08:52.827Z] INFO[2024-01-19 01:08:51] Actual:true, Expected: true [2024-01-19T01:08:52.827Z] 2024-01-19 01:08:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] Deleting user testuser1-1705625962 source and destination cluster from the user context [2024-01-19T01:08:52.827Z] INFO[2024-01-19 01:08:51] Verifying : Description : Verifying deletion of cluster [source-cluster] of the user testuser1-1705625965 [2024-01-19T01:08:52.827Z] INFO[2024-01-19 01:08:51] Actual:true, Expected: true [2024-01-19T01:08:52.827Z] INFO[2024-01-19 01:08:51] Verifying : Description : Verifying deletion of cluster [source-cluster] of the user testuser1-1705625968 [2024-01-19T01:08:52.827Z] INFO[2024-01-19 01:08:51] Actual:true, Expected: true [2024-01-19T01:08:52.827Z] INFO[2024-01-19 01:08:51] Verifying : Description : Verifying deletion of cluster [source-cluster] of the user testuser1-1705625962 [2024-01-19T01:08:52.827Z] INFO[2024-01-19 01:08:51] Actual:true, Expected: true [2024-01-19T01:08:52.827Z] INFO[2024-01-19 01:08:51] Verifying : Description : Verifying deletion of cluster [destination-cluster] of the user testuser1-1705625965 [2024-01-19T01:08:52.827Z] INFO[2024-01-19 01:08:51] Actual:true, Expected: true [2024-01-19T01:08:52.827Z] INFO[2024-01-19 01:08:51] Verifying : Description : Verifying deletion of cluster [destination-cluster] of the user testuser1-1705625968 [2024-01-19T01:08:52.827Z] INFO[2024-01-19 01:08:51] Actual:true, Expected: true [2024-01-19T01:08:52.827Z] INFO[2024-01-19 01:08:51] Verifying : Description : Verifying deletion of cluster [destination-cluster] of the user testuser1-1705625962 [2024-01-19T01:08:52.827Z] INFO[2024-01-19 01:08:51] Actual:true, Expected: true [2024-01-19T01:08:52.827Z] 2024-01-19 01:08:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] Creating source and destination cluster from the app-admin user testuser1-1705625950 [2024-01-19T01:08:52.827Z] INFO[2024-01-19 01:08:52] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T01:08:52.827Z] INFO[2024-01-19 01:08:52] Actual:true, Expected: true [2024-01-19T01:08:52.827Z] 2024-01-19 01:08:52 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T01:08:52.827Z] 2024-01-19 01:08:52 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T01:08:52.827Z] 2024-01-19 01:08:52 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T01:08:52.827Z] 2024-01-19 01:08:52 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T01:08:52.827Z] 2024-01-19 01:08:52 +0000:[INFO] [{VerifyRBACForAppAdmin}] Create cluster [source-cluster] in org [default] [2024-01-19T01:08:52.828Z] 2024-01-19 01:08:52 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T01:08:52.828Z] 2024-01-19 01:08:52 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T01:08:52.828Z] 2024-01-19 01:08:52 +0000:[INFO] [{VerifyRBACForAppAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:08:52.828Z] 2024-01-19 01:08:52 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T01:08:52.828Z] 2024-01-19 01:08:52 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T01:08:52.828Z] 2024-01-19 01:08:52 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-dqrci] for IBM [2024-01-19T01:08:52.828Z] 2024-01-19 01:08:52 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-dqrci] for org [default] with [ibm] as provider [2024-01-19T01:08:52.828Z] 2024-01-19 01:08:52 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:08:52.828Z] 2024-01-19 01:08:52 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:08:52.828Z] 2024-01-19 01:08:52 +0000:[WARNING] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-dqrci] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user px-backup-app.admin]] [2024-01-19T01:08:52.828Z] 2024-01-19 01:08:52 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user px-backup-app.admin] [2024-01-19T01:08:52.828Z] 2024-01-19 01:08:52 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-dqrci from admin context and sharing with all the users [2024-01-19T01:08:52.828Z] 2024-01-19 01:08:52 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-dqrci] for org [default] with [ibm] as provider [2024-01-19T01:08:52.828Z] 2024-01-19 01:08:52 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:08:52.828Z] 2024-01-19 01:08:52 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:08:52.829Z] 2024-01-19 01:08:52 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-dqrci] is present [2024-01-19T01:08:52.829Z] 2024-01-19 01:08:52 +0000:[INFO] [{VerifyRBACForAppAdmin}] Create cluster [source-cluster] in org [default] [2024-01-19T01:08:53.757Z] 2024-01-19 01:08:53 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-fkdjb] for IBM [2024-01-19T01:08:53.757Z] 2024-01-19 01:08:53 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-fkdjb] for org [default] with [ibm] as provider [2024-01-19T01:08:53.757Z] 2024-01-19 01:08:53 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:08:53.757Z] 2024-01-19 01:08:53 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:08:53.757Z] 2024-01-19 01:08:53 +0000:[WARNING] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-fkdjb] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user px-backup-app.admin]] [2024-01-19T01:08:53.757Z] 2024-01-19 01:08:53 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user px-backup-app.admin] [2024-01-19T01:08:54.015Z] 2024-01-19 01:08:53 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-fkdjb from admin context and sharing with all the users [2024-01-19T01:08:54.016Z] 2024-01-19 01:08:53 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-fkdjb] for org [default] with [ibm] as provider [2024-01-19T01:08:54.016Z] 2024-01-19 01:08:53 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:08:54.016Z] 2024-01-19 01:08:53 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:08:54.016Z] 2024-01-19 01:08:53 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-fkdjb] is present [2024-01-19T01:08:54.275Z] 2024-01-19 01:08:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:08:54.836Z] INFO[2024-01-19 01:08:54] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T01:08:54.836Z] INFO[2024-01-19 01:08:54] Actual:Online, Expected: Online [2024-01-19T01:08:54.836Z] INFO[2024-01-19 01:08:54] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T01:08:54.836Z] INFO[2024-01-19 01:08:54] Actual:true, Expected: true [2024-01-19T01:08:54.836Z] INFO[2024-01-19 01:08:54] Verifying : Description : Fetching [destination-cluster] cluster uid [2024-01-19T01:08:54.836Z] INFO[2024-01-19 01:08:54] Actual:true, Expected: true [2024-01-19T01:08:54.836Z] 2024-01-19 01:08:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] Taking backup of applications from the app-admin user testuser1-1705625950 [2024-01-19T01:08:55.093Z] 2024-01-19 01:08:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T01:08:55.093Z] 2024-01-19 01:08:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] Volume snapshot classes found in the cluster - [2024-01-19T01:08:55.093Z] 2024-01-19 01:08:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] ibmc-vpcblock-snapshot [2024-01-19T01:08:55.093Z] 2024-01-19 01:08:54 +0000:[INFO] [{VerifyRBACForAppAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T01:10:02.706Z] 2024-01-19 01:09:55 +0000:[INFO] [{VerifyRBACForAppAdmin}] first schedule backup for schedule name [backup-schedule-nxxtr] is [backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855] [2024-01-19T01:10:02.706Z] 2024/01/19 01:09:55 DoRetryWithTimeout - Error: {backup status for [backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:10:29.199Z] 2024/01/19 01:10:25 DoRetryWithTimeout - Error: {backup status for [backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:11:01.231Z] 2024-01-19 01:10:55 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating backup [backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855] in org [default] [2024-01-19T01:11:01.231Z] 2024-01-19 01:10:55 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855] [2024-01-19T01:11:01.231Z] 2024-01-19 01:10:55 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87888-01-19-00h57m39s] in backup [backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855] [2024-01-19T01:11:01.231Z] 2024-01-19 01:10:55 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87888-01-19-00h57m39s] in backup [backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855] [2024-01-19T01:11:01.231Z] 2024-01-19 01:10:55 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87888-01-19-00h57m39s] [2024-01-19T01:11:01.231Z] 2024-01-19 01:10:56 +0000:[WARNING] [{VerifyRBACForAppAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T01:11:01.231Z] 2024-01-19 01:10:56 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T01:11:01.231Z] 2024-01-19 01:10:56 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[mysql-data]] [2024-01-19T01:11:01.231Z] 2024-01-19 01:10:56 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T01:11:01.231Z] 2024-01-19 01:10:56 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating CR cleanup [2024-01-19T01:11:01.231Z] 2024-01-19 01:10:56 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T01:11:01.231Z] 2024-01-19 01:10:56 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] [2024-01-19T01:11:01.231Z] 2024-01-19 01:10:56 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:11:01.231Z] 2024/01/19 01:10:56 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:11:01.489Z] 2024-01-19 01:11:01 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] [2024-01-19T01:11:01.489Z] 2024-01-19 01:11:01 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:11:01.489Z] 2024/01/19 01:11:01 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:11:06.740Z] 2024-01-19 01:11:06 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] [2024-01-19T01:11:06.740Z] 2024-01-19 01:11:06 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:11:06.740Z] 2024/01/19 01:11:06 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:11:12.058Z] 2024-01-19 01:11:11 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] [2024-01-19T01:11:12.058Z] 2024-01-19 01:11:11 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:11:12.058Z] 2024/01/19 01:11:11 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:11:17.311Z] 2024-01-19 01:11:16 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] [2024-01-19T01:11:17.311Z] 2024-01-19 01:11:16 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:11:17.311Z] 2024/01/19 01:11:16 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:11:22.557Z] 2024-01-19 01:11:21 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] [2024-01-19T01:11:22.557Z] 2024-01-19 01:11:21 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:11:22.557Z] 2024/01/19 01:11:21 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:11:26.759Z] 2024-01-19 01:11:26 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] [2024-01-19T01:11:26.759Z] 2024-01-19 01:11:26 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:11:26.759Z] 2024/01/19 01:11:26 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:11:32.003Z] 2024-01-19 01:11:31 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] [2024-01-19T01:11:32.003Z] 2024-01-19 01:11:31 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:11:32.003Z] 2024/01/19 01:11:31 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:11:37.250Z] 2024-01-19 01:11:36 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] [2024-01-19T01:11:37.250Z] 2024-01-19 01:11:36 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:11:37.250Z] 2024/01/19 01:11:36 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:11:42.523Z] 2024-01-19 01:11:41 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] [2024-01-19T01:11:42.523Z] 2024-01-19 01:11:41 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:11:42.523Z] 2024/01/19 01:11:41 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:11:47.769Z] 2024-01-19 01:11:46 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] [2024-01-19T01:11:47.769Z] 2024-01-19 01:11:46 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:11:47.769Z] 2024/01/19 01:11:46 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:11:51.937Z] 2024-01-19 01:11:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] [2024-01-19T01:11:51.937Z] 2024-01-19 01:11:51 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:11:51.937Z] 2024/01/19 01:11:51 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:11:57.184Z] 2024-01-19 01:11:56 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] [2024-01-19T01:11:57.184Z] 2024-01-19 01:11:56 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:11:57.184Z] 2024/01/19 01:11:56 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:12:02.427Z] 2024-01-19 01:12:01 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] [2024-01-19T01:12:02.427Z] 2024-01-19 01:12:01 +0000:[INFO] [{VerifyRBACForAppAdmin}] CR found for [[backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]] under [mysql-ibm-pxb-0-87888-01-19-00h57m39s] namespace [2024-01-19T01:12:02.427Z] 2024/01/19 01:12:01 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855]}, Next try in [5s], timeout [5m0s] [2024-01-19T01:12:07.671Z] 2024-01-19 01:12:06 +0000:[INFO] [{VerifyRBACForAppAdmin}] All CRs in [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[]] [2024-01-19T01:12:07.671Z] INFO[2024-01-19 01:12:06] Verifying : Description : Verifying creation and validation of schedule backup with schedule name [backup-schedule-nxxtr] [2024-01-19T01:12:07.671Z] INFO[2024-01-19 01:12:06] Actual:true, Expected: true [2024-01-19T01:12:07.671Z] INFO[2024-01-19 01:12:07] Verifying : Description : Suspending Backup Schedule [backup-schedule-nxxtr] for user [testuser1-1705625950] [2024-01-19T01:12:07.671Z] INFO[2024-01-19 01:12:07] Actual:true, Expected: true [2024-01-19T01:12:07.671Z] 2024-01-19 01:12:07 +0000:[INFO] [{VerifyRBACForAppAdmin}] Taking restore of applications from the App-admin user testuser1-1705625950 [2024-01-19T01:12:07.930Z] 2024/01/19 01:12:07 DoRetryWithTimeout - Error: {restore status for [tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T01:12:39.997Z] 2024/01/19 01:12:37 DoRetryWithTimeout - Error: {restore status for [tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:13:12.031Z] 2024-01-19 01:13:07 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.CreateRestore:#936] - Restore [tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855] created successfully [2024-01-19T01:13:12.031Z] 2024-01-19 01:13:07 +0000:[INFO] [{VerifyRBACForAppAdmin}] Switching cluster context to cluster [destination-cluster] [2024-01-19T01:13:12.031Z] 2024-01-19 01:13:07 +0000:[INFO] [{VerifyRBACForAppAdmin}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T01:13:12.031Z] 2024-01-19 01:13:07 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-87888-01-19-00h57m39s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-19T01:13:12.031Z] 2024-01-19 01:13:07 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-87888-01-19-00h57m39s] [2024-01-19T01:13:12.031Z] 2024-01-19 01:13:07 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T01:13:12.031Z] 2024-01-19 01:13:07 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T01:13:12.031Z] 2024-01-19 01:13:07 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating restore [tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855] in org [default] [2024-01-19T01:13:12.031Z] 2024-01-19 01:13:07 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855] [2024-01-19T01:13:12.031Z] 2024-01-19 01:13:07 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855] [2024-01-19T01:13:12.032Z] 2024-01-19 01:13:07 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-87888-01-19-00h57m39s] in restore [tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855] [2024-01-19T01:13:12.032Z] 2024-01-19 01:13:07 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-87888-01-19-00h57m39s]) was found in restore [tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87888-01-19-00h57m39s] [2024-01-19T01:13:12.032Z] 2024-01-19 01:13:07 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-87888-01-19-00h57m39s]) found in the restore [tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855] was also present on the cluster/namespace [mysql-ibm-pxb-0-87888-01-19-00h57m39s] [2024-01-19T01:13:12.032Z] 2024-01-19 01:13:07 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-87888-01-19-00h57m39s]) was found in restore [tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87888-01-19-00h57m39s] [2024-01-19T01:13:12.032Z] 2024-01-19 01:13:07 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-87888-01-19-00h57m39s]) found in the restore [tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855] was also present on the cluster/namespace [mysql-ibm-pxb-0-87888-01-19-00h57m39s] [2024-01-19T01:13:12.032Z] 2024-01-19 01:13:07 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-87888-01-19-00h57m39s]) was found in restore [tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87888-01-19-00h57m39s] [2024-01-19T01:13:12.032Z] 2024-01-19 01:13:08 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-87888-01-19-00h57m39s]) found in the restore [tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855] was also present on the cluster/namespace [mysql-ibm-pxb-0-87888-01-19-00h57m39s] [2024-01-19T01:13:12.032Z] 2024-01-19 01:13:08 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-87888-01-19-00h57m39s]) was found in restore [tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87888-01-19-00h57m39s] [2024-01-19T01:13:12.032Z] 2024-01-19 01:13:08 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-87888-01-19-00h57m39s]) found in the restore [tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855] was also present on the cluster/namespace [mysql-ibm-pxb-0-87888-01-19-00h57m39s] [2024-01-19T01:13:12.032Z] 2024-01-19 01:13:08 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-87888-01-19-00h57m39s] in restore [tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855] [2024-01-19T01:13:12.032Z] 2024-01-19 01:13:08 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-87888-01-19-00h57m39s] in restore [tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855] [2024-01-19T01:13:12.032Z] 2024-01-19 01:13:08 +0000:[WARNING] [{VerifyRBACForAppAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T01:13:12.032Z] 2024-01-19 01:13:08 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T01:13:12.033Z] 2024-01-19 01:13:08 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87888-01-19-00h57m39s] are [[mysql-data]] [2024-01-19T01:13:12.033Z] 2024-01-19 01:13:08 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.ValidateRestore:#2240] - in restore [tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855], the status of the restored volume [pvc-0490dd20-9f7b-4ae8-9fd6-cf56ad4f540d] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-19T01:13:12.033Z] 2024-01-19 01:13:08 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validate applications in restored namespace [mysql-ibm-pxb-0-87888-01-19-00h57m39s] due to restore [tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855] [2024-01-19T01:13:12.033Z] 2024-01-19 01:13:08 +0000:[INFO] [{VerifyRBACForAppAdmin}] Validating mysql-ibm app [2024-01-19T01:13:12.033Z] 2024-01-19 01:13:08 +0000:[INFO] [{VerifyRBACForAppAdmin}] wait for mysql-ibm app to start running [2024-01-19T01:13:12.033Z] 2024-01-19 01:13:08 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T01:13:12.033Z] 2024-01-19 01:13:08 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T01:13:12.033Z] 2024-01-19 01:13:08 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T01:13:12.033Z] 2024-01-19 01:13:08 +0000:[INFO] [{VerifyRBACForAppAdmin}] Switching cluster context back to cluster path [/tmp/destination-config] [2024-01-19T01:13:12.033Z] 2024-01-19 01:13:08 +0000:[INFO] [{VerifyRBACForAppAdmin}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T01:13:12.033Z] INFO[2024-01-19 01:13:08] Verifying : Description : Verifying creation of restore tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855 of backup backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855 [2024-01-19T01:13:12.033Z] ERRO[2024-01-19 01:13:08] ValidateRestore Errors: {in restore [tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855], said restored volume [pvc-0490dd20-9f7b-4ae8-9fd6-cf56ad4f540d] cannot be found in the actual cluster [mysql-ibm-pxb-0-87888-01-19-00h57m39s]} [2024-01-19T01:13:12.033Z] 2024/01/19 01:13:08 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-nxxtr] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T01:13:20.122Z] 2024/01/19 01:13:18 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-nxxtr] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T01:13:30.068Z] 2024/01/19 01:13:28 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-nxxtr] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T01:13:40.022Z] 2024/01/19 01:13:38 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-nxxtr] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T01:13:49.966Z] 2024/01/19 01:13:48 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-nxxtr] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T01:13:59.922Z] 2024/01/19 01:13:58 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-nxxtr] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T01:14:09.866Z] 2024/01/19 01:14:08 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-nxxtr] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T01:14:19.844Z] 2024/01/19 01:14:18 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-nxxtr] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T01:14:29.828Z] 2024/01/19 01:14:29 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-nxxtr] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T01:14:39.781Z] 2024/01/19 01:14:39 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-nxxtr] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T01:14:41.669Z] INFO[2024-01-19 01:14:41] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T01:14:41.669Z] INFO[2024-01-19 01:14:41] sched stats: tick interval: 34841 < 1.2s (100%) [2024-01-19T01:14:41.669Z] INFO[2024-01-19 01:14:41] sched stats: tick process durations: 34840 < 500ms (100%) [2024-01-19T01:14:49.756Z] 2024/01/19 01:14:49 DoRetryWithTimeout - Error: {backupSchedule [backup-schedule-nxxtr] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T01:14:59.719Z] INFO[2024-01-19 01:14:59] Verifying : Description : Deleting Backup Schedule [backup-schedule-nxxtr] for user [testuser1-1705625950] [2024-01-19T01:14:59.719Z] INFO[2024-01-19 01:14:59] Actual:true, Expected: true [2024-01-19T01:14:59.719Z] 2024-01-19 01:14:59 +0000:[INFO] [{VerifyRBACForAppAdmin}] Deleting the deployed apps after the testcase [2024-01-19T01:14:59.719Z] 2024-01-19 01:14:59 +0000:[INFO] [{VerifyRBACForAppAdmin}] Destroying apps [2024-01-19T01:14:59.719Z] 2024-01-19 01:14:59 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T01:14:59.720Z] 2024-01-19 01:14:59 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T01:14:59.720Z] 2024-01-19 01:14:59 +0000:[INFO] [{VerifyRBACForAppAdmin}] start destroying mysql-ibm app [2024-01-19T01:14:59.720Z] 2024-01-19 01:14:59 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T01:14:59.720Z] 2024-01-19 01:14:59 +0000:[INFO] [{VerifyRBACForAppAdmin}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T01:14:59.720Z] 2024-01-19 01:14:59 +0000:[WARNING] [{VerifyRBACForAppAdmin}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T01:14:59.720Z] INFO[2024-01-19 01:14:59] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T01:14:59.720Z] INFO[2024-01-19 01:14:59] Actual:true, Expected: true [2024-01-19T01:14:59.720Z] 2024-01-19 01:14:59 +0000:[INFO] [{VerifyRBACForAppAdmin}] Deleting labels from namespaces - [mysql-ibm-pxb-0-87888-01-19-00h57m39s] [2024-01-19T01:14:59.720Z] INFO[2024-01-19 01:14:59] Verifying : Description : Deleting labels [map[0-f8cd8dc4-98ab-48f1-b5e4-477a5a43c57d:f8cd8dc4-98ab-48f1-b5e4-477a5a43c57d 1-f8cd8dc4-98ab-48f1-b5e4-477a5a43c57d:f8cd8dc4-98ab-48f1-b5e4-477a5a43c57d]] from namespaces [[mysql-ibm-pxb-0-87888-01-19-00h57m39s]] [2024-01-19T01:14:59.720Z] INFO[2024-01-19 01:14:59] Actual:true, Expected: true [2024-01-19T01:14:59.720Z] 2024-01-19 01:14:59 +0000:[INFO] [{VerifyRBACForAppAdmin}] Deleting the px-backup objects [2024-01-19T01:14:59.720Z] 2024-01-19 01:14:59 +0000:[INFO] [{VerifyRBACForAppAdmin}] Cleaning backup locations in map [map[bd1936a0-e91f-4f7c-b9b3-22720421495b:autogenerated-backup-location-tabml]], cloud credential [autogenerated-cred-mjwsl], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T01:14:59.976Z] INFO[2024-01-19 01:14:59] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-tabml] [2024-01-19T01:14:59.976Z] INFO[2024-01-19 01:14:59] Actual:true, Expected: true [2024-01-19T01:14:59.976Z] 2024-01-19 01:14:59 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-tabml] is present [2024-01-19T01:14:59.976Z] 2024/01/19 01:14:59 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-tabml is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T01:15:31.999Z] 2024-01-19 01:15:29 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-tabml] is present [2024-01-19T01:15:31.999Z] 2024/01/19 01:15:29 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-tabml is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T01:16:04.032Z] 2024-01-19 01:16:00 +0000:[INFO] [{VerifyRBACForAppAdmin}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 aws-global-aws-px-backup-system-test-4cac76dc-bl autogenerated-backup-location-dctzc] [2024-01-19T01:16:04.032Z] INFO[2024-01-19 01:16:00] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-tabml [2024-01-19T01:16:04.032Z] INFO[2024-01-19 01:16:00] Actual:true, Expected: true [2024-01-19T01:16:04.032Z] 2024-01-19 01:16:00 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-mjwsl] is present [2024-01-19T01:16:04.032Z] INFO[2024-01-19 01:16:00] Verifying : Description : Verifying if cloud cred [autogenerated-cred-mjwsl] is present [2024-01-19T01:16:04.032Z] INFO[2024-01-19 01:16:00] Actual:true, Expected: true [2024-01-19T01:16:04.032Z] INFO[2024-01-19 01:16:00] Verifying : Description : Verifying deletion of cloud cred [autogenerated-cred-mjwsl] [2024-01-19T01:16:04.032Z] ERRO[2024-01-19 01:16:00] Actual:rpc error: code = Unknown desc = backup location [autogenerated-backup-location-dctzc] has reference to cloud credential [autogenerated-cred-mjwsl:85d80493-e444-450d-8e2e-b57b72a3f791], Expected:  [2024-01-19T01:16:04.032Z] 2024-01-19 01:16:00 +0000:[INFO] [{VerifyRBACForAppAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T01:16:04.032Z] 2024-01-19 01:16:00 +0000:[INFO] [{VerifyRBACForAppAdmin}] Switching context to [/tmp/source-config] [2024-01-19T01:16:04.032Z] 2024-01-19 01:16:00 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T01:16:04.032Z] 2024-01-19 01:16:00 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T01:16:04.032Z] 2024-01-19 01:16:00 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T01:16:04.032Z] 2024-01-19 01:16:00 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T01:16:04.032Z] 2024-01-19 01:16:00 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T01:16:04.032Z] 2024-01-19 01:16:00 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T01:16:04.032Z] 2024-01-19 01:16:01 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T01:16:04.032Z] 2024-01-19 01:16:01 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T01:16:04.032Z] 2024-01-19 01:16:01 +0000:[INFO] [{VerifyRBACForAppAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T01:16:04.033Z] 2024/01/19 01:16:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:16:12.115Z] 2024/01/19 01:16:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:16:24.283Z] 2024/01/19 01:16:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:16:32.368Z] 2024/01/19 01:16:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:16:42.372Z] 2024/01/19 01:16:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:16:52.319Z] 2024/01/19 01:16:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:17:04.486Z] 2024/01/19 01:17:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:17:12.576Z] 2024/01/19 01:17:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:17:22.520Z] 2024/01/19 01:17:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:17:32.463Z] 2024/01/19 01:17:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:17:44.628Z] 2024/01/19 01:17:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:17:53.841Z] 2024/01/19 01:17:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:18:03.781Z] 2024/01/19 01:18:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:18:13.721Z] 2024/01/19 01:18:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:18:23.735Z] 2024/01/19 01:18:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:18:33.671Z] 2024/01/19 01:18:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:18:43.614Z] 2024/01/19 01:18:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:18:53.555Z] 2024/01/19 01:18:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:19:03.500Z] 2024/01/19 01:19:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:19:13.438Z] 2024/01/19 01:19:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:19:23.391Z] 2024/01/19 01:19:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:19:33.331Z] 2024/01/19 01:19:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:19:43.315Z] 2024/01/19 01:19:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:19:53.269Z] 2024/01/19 01:19:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:20:03.216Z] 2024/01/19 01:20:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:20:13.158Z] 2024/01/19 01:20:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:20:23.104Z] 2024/01/19 01:20:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:20:33.072Z] 2024/01/19 01:20:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:20:45.261Z] 2024/01/19 01:20:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:20:53.347Z] 2024/01/19 01:20:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:21:03.310Z] 2024-01-19 01:21:03 +0000:[ERROR] [{VerifyRBACForAppAdmin}] [backup.glob..func45.3.1:#1624] - Unable to switch context to source cluster [source-cluster]. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.310Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.310Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.310Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.310Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.310Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.311Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.311Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.311Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.311Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.311Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.311Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.311Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.311Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.311Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.311Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.311Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.311Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.311Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.311Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.311Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.311Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.311Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.311Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.311Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.311Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.311Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.311Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.311Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:21:03.312Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T01:21:03.312Z] [2024-01-19T01:21:03.312Z] ------------------------------ [2024-01-19T01:21:03.312Z] • Failure [1403.949 seconds] [2024-01-19T01:21:03.312Z] {VerifyRBACForAppAdmin} [2024-01-19T01:21:03.312Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_rbac_test.go:1094 [2024-01-19T01:21:03.312Z] Validates the RBAC operation for app-admin user [It] [2024-01-19T01:21:03.312Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_rbac_test.go:1154 [2024-01-19T01:21:03.312Z] [2024-01-19T01:21:03.312Z] Unexpected error: [2024-01-19T01:21:03.312Z] <*errors.errorString | 0xc0010fb6c0>: { [2024-01-19T01:21:03.312Z] s: "error: ValidateRestore Errors: {in restore [tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855], said restored volume [pvc-0490dd20-9f7b-4ae8-9fd6-cf56ad4f540d] cannot be found in the actual cluster [mysql-ibm-pxb-0-87888-01-19-00h57m39s]}, Description: Verifying creation of restore tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855 of backup backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855", [2024-01-19T01:21:03.312Z] } [2024-01-19T01:21:03.312Z] error: ValidateRestore Errors: {in restore [tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855], said restored volume [pvc-0490dd20-9f7b-4ae8-9fd6-cf56ad4f540d] cannot be found in the actual cluster [mysql-ibm-pxb-0-87888-01-19-00h57m39s]}, Description: Verifying creation of restore tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855 of backup backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855 [2024-01-19T01:21:03.312Z] occurred [2024-01-19T01:21:03.312Z] [2024-01-19T01:21:03.312Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-19T01:21:03.312Z] [2024-01-19T01:21:03.312Z] Full Stack Trace [2024-01-19T01:21:03.312Z] github.com/portworx/torpedo/pkg/aetosutil.(*Dashboard).VerifyFatal(0x7f8804b?, {0x702af20?, 0xc0010fb4f0}, {0x0?, 0x0}, {0xc001490160, 0xa4}) [2024-01-19T01:21:03.312Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 +0x143 [2024-01-19T01:21:03.312Z] github.com/portworx/torpedo/tests/backup.glob..func45.2.20() [2024-01-19T01:21:03.312Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_rbac_test.go:1467 +0x605 [2024-01-19T01:21:03.312Z] github.com/portworx/torpedo/tests/backup.glob..func45.2() [2024-01-19T01:21:03.312Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_rbac_test.go:1460 +0x186f [2024-01-19T01:21:03.312Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T01:21:03.312Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T01:21:03.312Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T01:21:03.312Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T01:21:03.312Z] created by testing.(*T).Run [2024-01-19T01:21:03.312Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T01:21:03.312Z] ------------------------------ [2024-01-19T01:21:03.312Z] STEP: schedule applications [2024-01-19T01:21:03.312Z] STEP: Validate applications [2024-01-19T01:21:03.312Z] STEP: validate applications [2024-01-19T01:21:03.312Z] STEP: validate mysql-ibm app's volumes [2024-01-19T01:21:03.312Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T01:21:03.312Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T01:21:03.312Z] STEP: get mysql-ibm app's volume: pvc-4bfa2c58-3699-4948-b1e7-a15450030f51 inspected by the volume driver [2024-01-19T01:21:03.312Z] STEP: wait for mysql-ibm app to start running [2024-01-19T01:21:03.312Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T01:21:03.312Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T01:21:03.312Z] STEP: Validate Px pod restart count [2024-01-19T01:21:03.312Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T01:21:03.312Z] STEP: Adding labels to namespaces [2024-01-19T01:21:03.312Z] STEP: Generating namespace label string from label map for namespaces [2024-01-19T01:21:03.312Z] STEP: Create a user with px-backup-app.admin role [2024-01-19T01:21:03.312Z] STEP: Verify App-Admin User doesnt have permission to create cloud credential [2024-01-19T01:21:03.312Z] STEP: Verify App-Admin User doesnt have permission to create role [2024-01-19T01:21:03.312Z] STEP: Adding Credentials and BackupLocation from px-admin user and making it public [2024-01-19T01:21:03.312Z] STEP: Verify App-Admin User can create backup location with cloud cred shared from px-admin [2024-01-19T01:21:03.312Z] STEP: Verify App-Admin User can't share backup location created with shared cloud cred from px-admin [2024-01-19T01:21:03.313Z] STEP: Verify App-Admin User has permission to create a schedule policy [2024-01-19T01:21:03.313Z] STEP: Verify App-Admin User has permission to create pre and post exec rules for applications [2024-01-19T01:21:03.313Z] STEP: Verify px-admin group user can list RBAC resources created by app-Admin User [2024-01-19T01:21:03.313Z] STEP: Create Users with Different types of roles [2024-01-19T01:21:03.313Z] STEP: Verify App-Admin User has permission to share schedule-policy and rules with non-admin users [2024-01-19T01:21:03.313Z] STEP: Create source and destination cluster from the user testuser1-1705625962 [2024-01-19T01:21:03.313Z] STEP: Create source and destination cluster from the user testuser1-1705625965 [2024-01-19T01:21:03.313Z] STEP: Create source and destination cluster from the user testuser1-1705625968 [2024-01-19T01:21:03.313Z] STEP: Take backup of applications from the user testuser1-1705625968 [2024-01-19T01:21:03.313Z] STEP: Take backup of applications from the user testuser1-1705625962 [2024-01-19T01:21:03.313Z] STEP: Take backup of applications from the user testuser1-1705625965 [2024-01-19T01:21:03.313Z] STEP: Take schedule backup of applications from the user testuser1-1705625968 [2024-01-19T01:21:03.313Z] STEP: Take schedule backup of applications from the user testuser1-1705625962 [2024-01-19T01:21:03.313Z] STEP: Take schedule backup of applications from the user testuser1-1705625965 [2024-01-19T01:21:03.313Z] STEP: Delete user testuser1-1705625968 backups from the user context [2024-01-19T01:21:03.313Z] STEP: Delete user testuser1-1705625962 backups from the user context [2024-01-19T01:21:03.313Z] STEP: Delete user testuser1-1705625965 backups from the user context [2024-01-19T01:21:03.313Z] STEP: Delete user testuser1-1705625962 backup schedule [2024-01-19T01:21:03.313Z] STEP: Delete user testuser1-1705625965 backup schedule [2024-01-19T01:21:03.313Z] STEP: Delete user testuser1-1705625968 backup schedule [2024-01-19T01:21:03.313Z] STEP: Delete user testuser1-1705625965 source and destination cluster from the user context [2024-01-19T01:21:03.313Z] STEP: Delete user testuser1-1705625968 source and destination cluster from the user context [2024-01-19T01:21:03.313Z] STEP: Delete user testuser1-1705625962 source and destination cluster from the user context [2024-01-19T01:21:03.313Z] STEP: Create source and destination cluster from the app-admin user testuser1-1705625950 [2024-01-19T01:21:03.313Z] STEP: Take backup of applications from the App-admin user testuser1-1705625950 [2024-01-19T01:21:03.313Z] STEP: Take restore of applications from the App-admin user testuser1-1705625950 [2024-01-19T01:21:03.313Z] STEP: validate mysql-ibm app's volumes [2024-01-19T01:21:03.313Z] STEP: wait for mysql-ibm app to start running [2024-01-19T01:21:03.313Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T01:21:03.313Z] STEP: Validate Px pod restart count [2024-01-19T01:21:03.313Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T01:21:03.313Z] STEP: destroy apps [2024-01-19T01:21:03.313Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T01:21:03.313Z] STEP: start destroying mysql-ibm app [2024-01-19T01:21:03.313Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T01:21:03.313Z] 2024-01-19 01:21:03 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Log Dir: /testresults//BackupLocationWithEncryptionKey.log [2024-01-19T01:21:03.314Z] INFO[2024-01-19 01:21:03] --------Test Start------ [2024-01-19T01:21:03.314Z] INFO[2024-01-19 01:21:03] #Test: BackupLocationWithEncryptionKey [2024-01-19T01:21:03.314Z] INFO[2024-01-19 01:21:03] #Description: Creating Backup Location with Encryption Keys [2024-01-19T01:21:03.314Z] INFO[2024-01-19 01:21:03] ------------------------ [2024-01-19T01:21:03.314Z] 2024-01-19 01:21:03 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Deploy applications [2024-01-19T01:21:03.314Z] 2024-01-19 01:21:03 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T01:21:03.314Z] 2024-01-19 01:21:03 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T01:21:03.314Z] 2024-01-19 01:21:03 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T01:21:03.314Z] 2024-01-19 01:21:03 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T01:21:03.314Z] 2024-01-19 01:21:03 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T01:21:03.314Z] 2024-01-19 01:21:03 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T01:21:03.315Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T01:21:03.575Z] 2024-01-19 01:21:03 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T01:21:03.575Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T01:21:03.575Z] 2024-01-19 01:21:03 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T01:21:03.575Z] 2024-01-19 01:21:03 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Validate applications [2024-01-19T01:21:03.575Z] 2024-01-19 01:21:03 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Validating mysql-ibm app [2024-01-19T01:21:03.575Z] 2024-01-19 01:21:03 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Validating mysql-ibm app's volumes [2024-01-19T01:21:03.575Z] 2024-01-19 01:21:03 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T01:21:03.576Z] 2024/01/19 01:21:03 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T01:21:13.525Z] 2024/01/19 01:21:13 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T01:21:25.692Z] 2024-01-19 01:21:23 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-79918-01-19-01h21m03s [2024-01-19T01:21:25.692Z] 2024-01-19 01:21:23 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T01:21:25.692Z] 2024-01-19 01:21:23 +0000:[WARNING] [{BackupLocationWithEncryptionKey}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T01:21:25.692Z] 2024-01-19 01:21:23 +0000:[INFO] [{BackupLocationWithEncryptionKey}] wait for mysql-ibm app to start running [2024-01-19T01:21:25.692Z] 2024-01-19 01:21:23 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T01:21:25.692Z] 2024/01/19 01:21:23 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T01:21:25.692Z] pod name:mysql-7d766cd65c-mxs52 namespace:mysql-ibm-pxb-0-79918-01-19-01h21m03s running:false ready:false node:10.240.64.8 [2024-01-19T01:21:25.692Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T01:21:35.313Z] 2024/01/19 01:21:33 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T01:21:35.313Z] pod name:mysql-7d766cd65c-mxs52 namespace:mysql-ibm-pxb-0-79918-01-19-01h21m03s running:true ready:false node:10.240.64.8 [2024-01-19T01:21:35.313Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T01:21:45.263Z] 2024/01/19 01:21:43 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T01:21:45.263Z] pod name:mysql-7d766cd65c-mxs52 namespace:mysql-ibm-pxb-0-79918-01-19-01h21m03s running:true ready:false node:10.240.64.8 [2024-01-19T01:21:45.263Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T01:21:55.215Z] 2024/01/19 01:21:54 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T01:21:55.215Z] pod name:mysql-7d766cd65c-mxs52 namespace:mysql-ibm-pxb-0-79918-01-19-01h21m03s running:true ready:false node:10.240.64.8 [2024-01-19T01:21:55.215Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T01:22:05.160Z] 2024/01/19 01:22:04 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T01:22:05.160Z] pod name:mysql-7d766cd65c-mxs52 namespace:mysql-ibm-pxb-0-79918-01-19-01h21m03s running:true ready:false node:10.240.64.8 [2024-01-19T01:22:05.160Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T01:22:15.113Z] 2024/01/19 01:22:14 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T01:22:15.113Z] pod name:mysql-7d766cd65c-mxs52 namespace:mysql-ibm-pxb-0-79918-01-19-01h21m03s running:true ready:false node:10.240.64.8 [2024-01-19T01:22:15.113Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T01:22:25.077Z] 2024/01/19 01:22:24 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T01:22:25.077Z] pod name:mysql-7d766cd65c-mxs52 namespace:mysql-ibm-pxb-0-79918-01-19-01h21m03s running:true ready:false node:10.240.64.8 [2024-01-19T01:22:25.077Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T01:22:35.133Z] 2024/01/19 01:22:34 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T01:22:35.133Z] pod name:mysql-7d766cd65c-mxs52 namespace:mysql-ibm-pxb-0-79918-01-19-01h21m03s running:true ready:false node:10.240.64.8 [2024-01-19T01:22:35.133Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T01:22:45.072Z] 2024-01-19 01:22:44 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T01:22:45.073Z] 2024-01-19 01:22:44 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T01:22:45.073Z] 2024-01-19 01:22:44 +0000:[INFO] [{BackupLocationWithEncryptionKey}] validate if mysql-ibm app's volumes are setup [2024-01-19T01:22:45.073Z] 2024-01-19 01:22:44 +0000:[WARNING] [{BackupLocationWithEncryptionKey}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T01:22:45.073Z] 2024-01-19 01:22:44 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T01:22:45.073Z] 2024-01-19 01:22:44 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T01:22:45.073Z] 2024-01-19 01:22:44 +0000:[WARNING] [{BackupLocationWithEncryptionKey}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T01:22:45.328Z] 2024-01-19 01:22:45 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Creating backup location and cloud setting [2024-01-19T01:22:45.329Z] 2024-01-19 01:22:45 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [cred-aws-1705627365] for org [default] with [aws] as provider [2024-01-19T01:22:45.329Z] 2024-01-19 01:22:45 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T01:22:45.584Z] 2024-01-19 01:22:45 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cred-aws-1705627365] is present [2024-01-19T01:22:45.584Z] INFO[2024-01-19 01:22:45] Verifying : Description : Verifying creation of cloud credential named [cred-aws-1705627365] for org [default] with [aws] as provider [2024-01-19T01:22:45.584Z] INFO[2024-01-19 01:22:45] Actual:true, Expected: true [2024-01-19T01:22:50.838Z] INFO[2024-01-19 01:22:49] Verifying : Description : Creating backup location [aws-global-aws-px-backup-system-test-4cac76dc-bl-1705627365] with CloudCred [cred-aws-1705627365] [2024-01-19T01:22:50.839Z] INFO[2024-01-19 01:22:49] Actual:true, Expected: true [2024-01-19T01:22:50.839Z] 2024-01-19 01:22:49 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Register clusters for backup [2024-01-19T01:22:50.839Z] INFO[2024-01-19 01:22:50] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T01:22:50.839Z] INFO[2024-01-19 01:22:50] Actual:true, Expected: true [2024-01-19T01:22:50.839Z] 2024-01-19 01:22:50 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T01:22:50.839Z] 2024-01-19 01:22:50 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T01:22:50.839Z] 2024-01-19 01:22:50 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T01:22:50.839Z] 2024-01-19 01:22:50 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T01:22:50.839Z] 2024-01-19 01:22:50 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Create cluster [source-cluster] in org [default] [2024-01-19T01:22:50.839Z] 2024-01-19 01:22:50 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T01:22:50.839Z] 2024-01-19 01:22:50 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T01:22:50.839Z] 2024-01-19 01:22:50 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:22:50.839Z] 2024-01-19 01:22:50 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T01:22:50.839Z] 2024-01-19 01:22:50 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T01:22:50.839Z] 2024-01-19 01:22:50 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-fyjpt] for IBM [2024-01-19T01:22:50.839Z] 2024-01-19 01:22:50 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-fyjpt] for org [default] with [ibm] as provider [2024-01-19T01:22:50.839Z] 2024-01-19 01:22:50 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:22:50.839Z] 2024-01-19 01:22:50 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:22:50.839Z] 2024-01-19 01:22:50 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-fyjpt] is present [2024-01-19T01:22:50.839Z] 2024-01-19 01:22:50 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Create cluster [source-cluster] in org [default] [2024-01-19T01:22:51.400Z] 2024-01-19 01:22:51 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-lrlgh] for IBM [2024-01-19T01:22:51.400Z] 2024-01-19 01:22:51 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-lrlgh] for org [default] with [ibm] as provider [2024-01-19T01:22:51.400Z] 2024-01-19 01:22:51 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:22:51.400Z] 2024-01-19 01:22:51 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:22:51.400Z] 2024-01-19 01:22:51 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-lrlgh] is present [2024-01-19T01:22:51.400Z] 2024-01-19 01:22:51 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:22:51.400Z] INFO[2024-01-19 01:22:51] Verifying : Description : Creating source and destination cluster [2024-01-19T01:22:51.400Z] INFO[2024-01-19 01:22:51] Actual:true, Expected: true [2024-01-19T01:22:51.401Z] INFO[2024-01-19 01:22:51] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T01:22:51.401Z] INFO[2024-01-19 01:22:51] Actual:Online, Expected: Online [2024-01-19T01:22:51.401Z] INFO[2024-01-19 01:22:51] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T01:22:51.401Z] INFO[2024-01-19 01:22:51] Actual:true, Expected: true [2024-01-19T01:22:51.401Z] 2024-01-19 01:22:51 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Taking backup of applications [2024-01-19T01:22:51.657Z] 2024-01-19 01:22:51 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T01:22:51.913Z] 2024-01-19 01:22:51 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Volume snapshot classes found in the cluster - [2024-01-19T01:22:51.913Z] 2024-01-19 01:22:51 +0000:[INFO] [{BackupLocationWithEncryptionKey}] ibmc-vpcblock-snapshot [2024-01-19T01:22:51.913Z] 2024-01-19 01:22:51 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T01:22:52.476Z] 2024/01/19 01:22:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-wwdiljjnxv] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T01:23:24.517Z] 2024/01/19 01:23:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-wwdiljjnxv] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:23:56.566Z] 2024-01-19 01:23:52 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.CreateBackup:#313] - Backup [tp-backup-wwdiljjnxv] created successfully [2024-01-19T01:23:56.566Z] 2024-01-19 01:23:52 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Validating backup [tp-backup-wwdiljjnxv] in org [default] [2024-01-19T01:23:56.566Z] 2024-01-19 01:23:52 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-wwdiljjnxv] [2024-01-19T01:23:56.566Z] 2024-01-19 01:23:52 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-79918-01-19-01h21m03s] in backup [tp-backup-wwdiljjnxv] [2024-01-19T01:23:56.566Z] 2024-01-19 01:23:52 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-79918-01-19-01h21m03s] in backup [tp-backup-wwdiljjnxv] [2024-01-19T01:23:56.566Z] 2024-01-19 01:23:52 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-79918-01-19-01h21m03s] [2024-01-19T01:23:56.566Z] 2024-01-19 01:23:52 +0000:[WARNING] [{BackupLocationWithEncryptionKey}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T01:23:56.566Z] 2024-01-19 01:23:52 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T01:23:56.567Z] 2024-01-19 01:23:52 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-79918-01-19-01h21m03s] are [[mysql-data]] [2024-01-19T01:23:56.567Z] 2024-01-19 01:23:52 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T01:23:56.567Z] 2024-01-19 01:23:52 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Validating CR cleanup [2024-01-19T01:23:56.567Z] 2024-01-19 01:23:52 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T01:23:56.567Z] 2024-01-19 01:23:52 +0000:[INFO] [{BackupLocationWithEncryptionKey}] All CRs in [mysql-ibm-pxb-0-79918-01-19-01h21m03s] are [[]] [2024-01-19T01:23:56.567Z] INFO[2024-01-19 01:23:52] Verifying : Description : Creation and Validation of backup [tp-backup-wwdiljjnxv] [2024-01-19T01:23:56.567Z] INFO[2024-01-19 01:23:52] Actual:true, Expected: true [2024-01-19T01:23:56.567Z] 2024-01-19 01:23:52 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Restoring the backed up application [2024-01-19T01:23:56.567Z] 2024/01/19 01:23:53 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-wwdiljjnxv-1705627432] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T01:24:28.595Z] 2024/01/19 01:24:23 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-wwdiljjnxv-1705627432] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:24:43.473Z] INFO[2024-01-19 01:24:42] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T01:24:43.474Z] INFO[2024-01-19 01:24:42] sched stats: tick interval: 35442 < 1.2s (100%) [2024-01-19T01:24:43.474Z] INFO[2024-01-19 01:24:42] sched stats: tick process durations: 35441 < 500ms (100%) [2024-01-19T01:24:55.651Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.CreateRestore:#936] - Restore [tp-restore-tp-backup-wwdiljjnxv-1705627432] created successfully [2024-01-19T01:24:55.651Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Switching cluster context to cluster [destination-cluster] [2024-01-19T01:24:55.651Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T01:24:55.651Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-79918-01-19-01h21m03s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-19T01:24:55.651Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-79918-01-19-01h21m03s] [2024-01-19T01:24:55.651Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T01:24:55.651Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T01:24:55.651Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Validating restore [tp-restore-tp-backup-wwdiljjnxv-1705627432] in org [default] [2024-01-19T01:24:55.651Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-restore-tp-backup-wwdiljjnxv-1705627432] [2024-01-19T01:24:55.651Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-restore-tp-backup-wwdiljjnxv-1705627432] [2024-01-19T01:24:55.651Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-79918-01-19-01h21m03s] in restore [tp-restore-tp-backup-wwdiljjnxv-1705627432] [2024-01-19T01:24:55.651Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-79918-01-19-01h21m03s]) was found in restore [tp-restore-tp-backup-wwdiljjnxv-1705627432], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-79918-01-19-01h21m03s] [2024-01-19T01:24:55.651Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-79918-01-19-01h21m03s]) found in the restore [tp-restore-tp-backup-wwdiljjnxv-1705627432] was also present on the cluster/namespace [mysql-ibm-pxb-0-79918-01-19-01h21m03s] [2024-01-19T01:24:55.651Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-79918-01-19-01h21m03s]) was found in restore [tp-restore-tp-backup-wwdiljjnxv-1705627432], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-79918-01-19-01h21m03s] [2024-01-19T01:24:55.651Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-79918-01-19-01h21m03s]) found in the restore [tp-restore-tp-backup-wwdiljjnxv-1705627432] was also present on the cluster/namespace [mysql-ibm-pxb-0-79918-01-19-01h21m03s] [2024-01-19T01:24:55.652Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-79918-01-19-01h21m03s]) was found in restore [tp-restore-tp-backup-wwdiljjnxv-1705627432], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-79918-01-19-01h21m03s] [2024-01-19T01:24:55.652Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-79918-01-19-01h21m03s]) found in the restore [tp-restore-tp-backup-wwdiljjnxv-1705627432] was also present on the cluster/namespace [mysql-ibm-pxb-0-79918-01-19-01h21m03s] [2024-01-19T01:24:55.652Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-79918-01-19-01h21m03s]) was found in restore [tp-restore-tp-backup-wwdiljjnxv-1705627432], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-79918-01-19-01h21m03s] [2024-01-19T01:24:55.652Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-79918-01-19-01h21m03s]) found in the restore [tp-restore-tp-backup-wwdiljjnxv-1705627432] was also present on the cluster/namespace [mysql-ibm-pxb-0-79918-01-19-01h21m03s] [2024-01-19T01:24:55.652Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-79918-01-19-01h21m03s] in restore [tp-restore-tp-backup-wwdiljjnxv-1705627432] [2024-01-19T01:24:55.652Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-79918-01-19-01h21m03s] in restore [tp-restore-tp-backup-wwdiljjnxv-1705627432] [2024-01-19T01:24:55.652Z] 2024-01-19 01:24:53 +0000:[WARNING] [{BackupLocationWithEncryptionKey}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T01:24:55.652Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T01:24:55.652Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-79918-01-19-01h21m03s] are [[mysql-data]] [2024-01-19T01:24:55.652Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.ValidateRestore:#2240] - in restore [tp-restore-tp-backup-wwdiljjnxv-1705627432], the status of the restored volume [pvc-04d0fc4b-4d25-4291-8530-bc94f3c056f7] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-19T01:24:55.652Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Validate applications in restored namespace [mysql-ibm-pxb-0-79918-01-19-01h21m03s] due to restore [tp-restore-tp-backup-wwdiljjnxv-1705627432] [2024-01-19T01:24:55.652Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Validating mysql-ibm app [2024-01-19T01:24:55.652Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] wait for mysql-ibm app to start running [2024-01-19T01:24:55.652Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T01:24:55.652Z] 2024-01-19 01:24:53 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T01:24:55.652Z] 2024-01-19 01:24:54 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T01:24:55.653Z] 2024-01-19 01:24:54 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Switching cluster context back to cluster path [/tmp/destination-config] [2024-01-19T01:24:55.653Z] 2024-01-19 01:24:54 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T01:24:55.653Z] 2024-01-19 01:24:54 +0000:[ERROR] [{BackupLocationWithEncryptionKey}] [backup.glob..func35.2.5:#118] - tp-restore-tp-backup-wwdiljjnxv-1705627432 restore failed. Err: ValidateRestore Errors: {in restore [tp-restore-tp-backup-wwdiljjnxv-1705627432], said restored volume [pvc-04d0fc4b-4d25-4291-8530-bc94f3c056f7] cannot be found in the actual cluster [mysql-ibm-pxb-0-79918-01-19-01h21m03s]} [2024-01-19T01:24:55.653Z] 2024-01-19 01:24:54 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.glob..func35.3:#124] - Deleting backup, restore and backup location, cloud account [2024-01-19T01:24:55.653Z] INFO[2024-01-19 01:24:54] Verifying : Description : Getting the backup driver [2024-01-19T01:24:55.653Z] INFO[2024-01-19 01:24:54] Actual:true, Expected: true [2024-01-19T01:24:55.653Z] INFO[2024-01-19 01:24:54] Verifying : Description : Deleting Restore tp-restore-tp-backup-wwdiljjnxv-1705627432 [2024-01-19T01:24:55.653Z] INFO[2024-01-19 01:24:54] Actual:true, Expected: true [2024-01-19T01:24:55.653Z] INFO[2024-01-19 01:24:54] Verifying : Description : Getting backup UID for backup tp-backup-wwdiljjnxv [2024-01-19T01:24:55.653Z] INFO[2024-01-19 01:24:54] Actual:true, Expected: true [2024-01-19T01:24:55.653Z] INFO[2024-01-19 01:24:54] Verifying : Description : Deleting backup tp-backup-wwdiljjnxv [2024-01-19T01:24:55.653Z] INFO[2024-01-19 01:24:54] Actual:true, Expected: true [2024-01-19T01:24:55.653Z] 2024-01-19 01:24:54 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Cleaning backup locations in map [map[0c30795f-6b61-4efc-bc8b-0a17182586d8:aws-global-aws-px-backup-system-test-4cac76dc-bl-1705627365]], cloud credential [cred-aws-1705627365], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T01:24:55.653Z] INFO[2024-01-19 01:24:54] Verifying : Description : Verifying deletion of backup location [aws-global-aws-px-backup-system-test-4cac76dc-bl-1705627365] [2024-01-19T01:24:55.653Z] INFO[2024-01-19 01:24:54] Actual:true, Expected: true [2024-01-19T01:24:55.653Z] 2024-01-19 01:24:54 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.IsBackupLocationPresent:#2442] - Backup location [aws-global-aws-px-backup-system-test-4cac76dc-bl-1705627365] is present [2024-01-19T01:24:55.653Z] 2024/01/19 01:24:54 DoRetryWithTimeout - Error: {backup location aws-global-aws-px-backup-system-test-4cac76dc-bl-1705627365 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T01:25:27.684Z] 2024-01-19 01:25:25 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.IsBackupLocationPresent:#2442] - Backup location [aws-global-aws-px-backup-system-test-4cac76dc-bl-1705627365] is present [2024-01-19T01:25:27.684Z] 2024/01/19 01:25:25 DoRetryWithTimeout - Error: {backup location aws-global-aws-px-backup-system-test-4cac76dc-bl-1705627365 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T01:25:59.717Z] 2024-01-19 01:25:55 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 aws-global-aws-px-backup-system-test-4cac76dc-bl autogenerated-backup-location-dctzc] [2024-01-19T01:25:59.717Z] INFO[2024-01-19 01:25:55] Verifying : Description : Verifying backup location deletion status aws-global-aws-px-backup-system-test-4cac76dc-bl-1705627365 [2024-01-19T01:25:59.717Z] INFO[2024-01-19 01:25:55] Actual:true, Expected: true [2024-01-19T01:25:59.717Z] 2024-01-19 01:25:55 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cred-aws-1705627365] is present [2024-01-19T01:25:59.717Z] INFO[2024-01-19 01:25:55] Verifying : Description : Verifying if cloud cred [cred-aws-1705627365] is present [2024-01-19T01:25:59.717Z] INFO[2024-01-19 01:25:55] Actual:true, Expected: true [2024-01-19T01:25:59.717Z] INFO[2024-01-19 01:25:55] Verifying : Description : Verifying deletion of cloud cred [cred-aws-1705627365] [2024-01-19T01:25:59.717Z] INFO[2024-01-19 01:25:55] Actual:true, Expected: true [2024-01-19T01:25:59.717Z] INFO[2024-01-19 01:25:55] Verifying : Description : Deleting cloud cred cred-aws-1705627365 [2024-01-19T01:25:59.717Z] INFO[2024-01-19 01:25:55] Actual:true, Expected: true [2024-01-19T01:25:59.717Z] INFO[2024-01-19 01:25:55] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T01:25:59.717Z] INFO[2024-01-19 01:25:55] Actual:true, Expected: true [2024-01-19T01:25:59.717Z] INFO[2024-01-19 01:25:55] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T01:25:59.717Z] ERRO[2024-01-19 01:25:55] Actual:rpc error: code = FailedPrecondition desc = cluster [destination-cluster] in use by backupschedule [tp-backup-schedule-1705605709], cannot be deleted, Expected:  [2024-01-19T01:25:59.717Z] INFO[2024-01-19 01:25:55] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-zbbvt] [2024-01-19T01:25:59.717Z] ERRO[2024-01-19 01:25:55] Actual:rpc error: code = Unknown desc = cluster [destination-cluster] has reference to cloud credential [ibm-destination-config-cloud-cred-zbbvt:a271ea5b-dc9d-40f1-a82a-651539c7e486], Expected:  [2024-01-19T01:25:59.717Z] INFO[2024-01-19 01:25:56] Verifying : Description : Deleting cluster source-cluster [2024-01-19T01:25:59.717Z] INFO[2024-01-19 01:25:56] Actual:true, Expected: true [2024-01-19T01:25:59.717Z] INFO[2024-01-19 01:25:56] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-dqrci] [2024-01-19T01:25:59.717Z] INFO[2024-01-19 01:25:56] Actual:true, Expected: true [2024-01-19T01:25:59.717Z] INFO[2024-01-19 01:25:56] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T01:25:59.717Z] INFO[2024-01-19 01:25:56] Actual:true, Expected: true [2024-01-19T01:25:59.717Z] INFO[2024-01-19 01:25:57] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-fkdjb] [2024-01-19T01:25:59.717Z] INFO[2024-01-19 01:25:57] Actual:true, Expected: true [2024-01-19T01:25:59.717Z] INFO[2024-01-19 01:25:57] Verifying : Description : Deleting cluster source-cluster [2024-01-19T01:25:59.717Z] INFO[2024-01-19 01:25:57] Actual:true, Expected: true [2024-01-19T01:25:59.718Z] INFO[2024-01-19 01:25:59] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-fyjpt] [2024-01-19T01:25:59.718Z] INFO[2024-01-19 01:25:59] Actual:true, Expected: true [2024-01-19T01:25:59.718Z] INFO[2024-01-19 01:25:59] --------Test End------ [2024-01-19T01:25:59.718Z] INFO[2024-01-19 01:25:59] #Test: [2024-01-19T01:25:59.718Z] INFO[2024-01-19 01:25:59] #Description: [2024-01-19T01:25:59.718Z] INFO[2024-01-19 01:25:59] ------------------------ [2024-01-19T01:25:59.718Z] 2024-01-19 01:25:59 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.EndPxBackupTorpedoTest:#7261] - >>>> FAILED TEST: {BackupLocationWithEncryptionKey} Creating Backup Location with Encryption Keys [2024-01-19T01:25:59.718Z] 2024-01-19 01:25:59 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-79918-01-19-01h21m03s] [2024-01-19T01:25:59.718Z] 2024-01-19 01:25:59 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T01:25:59.718Z] 2024-01-19 01:25:59 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T01:25:59.975Z] 2024-01-19 01:25:59 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-79918-01-19-01h21m03s] [2024-01-19T01:25:59.975Z] 2024-01-19 01:25:59 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {BackupLocationWithEncryptionKey} Creating Backup Location with Encryption Keys [2024-01-19T01:25:59.975Z] 2024-01-19 01:25:59 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T01:25:59.975Z] 2024/01/19 01:25:59 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T01:26:05.233Z] 2024/01/19 01:26:04 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T01:26:10.544Z] 2024/01/19 01:26:09 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T01:26:15.793Z] 2024/01/19 01:26:14 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T01:26:19.963Z] 2024/01/19 01:26:19 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T01:26:25.209Z] 2024/01/19 01:26:24 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T01:26:30.472Z] 2024/01/19 01:26:29 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T01:26:35.720Z] 2024/01/19 01:26:34 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T01:26:40.989Z] 2024/01/19 01:26:39 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.0.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.7 10.240.0.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T01:26:40.989Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T01:26:40.990Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T01:26:40.990Z] {dial tcp 10.240.0.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T01:26:45.161Z] 2024-01-19 01:26:44 +0000:[WARNING] [{BackupLocationWithEncryptionKey}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.7 10.240.0.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T01:26:45.161Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T01:26:45.161Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T01:26:45.161Z] {dial tcp 10.240.0.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T01:26:45.162Z] 2024-01-19 01:26:44 +0000:[ERROR] [{BackupLocationWithEncryptionKey}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.7 10.240.0.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T01:26:45.162Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T01:26:45.162Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T01:26:45.162Z] {dial tcp 10.240.0.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T01:26:45.162Z] 2024-01-19 01:26:44 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T01:26:45.162Z] 2024-01-19 01:26:44 +0000:[INFO] [{BackupLocationWithEncryptionKey}] Switching context to [/tmp/source-config] [2024-01-19T01:26:45.162Z] 2024-01-19 01:26:44 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T01:26:45.162Z] 2024-01-19 01:26:45 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T01:26:45.162Z] 2024-01-19 01:26:45 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T01:26:45.418Z] 2024-01-19 01:26:45 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T01:26:45.418Z] 2024-01-19 01:26:45 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T01:26:45.418Z] 2024-01-19 01:26:45 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T01:26:45.979Z] 2024-01-19 01:26:45 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T01:26:46.235Z] 2024-01-19 01:26:46 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T01:26:46.796Z] 2024-01-19 01:26:46 +0000:[INFO] [{BackupLocationWithEncryptionKey}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T01:26:46.797Z] 2024/01/19 01:26:46 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:26:58.981Z] 2024/01/19 01:26:56 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:27:07.063Z] 2024/01/19 01:27:06 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:27:17.024Z] 2024/01/19 01:27:16 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:27:26.985Z] 2024/01/19 01:27:26 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:27:39.163Z] 2024/01/19 01:27:36 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:27:47.267Z] 2024/01/19 01:27:46 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:27:57.213Z] 2024/01/19 01:27:56 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:28:07.166Z] 2024/01/19 01:28:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:28:17.105Z] 2024/01/19 01:28:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:28:29.290Z] 2024/01/19 01:28:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:28:37.381Z] 2024/01/19 01:28:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:28:47.335Z] 2024/01/19 01:28:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:28:57.295Z] 2024/01/19 01:28:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:29:09.497Z] 2024/01/19 01:29:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:29:17.580Z] 2024/01/19 01:29:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:29:29.083Z] 2024/01/19 01:29:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:29:39.022Z] 2024/01/19 01:29:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:29:48.965Z] 2024/01/19 01:29:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:29:58.930Z] 2024/01/19 01:29:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:30:08.877Z] 2024/01/19 01:30:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:30:18.903Z] 2024/01/19 01:30:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:30:28.845Z] 2024/01/19 01:30:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:30:38.838Z] 2024/01/19 01:30:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:30:48.783Z] 2024/01/19 01:30:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:30:58.734Z] 2024/01/19 01:30:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:31:08.678Z] 2024/01/19 01:31:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:31:18.771Z] 2024/01/19 01:31:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:31:28.722Z] 2024/01/19 01:31:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:31:38.665Z] 2024/01/19 01:31:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:31:48.606Z] 2024-01-19 01:31:47 +0000:[ERROR] [{BackupLocationWithEncryptionKey}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.606Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.606Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.606Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.606Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.607Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.607Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.607Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.607Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.607Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.607Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.607Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.607Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.607Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.607Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.607Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.607Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.607Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.607Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.607Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.607Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.607Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.607Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.607Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.607Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.607Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.607Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.607Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.608Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:31:48.608Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T01:31:48.608Z] • Failure [644.668 seconds] [2024-01-19T01:31:48.608Z] {BackupLocationWithEncryptionKey} [2024-01-19T01:31:48.608Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_portworx_test.go:28 [2024-01-19T01:31:48.608Z] Creating Backup Location with Encryption Keys [It] [2024-01-19T01:31:48.608Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_portworx_test.go:64 [2024-01-19T01:31:48.608Z] [2024-01-19T01:31:48.608Z] Unexpected error: [2024-01-19T01:31:48.608Z] <*errors.errorString | 0xc0016cb180>: { [2024-01-19T01:31:48.608Z] s: "ValidateRestore Errors: {in restore [tp-restore-tp-backup-wwdiljjnxv-1705627432], said restored volume [pvc-04d0fc4b-4d25-4291-8530-bc94f3c056f7] cannot be found in the actual cluster [mysql-ibm-pxb-0-79918-01-19-01h21m03s]}", [2024-01-19T01:31:48.608Z] } [2024-01-19T01:31:48.608Z] ValidateRestore Errors: {in restore [tp-restore-tp-backup-wwdiljjnxv-1705627432], said restored volume [pvc-04d0fc4b-4d25-4291-8530-bc94f3c056f7] cannot be found in the actual cluster [mysql-ibm-pxb-0-79918-01-19-01h21m03s]} [2024-01-19T01:31:48.608Z] occurred [2024-01-19T01:31:48.608Z] [2024-01-19T01:31:48.608Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T01:31:48.608Z] [2024-01-19T01:31:48.608Z] Full Stack Trace [2024-01-19T01:31:48.608Z] github.com/portworx/torpedo/pkg/log.FailOnError({0x88a5060, 0xc0016cb180}, {0x7ec4b21?, 0x7?}, {0xc0014f30a8?, 0x1?, 0x1?}) [2024-01-19T01:31:48.608Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 +0x290 [2024-01-19T01:31:48.608Z] github.com/portworx/torpedo/tests/backup.glob..func35.2.5() [2024-01-19T01:31:48.608Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_portworx_test.go:118 +0x452 [2024-01-19T01:31:48.608Z] github.com/portworx/torpedo/tests/backup.glob..func35.2() [2024-01-19T01:31:48.608Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_portworx_test.go:111 +0x4e9 [2024-01-19T01:31:48.608Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T01:31:48.608Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T01:31:48.608Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T01:31:48.608Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T01:31:48.608Z] created by testing.(*T).Run [2024-01-19T01:31:48.608Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T01:31:48.608Z] ------------------------------ [2024-01-19T01:31:48.608Z] STEP: schedule applications [2024-01-19T01:31:48.608Z] STEP: Validate applications [2024-01-19T01:31:48.608Z] STEP: validate applications [2024-01-19T01:31:48.608Z] STEP: validate mysql-ibm app's volumes [2024-01-19T01:31:48.608Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T01:31:48.608Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T01:31:48.608Z] STEP: get mysql-ibm app's volume: pvc-00b6754d-7824-4ced-adc9-a525f3073022 inspected by the volume driver [2024-01-19T01:31:48.608Z] STEP: wait for mysql-ibm app to start running [2024-01-19T01:31:48.608Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T01:31:48.608Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T01:31:48.608Z] STEP: Validate Px pod restart count [2024-01-19T01:31:48.608Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T01:31:48.608Z] STEP: Creating backup location and cloud setting [2024-01-19T01:31:48.608Z] STEP: Register clusters for backup [2024-01-19T01:31:48.608Z] STEP: Taking backup of applications [2024-01-19T01:31:48.608Z] STEP: Restoring the backed up application [2024-01-19T01:31:48.608Z] STEP: validate mysql-ibm app's volumes [2024-01-19T01:31:48.608Z] STEP: wait for mysql-ibm app to start running [2024-01-19T01:31:48.608Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T01:31:48.608Z] STEP: Validate Px pod restart count [2024-01-19T01:31:48.608Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T01:31:48.608Z] 2024-01-19 01:31:47 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Log Dir: /testresults//RestartBackupPodDuringBackupSharing.log [2024-01-19T01:31:48.608Z] INFO[2024-01-19 01:31:47] --------Test Start------ [2024-01-19T01:31:48.609Z] INFO[2024-01-19 01:31:47] #Test: RestartBackupPodDuringBackupSharing [2024-01-19T01:31:48.609Z] INFO[2024-01-19 01:31:47] #Description: Restart backup pod during backup sharing [2024-01-19T01:31:48.609Z] INFO[2024-01-19 01:31:47] ------------------------ [2024-01-19T01:31:48.609Z] 2024-01-19 01:31:47 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Deploy applications [2024-01-19T01:31:48.609Z] 2024-01-19 01:31:47 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T01:31:48.609Z] 2024-01-19 01:31:47 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T01:31:48.609Z] 2024-01-19 01:31:47 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T01:31:48.609Z] 2024-01-19 01:31:47 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T01:31:48.609Z] 2024-01-19 01:31:47 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T01:31:48.609Z] 2024-01-19 01:31:47 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T01:31:48.610Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T01:31:48.610Z] 2024-01-19 01:31:47 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T01:31:48.610Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T01:31:48.610Z] 2024-01-19 01:31:48 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T01:31:48.610Z] 2024-01-19 01:31:48 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Validate applications [2024-01-19T01:31:48.610Z] 2024-01-19 01:31:48 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Validating mysql-ibm app [2024-01-19T01:31:48.610Z] 2024-01-19 01:31:48 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Validating mysql-ibm app's volumes [2024-01-19T01:31:48.610Z] 2024-01-19 01:31:48 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T01:31:48.610Z] 2024/01/19 01:31:48 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T01:31:58.557Z] 2024/01/19 01:31:58 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T01:32:08.507Z] 2024-01-19 01:32:08 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-82948-01-19-01h31m47s [2024-01-19T01:32:08.508Z] 2024-01-19 01:32:08 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T01:32:08.508Z] 2024-01-19 01:32:08 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T01:32:08.508Z] 2024-01-19 01:32:08 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] wait for mysql-ibm app to start running [2024-01-19T01:32:08.508Z] 2024-01-19 01:32:08 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T01:32:08.508Z] 2024/01/19 01:32:08 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T01:32:08.508Z] pod name:mysql-7d766cd65c-qnkq9 namespace:mysql-ibm-pxb-0-82948-01-19-01h31m47s running:false ready:false node:10.240.64.8 [2024-01-19T01:32:08.508Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T01:32:20.679Z] 2024/01/19 01:32:18 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T01:32:20.679Z] pod name:mysql-7d766cd65c-qnkq9 namespace:mysql-ibm-pxb-0-82948-01-19-01h31m47s running:true ready:false node:10.240.64.8 [2024-01-19T01:32:20.679Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T01:32:28.771Z] 2024/01/19 01:32:28 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T01:32:28.771Z] pod name:mysql-7d766cd65c-qnkq9 namespace:mysql-ibm-pxb-0-82948-01-19-01h31m47s running:true ready:false node:10.240.64.8 [2024-01-19T01:32:28.771Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T01:32:40.944Z] 2024/01/19 01:32:38 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T01:32:40.944Z] pod name:mysql-7d766cd65c-qnkq9 namespace:mysql-ibm-pxb-0-82948-01-19-01h31m47s running:true ready:false node:10.240.64.8 [2024-01-19T01:32:40.944Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T01:32:49.035Z] 2024/01/19 01:32:48 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T01:32:49.035Z] pod name:mysql-7d766cd65c-qnkq9 namespace:mysql-ibm-pxb-0-82948-01-19-01h31m47s running:true ready:false node:10.240.64.8 [2024-01-19T01:32:49.035Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T01:33:01.207Z] 2024/01/19 01:32:59 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T01:33:01.207Z] pod name:mysql-7d766cd65c-qnkq9 namespace:mysql-ibm-pxb-0-82948-01-19-01h31m47s running:true ready:false node:10.240.64.8 [2024-01-19T01:33:01.207Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T01:33:09.295Z] 2024/01/19 01:33:09 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T01:33:09.295Z] pod name:mysql-7d766cd65c-qnkq9 namespace:mysql-ibm-pxb-0-82948-01-19-01h31m47s running:true ready:false node:10.240.64.8 [2024-01-19T01:33:09.295Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T01:33:21.467Z] 2024-01-19 01:33:19 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T01:33:21.467Z] 2024-01-19 01:33:19 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T01:33:21.467Z] 2024-01-19 01:33:19 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] validate if mysql-ibm app's volumes are setup [2024-01-19T01:33:21.467Z] 2024-01-19 01:33:19 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T01:33:21.467Z] 2024-01-19 01:33:19 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T01:33:21.467Z] 2024-01-19 01:33:19 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T01:33:21.467Z] 2024-01-19 01:33:19 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T01:33:21.467Z] 2024-01-19 01:33:19 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Creating cloud credentials [2024-01-19T01:33:21.467Z] 2024-01-19 01:33:19 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [cred-aws-1705592040] for org [default] with [aws] as provider [2024-01-19T01:33:21.467Z] 2024-01-19 01:33:19 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T01:33:21.467Z] 2024-01-19 01:33:20 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cred-aws-1705592040] is present [2024-01-19T01:33:21.467Z] INFO[2024-01-19 01:33:20] Verifying : Description : Verifying creation of cloud credential named [cred-aws-1705592040] for org [default] with [aws] as provider [2024-01-19T01:33:21.467Z] INFO[2024-01-19 01:33:20] Actual:true, Expected: true [2024-01-19T01:33:21.467Z] INFO[2024-01-19 01:33:20] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T01:33:21.467Z] INFO[2024-01-19 01:33:20] Actual:true, Expected: true [2024-01-19T01:33:21.467Z] 2024-01-19 01:33:20 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T01:33:21.467Z] 2024-01-19 01:33:21 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T01:33:21.467Z] 2024-01-19 01:33:21 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T01:33:21.467Z] 2024-01-19 01:33:21 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T01:33:21.467Z] 2024-01-19 01:33:21 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [source-cluster] in org [default] [2024-01-19T01:33:21.467Z] 2024-01-19 01:33:21 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T01:33:21.467Z] 2024-01-19 01:33:21 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T01:33:21.467Z] 2024-01-19 01:33:21 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:33:21.467Z] 2024-01-19 01:33:21 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T01:33:21.468Z] 2024-01-19 01:33:21 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T01:33:21.468Z] 2024-01-19 01:33:21 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-vtjgs] for IBM [2024-01-19T01:33:21.468Z] 2024-01-19 01:33:21 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-vtjgs] for org [default] with [ibm] as provider [2024-01-19T01:33:21.468Z] 2024-01-19 01:33:21 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:33:21.468Z] 2024-01-19 01:33:21 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:33:21.468Z] 2024-01-19 01:33:21 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-vtjgs] is present [2024-01-19T01:33:21.468Z] 2024-01-19 01:33:21 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [source-cluster] in org [default] [2024-01-19T01:33:22.031Z] 2024-01-19 01:33:21 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-pkgor] for IBM [2024-01-19T01:33:22.031Z] 2024-01-19 01:33:21 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-pkgor] for org [default] with [ibm] as provider [2024-01-19T01:33:22.031Z] 2024-01-19 01:33:21 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:33:22.031Z] 2024-01-19 01:33:21 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:33:22.296Z] 2024-01-19 01:33:22 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-pkgor] is present [2024-01-19T01:33:22.296Z] 2024-01-19 01:33:22 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:33:22.296Z] INFO[2024-01-19 01:33:22] Verifying : Description : Creating source source-cluster and destination destination-cluster cluster [2024-01-19T01:33:22.296Z] INFO[2024-01-19 01:33:22] Actual:true, Expected: true [2024-01-19T01:33:22.296Z] INFO[2024-01-19 01:33:22] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T01:33:22.296Z] INFO[2024-01-19 01:33:22] Actual:Online, Expected: Online [2024-01-19T01:33:22.296Z] INFO[2024-01-19 01:33:22] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T01:33:22.296Z] INFO[2024-01-19 01:33:22] Actual:true, Expected: true [2024-01-19T01:33:22.296Z] 2024-01-19 01:33:22 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Creating backup location [2024-01-19T01:33:24.812Z] INFO[2024-01-19 01:33:24] Verifying : Description : Creating backup location autogenerated-backup-location-1705628002 [2024-01-19T01:33:24.812Z] INFO[2024-01-19 01:33:24] Actual:true, Expected: true [2024-01-19T01:33:25.068Z] 2024-01-19 01:33:25 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T01:33:25.324Z] 2024-01-19 01:33:25 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Volume snapshot classes found in the cluster - [2024-01-19T01:33:25.324Z] 2024-01-19 01:33:25 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] ibmc-vpcblock-snapshot [2024-01-19T01:33:25.324Z] 2024-01-19 01:33:25 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T01:33:25.888Z] 2024/01/19 01:33:25 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705628005] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T01:33:57.915Z] 2024/01/19 01:33:55 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705628005] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:34:29.946Z] 2024/01/19 01:34:25 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705628005] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T01:34:44.788Z] INFO[2024-01-19 01:34:42] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T01:34:44.788Z] INFO[2024-01-19 01:34:42] sched stats: tick interval: 36042 < 1.2s (100%) [2024-01-19T01:34:44.788Z] INFO[2024-01-19 01:34:42] sched stats: tick process durations: 36041 < 500ms (100%) [2024-01-19T01:34:56.960Z] 2024-01-19 01:34:55 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.CreateBackup:#313] - Backup [tp-backup-1705628005] created successfully [2024-01-19T01:34:56.960Z] 2024-01-19 01:34:55 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Validating backup [tp-backup-1705628005] in org [default] [2024-01-19T01:34:56.960Z] 2024-01-19 01:34:55 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705628005] [2024-01-19T01:34:56.960Z] 2024-01-19 01:34:55 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82948-01-19-01h31m47s] in backup [tp-backup-1705628005] [2024-01-19T01:34:56.960Z] 2024-01-19 01:34:55 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82948-01-19-01h31m47s] in backup [tp-backup-1705628005] [2024-01-19T01:34:56.960Z] 2024-01-19 01:34:55 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82948-01-19-01h31m47s] [2024-01-19T01:34:56.960Z] 2024-01-19 01:34:55 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T01:34:56.960Z] 2024-01-19 01:34:55 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T01:34:56.960Z] 2024-01-19 01:34:55 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82948-01-19-01h31m47s] are [[mysql-data]] [2024-01-19T01:34:56.960Z] 2024-01-19 01:34:55 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T01:34:56.960Z] 2024-01-19 01:34:55 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Validating CR cleanup [2024-01-19T01:34:56.960Z] 2024-01-19 01:34:56 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T01:34:56.960Z] 2024-01-19 01:34:56 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] All CRs in [mysql-ibm-pxb-0-82948-01-19-01h31m47s] are [[]] [2024-01-19T01:34:56.960Z] INFO[2024-01-19 01:34:56] Verifying : Description : Creation and Validation of backup [tp-backup-1705628005] [2024-01-19T01:34:56.960Z] INFO[2024-01-19 01:34:56] Actual:true, Expected: true [2024-01-19T01:34:56.960Z] 2024-01-19 01:34:56 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Creating users [2024-01-19T01:34:56.960Z] 2024-01-19 01:34:56 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Creating 10 users [2024-01-19T01:34:57.522Z] I0119 01:34:57.358069 14 request.go:682] Waited for 1.16054422s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-19T01:35:00.790Z] 2024-01-19 01:35:00 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.AddUser:#728] - Response for user [testuser7-1705628096] creation - [2024-01-19T01:35:00.790Z] 2024-01-19 01:35:00 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.AddUser:#733] - User [testuser7-1705628096] created [2024-01-19T01:35:00.790Z] INFO[2024-01-19 01:35:00] Verifying : Description : Creating user - testuser7-1705628096 [2024-01-19T01:35:00.790Z] INFO[2024-01-19 01:35:00] Actual:true, Expected: true [2024-01-19T01:35:01.045Z] 2024-01-19 01:35:00 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.AddUser:#728] - Response for user [testuser2-1705628096] creation - [2024-01-19T01:35:01.045Z] 2024-01-19 01:35:00 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.AddUser:#733] - User [testuser2-1705628096] created [2024-01-19T01:35:01.045Z] INFO[2024-01-19 01:35:00] Verifying : Description : Creating user - testuser2-1705628096 [2024-01-19T01:35:01.045Z] INFO[2024-01-19 01:35:00] Actual:true, Expected: true [2024-01-19T01:35:01.301Z] 2024-01-19 01:35:01 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.AddUser:#728] - Response for user [testuser1-1705628096] creation - [2024-01-19T01:35:01.302Z] 2024-01-19 01:35:01 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.AddUser:#733] - User [testuser1-1705628096] created [2024-01-19T01:35:01.302Z] INFO[2024-01-19 01:35:01] Verifying : Description : Creating user - testuser1-1705628096 [2024-01-19T01:35:01.302Z] INFO[2024-01-19 01:35:01] Actual:true, Expected: true [2024-01-19T01:35:01.559Z] 2024-01-19 01:35:01 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.AddUser:#728] - Response for user [testuser5-1705628096] creation - [2024-01-19T01:35:01.559Z] 2024-01-19 01:35:01 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.AddUser:#733] - User [testuser5-1705628096] created [2024-01-19T01:35:01.559Z] INFO[2024-01-19 01:35:01] Verifying : Description : Creating user - testuser5-1705628096 [2024-01-19T01:35:01.559Z] INFO[2024-01-19 01:35:01] Actual:true, Expected: true [2024-01-19T01:35:01.559Z] 2024-01-19 01:35:01 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.AddUser:#728] - Response for user [testuser10-1705628096] creation - [2024-01-19T01:35:01.559Z] 2024-01-19 01:35:01 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.AddUser:#733] - User [testuser10-1705628096] created [2024-01-19T01:35:01.559Z] INFO[2024-01-19 01:35:01] Verifying : Description : Creating user - testuser10-1705628096 [2024-01-19T01:35:01.559Z] INFO[2024-01-19 01:35:01] Actual:true, Expected: true [2024-01-19T01:35:01.815Z] 2024-01-19 01:35:01 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.AddUser:#728] - Response for user [testuser4-1705628096] creation - [2024-01-19T01:35:01.815Z] 2024-01-19 01:35:01 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.AddUser:#733] - User [testuser4-1705628096] created [2024-01-19T01:35:01.815Z] INFO[2024-01-19 01:35:01] Verifying : Description : Creating user - testuser4-1705628096 [2024-01-19T01:35:01.815Z] INFO[2024-01-19 01:35:01] Actual:true, Expected: true [2024-01-19T01:35:02.071Z] 2024-01-19 01:35:01 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.AddUser:#728] - Response for user [testuser8-1705628096] creation - [2024-01-19T01:35:02.071Z] 2024-01-19 01:35:01 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.AddUser:#733] - User [testuser8-1705628096] created [2024-01-19T01:35:02.071Z] INFO[2024-01-19 01:35:01] Verifying : Description : Creating user - testuser8-1705628096 [2024-01-19T01:35:02.071Z] INFO[2024-01-19 01:35:01] Actual:true, Expected: true [2024-01-19T01:35:02.327Z] 2024-01-19 01:35:02 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.AddUser:#728] - Response for user [testuser6-1705628096] creation - [2024-01-19T01:35:02.327Z] 2024-01-19 01:35:02 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.AddUser:#733] - User [testuser6-1705628096] created [2024-01-19T01:35:02.327Z] INFO[2024-01-19 01:35:02] Verifying : Description : Creating user - testuser6-1705628096 [2024-01-19T01:35:02.327Z] INFO[2024-01-19 01:35:02] Actual:true, Expected: true [2024-01-19T01:35:02.327Z] 2024-01-19 01:35:02 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.AddUser:#728] - Response for user [testuser3-1705628096] creation - [2024-01-19T01:35:02.327Z] 2024-01-19 01:35:02 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.AddUser:#733] - User [testuser3-1705628096] created [2024-01-19T01:35:02.327Z] INFO[2024-01-19 01:35:02] Verifying : Description : Creating user - testuser3-1705628096 [2024-01-19T01:35:02.327Z] INFO[2024-01-19 01:35:02] Actual:true, Expected: true [2024-01-19T01:35:02.584Z] 2024-01-19 01:35:02 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.AddUser:#728] - Response for user [testuser9-1705628096] creation - [2024-01-19T01:35:02.584Z] 2024-01-19 01:35:02 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.AddUser:#733] - User [testuser9-1705628096] created [2024-01-19T01:35:02.584Z] INFO[2024-01-19 01:35:02] Verifying : Description : Creating user - testuser9-1705628096 [2024-01-19T01:35:02.584Z] INFO[2024-01-19 01:35:02] Actual:true, Expected: true [2024-01-19T01:35:02.584Z] 2024-01-19 01:35:02 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.glob..func49.2.6:#479] - Created 10 users and users list is [testuser7-1705628096 testuser2-1705628096 testuser1-1705628096 testuser5-1705628096 testuser10-1705628096 testuser4-1705628096 testuser8-1705628096 testuser6-1705628096 testuser3-1705628096 testuser9-1705628096] [2024-01-19T01:35:02.584Z] 2024-01-19 01:35:02 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Sharing Backup with multiple users [2024-01-19T01:35:03.951Z] 2024-01-19 01:35:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705628005 - 9ca140d9-0ea0-4038-8ddf-c99ec8d41591 [2024-01-19T01:35:04.880Z] 2024-01-19 01:35:04 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser7-1705628096 - fa58aa4c-f46b-454c-bd52-8370f548efa3 [2024-01-19T01:35:05.442Z] 2024-01-19 01:35:05 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2-1705628096 - 8f27b128-af90-4fa9-be36-35330680231b [2024-01-19T01:35:06.373Z] 2024-01-19 01:35:06 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705628096 - cc430c49-9272-41cd-b95e-4be0248d69bb [2024-01-19T01:35:07.299Z] 2024-01-19 01:35:06 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser5-1705628096 - 4e3b1848-48b5-46d0-9906-bd5e240ecc62 [2024-01-19T01:35:07.870Z] 2024-01-19 01:35:07 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser10-1705628096 - b35bb037-122f-498a-b15f-1688fe27cb60 [2024-01-19T01:35:08.795Z] 2024-01-19 01:35:08 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser4-1705628096 - 60bed100-ea29-40e1-b3ca-633f06aed5f7 [2024-01-19T01:35:09.722Z] 2024-01-19 01:35:09 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser8-1705628096 - 04d90090-0cca-4b75-b326-74a6f278659d [2024-01-19T01:35:10.283Z] 2024-01-19 01:35:10 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser6-1705628096 - 61c48609-132c-424f-a593-680e49e8a8cb [2024-01-19T01:35:11.228Z] 2024-01-19 01:35:10 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser3-1705628096 - 7a6999d9-4319-4019-9086-498f3352e9c0 [2024-01-19T01:35:12.155Z] 2024-01-19 01:35:11 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser9-1705628096 - fa63e049-e0a3-40a4-8318-732a19e20499 [2024-01-19T01:35:12.155Z] 2024-01-19 01:35:11 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Restart backup pod when backup sharing is in-progress [2024-01-19T01:35:12.155Z] INFO[2024-01-19 01:35:11] Verifying : Description : Getting px-backup namespace [2024-01-19T01:35:12.155Z] INFO[2024-01-19 01:35:11] Actual:true, Expected: true [2024-01-19T01:35:12.719Z] 2024-01-19 01:35:12 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.DeletePodWithWithoutLabelInNamespace:#1701] - Deleting pod px-backup-5d5b75cdb9-g7rmx with label map[app:px-backup] [2024-01-19T01:35:12.975Z] 2024/01/19 01:35:12 DoRetryWithTimeout - Error: {pod px-backup:px-backup-5d5b75cdb9-g7rmx (e0f3c53b-2b4c-403b-b6d4-250ddb9538d4) still present in the system}, Next try in [5s], timeout [5m0s] [2024-01-19T01:35:18.221Z] INFO[2024-01-19 01:35:17] Verifying : Description : Restart backup pod when backup sharing is in-progress [2024-01-19T01:35:18.221Z] INFO[2024-01-19 01:35:17] Actual:true, Expected: true [2024-01-19T01:35:18.221Z] 2024-01-19 01:35:17 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.ValidatePodByLabel:#3675] - Checking if pods with label map[app:px-backup] are running in namespace px-backup [2024-01-19T01:35:18.221Z] 2024/01/19 01:35:17 DoRetryWithTimeout - Error: {Pod px-backup-5d5b75cdb9-k9hqq, ID: e7af32f1-2e7e-4b8a-8c78-74d2b67d3053 is not ready. Status Running}, Next try in [30s], timeout [5m0s] [2024-01-19T01:35:50.244Z] 2024/01/19 01:35:47 DoRetryWithTimeout - Error: {Pod px-backup-5d5b75cdb9-k9hqq, ID: e7af32f1-2e7e-4b8a-8c78-74d2b67d3053 is not ready. Status Running}, Next try in [30s], timeout [5m0s] [2024-01-19T01:36:22.275Z] 2024-01-19 01:36:18 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Registering Source and Destination clusters from user context for user -testuser7-1705628096 [2024-01-19T01:36:22.275Z] INFO[2024-01-19 01:36:18] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T01:36:22.275Z] INFO[2024-01-19 01:36:18] Actual:true, Expected: true [2024-01-19T01:36:22.275Z] 2024-01-19 01:36:18 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T01:36:22.275Z] 2024-01-19 01:36:18 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T01:36:22.275Z] 2024-01-19 01:36:18 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T01:36:22.276Z] 2024-01-19 01:36:18 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T01:36:22.276Z] 2024-01-19 01:36:18 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [source-cluster] in org [default] [2024-01-19T01:36:22.276Z] 2024-01-19 01:36:18 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T01:36:22.276Z] 2024-01-19 01:36:18 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T01:36:22.276Z] 2024-01-19 01:36:18 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:36:22.276Z] 2024-01-19 01:36:18 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T01:36:22.276Z] 2024-01-19 01:36:18 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T01:36:22.276Z] 2024-01-19 01:36:18 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-liejf] for IBM [2024-01-19T01:36:22.276Z] 2024-01-19 01:36:18 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-liejf] for org [default] with [ibm] as provider [2024-01-19T01:36:22.276Z] 2024-01-19 01:36:18 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:22.276Z] 2024-01-19 01:36:18 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:22.276Z] 2024-01-19 01:36:18 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-liejf] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName7 LastName7 with roles: [px-backup-app.user]] [2024-01-19T01:36:22.276Z] 2024-01-19 01:36:18 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName7 LastName7 with roles: [px-backup-app.user] [2024-01-19T01:36:22.276Z] 2024-01-19 01:36:19 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-liejf from admin context and sharing with all the users [2024-01-19T01:36:22.277Z] 2024-01-19 01:36:19 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-liejf] for org [default] with [ibm] as provider [2024-01-19T01:36:22.277Z] 2024-01-19 01:36:19 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:22.277Z] 2024-01-19 01:36:19 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:22.277Z] 2024-01-19 01:36:19 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-liejf] is present [2024-01-19T01:36:22.277Z] 2024-01-19 01:36:19 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [source-cluster] in org [default] [2024-01-19T01:36:22.277Z] 2024-01-19 01:36:20 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-honeu] for IBM [2024-01-19T01:36:22.277Z] 2024-01-19 01:36:20 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-honeu] for org [default] with [ibm] as provider [2024-01-19T01:36:22.277Z] 2024-01-19 01:36:20 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:22.277Z] 2024-01-19 01:36:20 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:22.277Z] 2024-01-19 01:36:20 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-honeu] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName7 LastName7 with roles: [px-backup-app.user]] [2024-01-19T01:36:22.277Z] 2024-01-19 01:36:20 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName7 LastName7 with roles: [px-backup-app.user] [2024-01-19T01:36:22.277Z] 2024-01-19 01:36:20 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-honeu from admin context and sharing with all the users [2024-01-19T01:36:22.277Z] 2024-01-19 01:36:20 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-honeu] for org [default] with [ibm] as provider [2024-01-19T01:36:22.277Z] 2024-01-19 01:36:20 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:22.277Z] 2024-01-19 01:36:20 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:22.277Z] 2024-01-19 01:36:21 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-honeu] is present [2024-01-19T01:36:22.277Z] 2024-01-19 01:36:21 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:36:22.277Z] INFO[2024-01-19 01:36:21] Verifying : Description : Creating source and destination cluster for user testuser7-1705628096 [2024-01-19T01:36:22.277Z] INFO[2024-01-19 01:36:21] Actual:true, Expected: true [2024-01-19T01:36:22.278Z] INFO[2024-01-19 01:36:21] Verifying : Description : Verifying backup restore [tp-restore-1705628181] is not possible for backup [tp-backup-1705628005] with user [testuser7-1705628096] [2024-01-19T01:36:22.278Z] INFO[2024-01-19 01:36:21] Actual:true, Expected: true [2024-01-19T01:36:22.278Z] 2024-01-19 01:36:22 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Registering Source and Destination clusters from user context for user -testuser2-1705628096 [2024-01-19T01:36:22.278Z] INFO[2024-01-19 01:36:22] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T01:36:22.278Z] INFO[2024-01-19 01:36:22] Actual:true, Expected: true [2024-01-19T01:36:22.278Z] 2024-01-19 01:36:22 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T01:36:22.278Z] 2024-01-19 01:36:22 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T01:36:22.278Z] 2024-01-19 01:36:22 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T01:36:22.278Z] 2024-01-19 01:36:22 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T01:36:22.278Z] 2024-01-19 01:36:22 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [source-cluster] in org [default] [2024-01-19T01:36:22.278Z] 2024-01-19 01:36:22 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T01:36:22.278Z] 2024-01-19 01:36:22 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T01:36:22.278Z] 2024-01-19 01:36:22 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:36:22.278Z] 2024-01-19 01:36:22 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T01:36:22.278Z] 2024-01-19 01:36:22 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T01:36:22.278Z] 2024-01-19 01:36:22 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-prnvt] for IBM [2024-01-19T01:36:22.278Z] 2024-01-19 01:36:22 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-prnvt] for org [default] with [ibm] as provider [2024-01-19T01:36:22.278Z] 2024-01-19 01:36:22 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:22.278Z] 2024-01-19 01:36:22 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:22.278Z] 2024-01-19 01:36:22 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-prnvt] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName2 LastName2 with roles: [px-backup-app.user]] [2024-01-19T01:36:22.278Z] 2024-01-19 01:36:22 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName2 LastName2 with roles: [px-backup-app.user] [2024-01-19T01:36:23.206Z] 2024-01-19 01:36:22 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-prnvt from admin context and sharing with all the users [2024-01-19T01:36:23.206Z] 2024-01-19 01:36:22 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-prnvt] for org [default] with [ibm] as provider [2024-01-19T01:36:23.206Z] 2024-01-19 01:36:22 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:23.206Z] 2024-01-19 01:36:22 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:23.206Z] 2024-01-19 01:36:23 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-prnvt] is present [2024-01-19T01:36:23.463Z] 2024-01-19 01:36:23 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [source-cluster] in org [default] [2024-01-19T01:36:24.027Z] 2024-01-19 01:36:23 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-soitk] for IBM [2024-01-19T01:36:24.027Z] 2024-01-19 01:36:23 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-soitk] for org [default] with [ibm] as provider [2024-01-19T01:36:24.027Z] 2024-01-19 01:36:23 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:24.027Z] 2024-01-19 01:36:23 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:24.027Z] 2024-01-19 01:36:23 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-soitk] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName2 LastName2 with roles: [px-backup-app.user]] [2024-01-19T01:36:24.027Z] 2024-01-19 01:36:23 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName2 LastName2 with roles: [px-backup-app.user] [2024-01-19T01:36:24.591Z] 2024-01-19 01:36:24 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-soitk from admin context and sharing with all the users [2024-01-19T01:36:24.591Z] 2024-01-19 01:36:24 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-soitk] for org [default] with [ibm] as provider [2024-01-19T01:36:24.592Z] 2024-01-19 01:36:24 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:24.592Z] 2024-01-19 01:36:24 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:24.592Z] 2024-01-19 01:36:24 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-soitk] is present [2024-01-19T01:36:24.848Z] 2024-01-19 01:36:24 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:36:25.411Z] INFO[2024-01-19 01:36:25] Verifying : Description : Creating source and destination cluster for user testuser2-1705628096 [2024-01-19T01:36:25.411Z] INFO[2024-01-19 01:36:25] Actual:true, Expected: true [2024-01-19T01:36:25.411Z] INFO[2024-01-19 01:36:25] Verifying : Description : Verifying backup restore [tp-restore-1705628185] is not possible for backup [tp-backup-1705628005] with user [testuser2-1705628096] [2024-01-19T01:36:25.411Z] INFO[2024-01-19 01:36:25] Actual:true, Expected: true [2024-01-19T01:36:25.710Z] 2024-01-19 01:36:25 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Registering Source and Destination clusters from user context for user -testuser1-1705628096 [2024-01-19T01:36:25.710Z] INFO[2024-01-19 01:36:25] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T01:36:25.710Z] INFO[2024-01-19 01:36:25] Actual:true, Expected: true [2024-01-19T01:36:25.710Z] 2024-01-19 01:36:25 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T01:36:25.710Z] 2024-01-19 01:36:25 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T01:36:25.710Z] 2024-01-19 01:36:25 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T01:36:25.710Z] 2024-01-19 01:36:25 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T01:36:25.710Z] 2024-01-19 01:36:25 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [source-cluster] in org [default] [2024-01-19T01:36:25.710Z] 2024-01-19 01:36:25 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T01:36:25.710Z] 2024-01-19 01:36:25 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T01:36:25.710Z] 2024-01-19 01:36:25 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:36:25.710Z] 2024-01-19 01:36:25 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T01:36:25.710Z] 2024-01-19 01:36:25 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T01:36:25.710Z] 2024-01-19 01:36:25 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-ckdjo] for IBM [2024-01-19T01:36:25.710Z] 2024-01-19 01:36:25 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-ckdjo] for org [default] with [ibm] as provider [2024-01-19T01:36:25.710Z] 2024-01-19 01:36:25 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:25.711Z] 2024-01-19 01:36:25 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:25.711Z] 2024-01-19 01:36:25 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-ckdjo] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user]] [2024-01-19T01:36:25.711Z] 2024-01-19 01:36:25 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user] [2024-01-19T01:36:26.638Z] 2024-01-19 01:36:26 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-ckdjo from admin context and sharing with all the users [2024-01-19T01:36:26.639Z] 2024-01-19 01:36:26 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-ckdjo] for org [default] with [ibm] as provider [2024-01-19T01:36:26.639Z] 2024-01-19 01:36:26 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:26.639Z] 2024-01-19 01:36:26 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:26.639Z] 2024-01-19 01:36:26 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-ckdjo] is present [2024-01-19T01:36:26.896Z] 2024-01-19 01:36:26 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [source-cluster] in org [default] [2024-01-19T01:36:27.458Z] 2024-01-19 01:36:27 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-ldiyr] for IBM [2024-01-19T01:36:27.458Z] 2024-01-19 01:36:27 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-ldiyr] for org [default] with [ibm] as provider [2024-01-19T01:36:27.458Z] 2024-01-19 01:36:27 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:27.458Z] 2024-01-19 01:36:27 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:27.458Z] 2024-01-19 01:36:27 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-ldiyr] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user]] [2024-01-19T01:36:27.458Z] 2024-01-19 01:36:27 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user] [2024-01-19T01:36:28.021Z] 2024-01-19 01:36:27 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-ldiyr from admin context and sharing with all the users [2024-01-19T01:36:28.022Z] 2024-01-19 01:36:27 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-ldiyr] for org [default] with [ibm] as provider [2024-01-19T01:36:28.022Z] 2024-01-19 01:36:27 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:28.022Z] 2024-01-19 01:36:27 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:28.022Z] 2024-01-19 01:36:27 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-ldiyr] is present [2024-01-19T01:36:28.278Z] 2024-01-19 01:36:28 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:36:28.841Z] INFO[2024-01-19 01:36:28] Verifying : Description : Creating source and destination cluster for user testuser1-1705628096 [2024-01-19T01:36:28.841Z] INFO[2024-01-19 01:36:28] Actual:true, Expected: true [2024-01-19T01:36:28.841Z] INFO[2024-01-19 01:36:28] Verifying : Description : Verifying backup restore [tp-restore-1705628188] is not possible for backup [tp-backup-1705628005] with user [testuser1-1705628096] [2024-01-19T01:36:28.841Z] INFO[2024-01-19 01:36:28] Actual:true, Expected: true [2024-01-19T01:36:29.097Z] 2024-01-19 01:36:28 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Registering Source and Destination clusters from user context for user -testuser5-1705628096 [2024-01-19T01:36:29.097Z] INFO[2024-01-19 01:36:28] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T01:36:29.097Z] INFO[2024-01-19 01:36:28] Actual:true, Expected: true [2024-01-19T01:36:29.097Z] 2024-01-19 01:36:28 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T01:36:29.097Z] 2024-01-19 01:36:28 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T01:36:29.097Z] 2024-01-19 01:36:28 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T01:36:29.097Z] 2024-01-19 01:36:28 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T01:36:29.097Z] 2024-01-19 01:36:28 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [source-cluster] in org [default] [2024-01-19T01:36:29.097Z] 2024-01-19 01:36:28 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T01:36:29.097Z] 2024-01-19 01:36:28 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T01:36:29.097Z] 2024-01-19 01:36:28 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:36:29.097Z] 2024-01-19 01:36:28 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T01:36:29.098Z] 2024-01-19 01:36:28 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T01:36:29.098Z] 2024-01-19 01:36:28 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-qtwsj] for IBM [2024-01-19T01:36:29.098Z] 2024-01-19 01:36:28 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-qtwsj] for org [default] with [ibm] as provider [2024-01-19T01:36:29.098Z] 2024-01-19 01:36:28 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:29.098Z] 2024-01-19 01:36:28 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:29.098Z] 2024-01-19 01:36:28 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-qtwsj] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName5 LastName5 with roles: [px-backup-app.user]] [2024-01-19T01:36:29.098Z] 2024-01-19 01:36:28 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName5 LastName5 with roles: [px-backup-app.user] [2024-01-19T01:36:30.030Z] 2024-01-19 01:36:29 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-qtwsj from admin context and sharing with all the users [2024-01-19T01:36:30.030Z] 2024-01-19 01:36:29 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-qtwsj] for org [default] with [ibm] as provider [2024-01-19T01:36:30.030Z] 2024-01-19 01:36:29 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:30.030Z] 2024-01-19 01:36:29 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:30.030Z] 2024-01-19 01:36:29 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-qtwsj] is present [2024-01-19T01:36:30.285Z] 2024-01-19 01:36:30 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [source-cluster] in org [default] [2024-01-19T01:36:30.848Z] 2024-01-19 01:36:30 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-nlppm] for IBM [2024-01-19T01:36:30.848Z] 2024-01-19 01:36:30 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-nlppm] for org [default] with [ibm] as provider [2024-01-19T01:36:30.848Z] 2024-01-19 01:36:30 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:30.849Z] 2024-01-19 01:36:30 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:30.849Z] 2024-01-19 01:36:30 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-nlppm] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName5 LastName5 with roles: [px-backup-app.user]] [2024-01-19T01:36:30.849Z] 2024-01-19 01:36:30 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName5 LastName5 with roles: [px-backup-app.user] [2024-01-19T01:36:31.410Z] 2024-01-19 01:36:31 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-nlppm from admin context and sharing with all the users [2024-01-19T01:36:31.410Z] 2024-01-19 01:36:31 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-nlppm] for org [default] with [ibm] as provider [2024-01-19T01:36:31.410Z] 2024-01-19 01:36:31 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:31.410Z] 2024-01-19 01:36:31 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:31.410Z] 2024-01-19 01:36:31 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-nlppm] is present [2024-01-19T01:36:31.665Z] 2024-01-19 01:36:31 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:36:32.226Z] INFO[2024-01-19 01:36:32] Verifying : Description : Creating source and destination cluster for user testuser5-1705628096 [2024-01-19T01:36:32.226Z] INFO[2024-01-19 01:36:32] Actual:true, Expected: true [2024-01-19T01:36:32.482Z] INFO[2024-01-19 01:36:32] Verifying : Description : Verifying backup restore [tp-restore-1705628192] is not possible for backup [tp-backup-1705628005] with user [testuser5-1705628096] [2024-01-19T01:36:32.482Z] INFO[2024-01-19 01:36:32] Actual:true, Expected: true [2024-01-19T01:36:32.482Z] 2024-01-19 01:36:32 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Registering Source and Destination clusters from user context for user -testuser10-1705628096 [2024-01-19T01:36:32.482Z] INFO[2024-01-19 01:36:32] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T01:36:32.482Z] INFO[2024-01-19 01:36:32] Actual:true, Expected: true [2024-01-19T01:36:32.482Z] 2024-01-19 01:36:32 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T01:36:32.482Z] 2024-01-19 01:36:32 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T01:36:32.482Z] 2024-01-19 01:36:32 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T01:36:32.483Z] 2024-01-19 01:36:32 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T01:36:32.483Z] 2024-01-19 01:36:32 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [source-cluster] in org [default] [2024-01-19T01:36:32.483Z] 2024-01-19 01:36:32 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T01:36:32.483Z] 2024-01-19 01:36:32 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T01:36:32.483Z] 2024-01-19 01:36:32 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:36:32.483Z] 2024-01-19 01:36:32 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T01:36:32.483Z] 2024-01-19 01:36:32 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T01:36:32.483Z] 2024-01-19 01:36:32 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-jucrm] for IBM [2024-01-19T01:36:32.483Z] 2024-01-19 01:36:32 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-jucrm] for org [default] with [ibm] as provider [2024-01-19T01:36:32.483Z] 2024-01-19 01:36:32 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:32.483Z] 2024-01-19 01:36:32 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:32.483Z] 2024-01-19 01:36:32 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-jucrm] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName10 LastName10 with roles: [px-backup-app.user]] [2024-01-19T01:36:32.483Z] 2024-01-19 01:36:32 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName10 LastName10 with roles: [px-backup-app.user] [2024-01-19T01:36:33.411Z] 2024-01-19 01:36:33 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-jucrm from admin context and sharing with all the users [2024-01-19T01:36:33.411Z] 2024-01-19 01:36:33 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-jucrm] for org [default] with [ibm] as provider [2024-01-19T01:36:33.411Z] 2024-01-19 01:36:33 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:33.411Z] 2024-01-19 01:36:33 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:33.411Z] 2024-01-19 01:36:33 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-jucrm] is present [2024-01-19T01:36:33.666Z] 2024-01-19 01:36:33 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [source-cluster] in org [default] [2024-01-19T01:36:34.228Z] 2024-01-19 01:36:34 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-vrgdl] for IBM [2024-01-19T01:36:34.228Z] 2024-01-19 01:36:34 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-vrgdl] for org [default] with [ibm] as provider [2024-01-19T01:36:34.228Z] 2024-01-19 01:36:34 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:34.228Z] 2024-01-19 01:36:34 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:34.486Z] 2024-01-19 01:36:34 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-vrgdl] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName10 LastName10 with roles: [px-backup-app.user]] [2024-01-19T01:36:34.486Z] 2024-01-19 01:36:34 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName10 LastName10 with roles: [px-backup-app.user] [2024-01-19T01:36:34.751Z] 2024-01-19 01:36:34 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-vrgdl from admin context and sharing with all the users [2024-01-19T01:36:34.751Z] 2024-01-19 01:36:34 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-vrgdl] for org [default] with [ibm] as provider [2024-01-19T01:36:34.751Z] 2024-01-19 01:36:34 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:34.751Z] 2024-01-19 01:36:34 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:35.006Z] 2024-01-19 01:36:34 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-vrgdl] is present [2024-01-19T01:36:35.006Z] 2024-01-19 01:36:34 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:36:35.568Z] INFO[2024-01-19 01:36:35] Verifying : Description : Creating source and destination cluster for user testuser10-1705628096 [2024-01-19T01:36:35.568Z] INFO[2024-01-19 01:36:35] Actual:true, Expected: true [2024-01-19T01:36:35.823Z] INFO[2024-01-19 01:36:35] Verifying : Description : Verifying backup restore [tp-restore-1705628195] is not possible for backup [tp-backup-1705628005] with user [testuser10-1705628096] [2024-01-19T01:36:35.823Z] INFO[2024-01-19 01:36:35] Actual:true, Expected: true [2024-01-19T01:36:35.823Z] 2024-01-19 01:36:35 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Registering Source and Destination clusters from user context for user -testuser4-1705628096 [2024-01-19T01:36:35.823Z] INFO[2024-01-19 01:36:35] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T01:36:35.823Z] INFO[2024-01-19 01:36:35] Actual:true, Expected: true [2024-01-19T01:36:35.823Z] 2024-01-19 01:36:35 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T01:36:35.823Z] 2024-01-19 01:36:35 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T01:36:35.823Z] 2024-01-19 01:36:35 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T01:36:35.823Z] 2024-01-19 01:36:35 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T01:36:35.823Z] 2024-01-19 01:36:35 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [source-cluster] in org [default] [2024-01-19T01:36:35.823Z] 2024-01-19 01:36:35 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T01:36:35.823Z] 2024-01-19 01:36:35 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T01:36:35.823Z] 2024-01-19 01:36:35 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:36:35.823Z] 2024-01-19 01:36:35 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T01:36:35.823Z] 2024-01-19 01:36:35 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T01:36:35.823Z] 2024-01-19 01:36:35 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-bgkpv] for IBM [2024-01-19T01:36:35.824Z] 2024-01-19 01:36:35 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-bgkpv] for org [default] with [ibm] as provider [2024-01-19T01:36:35.824Z] 2024-01-19 01:36:35 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:35.824Z] 2024-01-19 01:36:35 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:36.080Z] 2024-01-19 01:36:35 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-bgkpv] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName4 LastName4 with roles: [px-backup-app.user]] [2024-01-19T01:36:36.080Z] 2024-01-19 01:36:35 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName4 LastName4 with roles: [px-backup-app.user] [2024-01-19T01:36:36.641Z] 2024-01-19 01:36:36 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-bgkpv from admin context and sharing with all the users [2024-01-19T01:36:36.641Z] 2024-01-19 01:36:36 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-bgkpv] for org [default] with [ibm] as provider [2024-01-19T01:36:36.641Z] 2024-01-19 01:36:36 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:36.641Z] 2024-01-19 01:36:36 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:36.898Z] 2024-01-19 01:36:36 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-bgkpv] is present [2024-01-19T01:36:36.898Z] 2024-01-19 01:36:36 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [source-cluster] in org [default] [2024-01-19T01:36:37.853Z] 2024-01-19 01:36:37 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-bnqjf] for IBM [2024-01-19T01:36:37.854Z] 2024-01-19 01:36:37 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-bnqjf] for org [default] with [ibm] as provider [2024-01-19T01:36:37.854Z] 2024-01-19 01:36:37 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:37.854Z] 2024-01-19 01:36:37 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:37.854Z] 2024-01-19 01:36:37 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-bnqjf] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName4 LastName4 with roles: [px-backup-app.user]] [2024-01-19T01:36:37.854Z] 2024-01-19 01:36:37 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName4 LastName4 with roles: [px-backup-app.user] [2024-01-19T01:36:38.111Z] 2024-01-19 01:36:37 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-bnqjf from admin context and sharing with all the users [2024-01-19T01:36:38.111Z] 2024-01-19 01:36:37 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-bnqjf] for org [default] with [ibm] as provider [2024-01-19T01:36:38.111Z] 2024-01-19 01:36:37 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:38.111Z] 2024-01-19 01:36:37 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:38.371Z] 2024-01-19 01:36:38 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-bnqjf] is present [2024-01-19T01:36:38.371Z] 2024-01-19 01:36:38 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:36:39.300Z] INFO[2024-01-19 01:36:38] Verifying : Description : Creating source and destination cluster for user testuser4-1705628096 [2024-01-19T01:36:39.300Z] INFO[2024-01-19 01:36:38] Actual:true, Expected: true [2024-01-19T01:36:39.300Z] INFO[2024-01-19 01:36:39] Verifying : Description : Verifying backup restore [tp-restore-1705628199] is not possible for backup [tp-backup-1705628005] with user [testuser4-1705628096] [2024-01-19T01:36:39.300Z] INFO[2024-01-19 01:36:39] Actual:true, Expected: true [2024-01-19T01:36:39.556Z] 2024-01-19 01:36:39 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Registering Source and Destination clusters from user context for user -testuser8-1705628096 [2024-01-19T01:36:39.556Z] INFO[2024-01-19 01:36:39] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T01:36:39.556Z] INFO[2024-01-19 01:36:39] Actual:true, Expected: true [2024-01-19T01:36:39.556Z] 2024-01-19 01:36:39 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T01:36:39.556Z] 2024-01-19 01:36:39 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T01:36:39.556Z] 2024-01-19 01:36:39 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T01:36:39.556Z] 2024-01-19 01:36:39 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T01:36:39.556Z] 2024-01-19 01:36:39 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [source-cluster] in org [default] [2024-01-19T01:36:39.556Z] 2024-01-19 01:36:39 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T01:36:39.556Z] 2024-01-19 01:36:39 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T01:36:39.556Z] 2024-01-19 01:36:39 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:36:39.556Z] 2024-01-19 01:36:39 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T01:36:39.556Z] 2024-01-19 01:36:39 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T01:36:39.556Z] 2024-01-19 01:36:39 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-oudxv] for IBM [2024-01-19T01:36:39.556Z] 2024-01-19 01:36:39 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-oudxv] for org [default] with [ibm] as provider [2024-01-19T01:36:39.557Z] 2024-01-19 01:36:39 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:39.557Z] 2024-01-19 01:36:39 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:39.557Z] 2024-01-19 01:36:39 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-oudxv] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName8 LastName8 with roles: [px-backup-app.user]] [2024-01-19T01:36:39.557Z] 2024-01-19 01:36:39 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName8 LastName8 with roles: [px-backup-app.user] [2024-01-19T01:36:40.118Z] 2024-01-19 01:36:39 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-oudxv from admin context and sharing with all the users [2024-01-19T01:36:40.118Z] 2024-01-19 01:36:39 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-oudxv] for org [default] with [ibm] as provider [2024-01-19T01:36:40.118Z] 2024-01-19 01:36:39 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:40.118Z] 2024-01-19 01:36:39 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:40.374Z] 2024-01-19 01:36:40 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-oudxv] is present [2024-01-19T01:36:40.374Z] 2024-01-19 01:36:40 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [source-cluster] in org [default] [2024-01-19T01:36:41.301Z] 2024-01-19 01:36:40 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-lkvsh] for IBM [2024-01-19T01:36:41.301Z] 2024-01-19 01:36:40 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-lkvsh] for org [default] with [ibm] as provider [2024-01-19T01:36:41.301Z] 2024-01-19 01:36:40 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:41.301Z] 2024-01-19 01:36:40 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:41.301Z] 2024-01-19 01:36:41 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-lkvsh] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName8 LastName8 with roles: [px-backup-app.user]] [2024-01-19T01:36:41.301Z] 2024-01-19 01:36:41 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName8 LastName8 with roles: [px-backup-app.user] [2024-01-19T01:36:41.560Z] 2024-01-19 01:36:41 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-lkvsh from admin context and sharing with all the users [2024-01-19T01:36:41.560Z] 2024-01-19 01:36:41 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-lkvsh] for org [default] with [ibm] as provider [2024-01-19T01:36:41.561Z] 2024-01-19 01:36:41 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:41.561Z] 2024-01-19 01:36:41 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:41.817Z] 2024-01-19 01:36:41 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-lkvsh] is present [2024-01-19T01:36:41.817Z] 2024-01-19 01:36:41 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:36:42.379Z] INFO[2024-01-19 01:36:42] Verifying : Description : Creating source and destination cluster for user testuser8-1705628096 [2024-01-19T01:36:42.379Z] INFO[2024-01-19 01:36:42] Actual:true, Expected: true [2024-01-19T01:36:42.641Z] INFO[2024-01-19 01:36:42] Verifying : Description : Verifying backup restore [tp-restore-1705628202] is not possible for backup [tp-backup-1705628005] with user [testuser8-1705628096] [2024-01-19T01:36:42.641Z] INFO[2024-01-19 01:36:42] Actual:true, Expected: true [2024-01-19T01:36:42.642Z] 2024-01-19 01:36:42 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Registering Source and Destination clusters from user context for user -testuser6-1705628096 [2024-01-19T01:36:42.642Z] INFO[2024-01-19 01:36:42] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T01:36:42.642Z] INFO[2024-01-19 01:36:42] Actual:true, Expected: true [2024-01-19T01:36:42.642Z] 2024-01-19 01:36:42 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T01:36:42.642Z] 2024-01-19 01:36:42 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T01:36:42.642Z] 2024-01-19 01:36:42 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T01:36:42.642Z] 2024-01-19 01:36:42 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T01:36:42.642Z] 2024-01-19 01:36:42 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [source-cluster] in org [default] [2024-01-19T01:36:42.642Z] 2024-01-19 01:36:42 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T01:36:42.642Z] 2024-01-19 01:36:42 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T01:36:42.642Z] 2024-01-19 01:36:42 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:36:42.642Z] 2024-01-19 01:36:42 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T01:36:42.642Z] 2024-01-19 01:36:42 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T01:36:42.642Z] 2024-01-19 01:36:42 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-wylyd] for IBM [2024-01-19T01:36:42.642Z] 2024-01-19 01:36:42 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-wylyd] for org [default] with [ibm] as provider [2024-01-19T01:36:42.642Z] 2024-01-19 01:36:42 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:42.642Z] 2024-01-19 01:36:42 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:42.642Z] 2024-01-19 01:36:42 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-wylyd] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName6 LastName6 with roles: [px-backup-app.user]] [2024-01-19T01:36:42.642Z] 2024-01-19 01:36:42 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName6 LastName6 with roles: [px-backup-app.user] [2024-01-19T01:36:43.572Z] 2024-01-19 01:36:43 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-wylyd from admin context and sharing with all the users [2024-01-19T01:36:43.572Z] 2024-01-19 01:36:43 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-wylyd] for org [default] with [ibm] as provider [2024-01-19T01:36:43.572Z] 2024-01-19 01:36:43 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:43.572Z] 2024-01-19 01:36:43 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:43.572Z] 2024-01-19 01:36:43 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-wylyd] is present [2024-01-19T01:36:43.828Z] 2024-01-19 01:36:43 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [source-cluster] in org [default] [2024-01-19T01:36:44.756Z] 2024-01-19 01:36:44 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-ptofl] for IBM [2024-01-19T01:36:44.756Z] 2024-01-19 01:36:44 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-ptofl] for org [default] with [ibm] as provider [2024-01-19T01:36:44.756Z] 2024-01-19 01:36:44 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:44.756Z] 2024-01-19 01:36:44 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:44.756Z] 2024-01-19 01:36:44 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-ptofl] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName6 LastName6 with roles: [px-backup-app.user]] [2024-01-19T01:36:44.756Z] 2024-01-19 01:36:44 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName6 LastName6 with roles: [px-backup-app.user] [2024-01-19T01:36:45.012Z] 2024-01-19 01:36:44 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-ptofl from admin context and sharing with all the users [2024-01-19T01:36:45.012Z] 2024-01-19 01:36:44 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-ptofl] for org [default] with [ibm] as provider [2024-01-19T01:36:45.012Z] 2024-01-19 01:36:44 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:45.012Z] 2024-01-19 01:36:44 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:45.012Z] 2024-01-19 01:36:44 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-ptofl] is present [2024-01-19T01:36:45.269Z] 2024-01-19 01:36:45 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:36:45.830Z] INFO[2024-01-19 01:36:45] Verifying : Description : Creating source and destination cluster for user testuser6-1705628096 [2024-01-19T01:36:45.830Z] INFO[2024-01-19 01:36:45] Actual:true, Expected: true [2024-01-19T01:36:46.086Z] INFO[2024-01-19 01:36:45] Verifying : Description : Verifying backup restore [tp-restore-1705628205] is not possible for backup [tp-backup-1705628005] with user [testuser6-1705628096] [2024-01-19T01:36:46.086Z] INFO[2024-01-19 01:36:45] Actual:true, Expected: true [2024-01-19T01:36:46.086Z] 2024-01-19 01:36:45 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Registering Source and Destination clusters from user context for user -testuser3-1705628096 [2024-01-19T01:36:46.086Z] INFO[2024-01-19 01:36:45] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T01:36:46.086Z] INFO[2024-01-19 01:36:45] Actual:true, Expected: true [2024-01-19T01:36:46.086Z] 2024-01-19 01:36:45 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T01:36:46.086Z] 2024-01-19 01:36:45 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T01:36:46.086Z] 2024-01-19 01:36:45 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T01:36:46.086Z] 2024-01-19 01:36:45 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T01:36:46.086Z] 2024-01-19 01:36:45 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [source-cluster] in org [default] [2024-01-19T01:36:46.087Z] 2024-01-19 01:36:45 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T01:36:46.087Z] 2024-01-19 01:36:45 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T01:36:46.087Z] 2024-01-19 01:36:45 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:36:46.087Z] 2024-01-19 01:36:45 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T01:36:46.087Z] 2024-01-19 01:36:45 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T01:36:46.087Z] 2024-01-19 01:36:45 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-liytn] for IBM [2024-01-19T01:36:46.087Z] 2024-01-19 01:36:45 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-liytn] for org [default] with [ibm] as provider [2024-01-19T01:36:46.087Z] 2024-01-19 01:36:45 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:46.087Z] 2024-01-19 01:36:45 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:46.087Z] 2024-01-19 01:36:45 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-liytn] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName3 LastName3 with roles: [px-backup-app.user]] [2024-01-19T01:36:46.087Z] 2024-01-19 01:36:45 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName3 LastName3 with roles: [px-backup-app.user] [2024-01-19T01:36:47.014Z] 2024-01-19 01:36:46 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-liytn from admin context and sharing with all the users [2024-01-19T01:36:47.014Z] 2024-01-19 01:36:46 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-liytn] for org [default] with [ibm] as provider [2024-01-19T01:36:47.014Z] 2024-01-19 01:36:46 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:47.014Z] 2024-01-19 01:36:46 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:47.014Z] 2024-01-19 01:36:46 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-liytn] is present [2024-01-19T01:36:47.270Z] 2024-01-19 01:36:47 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [source-cluster] in org [default] [2024-01-19T01:36:48.198Z] 2024-01-19 01:36:47 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-iqdxq] for IBM [2024-01-19T01:36:48.198Z] 2024-01-19 01:36:47 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-iqdxq] for org [default] with [ibm] as provider [2024-01-19T01:36:48.198Z] 2024-01-19 01:36:47 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:48.198Z] 2024-01-19 01:36:47 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:48.198Z] 2024-01-19 01:36:47 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-iqdxq] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName3 LastName3 with roles: [px-backup-app.user]] [2024-01-19T01:36:48.198Z] 2024-01-19 01:36:47 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName3 LastName3 with roles: [px-backup-app.user] [2024-01-19T01:36:48.455Z] 2024-01-19 01:36:48 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-iqdxq from admin context and sharing with all the users [2024-01-19T01:36:48.455Z] 2024-01-19 01:36:48 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-iqdxq] for org [default] with [ibm] as provider [2024-01-19T01:36:48.455Z] 2024-01-19 01:36:48 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:48.455Z] 2024-01-19 01:36:48 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:48.455Z] 2024-01-19 01:36:48 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-iqdxq] is present [2024-01-19T01:36:48.711Z] 2024-01-19 01:36:48 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:36:49.639Z] INFO[2024-01-19 01:36:49] Verifying : Description : Creating source and destination cluster for user testuser3-1705628096 [2024-01-19T01:36:49.639Z] INFO[2024-01-19 01:36:49] Actual:true, Expected: true [2024-01-19T01:36:49.639Z] INFO[2024-01-19 01:36:49] Verifying : Description : Verifying backup restore [tp-restore-1705628209] is not possible for backup [tp-backup-1705628005] with user [testuser3-1705628096] [2024-01-19T01:36:49.639Z] INFO[2024-01-19 01:36:49] Actual:true, Expected: true [2024-01-19T01:36:49.896Z] 2024-01-19 01:36:49 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Registering Source and Destination clusters from user context for user -testuser9-1705628096 [2024-01-19T01:36:49.896Z] INFO[2024-01-19 01:36:49] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T01:36:49.896Z] INFO[2024-01-19 01:36:49] Actual:true, Expected: true [2024-01-19T01:36:49.896Z] 2024-01-19 01:36:49 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T01:36:49.896Z] 2024-01-19 01:36:49 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T01:36:49.896Z] 2024-01-19 01:36:49 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T01:36:49.896Z] 2024-01-19 01:36:49 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T01:36:49.896Z] 2024-01-19 01:36:49 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [source-cluster] in org [default] [2024-01-19T01:36:49.897Z] 2024-01-19 01:36:49 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T01:36:49.897Z] 2024-01-19 01:36:49 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T01:36:49.897Z] 2024-01-19 01:36:49 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:36:49.897Z] 2024-01-19 01:36:49 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T01:36:49.897Z] 2024-01-19 01:36:49 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T01:36:49.897Z] 2024-01-19 01:36:49 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-phcef] for IBM [2024-01-19T01:36:49.897Z] 2024-01-19 01:36:49 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-phcef] for org [default] with [ibm] as provider [2024-01-19T01:36:49.897Z] 2024-01-19 01:36:49 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:49.897Z] 2024-01-19 01:36:49 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:49.897Z] 2024-01-19 01:36:49 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-phcef] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName9 LastName9 with roles: [px-backup-app.user]] [2024-01-19T01:36:49.897Z] 2024-01-19 01:36:49 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName9 LastName9 with roles: [px-backup-app.user] [2024-01-19T01:36:50.460Z] 2024-01-19 01:36:50 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-phcef from admin context and sharing with all the users [2024-01-19T01:36:50.460Z] 2024-01-19 01:36:50 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-phcef] for org [default] with [ibm] as provider [2024-01-19T01:36:50.460Z] 2024-01-19 01:36:50 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:50.460Z] 2024-01-19 01:36:50 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:50.460Z] 2024-01-19 01:36:50 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-phcef] is present [2024-01-19T01:36:50.719Z] 2024-01-19 01:36:50 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [source-cluster] in org [default] [2024-01-19T01:36:51.650Z] 2024-01-19 01:36:51 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-posci] for IBM [2024-01-19T01:36:51.650Z] 2024-01-19 01:36:51 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-posci] for org [default] with [ibm] as provider [2024-01-19T01:36:51.650Z] 2024-01-19 01:36:51 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:51.650Z] 2024-01-19 01:36:51 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:51.650Z] 2024-01-19 01:36:51 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-posci] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName9 LastName9 with roles: [px-backup-app.user]] [2024-01-19T01:36:51.650Z] 2024-01-19 01:36:51 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName9 LastName9 with roles: [px-backup-app.user] [2024-01-19T01:36:51.651Z] 2024-01-19 01:36:51 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-posci from admin context and sharing with all the users [2024-01-19T01:36:51.651Z] 2024-01-19 01:36:51 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-posci] for org [default] with [ibm] as provider [2024-01-19T01:36:51.651Z] 2024-01-19 01:36:51 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:36:51.651Z] 2024-01-19 01:36:51 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:36:51.907Z] 2024-01-19 01:36:51 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-posci] is present [2024-01-19T01:36:51.907Z] 2024-01-19 01:36:51 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:36:52.835Z] INFO[2024-01-19 01:36:52] Verifying : Description : Creating source and destination cluster for user testuser9-1705628096 [2024-01-19T01:36:52.836Z] INFO[2024-01-19 01:36:52] Actual:true, Expected: true [2024-01-19T01:36:52.836Z] INFO[2024-01-19 01:36:52] Verifying : Description : Verifying backup restore [tp-restore-1705628212] is not possible for backup [tp-backup-1705628005] with user [testuser9-1705628096] [2024-01-19T01:36:52.836Z] INFO[2024-01-19 01:36:52] Actual:true, Expected: true [2024-01-19T01:36:52.836Z] 2024-01-19 01:36:52 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Sharing Backup with multiple users [2024-01-19T01:36:53.093Z] 2024-01-19 01:36:53 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705628005 - 9ca140d9-0ea0-4038-8ddf-c99ec8d41591 [2024-01-19T01:36:54.025Z] 2024-01-19 01:36:53 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser7-1705628096 - fa58aa4c-f46b-454c-bd52-8370f548efa3 [2024-01-19T01:36:54.954Z] 2024-01-19 01:36:54 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2-1705628096 - 8f27b128-af90-4fa9-be36-35330680231b [2024-01-19T01:36:55.516Z] 2024-01-19 01:36:55 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705628096 - cc430c49-9272-41cd-b95e-4be0248d69bb [2024-01-19T01:36:56.443Z] 2024-01-19 01:36:56 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser5-1705628096 - 4e3b1848-48b5-46d0-9906-bd5e240ecc62 [2024-01-19T01:36:57.373Z] 2024-01-19 01:36:57 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser10-1705628096 - b35bb037-122f-498a-b15f-1688fe27cb60 [2024-01-19T01:36:57.935Z] 2024-01-19 01:36:57 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser4-1705628096 - 60bed100-ea29-40e1-b3ca-633f06aed5f7 [2024-01-19T01:36:58.862Z] 2024-01-19 01:36:58 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser8-1705628096 - 04d90090-0cca-4b75-b326-74a6f278659d [2024-01-19T01:36:59.790Z] 2024-01-19 01:36:59 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser6-1705628096 - 61c48609-132c-424f-a593-680e49e8a8cb [2024-01-19T01:37:00.351Z] 2024-01-19 01:37:00 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser3-1705628096 - 7a6999d9-4319-4019-9086-498f3352e9c0 [2024-01-19T01:37:01.278Z] 2024-01-19 01:37:01 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser9-1705628096 - fa63e049-e0a3-40a4-8318-732a19e20499 [2024-01-19T01:37:01.278Z] 2024-01-19 01:37:01 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Restart mongo pod when backup sharing is in-progress [2024-01-19T01:37:01.278Z] INFO[2024-01-19 01:37:01] Verifying : Description : Getting px-backup namespace [2024-01-19T01:37:01.278Z] INFO[2024-01-19 01:37:01] Actual:true, Expected: true [2024-01-19T01:37:01.853Z] 2024-01-19 01:37:01 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.DeletePodWithWithoutLabelInNamespace:#1701] - Deleting pod pxc-backup-mongodb-0 with label map[app.kubernetes.io/component:pxc-backup-mongodb] [2024-01-19T01:37:02.109Z] 2024/01/19 01:37:02 DoRetryWithTimeout - Error: {pod px-backup:pxc-backup-mongodb-0 (14e70588-37fa-4241-b9ab-01c801aa97b2) still present in the system}, Next try in [5s], timeout [5m0s] [2024-01-19T01:37:07.354Z] 2024/01/19 01:37:07 DoRetryWithTimeout - Error: {pod px-backup:pxc-backup-mongodb-0 (14e70588-37fa-4241-b9ab-01c801aa97b2) still present in the system}, Next try in [5s], timeout [5m0s] [2024-01-19T01:37:12.599Z] 2024/01/19 01:37:12 DoRetryWithTimeout - Error: {pod px-backup:pxc-backup-mongodb-0 (14e70588-37fa-4241-b9ab-01c801aa97b2) still present in the system}, Next try in [5s], timeout [5m0s] [2024-01-19T01:37:17.843Z] 2024/01/19 01:37:17 DoRetryWithTimeout - Error: {pod px-backup:pxc-backup-mongodb-0 (14e70588-37fa-4241-b9ab-01c801aa97b2) still present in the system}, Next try in [5s], timeout [5m0s] [2024-01-19T01:37:23.099Z] 2024-01-19 01:37:22 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.DeletePodWithWithoutLabelInNamespace:#1701] - Deleting pod pxc-backup-mongodb-1 with label map[app.kubernetes.io/component:pxc-backup-mongodb] [2024-01-19T01:37:23.099Z] 2024/01/19 01:37:22 DoRetryWithTimeout - Error: {pod px-backup:pxc-backup-mongodb-1 (b429ea13-c5a0-4897-8be4-69a3a40b6ced) still present in the system}, Next try in [5s], timeout [5m0s] [2024-01-19T01:37:28.347Z] 2024/01/19 01:37:27 DoRetryWithTimeout - Error: {pod px-backup:pxc-backup-mongodb-1 (b429ea13-c5a0-4897-8be4-69a3a40b6ced) still present in the system}, Next try in [5s], timeout [5m0s] [2024-01-19T01:37:32.511Z] 2024/01/19 01:37:32 DoRetryWithTimeout - Error: {pod px-backup:pxc-backup-mongodb-1 (b429ea13-c5a0-4897-8be4-69a3a40b6ced) still present in the system}, Next try in [5s], timeout [5m0s] [2024-01-19T01:37:37.861Z] 2024/01/19 01:37:37 DoRetryWithTimeout - Error: {pod px-backup:pxc-backup-mongodb-1 (b429ea13-c5a0-4897-8be4-69a3a40b6ced) still present in the system}, Next try in [5s], timeout [5m0s] [2024-01-19T01:37:43.106Z] 2024-01-19 01:37:42 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.DeletePodWithWithoutLabelInNamespace:#1701] - Deleting pod pxc-backup-mongodb-2 with label map[app.kubernetes.io/component:pxc-backup-mongodb] [2024-01-19T01:37:43.106Z] 2024/01/19 01:37:42 DoRetryWithTimeout - Error: {pod px-backup:pxc-backup-mongodb-2 (5936ba96-b7a8-43f9-821a-68892b54bb25) still present in the system}, Next try in [5s], timeout [5m0s] [2024-01-19T01:37:48.350Z] 2024/01/19 01:37:47 DoRetryWithTimeout - Error: {pod px-backup:pxc-backup-mongodb-2 (5936ba96-b7a8-43f9-821a-68892b54bb25) still present in the system}, Next try in [5s], timeout [5m0s] [2024-01-19T01:37:53.592Z] 2024/01/19 01:37:52 DoRetryWithTimeout - Error: {pod px-backup:pxc-backup-mongodb-2 (5936ba96-b7a8-43f9-821a-68892b54bb25) still present in the system}, Next try in [5s], timeout [5m0s] [2024-01-19T01:37:57.760Z] 2024/01/19 01:37:57 DoRetryWithTimeout - Error: {pod px-backup:pxc-backup-mongodb-2 (5936ba96-b7a8-43f9-821a-68892b54bb25) still present in the system}, Next try in [5s], timeout [5m0s] [2024-01-19T01:38:03.019Z] INFO[2024-01-19 01:38:02] Verifying : Description : Restart mongo pod when backup sharing is in-progress [2024-01-19T01:38:03.019Z] INFO[2024-01-19 01:38:02] Actual:true, Expected: true [2024-01-19T01:38:03.019Z] 2024-01-19 01:38:02 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.IsMongoDBReady:#3691] - Verify that at least 2 mongodb pods are in Ready state at the end of the testcase [2024-01-19T01:38:03.019Z] 2024/01/19 01:38:02 DoRetryWithTimeout - Error: {mongodb pods are not ready yet. expected ready pods - 3, actual ready pods - 0}, Next try in [30s], timeout [30m0s] [2024-01-19T01:38:35.062Z] 2024/01/19 01:38:32 DoRetryWithTimeout - Error: {mongodb pods are not ready yet. expected ready pods - 3, actual ready pods - 1}, Next try in [30s], timeout [30m0s] [2024-01-19T01:39:07.111Z] 2024/01/19 01:39:02 DoRetryWithTimeout - Error: {mongodb pods are not ready yet. expected ready pods - 3, actual ready pods - 2}, Next try in [30s], timeout [30m0s] [2024-01-19T01:39:33.616Z] 2024-01-19 01:39:32 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.IsMongoDBReady:#3731] - Number of mongodb pods in Ready state are 3 [2024-01-19T01:39:33.617Z] 2024/01/19 01:39:33 DoRetryWithTimeout - Error: {restore status for [tp-restore-1705628372] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T01:40:05.697Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.CreateRestore:#936] - Restore [tp-restore-1705628372] created successfully [2024-01-19T01:40:05.697Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Switching cluster context to cluster [destination-cluster] [2024-01-19T01:40:05.697Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T01:40:05.697Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-82948-01-19-01h31m47s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-19T01:40:05.697Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-82948-01-19-01h31m47s] [2024-01-19T01:40:05.697Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T01:40:05.697Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T01:40:05.697Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Validating restore [tp-restore-1705628372] in org [default] [2024-01-19T01:40:05.697Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-restore-1705628372] [2024-01-19T01:40:05.697Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-restore-1705628372] [2024-01-19T01:40:05.697Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-82948-01-19-01h31m47s] in restore [tp-restore-1705628372] [2024-01-19T01:40:05.697Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-82948-01-19-01h31m47s]) was found in restore [tp-restore-1705628372], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82948-01-19-01h31m47s] [2024-01-19T01:40:05.697Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-82948-01-19-01h31m47s]) found in the restore [tp-restore-1705628372] was also present on the cluster/namespace [mysql-ibm-pxb-0-82948-01-19-01h31m47s] [2024-01-19T01:40:05.697Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-82948-01-19-01h31m47s]) was found in restore [tp-restore-1705628372], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82948-01-19-01h31m47s] [2024-01-19T01:40:05.697Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-82948-01-19-01h31m47s]) found in the restore [tp-restore-1705628372] was also present on the cluster/namespace [mysql-ibm-pxb-0-82948-01-19-01h31m47s] [2024-01-19T01:40:05.698Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-82948-01-19-01h31m47s]) was found in restore [tp-restore-1705628372], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82948-01-19-01h31m47s] [2024-01-19T01:40:05.698Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-82948-01-19-01h31m47s]) found in the restore [tp-restore-1705628372] was also present on the cluster/namespace [mysql-ibm-pxb-0-82948-01-19-01h31m47s] [2024-01-19T01:40:05.698Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-82948-01-19-01h31m47s]) was found in restore [tp-restore-1705628372], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82948-01-19-01h31m47s] [2024-01-19T01:40:05.698Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-82948-01-19-01h31m47s]) found in the restore [tp-restore-1705628372] was also present on the cluster/namespace [mysql-ibm-pxb-0-82948-01-19-01h31m47s] [2024-01-19T01:40:05.698Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-82948-01-19-01h31m47s] in restore [tp-restore-1705628372] [2024-01-19T01:40:05.698Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-82948-01-19-01h31m47s] in restore [tp-restore-1705628372] [2024-01-19T01:40:05.698Z] 2024-01-19 01:40:03 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T01:40:05.698Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T01:40:05.698Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82948-01-19-01h31m47s] are [[mysql-data]] [2024-01-19T01:40:05.698Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [backup.ValidateRestore:#2240] - in restore [tp-restore-1705628372], the status of the restored volume [pvc-6cdf1d06-9553-4611-9633-f41ceabed9df] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-19T01:40:05.698Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Validate applications in restored namespace [mysql-ibm-pxb-0-82948-01-19-01h31m47s] due to restore [tp-restore-1705628372] [2024-01-19T01:40:05.698Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Validating mysql-ibm app [2024-01-19T01:40:05.698Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] wait for mysql-ibm app to start running [2024-01-19T01:40:05.698Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T01:40:05.698Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T01:40:05.699Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T01:40:05.699Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Switching cluster context back to cluster path [/tmp/destination-config] [2024-01-19T01:40:05.699Z] 2024-01-19 01:40:03 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T01:40:05.699Z] INFO[2024-01-19 01:40:03] Verifying : Description : Restore the backup tp-backup-1705628005 for user testuser7-1705628096 [2024-01-19T01:40:05.699Z] ERRO[2024-01-19 01:40:03] ValidateRestore Errors: {in restore [tp-restore-1705628372], said restored volume [pvc-6cdf1d06-9553-4611-9633-f41ceabed9df] cannot be found in the actual cluster [mysql-ibm-pxb-0-82948-01-19-01h31m47s]} [2024-01-19T01:40:05.699Z] 2024-01-19 01:40:04 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Deleting the deployed apps after the testcase [2024-01-19T01:40:05.699Z] 2024-01-19 01:40:04 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Destroying apps [2024-01-19T01:40:05.699Z] 2024-01-19 01:40:04 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T01:40:05.699Z] 2024-01-19 01:40:04 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T01:40:05.699Z] 2024-01-19 01:40:04 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] start destroying mysql-ibm app [2024-01-19T01:40:05.699Z] 2024-01-19 01:40:04 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T01:40:05.699Z] 2024-01-19 01:40:04 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T01:40:05.699Z] 2024-01-19 01:40:04 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T01:40:05.699Z] INFO[2024-01-19 01:40:04] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T01:40:05.699Z] INFO[2024-01-19 01:40:04] Actual:true, Expected: true [2024-01-19T01:40:05.699Z] 2024-01-19 01:40:04 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Deleting the backups [2024-01-19T01:40:05.699Z] INFO[2024-01-19 01:40:04] Verifying : Description : Deleting the backup tp-backup-1705628005 [2024-01-19T01:40:05.699Z] INFO[2024-01-19 01:40:04] Actual:true, Expected: true [2024-01-19T01:40:05.699Z] 2024-01-19 01:40:04 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Cleaning backup locations in map [map[4b199f31-3f2e-4a73-a7ab-aeb20f3ab33a:autogenerated-backup-location-1705628002]], cloud credential [cred-aws-1705592040], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T01:40:05.699Z] INFO[2024-01-19 01:40:04] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-1705628002] [2024-01-19T01:40:05.699Z] ERRO[2024-01-19 01:40:04] Actual:rpc error: code = Unknown desc = restore object [tp-restore-1705628372] has reference to backuplocation [autogenerated-backup-location-1705628002:4b199f31-3f2e-4a73-a7ab-aeb20f3ab33a], Expected:  [2024-01-19T01:40:05.699Z] INFO[2024-01-19 01:40:04] --------Test End------ [2024-01-19T01:40:05.699Z] INFO[2024-01-19 01:40:04] #Test: [2024-01-19T01:40:05.699Z] INFO[2024-01-19 01:40:04] #Description: [2024-01-19T01:40:05.699Z] INFO[2024-01-19 01:40:04] ------------------------ [2024-01-19T01:40:05.699Z] 2024-01-19 01:40:04 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.EndPxBackupTorpedoTest:#7261] - >>>> FAILED TEST: {RestartBackupPodDuringBackupSharing} Restart backup pod during backup sharing [2024-01-19T01:40:05.699Z] 2024-01-19 01:40:04 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-82948-01-19-01h31m47s] [2024-01-19T01:40:05.700Z] 2024-01-19 01:40:05 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T01:40:05.700Z] 2024-01-19 01:40:05 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T01:40:05.700Z] 2024-01-19 01:40:05 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-82948-01-19-01h31m47s] [2024-01-19T01:40:05.700Z] 2024-01-19 01:40:05 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {RestartBackupPodDuringBackupSharing} Restart backup pod during backup sharing [2024-01-19T01:40:05.700Z] 2024-01-19 01:40:05 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T01:40:05.700Z] 2024/01/19 01:40:05 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T01:40:10.946Z] 2024/01/19 01:40:10 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T01:40:16.187Z] 2024/01/19 01:40:15 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T01:40:21.437Z] 2024/01/19 01:40:20 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T01:40:25.598Z] 2024/01/19 01:40:25 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T01:40:30.845Z] 2024/01/19 01:40:30 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T01:40:36.087Z] 2024/01/19 01:40:35 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T01:40:41.333Z] 2024/01/19 01:40:40 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T01:40:45.497Z] 2024/01/19 01:40:45 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.128.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.7 10.240.128.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T01:40:45.497Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T01:40:45.497Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T01:40:45.497Z] {dial tcp 10.240.128.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T01:40:50.740Z] 2024-01-19 01:40:50 +0000:[WARNING] [{RestartBackupPodDuringBackupSharing}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.7 10.240.128.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T01:40:50.740Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T01:40:50.740Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T01:40:50.740Z] {dial tcp 10.240.128.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T01:40:50.741Z] 2024-01-19 01:40:50 +0000:[ERROR] [{RestartBackupPodDuringBackupSharing}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.7 10.240.128.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T01:40:50.741Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T01:40:50.741Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T01:40:50.741Z] {dial tcp 10.240.128.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T01:40:50.741Z] 2024-01-19 01:40:50 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T01:40:50.741Z] 2024-01-19 01:40:50 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] Switching context to [/tmp/source-config] [2024-01-19T01:40:50.741Z] 2024-01-19 01:40:50 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T01:40:50.741Z] 2024-01-19 01:40:50 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T01:40:50.741Z] 2024-01-19 01:40:50 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T01:40:50.741Z] 2024-01-19 01:40:50 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T01:40:50.741Z] 2024-01-19 01:40:50 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T01:40:50.998Z] 2024-01-19 01:40:50 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T01:40:51.254Z] 2024-01-19 01:40:51 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T01:40:51.817Z] 2024-01-19 01:40:51 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T01:40:52.073Z] 2024-01-19 01:40:51 +0000:[INFO] [{RestartBackupPodDuringBackupSharing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T01:40:52.331Z] 2024/01/19 01:40:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:41:02.269Z] 2024/01/19 01:41:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:41:14.443Z] 2024/01/19 01:41:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:41:22.533Z] 2024/01/19 01:41:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:41:32.486Z] 2024/01/19 01:41:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:41:42.429Z] 2024/01/19 01:41:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:41:54.598Z] 2024/01/19 01:41:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:42:02.688Z] 2024/01/19 01:42:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:42:12.627Z] 2024/01/19 01:42:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:42:22.568Z] 2024/01/19 01:42:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:42:32.509Z] 2024/01/19 01:42:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:42:44.683Z] 2024/01/19 01:42:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:42:52.780Z] 2024/01/19 01:42:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:43:02.722Z] 2024/01/19 01:43:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:43:12.668Z] 2024/01/19 01:43:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:43:24.838Z] 2024/01/19 01:43:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:43:32.930Z] 2024/01/19 01:43:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:43:42.876Z] 2024/01/19 01:43:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:43:52.822Z] 2024/01/19 01:43:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:44:05.027Z] 2024/01/19 01:44:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:44:13.118Z] 2024/01/19 01:44:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:44:23.062Z] 2024/01/19 01:44:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:44:33.027Z] 2024/01/19 01:44:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:44:42.968Z] 2024/01/19 01:44:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:44:43.897Z] INFO[2024-01-19 01:44:43] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T01:44:43.897Z] INFO[2024-01-19 01:44:43] sched stats: tick interval: 36643 < 1.2s (100%) [2024-01-19T01:44:43.897Z] INFO[2024-01-19 01:44:43] sched stats: tick process durations: 36642 < 500ms (100%) [2024-01-19T01:44:53.838Z] 2024/01/19 01:44:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:45:03.781Z] 2024/01/19 01:45:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:45:13.742Z] 2024/01/19 01:45:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:45:23.685Z] 2024/01/19 01:45:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:45:33.626Z] 2024/01/19 01:45:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:45:43.572Z] 2024/01/19 01:45:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:45:53.526Z] 2024-01-19 01:45:53 +0000:[ERROR] [{RestartBackupPodDuringBackupSharing}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.526Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.526Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.526Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.526Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.526Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.526Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.526Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.526Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.526Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.526Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.526Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.527Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.527Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.527Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.527Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.527Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.527Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.527Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.527Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.527Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.527Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.527Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.527Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.527Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.527Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.527Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.527Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.527Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:45:53.527Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T01:45:53.527Z] • Failure [845.294 seconds] [2024-01-19T01:45:53.527Z] {RestartBackupPodDuringBackupSharing} [2024-01-19T01:45:53.527Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_resiliency_test.go:386 [2024-01-19T01:45:53.527Z] Restart backup pod during backup sharing [It] [2024-01-19T01:45:53.527Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_resiliency_test.go:422 [2024-01-19T01:45:53.527Z] [2024-01-19T01:45:53.527Z] Unexpected error: [2024-01-19T01:45:53.527Z] <*errors.errorString | 0xc0010be230>: { [2024-01-19T01:45:53.527Z] s: "error: ValidateRestore Errors: {in restore [tp-restore-1705628372], said restored volume [pvc-6cdf1d06-9553-4611-9633-f41ceabed9df] cannot be found in the actual cluster [mysql-ibm-pxb-0-82948-01-19-01h31m47s]}, Description: Restore the backup tp-backup-1705628005 for user testuser7-1705628096", [2024-01-19T01:45:53.527Z] } [2024-01-19T01:45:53.528Z] error: ValidateRestore Errors: {in restore [tp-restore-1705628372], said restored volume [pvc-6cdf1d06-9553-4611-9633-f41ceabed9df] cannot be found in the actual cluster [mysql-ibm-pxb-0-82948-01-19-01h31m47s]}, Description: Restore the backup tp-backup-1705628005 for user testuser7-1705628096 [2024-01-19T01:45:53.528Z] occurred [2024-01-19T01:45:53.528Z] [2024-01-19T01:45:53.528Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-19T01:45:53.528Z] [2024-01-19T01:45:53.528Z] Full Stack Trace [2024-01-19T01:45:53.528Z] github.com/portworx/torpedo/pkg/aetosutil.(*Dashboard).VerifyFatal(0x7f3430f?, {0x702af20?, 0xc0010be000}, {0x0?, 0x0}, {0xc0016efea0, 0x45}) [2024-01-19T01:45:53.528Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 +0x143 [2024-01-19T01:45:53.528Z] github.com/portworx/torpedo/tests/backup.glob..func49.2.12() [2024-01-19T01:45:53.528Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_resiliency_test.go:561 +0x418 [2024-01-19T01:45:53.528Z] github.com/portworx/torpedo/tests/backup.glob..func49.2() [2024-01-19T01:45:53.528Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_resiliency_test.go:551 +0x969 [2024-01-19T01:45:53.528Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T01:45:53.528Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T01:45:53.528Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T01:45:53.528Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T01:45:53.528Z] created by testing.(*T).Run [2024-01-19T01:45:53.528Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T01:45:53.528Z] ------------------------------ [2024-01-19T01:45:53.528Z] STEP: schedule applications [2024-01-19T01:45:53.528Z] STEP: Validate applications [2024-01-19T01:45:53.528Z] STEP: validate applications [2024-01-19T01:45:53.528Z] STEP: validate mysql-ibm app's volumes [2024-01-19T01:45:53.528Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T01:45:53.528Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T01:45:53.528Z] STEP: get mysql-ibm app's volume: pvc-27b20202-3340-4d3b-b819-728bfec2a5b8 inspected by the volume driver [2024-01-19T01:45:53.528Z] STEP: wait for mysql-ibm app to start running [2024-01-19T01:45:53.528Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T01:45:53.528Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T01:45:53.528Z] STEP: Validate Px pod restart count [2024-01-19T01:45:53.528Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T01:45:53.528Z] STEP: Creating cloud credentials [2024-01-19T01:45:53.528Z] STEP: Register cluster for backup [2024-01-19T01:45:53.528Z] STEP: Creating backup location [2024-01-19T01:45:53.528Z] STEP: Start backup of application to bucket [2024-01-19T01:45:53.528Z] STEP: Create users [2024-01-19T01:45:53.528Z] STEP: Share Backup with multiple users [2024-01-19T01:45:53.528Z] STEP: Restart backup pod when backup sharing is in-progress [2024-01-19T01:45:53.528Z] STEP: Validate the shared backup with users [2024-01-19T01:45:53.528Z] STEP: Share Backup with multiple users [2024-01-19T01:45:53.529Z] STEP: Restart mongo pods when backup sharing is in-progress [2024-01-19T01:45:53.529Z] STEP: Validate the shared backup with users [2024-01-19T01:45:53.529Z] STEP: validate mysql-ibm app's volumes [2024-01-19T01:45:53.529Z] STEP: wait for mysql-ibm app to start running [2024-01-19T01:45:53.529Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T01:45:53.529Z] STEP: Validate Px pod restart count [2024-01-19T01:45:53.529Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T01:45:53.529Z] STEP: destroy apps [2024-01-19T01:45:53.529Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T01:45:53.529Z] STEP: start destroying mysql-ibm app [2024-01-19T01:45:53.529Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T01:45:53.529Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] Log Dir: /testresults//NodeCountForLicensing.log [2024-01-19T01:45:53.529Z] INFO[2024-01-19 01:45:53] --------Test Start------ [2024-01-19T01:45:53.529Z] INFO[2024-01-19 01:45:53] #Test: NodeCountForLicensing [2024-01-19T01:45:53.529Z] INFO[2024-01-19 01:45:53] #Description: Verify worker node on application cluster with label portworx.io/nobackup=true is not counted for licensing [2024-01-19T01:45:53.529Z] INFO[2024-01-19 01:45:53] ------------------------ [2024-01-19T01:45:53.529Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] Adding source and destination cluster for backup [2024-01-19T01:45:53.529Z] INFO[2024-01-19 01:45:53] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T01:45:53.529Z] INFO[2024-01-19 01:45:53] Actual:true, Expected: true [2024-01-19T01:45:53.529Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T01:45:53.529Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T01:45:53.529Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T01:45:53.529Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T01:45:53.529Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] Create cluster [source-cluster] in org [default] [2024-01-19T01:45:53.529Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T01:45:53.529Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T01:45:53.529Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:45:53.529Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T01:45:53.529Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T01:45:53.529Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-sejkd] for IBM [2024-01-19T01:45:53.529Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-sejkd] for org [default] with [ibm] as provider [2024-01-19T01:45:53.529Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:45:53.529Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:45:53.530Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-sejkd] is present [2024-01-19T01:45:53.530Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] Create cluster [source-cluster] in org [default] [2024-01-19T01:45:53.786Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-dapwp] for IBM [2024-01-19T01:45:53.786Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-dapwp] for org [default] with [ibm] as provider [2024-01-19T01:45:53.786Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T01:45:53.786Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T01:45:53.786Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-dapwp] is present [2024-01-19T01:45:53.786Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] Create cluster [destination-cluster] in org [default] [2024-01-19T01:45:53.786Z] INFO[2024-01-19 01:45:53] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T01:45:53.786Z] INFO[2024-01-19 01:45:53] Actual:Online, Expected: Online [2024-01-19T01:45:53.786Z] INFO[2024-01-19 01:45:53] Verifying : Description : Verifying if [destination-cluster] cluster is online [2024-01-19T01:45:53.786Z] INFO[2024-01-19 01:45:53] Actual:Online, Expected: Online [2024-01-19T01:45:53.786Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] Getting the total number of worker nodes in source and destination cluster [2024-01-19T01:45:53.786Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] Total number of worker nodes in source cluster are 0 [2024-01-19T01:45:53.786Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] Switching cluster context to destination cluster [2024-01-19T01:45:53.786Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T01:45:53.786Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T01:45:53.786Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] Total number of worker nodes in destination cluster are 0 [2024-01-19T01:45:53.787Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] Total number of worker nodes in source and destination cluster are 0 [2024-01-19T01:45:53.787Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] Switching cluster context back to source cluster [2024-01-19T01:45:53.787Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T01:45:53.787Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] Switching context to [/tmp/source-config] [2024-01-19T01:45:54.043Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T01:45:54.044Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T01:45:54.044Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T01:45:54.044Z] 2024-01-19 01:45:53 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T01:45:54.302Z] 2024-01-19 01:45:54 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T01:45:54.302Z] 2024-01-19 01:45:54 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T01:45:54.865Z] 2024-01-19 01:45:54 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T01:45:55.121Z] 2024-01-19 01:45:54 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T01:45:55.683Z] 2024-01-19 01:45:55 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T01:45:55.683Z] 2024/01/19 01:45:55 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:46:07.854Z] 2024/01/19 01:46:05 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:46:15.946Z] 2024/01/19 01:46:15 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:46:25.894Z] 2024/01/19 01:46:25 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:46:37.040Z] 2024/01/19 01:46:35 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:46:46.987Z] 2024/01/19 01:46:45 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:46:56.928Z] 2024/01/19 01:46:55 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:47:06.873Z] 2024/01/19 01:47:05 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:47:16.815Z] 2024/01/19 01:47:15 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:47:26.768Z] 2024/01/19 01:47:25 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:47:36.713Z] 2024/01/19 01:47:35 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:47:46.659Z] 2024/01/19 01:47:45 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:47:56.600Z] 2024/01/19 01:47:55 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:48:06.556Z] 2024/01/19 01:48:05 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:48:16.507Z] 2024/01/19 01:48:15 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:48:26.451Z] 2024/01/19 01:48:25 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:48:36.396Z] 2024/01/19 01:48:36 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:48:46.339Z] 2024/01/19 01:48:46 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:48:56.285Z] 2024/01/19 01:48:56 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:49:06.231Z] 2024/01/19 01:49:06 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:49:16.172Z] 2024/01/19 01:49:16 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:49:26.159Z] 2024/01/19 01:49:26 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:49:38.333Z] 2024/01/19 01:49:36 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:49:46.412Z] 2024/01/19 01:49:46 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:49:56.351Z] 2024/01/19 01:49:56 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:50:06.291Z] 2024/01/19 01:50:06 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:50:18.461Z] 2024/01/19 01:50:16 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:50:26.541Z] 2024/01/19 01:50:26 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:50:36.485Z] 2024/01/19 01:50:36 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:50:46.443Z] 2024/01/19 01:50:46 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:50:56.404Z] 2024-01-19 01:50:56 +0000:[ERROR] [{NodeCountForLicensing}] [backup.glob..func11.2.2:#60] - Switching context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.404Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.404Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.404Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.404Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.404Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.405Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.405Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.405Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.405Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.405Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.405Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.405Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.405Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.405Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.405Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.405Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.405Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.405Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.405Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.405Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.405Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.405Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.405Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.405Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.405Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.406Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.406Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.406Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:50:56.406Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T01:50:56.406Z] 2024-01-19 01:50:56 +0000:[INFO] [{NodeCountForLicensing}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T01:50:56.406Z] 2024-01-19 01:50:56 +0000:[INFO] [{NodeCountForLicensing}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T01:50:56.406Z] INFO[2024-01-19 01:50:56] Verifying : Description : Switching context to destination cluster [2024-01-19T01:50:56.406Z] INFO[2024-01-19 01:50:56] Actual:true, Expected: true [2024-01-19T01:50:56.406Z] 2024-01-19 01:50:56 +0000:[INFO] [{NodeCountForLicensing}] Removing label portworx.io/nobackup=true from all worker nodes on destination cluster if present [2024-01-19T01:50:56.406Z] 2024-01-19 01:50:56 +0000:[INFO] [{NodeCountForLicensing}] Switching cluster context back to source cluster [2024-01-19T01:50:56.406Z] 2024-01-19 01:50:56 +0000:[INFO] [{NodeCountForLicensing}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T01:50:56.406Z] 2024-01-19 01:50:56 +0000:[INFO] [{NodeCountForLicensing}] Switching context to [/tmp/source-config] [2024-01-19T01:50:56.668Z] 2024-01-19 01:50:56 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T01:50:56.668Z] 2024-01-19 01:50:56 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T01:50:56.668Z] 2024-01-19 01:50:56 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T01:50:56.668Z] 2024-01-19 01:50:56 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T01:50:56.668Z] 2024-01-19 01:50:56 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T01:50:56.924Z] 2024-01-19 01:50:56 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T01:50:57.493Z] 2024-01-19 01:50:57 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T01:50:57.749Z] 2024-01-19 01:50:57 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T01:50:58.311Z] 2024-01-19 01:50:57 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T01:50:58.311Z] 2024/01/19 01:50:58 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:51:08.250Z] 2024/01/19 01:51:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:51:20.421Z] 2024/01/19 01:51:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:51:28.502Z] 2024/01/19 01:51:28 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:51:38.436Z] 2024/01/19 01:51:38 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:51:48.380Z] 2024/01/19 01:51:48 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:52:00.576Z] 2024/01/19 01:51:58 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:52:08.662Z] 2024/01/19 01:52:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:52:18.599Z] 2024/01/19 01:52:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:52:28.542Z] 2024/01/19 01:52:28 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:52:40.704Z] 2024/01/19 01:52:38 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:52:48.799Z] 2024/01/19 01:52:48 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:52:58.790Z] 2024/01/19 01:52:58 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:53:08.732Z] 2024/01/19 01:53:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:53:18.726Z] 2024/01/19 01:53:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:53:28.667Z] 2024/01/19 01:53:28 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:53:40.838Z] 2024/01/19 01:53:38 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:53:48.921Z] 2024/01/19 01:53:48 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:53:58.864Z] 2024/01/19 01:53:58 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:54:08.802Z] 2024/01/19 01:54:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:54:20.978Z] 2024/01/19 01:54:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:54:29.085Z] 2024/01/19 01:54:28 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:54:39.024Z] 2024/01/19 01:54:38 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:54:45.555Z] INFO[2024-01-19 01:54:44] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T01:54:45.555Z] INFO[2024-01-19 01:54:44] sched stats: tick interval: 37244 < 1.2s (100%) [2024-01-19T01:54:45.555Z] INFO[2024-01-19 01:54:44] sched stats: tick process durations: 37243 < 500ms (100%) [2024-01-19T01:54:49.719Z] 2024/01/19 01:54:48 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:54:59.660Z] 2024/01/19 01:54:58 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:55:09.606Z] 2024/01/19 01:55:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:55:19.545Z] 2024/01/19 01:55:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:55:29.495Z] 2024/01/19 01:55:28 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:55:39.432Z] 2024/01/19 01:55:38 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:55:49.372Z] 2024/01/19 01:55:49 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:55:59.314Z] INFO[2024-01-19 01:55:59] Verifying : Description : Switching context to source cluster [2024-01-19T01:55:59.314Z] ERRO[2024-01-19 01:55:59] failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.314Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.314Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.314Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.314Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.314Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.314Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.314Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.314Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.314Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.314Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.314Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.314Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.315Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.315Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.315Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.315Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.315Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.315Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.315Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.315Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.315Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.315Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.315Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.315Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.315Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.315Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.315Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.315Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T01:55:59.316Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T01:55:59.316Z] 2024-01-19 01:55:59 +0000:[INFO] [{NodeCountForLicensing}] Removing label portworx.io/nobackup=true from all worker nodes on source cluster if present [2024-01-19T01:55:59.316Z] 2024-01-19 01:55:59 +0000:[INFO] [{NodeCountForLicensing}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T01:55:59.316Z] INFO[2024-01-19 01:55:59] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T01:55:59.316Z] INFO[2024-01-19 01:55:59] Actual:true, Expected: true [2024-01-19T01:55:59.573Z] INFO[2024-01-19 01:55:59] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T01:55:59.573Z] ERRO[2024-01-19 01:55:59] Actual:rpc error: code = FailedPrecondition desc = cluster [destination-cluster] in use by backupschedule [tp-backup-schedule-1705605709], cannot be deleted, Expected:  [2024-01-19T01:56:00.135Z] INFO[2024-01-19 01:55:59] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-zbbvt] [2024-01-19T01:56:00.135Z] ERRO[2024-01-19 01:55:59] Actual:rpc error: code = Unknown desc = cluster [destination-cluster] has reference to cloud credential [ibm-destination-config-cloud-cred-zbbvt:a271ea5b-dc9d-40f1-a82a-651539c7e486], Expected:  [2024-01-19T01:56:00.391Z] INFO[2024-01-19 01:56:00] Verifying : Description : Deleting cluster source-cluster [2024-01-19T01:56:00.391Z] INFO[2024-01-19 01:56:00] Actual:true, Expected: true [2024-01-19T01:56:01.754Z] INFO[2024-01-19 01:56:01] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-vtjgs] [2024-01-19T01:56:01.754Z] INFO[2024-01-19 01:56:01] Actual:true, Expected: true [2024-01-19T01:56:02.013Z] INFO[2024-01-19 01:56:01] Verifying : Description : Deleting cluster source-cluster [2024-01-19T01:56:02.013Z] INFO[2024-01-19 01:56:01] Actual:true, Expected: true [2024-01-19T01:56:02.939Z] INFO[2024-01-19 01:56:02] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-liejf] [2024-01-19T01:56:02.939Z] INFO[2024-01-19 01:56:02] Actual:true, Expected: true [2024-01-19T01:56:03.196Z] INFO[2024-01-19 01:56:03] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T01:56:03.196Z] INFO[2024-01-19 01:56:03] Actual:true, Expected: true [2024-01-19T01:56:04.559Z] INFO[2024-01-19 01:56:04] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-honeu] [2024-01-19T01:56:04.559Z] INFO[2024-01-19 01:56:04] Actual:true, Expected: true [2024-01-19T01:56:04.559Z] INFO[2024-01-19 01:56:04] Verifying : Description : Deleting cluster source-cluster [2024-01-19T01:56:04.559Z] INFO[2024-01-19 01:56:04] Actual:true, Expected: true [2024-01-19T01:56:05.923Z] INFO[2024-01-19 01:56:05] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-prnvt] [2024-01-19T01:56:05.923Z] INFO[2024-01-19 01:56:05] Actual:true, Expected: true [2024-01-19T01:56:06.179Z] INFO[2024-01-19 01:56:05] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T01:56:06.179Z] INFO[2024-01-19 01:56:05] Actual:true, Expected: true [2024-01-19T01:56:07.106Z] INFO[2024-01-19 01:56:06] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-soitk] [2024-01-19T01:56:07.106Z] INFO[2024-01-19 01:56:06] Actual:true, Expected: true [2024-01-19T01:56:07.364Z] INFO[2024-01-19 01:56:07] Verifying : Description : Deleting cluster source-cluster [2024-01-19T01:56:07.364Z] INFO[2024-01-19 01:56:07] Actual:true, Expected: true [2024-01-19T01:56:08.740Z] INFO[2024-01-19 01:56:08] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-ckdjo] [2024-01-19T01:56:08.740Z] INFO[2024-01-19 01:56:08] Actual:true, Expected: true [2024-01-19T01:56:08.740Z] INFO[2024-01-19 01:56:08] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T01:56:08.740Z] INFO[2024-01-19 01:56:08] Actual:true, Expected: true [2024-01-19T01:56:10.106Z] INFO[2024-01-19 01:56:09] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-ldiyr] [2024-01-19T01:56:10.106Z] INFO[2024-01-19 01:56:09] Actual:true, Expected: true [2024-01-19T01:56:10.106Z] INFO[2024-01-19 01:56:10] Verifying : Description : Deleting cluster source-cluster [2024-01-19T01:56:10.106Z] INFO[2024-01-19 01:56:10] Actual:true, Expected: true [2024-01-19T01:56:11.473Z] INFO[2024-01-19 01:56:11] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-qtwsj] [2024-01-19T01:56:11.473Z] INFO[2024-01-19 01:56:11] Actual:true, Expected: true [2024-01-19T01:56:11.728Z] INFO[2024-01-19 01:56:11] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T01:56:11.728Z] INFO[2024-01-19 01:56:11] Actual:true, Expected: true [2024-01-19T01:56:12.656Z] INFO[2024-01-19 01:56:12] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-nlppm] [2024-01-19T01:56:12.656Z] INFO[2024-01-19 01:56:12] Actual:true, Expected: true [2024-01-19T01:56:12.912Z] INFO[2024-01-19 01:56:12] Verifying : Description : Deleting cluster source-cluster [2024-01-19T01:56:12.912Z] INFO[2024-01-19 01:56:12] Actual:true, Expected: true [2024-01-19T01:56:14.278Z] INFO[2024-01-19 01:56:13] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-jucrm] [2024-01-19T01:56:14.278Z] INFO[2024-01-19 01:56:13] Actual:true, Expected: true [2024-01-19T01:56:14.278Z] INFO[2024-01-19 01:56:14] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T01:56:14.278Z] INFO[2024-01-19 01:56:14] Actual:true, Expected: true [2024-01-19T01:56:15.645Z] INFO[2024-01-19 01:56:15] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-vrgdl] [2024-01-19T01:56:15.645Z] INFO[2024-01-19 01:56:15] Actual:true, Expected: true [2024-01-19T01:56:15.900Z] INFO[2024-01-19 01:56:15] Verifying : Description : Deleting cluster source-cluster [2024-01-19T01:56:15.900Z] INFO[2024-01-19 01:56:15] Actual:true, Expected: true [2024-01-19T01:56:16.827Z] INFO[2024-01-19 01:56:16] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-bgkpv] [2024-01-19T01:56:16.827Z] INFO[2024-01-19 01:56:16] Actual:true, Expected: true [2024-01-19T01:56:17.083Z] INFO[2024-01-19 01:56:17] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T01:56:17.083Z] INFO[2024-01-19 01:56:17] Actual:true, Expected: true [2024-01-19T01:56:18.450Z] INFO[2024-01-19 01:56:18] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-bnqjf] [2024-01-19T01:56:18.450Z] INFO[2024-01-19 01:56:18] Actual:true, Expected: true [2024-01-19T01:56:18.706Z] INFO[2024-01-19 01:56:18] Verifying : Description : Deleting cluster source-cluster [2024-01-19T01:56:18.706Z] INFO[2024-01-19 01:56:18] Actual:true, Expected: true [2024-01-19T01:56:19.633Z] INFO[2024-01-19 01:56:19] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-oudxv] [2024-01-19T01:56:19.633Z] INFO[2024-01-19 01:56:19] Actual:true, Expected: true [2024-01-19T01:56:19.890Z] INFO[2024-01-19 01:56:19] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T01:56:19.890Z] INFO[2024-01-19 01:56:19] Actual:true, Expected: true [2024-01-19T01:56:21.254Z] INFO[2024-01-19 01:56:20] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-lkvsh] [2024-01-19T01:56:21.254Z] INFO[2024-01-19 01:56:20] Actual:true, Expected: true [2024-01-19T01:56:21.509Z] INFO[2024-01-19 01:56:21] Verifying : Description : Deleting cluster source-cluster [2024-01-19T01:56:21.509Z] INFO[2024-01-19 01:56:21] Actual:true, Expected: true [2024-01-19T01:56:22.437Z] INFO[2024-01-19 01:56:22] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-wylyd] [2024-01-19T01:56:22.437Z] INFO[2024-01-19 01:56:22] Actual:true, Expected: true [2024-01-19T01:56:22.693Z] INFO[2024-01-19 01:56:22] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T01:56:22.693Z] INFO[2024-01-19 01:56:22] Actual:true, Expected: true [2024-01-19T01:56:24.056Z] INFO[2024-01-19 01:56:23] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-ptofl] [2024-01-19T01:56:24.056Z] INFO[2024-01-19 01:56:23] Actual:true, Expected: true [2024-01-19T01:56:24.056Z] INFO[2024-01-19 01:56:23] Verifying : Description : Deleting cluster source-cluster [2024-01-19T01:56:24.056Z] INFO[2024-01-19 01:56:23] Actual:true, Expected: true [2024-01-19T01:56:25.422Z] INFO[2024-01-19 01:56:25] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-liytn] [2024-01-19T01:56:25.422Z] INFO[2024-01-19 01:56:25] Actual:true, Expected: true [2024-01-19T01:56:25.422Z] INFO[2024-01-19 01:56:25] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T01:56:25.422Z] INFO[2024-01-19 01:56:25] Actual:true, Expected: true [2024-01-19T01:56:26.791Z] INFO[2024-01-19 01:56:26] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-iqdxq] [2024-01-19T01:56:26.791Z] INFO[2024-01-19 01:56:26] Actual:true, Expected: true [2024-01-19T01:56:26.791Z] INFO[2024-01-19 01:56:26] Verifying : Description : Deleting cluster source-cluster [2024-01-19T01:56:26.791Z] INFO[2024-01-19 01:56:26] Actual:true, Expected: true [2024-01-19T01:56:28.155Z] INFO[2024-01-19 01:56:27] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-phcef] [2024-01-19T01:56:28.155Z] INFO[2024-01-19 01:56:27] Actual:true, Expected: true [2024-01-19T01:56:28.155Z] INFO[2024-01-19 01:56:28] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T01:56:28.155Z] INFO[2024-01-19 01:56:28] Actual:true, Expected: true [2024-01-19T01:56:29.521Z] INFO[2024-01-19 01:56:29] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-posci] [2024-01-19T01:56:29.521Z] INFO[2024-01-19 01:56:29] Actual:true, Expected: true [2024-01-19T01:56:29.521Z] INFO[2024-01-19 01:56:29] --------Test End------ [2024-01-19T01:56:29.522Z] INFO[2024-01-19 01:56:29] #Test: [2024-01-19T01:56:29.522Z] INFO[2024-01-19 01:56:29] #Description: [2024-01-19T01:56:29.522Z] INFO[2024-01-19 01:56:29] ------------------------ [2024-01-19T01:56:29.522Z] 2024-01-19 01:56:29 +0000:[INFO] [{NodeCountForLicensing}] [tests.EndPxBackupTorpedoTest:#7261] - >>>> FAILED TEST: {NodeCountForLicensing} Verify worker node on application cluster with label portworx.io/nobackup=true is not counted for licensing [2024-01-19T01:56:29.776Z] 2024-01-19 01:56:29 +0000:[INFO] [{NodeCountForLicensing}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T01:56:29.776Z] 2024-01-19 01:56:29 +0000:[INFO] [{NodeCountForLicensing}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T01:56:29.776Z] 2024-01-19 01:56:29 +0000:[INFO] [{NodeCountForLicensing}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {NodeCountForLicensing} Verify worker node on application cluster with label portworx.io/nobackup=true is not counted for licensing [2024-01-19T01:56:29.777Z] 2024-01-19 01:56:29 +0000:[INFO] [{NodeCountForLicensing}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T01:56:29.777Z] 2024/01/19 01:56:29 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T01:56:35.024Z] 2024/01/19 01:56:34 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T01:56:40.265Z] 2024/01/19 01:56:39 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T01:56:45.513Z] 2024/01/19 01:56:44 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T01:56:50.761Z] 2024/01/19 01:56:49 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T01:56:54.925Z] 2024/01/19 01:56:54 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T01:57:00.171Z] 2024/01/19 01:56:59 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T01:57:05.417Z] 2024/01/19 01:57:04 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T01:57:10.673Z] 2024/01/19 01:57:09 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.0.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.7 10.240.0.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T01:57:10.673Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T01:57:10.673Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T01:57:10.673Z] {dial tcp 10.240.0.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T01:57:14.850Z] 2024-01-19 01:57:14 +0000:[WARNING] [{NodeCountForLicensing}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.7 10.240.0.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T01:57:14.850Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T01:57:14.850Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T01:57:14.850Z] {dial tcp 10.240.0.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T01:57:14.850Z] 2024-01-19 01:57:14 +0000:[ERROR] [{NodeCountForLicensing}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.7 10.240.0.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T01:57:14.850Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T01:57:14.850Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T01:57:14.850Z] {dial tcp 10.240.0.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T01:57:14.850Z] 2024-01-19 01:57:14 +0000:[INFO] [{NodeCountForLicensing}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T01:57:14.850Z] 2024-01-19 01:57:14 +0000:[INFO] [{NodeCountForLicensing}] Switching context to [/tmp/source-config] [2024-01-19T01:57:14.850Z] 2024-01-19 01:57:14 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T01:57:15.107Z] 2024-01-19 01:57:14 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T01:57:15.107Z] 2024-01-19 01:57:14 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T01:57:15.107Z] 2024-01-19 01:57:14 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T01:57:15.107Z] 2024-01-19 01:57:15 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T01:57:15.363Z] 2024-01-19 01:57:15 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T01:57:15.619Z] 2024-01-19 01:57:15 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T01:57:16.180Z] 2024-01-19 01:57:15 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T01:57:16.441Z] 2024-01-19 01:57:16 +0000:[INFO] [{NodeCountForLicensing}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T01:57:16.697Z] 2024/01/19 01:57:16 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:57:26.632Z] 2024/01/19 01:57:26 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:57:38.796Z] 2024/01/19 01:57:36 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:57:46.881Z] 2024/01/19 01:57:46 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:57:56.820Z] 2024/01/19 01:57:56 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:58:06.784Z] 2024/01/19 01:58:06 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:58:18.955Z] 2024/01/19 01:58:16 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:58:27.160Z] 2024/01/19 01:58:26 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:58:37.126Z] 2024/01/19 01:58:36 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:58:47.086Z] 2024/01/19 01:58:46 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:58:57.052Z] 2024/01/19 01:58:56 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:59:06.993Z] 2024/01/19 01:59:06 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:59:16.939Z] 2024/01/19 01:59:16 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:59:29.111Z] 2024/01/19 01:59:26 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:59:37.197Z] 2024/01/19 01:59:36 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:59:47.164Z] 2024/01/19 01:59:46 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T01:59:57.108Z] 2024/01/19 01:59:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:00:07.081Z] 2024/01/19 02:00:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:00:18.631Z] 2024/01/19 02:00:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:00:28.571Z] 2024/01/19 02:00:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:00:38.510Z] 2024/01/19 02:00:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:00:48.451Z] 2024/01/19 02:00:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:00:58.403Z] 2024/01/19 02:00:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:01:08.342Z] 2024/01/19 02:01:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:01:18.292Z] 2024/01/19 02:01:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:01:28.240Z] 2024/01/19 02:01:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:01:38.183Z] 2024/01/19 02:01:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:01:48.150Z] 2024/01/19 02:01:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:01:58.173Z] 2024/01/19 02:01:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:02:08.110Z] 2024/01/19 02:02:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:02:18.048Z] 2024-01-19 02:02:17 +0000:[ERROR] [{NodeCountForLicensing}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.048Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.048Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.048Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.048Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.049Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.049Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.049Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.049Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.049Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.049Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.049Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.049Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.049Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.049Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.049Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.049Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.049Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.049Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.049Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.049Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.049Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.049Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.049Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.049Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.049Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.049Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.050Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.050Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.050Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T02:02:18.050Z] • Failure [984.310 seconds] [2024-01-19T02:02:18.050Z] {NodeCountForLicensing} [2024-01-19T02:02:18.050Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_license_test.go:18 [2024-01-19T02:02:18.050Z] Verify worker node on application cluster with label portworx.io/nobackup=true is not counted for licensing [It] [2024-01-19T02:02:18.050Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_license_test.go:32 [2024-01-19T02:02:18.050Z] [2024-01-19T02:02:18.050Z] Unexpected error: [2024-01-19T02:02:18.050Z] <*errors.errorString | 0xc000f591e0>: { [2024-01-19T02:02:18.051Z] s: "failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{No... [2024-01-19T02:02:18.051Z] [2024-01-19T02:02:18.051Z] Gomega truncated this representation as it exceeds 'format.MaxLength'. [2024-01-19T02:02:18.051Z] Consider having the object provide a custom 'GomegaStringer' representation [2024-01-19T02:02:18.051Z] or adjust the parameters in Gomega's 'format' package. [2024-01-19T02:02:18.051Z] [2024-01-19T02:02:18.051Z] Learn more here: https://onsi.github.io/gomega/#adjusting-output [2024-01-19T02:02:18.051Z] [2024-01-19T02:02:18.051Z] failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.051Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.051Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.051Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.051Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.051Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.051Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.051Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.051Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.051Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.051Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.051Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.052Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.052Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.052Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.052Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.052Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.052Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.052Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.052Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.052Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.052Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.052Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.052Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.052Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.052Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.052Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.052Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.052Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:02:18.052Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T02:02:18.052Z] occurred [2024-01-19T02:02:18.052Z] [2024-01-19T02:02:18.052Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T02:02:18.052Z] [2024-01-19T02:02:18.052Z] Full Stack Trace [2024-01-19T02:02:18.052Z] github.com/portworx/torpedo/pkg/log.FailOnError({0x88a5060, 0xc000f591e0}, {0x7f42292?, 0x0?}, {0x0?, 0xc000ef3008?, 0x3dde6ed?}) [2024-01-19T02:02:18.052Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 +0x290 [2024-01-19T02:02:18.052Z] github.com/portworx/torpedo/tests/backup.glob..func11.2.2() [2024-01-19T02:02:18.052Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_license_test.go:60 +0x773 [2024-01-19T02:02:18.052Z] github.com/portworx/torpedo/tests/backup.glob..func11.2() [2024-01-19T02:02:18.052Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_license_test.go:46 +0x195 [2024-01-19T02:02:18.053Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T02:02:18.053Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T02:02:18.053Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T02:02:18.053Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T02:02:18.053Z] created by testing.(*T).Run [2024-01-19T02:02:18.053Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T02:02:18.053Z] ------------------------------ [2024-01-19T02:02:18.053Z] STEP: Adding source and destination cluster for backup [2024-01-19T02:02:18.053Z] STEP: Getting the total number of worker nodes in source and destination cluster [2024-01-19T02:02:18.053Z] SSSS2024-01-19 02:02:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Log Dir: /testresults//MultipleInPlaceRestoreSameTime.log [2024-01-19T02:02:18.053Z] INFO[2024-01-19 02:02:17] --------Test Start------ [2024-01-19T02:02:18.053Z] INFO[2024-01-19 02:02:17] #Test: MultipleInPlaceRestoreSameTime [2024-01-19T02:02:18.053Z] INFO[2024-01-19 02:02:17] #Description: Issue multiple in-place restores at the same time [2024-01-19T02:02:18.053Z] INFO[2024-01-19 02:02:17] ------------------------ [2024-01-19T02:02:18.053Z] 2024-01-19 02:02:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Deploy applications needed for backup [2024-01-19T02:02:18.053Z] 2024-01-19 02:02:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T02:02:18.053Z] 2024-01-19 02:02:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T02:02:18.053Z] 2024-01-19 02:02:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T02:02:18.053Z] 2024-01-19 02:02:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T02:02:18.053Z] 2024-01-19 02:02:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T02:02:18.053Z] 2024-01-19 02:02:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T02:02:18.054Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T02:02:18.055Z] 2024-01-19 02:02:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T02:02:18.055Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T02:02:18.055Z] 2024-01-19 02:02:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T02:02:18.055Z] 2024-01-19 02:02:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T02:02:18.055Z] 2024-01-19 02:02:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T02:02:18.055Z] 2024-01-19 02:02:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T02:02:18.055Z] 2024-01-19 02:02:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T02:02:18.055Z] 2024-01-19 02:02:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T02:02:18.055Z] 2024-01-19 02:02:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T02:02:18.056Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T02:02:18.056Z] 2024-01-19 02:02:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T02:02:18.057Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T02:02:18.057Z] 2024-01-19 02:02:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T02:02:18.057Z] 2024-01-19 02:02:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T02:02:18.057Z] 2024-01-19 02:02:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T02:02:18.057Z] 2024-01-19 02:02:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T02:02:18.314Z] 2024-01-19 02:02:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T02:02:18.314Z] 2024-01-19 02:02:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T02:02:18.570Z] 2024-01-19 02:02:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T02:02:18.571Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T02:02:18.571Z] 2024-01-19 02:02:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T02:02:18.572Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T02:02:18.572Z] 2024-01-19 02:02:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T02:02:18.572Z] 2024-01-19 02:02:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T02:02:18.572Z] 2024-01-19 02:02:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T02:02:18.572Z] 2024-01-19 02:02:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T02:02:18.572Z] 2024-01-19 02:02:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T02:02:18.572Z] 2024-01-19 02:02:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T02:02:18.572Z] 2024-01-19 02:02:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T02:02:18.573Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T02:02:18.829Z] 2024-01-19 02:02:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T02:02:18.829Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T02:02:18.829Z] 2024-01-19 02:02:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T02:02:18.829Z] 2024-01-19 02:02:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T02:02:18.829Z] 2024-01-19 02:02:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T02:02:18.829Z] 2024-01-19 02:02:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T02:02:18.829Z] 2024-01-19 02:02:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T02:02:18.829Z] 2024-01-19 02:02:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T02:02:19.089Z] 2024-01-19 02:02:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T02:02:19.091Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T02:02:19.091Z] 2024-01-19 02:02:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T02:02:19.091Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T02:02:19.091Z] 2024-01-19 02:02:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T02:02:20.018Z] 2024-01-19 02:02:19 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validate applications [2024-01-19T02:02:20.018Z] 2024-01-19 02:02:19 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validate applications [2024-01-19T02:02:20.018Z] 2024-01-19 02:02:19 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating mysql-ibm app [2024-01-19T02:02:20.018Z] 2024-01-19 02:02:19 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating mysql-ibm app's volumes [2024-01-19T02:02:20.273Z] 2024-01-19 02:02:19 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:02:20.273Z] 2024/01/19 02:02:20 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T02:02:32.442Z] 2024-01-19 02:02:30 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-58051-01-19-02h02m17s [2024-01-19T02:02:32.442Z] 2024-01-19 02:02:30 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T02:02:32.442Z] 2024-01-19 02:02:30 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T02:02:32.442Z] 2024-01-19 02:02:30 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] wait for mysql-ibm app to start running [2024-01-19T02:02:32.442Z] 2024-01-19 02:02:30 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T02:02:32.442Z] 2024/01/19 02:02:30 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:02:32.442Z] pod name:mysql-7d766cd65c-6xwkx namespace:mysql-ibm-pxb-0-58051-01-19-02h02m17s running:true ready:false node: [2024-01-19T02:02:32.442Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:02:42.391Z] 2024/01/19 02:02:40 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:02:42.391Z] pod name:mysql-7d766cd65c-6xwkx namespace:mysql-ibm-pxb-0-58051-01-19-02h02m17s running:false ready:false node:10.240.64.8 [2024-01-19T02:02:42.391Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:02:52.329Z] 2024/01/19 02:02:50 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:02:52.329Z] pod name:mysql-7d766cd65c-6xwkx namespace:mysql-ibm-pxb-0-58051-01-19-02h02m17s running:true ready:false node:10.240.64.8 [2024-01-19T02:02:52.329Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:03:02.273Z] 2024/01/19 02:03:00 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:03:02.273Z] pod name:mysql-7d766cd65c-6xwkx namespace:mysql-ibm-pxb-0-58051-01-19-02h02m17s running:true ready:false node:10.240.64.8 [2024-01-19T02:03:02.273Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:03:12.220Z] 2024/01/19 02:03:10 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:03:12.220Z] pod name:mysql-7d766cd65c-6xwkx namespace:mysql-ibm-pxb-0-58051-01-19-02h02m17s running:true ready:false node:10.240.64.8 [2024-01-19T02:03:12.220Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:03:22.170Z] 2024/01/19 02:03:20 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:03:22.170Z] pod name:mysql-7d766cd65c-6xwkx namespace:mysql-ibm-pxb-0-58051-01-19-02h02m17s running:true ready:false node:10.240.64.8 [2024-01-19T02:03:22.170Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:03:32.164Z] 2024/01/19 02:03:30 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:03:32.164Z] pod name:mysql-7d766cd65c-6xwkx namespace:mysql-ibm-pxb-0-58051-01-19-02h02m17s running:true ready:false node:10.240.64.8 [2024-01-19T02:03:32.164Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:03:42.113Z] 2024/01/19 02:03:41 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:03:42.113Z] pod name:mysql-7d766cd65c-6xwkx namespace:mysql-ibm-pxb-0-58051-01-19-02h02m17s running:true ready:false node:10.240.64.8 [2024-01-19T02:03:42.113Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:03:52.060Z] 2024/01/19 02:03:51 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:03:52.060Z] pod name:mysql-7d766cd65c-6xwkx namespace:mysql-ibm-pxb-0-58051-01-19-02h02m17s running:true ready:false node:10.240.64.8 [2024-01-19T02:03:52.060Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:04:02.001Z] 2024-01-19 02:04:01 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T02:04:02.001Z] 2024-01-19 02:04:01 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T02:04:02.001Z] 2024-01-19 02:04:01 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] validate if mysql-ibm app's volumes are setup [2024-01-19T02:04:02.002Z] 2024-01-19 02:04:01 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:04:02.002Z] 2024-01-19 02:04:01 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:04:02.002Z] 2024-01-19 02:04:01 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T02:04:02.002Z] 2024-01-19 02:04:01 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T02:04:02.002Z] 2024-01-19 02:04:01 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating mysql-ibm app [2024-01-19T02:04:02.002Z] 2024-01-19 02:04:01 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating mysql-ibm app's volumes [2024-01-19T02:04:02.002Z] 2024-01-19 02:04:01 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:04:02.002Z] 2024-01-19 02:04:01 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:04:02.002Z] 2024-01-19 02:04:01 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-1-58051-01-19-02h02m17s [2024-01-19T02:04:02.002Z] 2024-01-19 02:04:01 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T02:04:02.002Z] 2024-01-19 02:04:01 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T02:04:02.002Z] 2024-01-19 02:04:01 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] wait for mysql-ibm app to start running [2024-01-19T02:04:02.002Z] 2024-01-19 02:04:01 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T02:04:02.002Z] 2024/01/19 02:04:01 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:04:02.002Z] pod name:mysql-7d766cd65c-zgmgh namespace:mysql-ibm-pxb-1-58051-01-19-02h02m17s running:true ready:true node:10.240.64.27 [2024-01-19T02:04:02.002Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:04:11.950Z] 2024-01-19 02:04:11 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T02:04:12.205Z] 2024-01-19 02:04:11 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T02:04:12.205Z] 2024-01-19 02:04:12 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] validate if mysql-ibm app's volumes are setup [2024-01-19T02:04:12.205Z] 2024-01-19 02:04:12 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:04:12.206Z] 2024-01-19 02:04:12 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:04:12.206Z] 2024-01-19 02:04:12 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T02:04:12.206Z] 2024-01-19 02:04:12 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T02:04:12.461Z] 2024-01-19 02:04:12 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating mysql-ibm app [2024-01-19T02:04:12.462Z] 2024-01-19 02:04:12 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating mysql-ibm app's volumes [2024-01-19T02:04:12.462Z] 2024-01-19 02:04:12 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:04:12.462Z] 2024-01-19 02:04:12 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:04:12.462Z] 2024-01-19 02:04:12 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-2-58051-01-19-02h02m17s [2024-01-19T02:04:12.462Z] 2024-01-19 02:04:12 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T02:04:12.462Z] 2024-01-19 02:04:12 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T02:04:12.462Z] 2024-01-19 02:04:12 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] wait for mysql-ibm app to start running [2024-01-19T02:04:12.462Z] 2024-01-19 02:04:12 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T02:04:12.720Z] 2024-01-19 02:04:12 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T02:04:13.284Z] 2024-01-19 02:04:13 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T02:04:14.216Z] 2024-01-19 02:04:13 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] validate if mysql-ibm app's volumes are setup [2024-01-19T02:04:14.216Z] 2024-01-19 02:04:13 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:04:14.216Z] 2024-01-19 02:04:13 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:04:14.216Z] 2024-01-19 02:04:13 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T02:04:14.216Z] 2024-01-19 02:04:13 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T02:04:14.216Z] 2024-01-19 02:04:13 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating mysql-ibm app [2024-01-19T02:04:14.216Z] 2024-01-19 02:04:13 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating mysql-ibm app's volumes [2024-01-19T02:04:14.216Z] 2024-01-19 02:04:13 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:04:14.216Z] 2024-01-19 02:04:13 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:04:14.216Z] 2024-01-19 02:04:13 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-3-58051-01-19-02h02m17s [2024-01-19T02:04:14.216Z] 2024-01-19 02:04:14 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T02:04:14.217Z] 2024-01-19 02:04:14 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T02:04:14.217Z] 2024-01-19 02:04:14 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] wait for mysql-ibm app to start running [2024-01-19T02:04:14.217Z] 2024-01-19 02:04:14 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T02:04:14.778Z] 2024-01-19 02:04:14 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T02:04:15.341Z] 2024-01-19 02:04:15 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T02:04:15.905Z] 2024-01-19 02:04:15 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] validate if mysql-ibm app's volumes are setup [2024-01-19T02:04:16.161Z] 2024-01-19 02:04:15 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:04:16.161Z] 2024-01-19 02:04:15 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:04:16.161Z] 2024-01-19 02:04:15 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T02:04:16.161Z] 2024-01-19 02:04:15 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T02:04:16.161Z] 2024-01-19 02:04:15 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating mysql-ibm app [2024-01-19T02:04:16.161Z] 2024-01-19 02:04:15 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating mysql-ibm app's volumes [2024-01-19T02:04:16.161Z] 2024-01-19 02:04:16 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:04:16.161Z] 2024-01-19 02:04:16 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:04:16.161Z] 2024-01-19 02:04:16 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-4-58051-01-19-02h02m17s [2024-01-19T02:04:16.417Z] 2024-01-19 02:04:16 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T02:04:16.417Z] 2024-01-19 02:04:16 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T02:04:16.417Z] 2024-01-19 02:04:16 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] wait for mysql-ibm app to start running [2024-01-19T02:04:16.417Z] 2024-01-19 02:04:16 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T02:04:16.673Z] 2024-01-19 02:04:16 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T02:04:17.240Z] 2024-01-19 02:04:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T02:04:18.168Z] 2024-01-19 02:04:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] validate if mysql-ibm app's volumes are setup [2024-01-19T02:04:18.168Z] 2024-01-19 02:04:17 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:04:18.168Z] 2024-01-19 02:04:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:04:18.169Z] 2024-01-19 02:04:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T02:04:18.169Z] 2024-01-19 02:04:17 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T02:04:18.169Z] INFO[2024-01-19 02:04:17] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T02:04:18.169Z] INFO[2024-01-19 02:04:17] Actual:true, Expected: true [2024-01-19T02:04:18.169Z] 2024-01-19 02:04:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T02:04:18.169Z] 2024-01-19 02:04:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T02:04:18.169Z] 2024-01-19 02:04:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T02:04:18.169Z] 2024-01-19 02:04:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T02:04:18.169Z] 2024-01-19 02:04:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Create cluster [source-cluster] in org [default] [2024-01-19T02:04:18.169Z] 2024-01-19 02:04:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T02:04:18.169Z] 2024-01-19 02:04:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T02:04:18.169Z] 2024-01-19 02:04:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Create cluster [destination-cluster] in org [default] [2024-01-19T02:04:18.169Z] 2024-01-19 02:04:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T02:04:18.169Z] 2024-01-19 02:04:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T02:04:18.169Z] 2024-01-19 02:04:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-pmzso] for IBM [2024-01-19T02:04:18.169Z] 2024-01-19 02:04:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-pmzso] for org [default] with [ibm] as provider [2024-01-19T02:04:18.169Z] 2024-01-19 02:04:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T02:04:18.169Z] 2024-01-19 02:04:17 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T02:04:18.169Z] 2024-01-19 02:04:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-pmzso] is present [2024-01-19T02:04:18.169Z] 2024-01-19 02:04:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Create cluster [source-cluster] in org [default] [2024-01-19T02:04:18.731Z] 2024-01-19 02:04:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-gzpiz] for IBM [2024-01-19T02:04:18.731Z] 2024-01-19 02:04:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-gzpiz] for org [default] with [ibm] as provider [2024-01-19T02:04:18.731Z] 2024-01-19 02:04:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T02:04:18.731Z] 2024-01-19 02:04:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T02:04:18.987Z] 2024-01-19 02:04:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-gzpiz] is present [2024-01-19T02:04:18.987Z] 2024-01-19 02:04:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Create cluster [destination-cluster] in org [default] [2024-01-19T02:04:18.987Z] INFO[2024-01-19 02:04:18] Verifying : Description : Creating source and destination cluster [2024-01-19T02:04:18.987Z] INFO[2024-01-19 02:04:18] Actual:true, Expected: true [2024-01-19T02:04:18.987Z] INFO[2024-01-19 02:04:18] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T02:04:18.987Z] INFO[2024-01-19 02:04:18] Actual:Online, Expected: Online [2024-01-19T02:04:18.987Z] INFO[2024-01-19 02:04:18] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T02:04:18.987Z] INFO[2024-01-19 02:04:18] Actual:true, Expected: true [2024-01-19T02:04:18.987Z] 2024-01-19 02:04:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Creating backup location and cloud setting [2024-01-19T02:04:18.987Z] 2024-01-19 02:04:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [cred-aws-zlihj] for org [default] with [aws] as provider [2024-01-19T02:04:18.987Z] 2024-01-19 02:04:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T02:04:18.987Z] 2024-01-19 02:04:18 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cred-aws-zlihj] is present [2024-01-19T02:04:18.987Z] INFO[2024-01-19 02:04:18] Verifying : Description : Verifying creation of cloud credential named [cred-aws-zlihj] for org [default] with [aws] as provider [2024-01-19T02:04:18.987Z] INFO[2024-01-19 02:04:18] Actual:true, Expected: true [2024-01-19T02:04:21.549Z] INFO[2024-01-19 02:04:21] Verifying : Description : Creating backup location autogenerated-backup-location-exgca [2024-01-19T02:04:21.549Z] INFO[2024-01-19 02:04:21] Actual:true, Expected: true [2024-01-19T02:04:21.549Z] 2024-01-19 02:04:21 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Taking backup of application for different combination of restores [2024-01-19T02:04:21.549Z] 2024-01-19 02:04:21 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T02:04:21.549Z] 2024-01-19 02:04:21 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T02:04:21.549Z] 2024-01-19 02:04:21 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T02:04:21.549Z] 2024-01-19 02:04:21 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T02:04:21.549Z] 2024-01-19 02:04:21 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T02:04:21.549Z] 2024-01-19 02:04:21 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Volume snapshot classes found in the cluster - [2024-01-19T02:04:21.549Z] 2024-01-19 02:04:21 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] ibmc-vpcblock-snapshot [2024-01-19T02:04:21.549Z] 2024-01-19 02:04:21 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T02:04:21.549Z] 2024-01-19 02:04:21 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Volume snapshot classes found in the cluster - [2024-01-19T02:04:21.549Z] 2024-01-19 02:04:21 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] ibmc-vpcblock-snapshot [2024-01-19T02:04:21.549Z] 2024-01-19 02:04:21 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T02:04:21.549Z] 2024-01-19 02:04:21 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Volume snapshot classes found in the cluster - [2024-01-19T02:04:21.549Z] 2024-01-19 02:04:21 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] ibmc-vpcblock-snapshot [2024-01-19T02:04:21.549Z] 2024-01-19 02:04:21 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T02:04:21.550Z] 2024-01-19 02:04:21 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Volume snapshot classes found in the cluster - [2024-01-19T02:04:21.550Z] 2024-01-19 02:04:21 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] ibmc-vpcblock-snapshot [2024-01-19T02:04:21.550Z] 2024-01-19 02:04:21 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T02:04:21.805Z] 2024-01-19 02:04:21 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Volume snapshot classes found in the cluster - [2024-01-19T02:04:21.805Z] 2024-01-19 02:04:21 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] ibmc-vpcblock-snapshot [2024-01-19T02:04:21.805Z] 2024-01-19 02:04:21 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T02:04:22.061Z] 2024/01/19 02:04:21 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-okdxe] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T02:04:22.316Z] 2024/01/19 02:04:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-elsrs] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T02:04:22.317Z] 2024/01/19 02:04:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ozbos] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T02:04:22.317Z] 2024/01/19 02:04:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-owsih] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T02:04:22.317Z] 2024/01/19 02:04:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-quypf] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T02:04:48.812Z] INFO[2024-01-19 02:04:45] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T02:04:48.812Z] INFO[2024-01-19 02:04:45] sched stats: tick interval: 37845 < 1.2s (100%) [2024-01-19T02:04:48.812Z] INFO[2024-01-19 02:04:45] sched stats: tick process durations: 37844 < 500ms (100%) [2024-01-19T02:04:52.101Z] 2024/01/19 02:04:51 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-okdxe] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:04:52.101Z] 2024/01/19 02:04:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-elsrs] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:04:52.101Z] 2024/01/19 02:04:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ozbos] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:04:52.357Z] 2024/01/19 02:04:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-owsih] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:04:52.357Z] 2024/01/19 02:04:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-quypf] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:05:24.379Z] 2024/01/19 02:05:21 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-okdxe] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:05:24.379Z] 2024/01/19 02:05:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-elsrs] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:05:24.379Z] 2024/01/19 02:05:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ozbos] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:05:24.379Z] 2024/01/19 02:05:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-owsih] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:05:24.379Z] 2024/01/19 02:05:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-quypf] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:05:56.411Z] 2024/01/19 02:05:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-okdxe] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:05:56.411Z] 2024/01/19 02:05:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-elsrs] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:05:56.411Z] 2024/01/19 02:05:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ozbos] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:05:56.411Z] 2024/01/19 02:05:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-owsih] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:05:56.411Z] 2024/01/19 02:05:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-quypf] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:06:22.926Z] 2024/01/19 02:06:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-okdxe] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:06:22.926Z] 2024/01/19 02:06:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-elsrs] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:06:22.926Z] 2024/01/19 02:06:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ozbos] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:06:22.926Z] 2024/01/19 02:06:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-owsih] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:06:22.926Z] 2024/01/19 02:06:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-quypf] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:06:54.957Z] 2024/01/19 02:06:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-okdxe] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:06:54.957Z] 2024/01/19 02:06:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ozbos] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:06:54.958Z] 2024/01/19 02:06:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-elsrs] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:06:54.958Z] 2024/01/19 02:06:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-owsih] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:06:54.958Z] 2024/01/19 02:06:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-quypf] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:07:26.983Z] 2024/01/19 02:07:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-okdxe] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:07:26.983Z] 2024/01/19 02:07:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ozbos] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:07:26.983Z] 2024/01/19 02:07:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-elsrs] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:07:26.983Z] 2024/01/19 02:07:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-owsih] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:07:26.983Z] 2024/01/19 02:07:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-quypf] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:07:53.509Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CreateBackup:#313] - Backup [tp-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-okdxe] created successfully [2024-01-19T02:07:53.509Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating backup [tp-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-okdxe] in org [default] [2024-01-19T02:07:53.509Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-okdxe] [2024-01-19T02:07:53.509Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-4-58051-01-19-02h02m17s] in backup [tp-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-okdxe] [2024-01-19T02:07:53.509Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-4-58051-01-19-02h02m17s] in backup [tp-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-okdxe] [2024-01-19T02:07:53.509Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-4-58051-01-19-02h02m17s] [2024-01-19T02:07:53.509Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CreateBackup:#313] - Backup [tp-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ozbos] created successfully [2024-01-19T02:07:53.509Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating backup [tp-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ozbos] in org [default] [2024-01-19T02:07:53.510Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ozbos] [2024-01-19T02:07:53.510Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CreateBackup:#313] - Backup [tp-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-elsrs] created successfully [2024-01-19T02:07:53.510Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating backup [tp-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-elsrs] in org [default] [2024-01-19T02:07:53.510Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-elsrs] [2024-01-19T02:07:53.510Z] 2024-01-19 02:07:52 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:07:53.510Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:07:53.510Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-4-58051-01-19-02h02m17s] are [[mysql-data]] [2024-01-19T02:07:53.510Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T02:07:53.510Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating CR cleanup [2024-01-19T02:07:53.510Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-58051-01-19-02h02m17s] in backup [tp-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-elsrs] [2024-01-19T02:07:53.510Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-58051-01-19-02h02m17s] in backup [tp-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-elsrs] [2024-01-19T02:07:53.510Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-58051-01-19-02h02m17s] [2024-01-19T02:07:53.510Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-58051-01-19-02h02m17s] in backup [tp-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ozbos] [2024-01-19T02:07:53.510Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-58051-01-19-02h02m17s] in backup [tp-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ozbos] [2024-01-19T02:07:53.510Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-2-58051-01-19-02h02m17s] [2024-01-19T02:07:53.510Z] 2024-01-19 02:07:52 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:07:53.510Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:07:53.510Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-58051-01-19-02h02m17s] are [[mysql-data]] [2024-01-19T02:07:53.510Z] 2024-01-19 02:07:52 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:07:53.511Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:07:53.511Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-2-58051-01-19-02h02m17s] are [[mysql-data]] [2024-01-19T02:07:53.511Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T02:07:53.511Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating CR cleanup [2024-01-19T02:07:53.511Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T02:07:53.511Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating CR cleanup [2024-01-19T02:07:53.511Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CreateBackup:#313] - Backup [tp-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-owsih] created successfully [2024-01-19T02:07:53.511Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating backup [tp-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-owsih] in org [default] [2024-01-19T02:07:53.511Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-owsih] [2024-01-19T02:07:53.511Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CreateBackup:#313] - Backup [tp-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-quypf] created successfully [2024-01-19T02:07:53.511Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating backup [tp-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-quypf] in org [default] [2024-01-19T02:07:53.511Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-quypf] [2024-01-19T02:07:53.511Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T02:07:53.511Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T02:07:53.511Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-3-58051-01-19-02h02m17s] in backup [tp-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-owsih] [2024-01-19T02:07:53.511Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-3-58051-01-19-02h02m17s] in backup [tp-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-owsih] [2024-01-19T02:07:53.511Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-3-58051-01-19-02h02m17s] [2024-01-19T02:07:53.511Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T02:07:53.511Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58051-01-19-02h02m17s] in backup [tp-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-quypf] [2024-01-19T02:07:53.511Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58051-01-19-02h02m17s] in backup [tp-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-quypf] [2024-01-19T02:07:53.512Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58051-01-19-02h02m17s] [2024-01-19T02:07:53.512Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] All CRs in [mysql-ibm-pxb-4-58051-01-19-02h02m17s] are [[]] [2024-01-19T02:07:53.512Z] INFO[2024-01-19 02:07:52] Verifying : Description : Creation and Validation of backup [tp-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-okdxe] of namespace (scheduled Context) [mysql-ibm-pxb-4-58051-01-19-02h02m17s] [2024-01-19T02:07:53.512Z] INFO[2024-01-19 02:07:52] Actual:true, Expected: true [2024-01-19T02:07:53.512Z] 2024-01-19 02:07:52 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:07:53.512Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:07:53.512Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-3-58051-01-19-02h02m17s] are [[mysql-data]] [2024-01-19T02:07:53.512Z] 2024-01-19 02:07:52 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:07:53.512Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:07:53.512Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-58051-01-19-02h02m17s] are [[mysql-data]] [2024-01-19T02:07:53.512Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] All CRs in [mysql-ibm-pxb-1-58051-01-19-02h02m17s] are [[]] [2024-01-19T02:07:53.512Z] INFO[2024-01-19 02:07:52] Verifying : Description : Creation and Validation of backup [tp-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-elsrs] of namespace (scheduled Context) [mysql-ibm-pxb-1-58051-01-19-02h02m17s] [2024-01-19T02:07:53.512Z] INFO[2024-01-19 02:07:52] Actual:true, Expected: true [2024-01-19T02:07:53.512Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] All CRs in [mysql-ibm-pxb-2-58051-01-19-02h02m17s] are [[]] [2024-01-19T02:07:53.512Z] INFO[2024-01-19 02:07:52] Verifying : Description : Creation and Validation of backup [tp-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ozbos] of namespace (scheduled Context) [mysql-ibm-pxb-2-58051-01-19-02h02m17s] [2024-01-19T02:07:53.512Z] INFO[2024-01-19 02:07:52] Actual:true, Expected: true [2024-01-19T02:07:53.512Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T02:07:53.512Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating CR cleanup [2024-01-19T02:07:53.512Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T02:07:53.512Z] 2024-01-19 02:07:52 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating CR cleanup [2024-01-19T02:07:53.512Z] 2024-01-19 02:07:53 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T02:07:53.512Z] 2024-01-19 02:07:53 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T02:07:53.512Z] 2024-01-19 02:07:53 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] All CRs in [mysql-ibm-pxb-3-58051-01-19-02h02m17s] are [[]] [2024-01-19T02:07:53.513Z] INFO[2024-01-19 02:07:53] Verifying : Description : Creation and Validation of backup [tp-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-owsih] of namespace (scheduled Context) [mysql-ibm-pxb-3-58051-01-19-02h02m17s] [2024-01-19T02:07:53.513Z] INFO[2024-01-19 02:07:53] Actual:true, Expected: true [2024-01-19T02:07:53.513Z] 2024-01-19 02:07:53 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] All CRs in [mysql-ibm-pxb-0-58051-01-19-02h02m17s] are [[]] [2024-01-19T02:07:53.513Z] INFO[2024-01-19 02:07:53] Verifying : Description : Creation and Validation of backup [tp-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-quypf] of namespace (scheduled Context) [mysql-ibm-pxb-0-58051-01-19-02h02m17s] [2024-01-19T02:07:53.513Z] INFO[2024-01-19 02:07:53] Actual:true, Expected: true [2024-01-19T02:07:53.513Z] 2024-01-19 02:07:53 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Issuing multiple in-place restore at the same time [2024-01-19T02:07:55.411Z] 2024/01/19 02:07:55 DoRetryWithTimeout - Error: {restore status for [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-azjsc] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T02:07:55.412Z] 2024/01/19 02:07:55 DoRetryWithTimeout - Error: {restore status for [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ikbki] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T02:07:55.412Z] 2024/01/19 02:07:55 DoRetryWithTimeout - Error: {restore status for [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-edflu] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T02:07:55.412Z] 2024/01/19 02:07:55 DoRetryWithTimeout - Error: {restore status for [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-akbki] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T02:07:55.412Z] 2024/01/19 02:07:55 DoRetryWithTimeout - Error: {restore status for [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-aulqy] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T02:08:27.456Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CreateRestore:#936] - Restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-azjsc] created successfully [2024-01-19T02:08:27.456Z] INFO[2024-01-19 02:08:25] Verifying : Description : Restoring backup tp-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-okdxe into namespce mysql-ibm-pxb-4-58051-01-19-02h02m17s with replacing existing resources [2024-01-19T02:08:27.456Z] INFO[2024-01-19 02:08:25] Actual:true, Expected: true [2024-01-19T02:08:27.456Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CreateRestore:#936] - Restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ikbki] created successfully [2024-01-19T02:08:27.456Z] INFO[2024-01-19 02:08:25] Verifying : Description : Restoring backup tp-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ozbos into namespce mysql-ibm-pxb-2-58051-01-19-02h02m17s with replacing existing resources [2024-01-19T02:08:27.456Z] INFO[2024-01-19 02:08:25] Actual:true, Expected: true [2024-01-19T02:08:27.456Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CreateRestore:#936] - Restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-edflu] created successfully [2024-01-19T02:08:27.456Z] INFO[2024-01-19 02:08:25] Verifying : Description : Restoring backup tp-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-owsih into namespce mysql-ibm-pxb-3-58051-01-19-02h02m17s with replacing existing resources [2024-01-19T02:08:27.456Z] INFO[2024-01-19 02:08:25] Actual:true, Expected: true [2024-01-19T02:08:27.456Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CreateRestore:#936] - Restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-akbki] created successfully [2024-01-19T02:08:27.456Z] INFO[2024-01-19 02:08:25] Verifying : Description : Restoring backup tp-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-quypf into namespce mysql-ibm-pxb-0-58051-01-19-02h02m17s with replacing existing resources [2024-01-19T02:08:27.456Z] INFO[2024-01-19 02:08:25] Actual:true, Expected: true [2024-01-19T02:08:27.456Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CreateRestore:#936] - Restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-aulqy] created successfully [2024-01-19T02:08:27.456Z] INFO[2024-01-19 02:08:25] Verifying : Description : Restoring backup tp-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-elsrs into namespce mysql-ibm-pxb-1-58051-01-19-02h02m17s with replacing existing resources [2024-01-19T02:08:27.456Z] INFO[2024-01-19 02:08:25] Actual:true, Expected: true [2024-01-19T02:08:27.456Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating multiple in-place restore at the same time [2024-01-19T02:08:27.456Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-azjsc] [2024-01-19T02:08:27.456Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-4-58051-01-19-02h02m17s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-19T02:08:27.457Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-4-58051-01-19-02h02m17s] [2024-01-19T02:08:27.457Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T02:08:27.457Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T02:08:27.457Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-azjsc] in org [default] [2024-01-19T02:08:27.457Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-azjsc] [2024-01-19T02:08:27.457Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-aulqy] [2024-01-19T02:08:27.457Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-1-58051-01-19-02h02m17s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-19T02:08:27.457Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-1-58051-01-19-02h02m17s] [2024-01-19T02:08:27.457Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ikbki] [2024-01-19T02:08:27.457Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-2-58051-01-19-02h02m17s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-19T02:08:27.457Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T02:08:27.457Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T02:08:27.457Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-aulqy] in org [default] [2024-01-19T02:08:27.457Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-aulqy] [2024-01-19T02:08:27.457Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-2-58051-01-19-02h02m17s] [2024-01-19T02:08:27.457Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T02:08:27.457Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T02:08:27.457Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ikbki] in org [default] [2024-01-19T02:08:27.457Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ikbki] [2024-01-19T02:08:27.457Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-akbki] [2024-01-19T02:08:27.458Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-58051-01-19-02h02m17s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-19T02:08:27.458Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-58051-01-19-02h02m17s] [2024-01-19T02:08:27.458Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T02:08:27.458Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T02:08:27.458Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-akbki] in org [default] [2024-01-19T02:08:27.458Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-akbki] [2024-01-19T02:08:27.458Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-edflu] [2024-01-19T02:08:27.458Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-3-58051-01-19-02h02m17s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-19T02:08:27.458Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-3-58051-01-19-02h02m17s] [2024-01-19T02:08:27.458Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T02:08:27.458Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T02:08:27.458Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-edflu] in org [default] [2024-01-19T02:08:27.458Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-edflu] [2024-01-19T02:08:27.458Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2110] - Checking status of restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-aulqy] [2024-01-19T02:08:27.458Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-1-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-aulqy] [2024-01-19T02:08:27.458Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2110] - Checking status of restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ikbki] [2024-01-19T02:08:27.458Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-2-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ikbki] [2024-01-19T02:08:27.459Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-2-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ikbki], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-2-58051-01-19-02h02m17s] [2024-01-19T02:08:27.459Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2110] - Checking status of restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-akbki] [2024-01-19T02:08:27.459Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-akbki] [2024-01-19T02:08:27.459Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2110] - Checking status of restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-edflu] [2024-01-19T02:08:27.459Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-3-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-edflu] [2024-01-19T02:08:27.459Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-3-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-edflu], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-3-58051-01-19-02h02m17s] [2024-01-19T02:08:27.459Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-1-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-aulqy], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-1-58051-01-19-02h02m17s] [2024-01-19T02:08:27.459Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2110] - Checking status of restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-azjsc] [2024-01-19T02:08:27.459Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-akbki], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-58051-01-19-02h02m17s] [2024-01-19T02:08:27.459Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-4-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-azjsc] [2024-01-19T02:08:27.459Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-4-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-azjsc], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-4-58051-01-19-02h02m17s] [2024-01-19T02:08:27.459Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-3-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-edflu] was also present on the cluster/namespace [mysql-ibm-pxb-3-58051-01-19-02h02m17s] [2024-01-19T02:08:27.460Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-3-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-edflu], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-3-58051-01-19-02h02m17s] [2024-01-19T02:08:27.460Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-1-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-aulqy] was also present on the cluster/namespace [mysql-ibm-pxb-1-58051-01-19-02h02m17s] [2024-01-19T02:08:27.460Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-2-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ikbki] was also present on the cluster/namespace [mysql-ibm-pxb-2-58051-01-19-02h02m17s] [2024-01-19T02:08:27.460Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-2-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ikbki], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-2-58051-01-19-02h02m17s] [2024-01-19T02:08:27.460Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-akbki] was also present on the cluster/namespace [mysql-ibm-pxb-0-58051-01-19-02h02m17s] [2024-01-19T02:08:27.460Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-akbki], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-58051-01-19-02h02m17s] [2024-01-19T02:08:27.460Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-1-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-aulqy], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-1-58051-01-19-02h02m17s] [2024-01-19T02:08:27.460Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-4-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-azjsc] was also present on the cluster/namespace [mysql-ibm-pxb-4-58051-01-19-02h02m17s] [2024-01-19T02:08:27.460Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-4-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-azjsc], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-4-58051-01-19-02h02m17s] [2024-01-19T02:08:27.460Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-3-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-edflu] was also present on the cluster/namespace [mysql-ibm-pxb-3-58051-01-19-02h02m17s] [2024-01-19T02:08:27.461Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-3-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-edflu], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-3-58051-01-19-02h02m17s] [2024-01-19T02:08:27.461Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-3-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-edflu] was also present on the cluster/namespace [mysql-ibm-pxb-3-58051-01-19-02h02m17s] [2024-01-19T02:08:27.461Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-3-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-edflu], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-3-58051-01-19-02h02m17s] [2024-01-19T02:08:27.461Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-3-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-edflu] was also present on the cluster/namespace [mysql-ibm-pxb-3-58051-01-19-02h02m17s] [2024-01-19T02:08:27.461Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-3-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-edflu] [2024-01-19T02:08:27.461Z] 2024-01-19 02:08:25 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-3-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-edflu] [2024-01-19T02:08:27.461Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-2-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ikbki] was also present on the cluster/namespace [mysql-ibm-pxb-2-58051-01-19-02h02m17s] [2024-01-19T02:08:27.461Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-2-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ikbki], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-2-58051-01-19-02h02m17s] [2024-01-19T02:08:27.461Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-2-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ikbki] was also present on the cluster/namespace [mysql-ibm-pxb-2-58051-01-19-02h02m17s] [2024-01-19T02:08:27.461Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-2-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ikbki], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-2-58051-01-19-02h02m17s] [2024-01-19T02:08:27.462Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-2-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ikbki] was also present on the cluster/namespace [mysql-ibm-pxb-2-58051-01-19-02h02m17s] [2024-01-19T02:08:27.462Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-2-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ikbki] [2024-01-19T02:08:27.462Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-2-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ikbki] [2024-01-19T02:08:27.462Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-akbki] was also present on the cluster/namespace [mysql-ibm-pxb-0-58051-01-19-02h02m17s] [2024-01-19T02:08:27.462Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-akbki], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-58051-01-19-02h02m17s] [2024-01-19T02:08:27.462Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-akbki] was also present on the cluster/namespace [mysql-ibm-pxb-0-58051-01-19-02h02m17s] [2024-01-19T02:08:27.462Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-akbki], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-58051-01-19-02h02m17s] [2024-01-19T02:08:27.462Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-akbki] was also present on the cluster/namespace [mysql-ibm-pxb-0-58051-01-19-02h02m17s] [2024-01-19T02:08:27.462Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-akbki] [2024-01-19T02:08:27.463Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-akbki] [2024-01-19T02:08:27.463Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-1-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-aulqy] was also present on the cluster/namespace [mysql-ibm-pxb-1-58051-01-19-02h02m17s] [2024-01-19T02:08:27.463Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-1-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-aulqy], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-1-58051-01-19-02h02m17s] [2024-01-19T02:08:27.463Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-1-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-aulqy] was also present on the cluster/namespace [mysql-ibm-pxb-1-58051-01-19-02h02m17s] [2024-01-19T02:08:27.463Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-1-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-aulqy], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-1-58051-01-19-02h02m17s] [2024-01-19T02:08:27.463Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-1-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-aulqy] was also present on the cluster/namespace [mysql-ibm-pxb-1-58051-01-19-02h02m17s] [2024-01-19T02:08:27.463Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-1-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-aulqy] [2024-01-19T02:08:27.463Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-1-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-aulqy] [2024-01-19T02:08:27.463Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-4-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-azjsc] was also present on the cluster/namespace [mysql-ibm-pxb-4-58051-01-19-02h02m17s] [2024-01-19T02:08:27.463Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-4-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-azjsc], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-4-58051-01-19-02h02m17s] [2024-01-19T02:08:27.463Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-4-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-azjsc] was also present on the cluster/namespace [mysql-ibm-pxb-4-58051-01-19-02h02m17s] [2024-01-19T02:08:27.464Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-4-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-azjsc], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-4-58051-01-19-02h02m17s] [2024-01-19T02:08:27.464Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-4-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-azjsc] was also present on the cluster/namespace [mysql-ibm-pxb-4-58051-01-19-02h02m17s] [2024-01-19T02:08:27.464Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-4-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-azjsc] [2024-01-19T02:08:27.464Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-4-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-azjsc] [2024-01-19T02:08:27.464Z] 2024-01-19 02:08:26 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:08:27.464Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:08:27.464Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-3-58051-01-19-02h02m17s] are [[mysql-data]] [2024-01-19T02:08:27.464Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2243] - in restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-edflu], the status of the restored volume [pvc-8392530a-d166-4d47-a3b0-d43928fa9c8b] was not Success. It was [Retained] with reason [Skipped from volume restore as policy is set to Retain and pvc already exists] [2024-01-19T02:08:27.464Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validate applications in restored namespace [mysql-ibm-pxb-3-58051-01-19-02h02m17s] due to restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-edflu] [2024-01-19T02:08:27.464Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating mysql-ibm app [2024-01-19T02:08:27.464Z] 2024-01-19 02:08:26 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] wait for mysql-ibm app to start running [2024-01-19T02:08:27.464Z] 2024-01-19 02:08:27 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:08:27.464Z] 2024-01-19 02:08:27 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:08:27.464Z] 2024-01-19 02:08:27 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-2-58051-01-19-02h02m17s] are [[mysql-data]] [2024-01-19T02:08:27.464Z] 2024-01-19 02:08:27 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2243] - in restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ikbki], the status of the restored volume [pvc-54d25d3d-581b-45da-b799-b682a5ff95a1] was not Success. It was [Retained] with reason [Skipped from volume restore as policy is set to Retain and pvc already exists] [2024-01-19T02:08:27.465Z] 2024-01-19 02:08:27 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validate applications in restored namespace [mysql-ibm-pxb-2-58051-01-19-02h02m17s] due to restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ikbki] [2024-01-19T02:08:27.465Z] 2024-01-19 02:08:27 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating mysql-ibm app [2024-01-19T02:08:27.465Z] 2024-01-19 02:08:27 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] wait for mysql-ibm app to start running [2024-01-19T02:08:27.465Z] 2024-01-19 02:08:27 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:08:27.465Z] 2024-01-19 02:08:27 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:08:27.465Z] 2024-01-19 02:08:27 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-58051-01-19-02h02m17s] are [[mysql-data]] [2024-01-19T02:08:27.465Z] 2024-01-19 02:08:27 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2243] - in restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-akbki], the status of the restored volume [pvc-e05c8f60-e1c6-4309-914d-535f4f735636] was not Success. It was [Retained] with reason [Skipped from volume restore as policy is set to Retain and pvc already exists] [2024-01-19T02:08:27.465Z] 2024-01-19 02:08:27 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validate applications in restored namespace [mysql-ibm-pxb-0-58051-01-19-02h02m17s] due to restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-akbki] [2024-01-19T02:08:27.465Z] 2024-01-19 02:08:27 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating mysql-ibm app [2024-01-19T02:08:27.465Z] 2024-01-19 02:08:27 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] wait for mysql-ibm app to start running [2024-01-19T02:08:27.722Z] 2024-01-19 02:08:27 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:08:27.722Z] 2024-01-19 02:08:27 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:08:27.722Z] 2024-01-19 02:08:27 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-58051-01-19-02h02m17s] are [[mysql-data]] [2024-01-19T02:08:27.722Z] 2024-01-19 02:08:27 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2243] - in restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-aulqy], the status of the restored volume [pvc-b354e5fc-cf44-4fdc-998e-c48e6f53ac75] was not Success. It was [Retained] with reason [Skipped from volume restore as policy is set to Retain and pvc already exists] [2024-01-19T02:08:27.722Z] 2024-01-19 02:08:27 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validate applications in restored namespace [mysql-ibm-pxb-1-58051-01-19-02h02m17s] due to restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-aulqy] [2024-01-19T02:08:27.722Z] 2024-01-19 02:08:27 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating mysql-ibm app [2024-01-19T02:08:27.722Z] 2024-01-19 02:08:27 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] wait for mysql-ibm app to start running [2024-01-19T02:08:27.722Z] 2024-01-19 02:08:27 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:08:27.722Z] 2024-01-19 02:08:27 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:08:27.722Z] 2024-01-19 02:08:27 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-4-58051-01-19-02h02m17s] are [[mysql-data]] [2024-01-19T02:08:27.722Z] 2024-01-19 02:08:27 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2243] - in restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-azjsc], the status of the restored volume [pvc-d8cc3133-1781-4ccc-945c-d44343d98501] was not Success. It was [Retained] with reason [Skipped from volume restore as policy is set to Retain and pvc already exists] [2024-01-19T02:08:27.722Z] 2024-01-19 02:08:27 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validate applications in restored namespace [mysql-ibm-pxb-4-58051-01-19-02h02m17s] due to restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-azjsc] [2024-01-19T02:08:27.722Z] 2024-01-19 02:08:27 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating mysql-ibm app [2024-01-19T02:08:27.722Z] 2024-01-19 02:08:27 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] wait for mysql-ibm app to start running [2024-01-19T02:08:27.979Z] 2024-01-19 02:08:27 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T02:08:28.235Z] 2024-01-19 02:08:27 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T02:08:28.235Z] 2024-01-19 02:08:28 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T02:08:28.235Z] 2024-01-19 02:08:28 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T02:08:28.491Z] 2024-01-19 02:08:28 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T02:08:28.491Z] 2024-01-19 02:08:28 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T02:08:28.492Z] 2024-01-19 02:08:28 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T02:08:28.747Z] 2024-01-19 02:08:28 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T02:08:29.680Z] 2024-01-19 02:08:29 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating CR cleanup [2024-01-19T02:08:29.681Z] 2024-01-19 02:08:29 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T02:08:29.681Z] 2024-01-19 02:08:29 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] All CRs in [mysql-ibm-pxb-3-58051-01-19-02h02m17s] are [[]] [2024-01-19T02:08:30.242Z] 2024-01-19 02:08:29 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T02:08:30.806Z] 2024-01-19 02:08:30 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T02:08:31.065Z] 2024-01-19 02:08:30 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T02:08:31.323Z] 2024-01-19 02:08:31 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T02:08:33.218Z] 2024-01-19 02:08:32 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T02:08:33.218Z] 2024-01-19 02:08:33 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating CR cleanup [2024-01-19T02:08:33.478Z] 2024-01-19 02:08:33 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T02:08:33.479Z] 2024-01-19 02:08:33 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T02:08:33.479Z] 2024-01-19 02:08:33 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] All CRs in [mysql-ibm-pxb-2-58051-01-19-02h02m17s] are [[]] [2024-01-19T02:08:33.735Z] 2024-01-19 02:08:33 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T02:08:35.623Z] 2024-01-19 02:08:35 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating CR cleanup [2024-01-19T02:08:35.879Z] 2024-01-19 02:08:35 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T02:08:35.879Z] 2024-01-19 02:08:35 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating CR cleanup [2024-01-19T02:08:35.879Z] 2024-01-19 02:08:35 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] All CRs in [mysql-ibm-pxb-0-58051-01-19-02h02m17s] are [[]] [2024-01-19T02:08:36.135Z] 2024-01-19 02:08:35 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T02:08:36.135Z] 2024-01-19 02:08:35 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating CR cleanup [2024-01-19T02:08:36.135Z] 2024-01-19 02:08:36 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] All CRs in [mysql-ibm-pxb-1-58051-01-19-02h02m17s] are [[]] [2024-01-19T02:08:36.135Z] 2024-01-19 02:08:36 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T02:08:36.391Z] 2024-01-19 02:08:36 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] All CRs in [mysql-ibm-pxb-4-58051-01-19-02h02m17s] are [[]] [2024-01-19T02:08:36.391Z] INFO[2024-01-19 02:08:36] Verifying : Description : Errors generated while validating in-place restore - [2024-01-19T02:08:36.391Z]  [2024-01-19T02:08:36.391Z] INFO[2024-01-19 02:08:36] Actual:0, Expected: 0 [2024-01-19T02:08:36.392Z] 2024-01-19 02:08:36 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Issuing multiple in-place restore at the same time with replace existing resources [2024-01-19T02:08:36.953Z] 2024-01-19 02:08:36 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CreateRestoreWithReplacePolicy:#1009] - Getting the UID of the backup tp-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-okdxe needed to be restored [2024-01-19T02:08:36.953Z] 2024-01-19 02:08:36 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CreateRestoreWithReplacePolicy:#1009] - Getting the UID of the backup tp-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ozbos needed to be restored [2024-01-19T02:08:36.953Z] 2024-01-19 02:08:36 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CreateRestoreWithReplacePolicy:#1009] - Getting the UID of the backup tp-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-owsih needed to be restored [2024-01-19T02:08:36.953Z] 2024-01-19 02:08:36 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CreateRestoreWithReplacePolicy:#1009] - Getting the UID of the backup tp-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-quypf needed to be restored [2024-01-19T02:08:36.953Z] 2024-01-19 02:08:36 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CreateRestoreWithReplacePolicy:#1009] - Getting the UID of the backup tp-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-elsrs needed to be restored [2024-01-19T02:08:37.513Z] 2024/01/19 02:08:37 DoRetryWithTimeout - Error: {restore status for [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-demhz] expected was [Success] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T02:08:37.514Z] 2024/01/19 02:08:37 DoRetryWithTimeout - Error: {restore status for [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-gwqmc] expected was [Success] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T02:08:37.769Z] 2024/01/19 02:08:37 DoRetryWithTimeout - Error: {restore status for [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-qdqei] expected was [Success] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T02:08:37.769Z] 2024/01/19 02:08:37 DoRetryWithTimeout - Error: {restore status for [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-hlbln] expected was [Success] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T02:08:37.769Z] 2024/01/19 02:08:37 DoRetryWithTimeout - Error: {restore status for [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-cfhmj] expected was [Success] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T02:09:10.899Z] 2024/01/19 02:09:07 DoRetryWithTimeout - Error: {restore status for [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-demhz] expected was [Success] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:09:10.899Z] 2024/01/19 02:09:07 DoRetryWithTimeout - Error: {restore status for [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-gwqmc] expected was [Success] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:09:10.900Z] 2024/01/19 02:09:07 DoRetryWithTimeout - Error: {restore status for [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-qdqei] expected was [Success] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:09:10.900Z] 2024/01/19 02:09:07 DoRetryWithTimeout - Error: {restore status for [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-hlbln] expected was [Success] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:09:10.900Z] 2024/01/19 02:09:07 DoRetryWithTimeout - Error: {restore status for [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-cfhmj] expected was [Success] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:09:37.439Z] 2024-01-19 02:09:37 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CreateRestoreWithReplacePolicy:#1045] - Restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-demhz] created successfully [2024-01-19T02:09:37.439Z] INFO[2024-01-19 02:09:37] Verifying : Description : Restoring backup tp-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-quypf into namespce mysql-ibm-pxb-0-58051-01-19-02h02m17s with replacing existing resources [2024-01-19T02:09:37.439Z] INFO[2024-01-19 02:09:37] Actual:true, Expected: true [2024-01-19T02:09:37.703Z] 2024-01-19 02:09:37 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CreateRestoreWithReplacePolicy:#1045] - Restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-gwqmc] created successfully [2024-01-19T02:09:37.703Z] INFO[2024-01-19 02:09:37] Verifying : Description : Restoring backup tp-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-owsih into namespce mysql-ibm-pxb-3-58051-01-19-02h02m17s with replacing existing resources [2024-01-19T02:09:37.703Z] INFO[2024-01-19 02:09:37] Actual:true, Expected: true [2024-01-19T02:09:37.703Z] 2024/01/19 02:09:37 DoRetryWithTimeout - Error: {restore status for [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-qdqei] expected was [Success] but got [InProgress] because of [Application resources restore is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:09:37.703Z] 2024/01/19 02:09:37 DoRetryWithTimeout - Error: {restore status for [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-hlbln] expected was [Success] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:09:37.703Z] 2024/01/19 02:09:37 DoRetryWithTimeout - Error: {restore status for [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-cfhmj] expected was [Success] but got [InProgress] because of [Application resources restore is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:10:09.744Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CreateRestoreWithReplacePolicy:#1045] - Restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-qdqei] created successfully [2024-01-19T02:10:09.744Z] INFO[2024-01-19 02:10:07] Verifying : Description : Restoring backup tp-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ozbos into namespce mysql-ibm-pxb-2-58051-01-19-02h02m17s with replacing existing resources [2024-01-19T02:10:09.744Z] INFO[2024-01-19 02:10:07] Actual:true, Expected: true [2024-01-19T02:10:09.744Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CreateRestoreWithReplacePolicy:#1045] - Restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-hlbln] created successfully [2024-01-19T02:10:09.744Z] INFO[2024-01-19 02:10:07] Verifying : Description : Restoring backup tp-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-elsrs into namespce mysql-ibm-pxb-1-58051-01-19-02h02m17s with replacing existing resources [2024-01-19T02:10:09.744Z] INFO[2024-01-19 02:10:07] Actual:true, Expected: true [2024-01-19T02:10:09.744Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CreateRestoreWithReplacePolicy:#1045] - Restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-cfhmj] created successfully [2024-01-19T02:10:09.744Z] INFO[2024-01-19 02:10:07] Verifying : Description : Restoring backup tp-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-okdxe into namespce mysql-ibm-pxb-4-58051-01-19-02h02m17s with replacing existing resources [2024-01-19T02:10:09.744Z] INFO[2024-01-19 02:10:07] Actual:true, Expected: true [2024-01-19T02:10:09.744Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating multiple in-place restore with replace existing resources [2024-01-19T02:10:09.744Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating restore with replace policy [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-cfhmj] [2024-01-19T02:10:09.744Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-4-58051-01-19-02h02m17s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-19T02:10:09.744Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-4-58051-01-19-02h02m17s] [2024-01-19T02:10:09.744Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T02:10:09.745Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T02:10:09.745Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-cfhmj] in org [default] [2024-01-19T02:10:09.745Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-cfhmj] [2024-01-19T02:10:09.745Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating restore with replace policy [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-demhz] [2024-01-19T02:10:09.745Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-58051-01-19-02h02m17s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-19T02:10:09.745Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-58051-01-19-02h02m17s] [2024-01-19T02:10:09.745Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T02:10:09.745Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T02:10:09.745Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-demhz] in org [default] [2024-01-19T02:10:09.745Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-demhz] [2024-01-19T02:10:09.745Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating restore with replace policy [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-gwqmc] [2024-01-19T02:10:09.745Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-3-58051-01-19-02h02m17s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-19T02:10:09.745Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-3-58051-01-19-02h02m17s] [2024-01-19T02:10:09.745Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T02:10:09.745Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T02:10:09.745Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-gwqmc] in org [default] [2024-01-19T02:10:09.745Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-gwqmc] [2024-01-19T02:10:09.745Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating restore with replace policy [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-qdqei] [2024-01-19T02:10:09.745Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-2-58051-01-19-02h02m17s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-19T02:10:09.746Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-2-58051-01-19-02h02m17s] [2024-01-19T02:10:09.746Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T02:10:09.746Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T02:10:09.746Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-qdqei] in org [default] [2024-01-19T02:10:09.746Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-qdqei] [2024-01-19T02:10:09.746Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating restore with replace policy [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-hlbln] [2024-01-19T02:10:09.746Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-1-58051-01-19-02h02m17s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-19T02:10:09.746Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-1-58051-01-19-02h02m17s] [2024-01-19T02:10:09.746Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T02:10:09.746Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T02:10:09.746Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-hlbln] in org [default] [2024-01-19T02:10:09.746Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-hlbln] [2024-01-19T02:10:09.746Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2110] - Checking status of restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-cfhmj] [2024-01-19T02:10:09.746Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-4-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-cfhmj] [2024-01-19T02:10:09.746Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-4-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-cfhmj], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-4-58051-01-19-02h02m17s] [2024-01-19T02:10:09.746Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2110] - Checking status of restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-demhz] [2024-01-19T02:10:09.746Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-demhz] [2024-01-19T02:10:09.747Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-demhz], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-58051-01-19-02h02m17s] [2024-01-19T02:10:09.747Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2110] - Checking status of restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-gwqmc] [2024-01-19T02:10:09.747Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-3-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-gwqmc] [2024-01-19T02:10:09.747Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-3-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-gwqmc], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-3-58051-01-19-02h02m17s] [2024-01-19T02:10:09.747Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2110] - Checking status of restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-qdqei] [2024-01-19T02:10:09.747Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-2-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-qdqei] [2024-01-19T02:10:09.747Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-2-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-qdqei], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-2-58051-01-19-02h02m17s] [2024-01-19T02:10:09.747Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2110] - Checking status of restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-hlbln] [2024-01-19T02:10:09.747Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-1-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-hlbln] [2024-01-19T02:10:09.747Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-1-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-hlbln], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-1-58051-01-19-02h02m17s] [2024-01-19T02:10:09.747Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-4-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-cfhmj] was also present on the cluster/namespace [mysql-ibm-pxb-4-58051-01-19-02h02m17s] [2024-01-19T02:10:09.747Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-4-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-cfhmj], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-4-58051-01-19-02h02m17s] [2024-01-19T02:10:09.748Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-demhz] was also present on the cluster/namespace [mysql-ibm-pxb-0-58051-01-19-02h02m17s] [2024-01-19T02:10:09.748Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-demhz], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-58051-01-19-02h02m17s] [2024-01-19T02:10:09.748Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-3-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-gwqmc] was also present on the cluster/namespace [mysql-ibm-pxb-3-58051-01-19-02h02m17s] [2024-01-19T02:10:09.748Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-3-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-gwqmc], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-3-58051-01-19-02h02m17s] [2024-01-19T02:10:09.748Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-2-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-qdqei] was also present on the cluster/namespace [mysql-ibm-pxb-2-58051-01-19-02h02m17s] [2024-01-19T02:10:09.748Z] 2024-01-19 02:10:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-2-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-qdqei], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-2-58051-01-19-02h02m17s] [2024-01-19T02:10:09.748Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-1-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-hlbln] was also present on the cluster/namespace [mysql-ibm-pxb-1-58051-01-19-02h02m17s] [2024-01-19T02:10:09.749Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-1-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-hlbln], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-1-58051-01-19-02h02m17s] [2024-01-19T02:10:09.749Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-4-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-cfhmj] was also present on the cluster/namespace [mysql-ibm-pxb-4-58051-01-19-02h02m17s] [2024-01-19T02:10:09.749Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-4-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-cfhmj], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-4-58051-01-19-02h02m17s] [2024-01-19T02:10:09.749Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-4-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-cfhmj] was also present on the cluster/namespace [mysql-ibm-pxb-4-58051-01-19-02h02m17s] [2024-01-19T02:10:09.749Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-4-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-cfhmj], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-4-58051-01-19-02h02m17s] [2024-01-19T02:10:09.749Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-4-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-cfhmj] was also present on the cluster/namespace [mysql-ibm-pxb-4-58051-01-19-02h02m17s] [2024-01-19T02:10:09.749Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-4-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-cfhmj] [2024-01-19T02:10:09.750Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-4-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-cfhmj] [2024-01-19T02:10:09.750Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-demhz] was also present on the cluster/namespace [mysql-ibm-pxb-0-58051-01-19-02h02m17s] [2024-01-19T02:10:09.750Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-demhz], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-58051-01-19-02h02m17s] [2024-01-19T02:10:09.750Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-demhz] was also present on the cluster/namespace [mysql-ibm-pxb-0-58051-01-19-02h02m17s] [2024-01-19T02:10:09.750Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-demhz], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-58051-01-19-02h02m17s] [2024-01-19T02:10:09.750Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-demhz] was also present on the cluster/namespace [mysql-ibm-pxb-0-58051-01-19-02h02m17s] [2024-01-19T02:10:09.750Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-demhz] [2024-01-19T02:10:09.751Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-demhz] [2024-01-19T02:10:09.751Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-3-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-gwqmc] was also present on the cluster/namespace [mysql-ibm-pxb-3-58051-01-19-02h02m17s] [2024-01-19T02:10:09.751Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-3-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-gwqmc], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-3-58051-01-19-02h02m17s] [2024-01-19T02:10:09.751Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-3-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-gwqmc] was also present on the cluster/namespace [mysql-ibm-pxb-3-58051-01-19-02h02m17s] [2024-01-19T02:10:09.751Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-3-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-gwqmc], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-3-58051-01-19-02h02m17s] [2024-01-19T02:10:09.751Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-3-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-gwqmc] was also present on the cluster/namespace [mysql-ibm-pxb-3-58051-01-19-02h02m17s] [2024-01-19T02:10:09.751Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-3-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-gwqmc] [2024-01-19T02:10:09.752Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-3-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-gwqmc] [2024-01-19T02:10:09.752Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-2-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-qdqei] was also present on the cluster/namespace [mysql-ibm-pxb-2-58051-01-19-02h02m17s] [2024-01-19T02:10:09.752Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-2-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-qdqei], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-2-58051-01-19-02h02m17s] [2024-01-19T02:10:09.752Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-2-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-qdqei] was also present on the cluster/namespace [mysql-ibm-pxb-2-58051-01-19-02h02m17s] [2024-01-19T02:10:09.752Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-2-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-qdqei], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-2-58051-01-19-02h02m17s] [2024-01-19T02:10:09.752Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-2-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-qdqei] was also present on the cluster/namespace [mysql-ibm-pxb-2-58051-01-19-02h02m17s] [2024-01-19T02:10:09.752Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-2-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-qdqei] [2024-01-19T02:10:09.752Z] 2024-01-19 02:10:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-2-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-qdqei] [2024-01-19T02:10:09.753Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-1-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-hlbln] was also present on the cluster/namespace [mysql-ibm-pxb-1-58051-01-19-02h02m17s] [2024-01-19T02:10:09.753Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-1-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-hlbln], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-1-58051-01-19-02h02m17s] [2024-01-19T02:10:09.753Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-1-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-hlbln] was also present on the cluster/namespace [mysql-ibm-pxb-1-58051-01-19-02h02m17s] [2024-01-19T02:10:09.753Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-1-58051-01-19-02h02m17s]) was found in restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-hlbln], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-1-58051-01-19-02h02m17s] [2024-01-19T02:10:09.753Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-1-58051-01-19-02h02m17s]) found in the restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-hlbln] was also present on the cluster/namespace [mysql-ibm-pxb-1-58051-01-19-02h02m17s] [2024-01-19T02:10:09.753Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-1-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-hlbln] [2024-01-19T02:10:09.753Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-1-58051-01-19-02h02m17s] in restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-hlbln] [2024-01-19T02:10:09.753Z] 2024-01-19 02:10:09 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:10:09.754Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:10:09.754Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-4-58051-01-19-02h02m17s] are [[mysql-data]] [2024-01-19T02:10:09.754Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2240] - in restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-cfhmj], the status of the restored volume [pvc-cbc72f26-5bc0-4257-80fb-b71746d13633] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-19T02:10:09.754Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validate applications in restored namespace [mysql-ibm-pxb-4-58051-01-19-02h02m17s] due to restore [test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-cfhmj] [2024-01-19T02:10:09.754Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating mysql-ibm app [2024-01-19T02:10:09.754Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] wait for mysql-ibm app to start running [2024-01-19T02:10:09.754Z] 2024-01-19 02:10:09 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:10:09.754Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:10:09.754Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-58051-01-19-02h02m17s] are [[mysql-data]] [2024-01-19T02:10:09.754Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2240] - in restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-demhz], the status of the restored volume [pvc-91c8a31d-0a15-49fe-be42-b7840d5df313] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-19T02:10:09.754Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validate applications in restored namespace [mysql-ibm-pxb-0-58051-01-19-02h02m17s] due to restore [test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-demhz] [2024-01-19T02:10:09.755Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating mysql-ibm app [2024-01-19T02:10:09.755Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] wait for mysql-ibm app to start running [2024-01-19T02:10:09.755Z] 2024-01-19 02:10:09 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:10:09.755Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:10:09.755Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-3-58051-01-19-02h02m17s] are [[mysql-data]] [2024-01-19T02:10:09.755Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2240] - in restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-gwqmc], the status of the restored volume [pvc-f7960376-2f87-489d-9d28-f58b5b042538] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-19T02:10:09.755Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validate applications in restored namespace [mysql-ibm-pxb-3-58051-01-19-02h02m17s] due to restore [test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-gwqmc] [2024-01-19T02:10:09.755Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating mysql-ibm app [2024-01-19T02:10:09.755Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] wait for mysql-ibm app to start running [2024-01-19T02:10:10.013Z] 2024-01-19 02:10:09 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:10:10.013Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:10:10.013Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-2-58051-01-19-02h02m17s] are [[mysql-data]] [2024-01-19T02:10:10.013Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2240] - in restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-qdqei], the status of the restored volume [pvc-31879f3a-056b-478a-af54-3d19f45bc5a2] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-19T02:10:10.013Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validate applications in restored namespace [mysql-ibm-pxb-2-58051-01-19-02h02m17s] due to restore [test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-qdqei] [2024-01-19T02:10:10.013Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating mysql-ibm app [2024-01-19T02:10:10.013Z] 2024-01-19 02:10:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] wait for mysql-ibm app to start running [2024-01-19T02:10:10.269Z] 2024-01-19 02:10:10 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:10:10.269Z] 2024-01-19 02:10:10 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:10:10.269Z] 2024-01-19 02:10:10 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-58051-01-19-02h02m17s] are [[mysql-data]] [2024-01-19T02:10:10.270Z] 2024-01-19 02:10:10 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.ValidateRestore:#2240] - in restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-hlbln], the status of the restored volume [pvc-058f8fc2-466e-4572-a247-199b46bee37d] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-19T02:10:10.270Z] 2024-01-19 02:10:10 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validate applications in restored namespace [mysql-ibm-pxb-1-58051-01-19-02h02m17s] due to restore [test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-hlbln] [2024-01-19T02:10:10.270Z] 2024-01-19 02:10:10 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating mysql-ibm app [2024-01-19T02:10:10.270Z] 2024-01-19 02:10:10 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] wait for mysql-ibm app to start running [2024-01-19T02:10:10.533Z] 2024-01-19 02:10:10 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T02:10:10.533Z] 2024/01/19 02:10:10 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:10:10.533Z] pod name:mysql-7d766cd65c-mwbrg namespace:mysql-ibm-pxb-4-58051-01-19-02h02m17s running:true ready:false node:10.240.64.27 [2024-01-19T02:10:10.533Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:10:10.533Z] 2024-01-19 02:10:10 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T02:10:10.790Z] 2024/01/19 02:10:10 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:10:10.790Z] pod name:mysql-7d766cd65c-wwl6h namespace:mysql-ibm-pxb-0-58051-01-19-02h02m17s running:true ready:false node:10.240.64.8 [2024-01-19T02:10:10.790Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:10:10.790Z] 2024-01-19 02:10:10 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T02:10:10.790Z] 2024/01/19 02:10:10 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:10:10.790Z] pod name:mysql-7d766cd65c-pcrlf namespace:mysql-ibm-pxb-3-58051-01-19-02h02m17s running:true ready:false node:10.240.64.8 [2024-01-19T02:10:10.790Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:10:11.047Z] 2024-01-19 02:10:10 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T02:10:11.047Z] 2024/01/19 02:10:10 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:10:11.047Z] pod name:mysql-7d766cd65c-8ft98 namespace:mysql-ibm-pxb-2-58051-01-19-02h02m17s running:true ready:false node:10.240.64.8 [2024-01-19T02:10:11.047Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:10:11.304Z] 2024-01-19 02:10:11 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T02:10:11.560Z] 2024/01/19 02:10:11 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:10:11.560Z] pod name:mysql-7d766cd65c-j6htc namespace:mysql-ibm-pxb-1-58051-01-19-02h02m17s running:true ready:false node:10.240.64.27 [2024-01-19T02:10:11.560Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:10:21.502Z] 2024/01/19 02:10:20 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:10:21.502Z] pod name:mysql-7d766cd65c-mwbrg namespace:mysql-ibm-pxb-4-58051-01-19-02h02m17s running:true ready:false node:10.240.64.27 [2024-01-19T02:10:21.503Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:10:21.503Z] 2024/01/19 02:10:20 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:10:21.503Z] pod name:mysql-7d766cd65c-wwl6h namespace:mysql-ibm-pxb-0-58051-01-19-02h02m17s running:true ready:false node:10.240.64.8 [2024-01-19T02:10:21.503Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:10:21.503Z] 2024/01/19 02:10:20 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:10:21.503Z] pod name:mysql-7d766cd65c-pcrlf namespace:mysql-ibm-pxb-3-58051-01-19-02h02m17s running:true ready:false node:10.240.64.8 [2024-01-19T02:10:21.503Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:10:21.503Z] 2024/01/19 02:10:21 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:10:21.503Z] pod name:mysql-7d766cd65c-8ft98 namespace:mysql-ibm-pxb-2-58051-01-19-02h02m17s running:true ready:false node:10.240.64.8 [2024-01-19T02:10:21.503Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:10:21.503Z] 2024/01/19 02:10:21 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:10:21.503Z] pod name:mysql-7d766cd65c-j6htc namespace:mysql-ibm-pxb-1-58051-01-19-02h02m17s running:true ready:false node:10.240.64.27 [2024-01-19T02:10:21.503Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:10:31.503Z] 2024/01/19 02:10:30 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:10:31.503Z] pod name:mysql-7d766cd65c-mwbrg namespace:mysql-ibm-pxb-4-58051-01-19-02h02m17s running:true ready:false node:10.240.64.27 [2024-01-19T02:10:31.503Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:10:31.503Z] 2024/01/19 02:10:30 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:10:31.503Z] pod name:mysql-7d766cd65c-wwl6h namespace:mysql-ibm-pxb-0-58051-01-19-02h02m17s running:true ready:false node:10.240.64.8 [2024-01-19T02:10:31.503Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:10:31.503Z] 2024/01/19 02:10:30 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:10:31.503Z] pod name:mysql-7d766cd65c-pcrlf namespace:mysql-ibm-pxb-3-58051-01-19-02h02m17s running:true ready:false node:10.240.64.8 [2024-01-19T02:10:31.503Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:10:31.503Z] 2024/01/19 02:10:31 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:10:31.503Z] pod name:mysql-7d766cd65c-8ft98 namespace:mysql-ibm-pxb-2-58051-01-19-02h02m17s running:true ready:false node:10.240.64.8 [2024-01-19T02:10:31.503Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:10:31.761Z] 2024/01/19 02:10:31 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:10:31.761Z] pod name:mysql-7d766cd65c-j6htc namespace:mysql-ibm-pxb-1-58051-01-19-02h02m17s running:true ready:false node:10.240.64.27 [2024-01-19T02:10:31.761Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:10:41.707Z] 2024/01/19 02:10:40 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:10:41.707Z] pod name:mysql-7d766cd65c-mwbrg namespace:mysql-ibm-pxb-4-58051-01-19-02h02m17s running:true ready:false node:10.240.64.27 [2024-01-19T02:10:41.707Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:10:41.707Z] 2024/01/19 02:10:40 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:10:41.707Z] pod name:mysql-7d766cd65c-wwl6h namespace:mysql-ibm-pxb-0-58051-01-19-02h02m17s running:true ready:false node:10.240.64.8 [2024-01-19T02:10:41.707Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:10:41.707Z] 2024/01/19 02:10:40 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:10:41.707Z] pod name:mysql-7d766cd65c-pcrlf namespace:mysql-ibm-pxb-3-58051-01-19-02h02m17s running:true ready:false node:10.240.64.8 [2024-01-19T02:10:41.707Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:10:41.707Z] 2024/01/19 02:10:41 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:10:41.707Z] pod name:mysql-7d766cd65c-8ft98 namespace:mysql-ibm-pxb-2-58051-01-19-02h02m17s running:true ready:false node:10.240.64.8 [2024-01-19T02:10:41.707Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:10:41.707Z] 2024/01/19 02:10:41 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:10:41.707Z] pod name:mysql-7d766cd65c-j6htc namespace:mysql-ibm-pxb-1-58051-01-19-02h02m17s running:true ready:false node:10.240.64.27 [2024-01-19T02:10:41.707Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:10:51.651Z] 2024/01/19 02:10:50 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:10:51.651Z] pod name:mysql-7d766cd65c-mwbrg namespace:mysql-ibm-pxb-4-58051-01-19-02h02m17s running:true ready:false node:10.240.64.27 [2024-01-19T02:10:51.651Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:10:51.651Z] 2024/01/19 02:10:50 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:10:51.651Z] pod name:mysql-7d766cd65c-wwl6h namespace:mysql-ibm-pxb-0-58051-01-19-02h02m17s running:true ready:false node:10.240.64.8 [2024-01-19T02:10:51.651Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:10:51.651Z] 2024/01/19 02:10:51 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:10:51.651Z] pod name:mysql-7d766cd65c-pcrlf namespace:mysql-ibm-pxb-3-58051-01-19-02h02m17s running:true ready:false node:10.240.64.8 [2024-01-19T02:10:51.651Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:10:51.651Z] 2024-01-19 02:10:51 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T02:10:51.906Z] 2024-01-19 02:10:51 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T02:10:52.832Z] 2024/01/19 02:10:52 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:10:52.832Z] pod name:mysql-7d766cd65c-j6htc namespace:mysql-ibm-pxb-1-58051-01-19-02h02m17s running:true ready:false node:10.240.64.27 [2024-01-19T02:10:52.832Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:10:53.393Z] 2024-01-19 02:10:53 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating CR cleanup [2024-01-19T02:10:53.393Z] 2024-01-19 02:10:53 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T02:10:53.393Z] 2024-01-19 02:10:53 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] All CRs in [mysql-ibm-pxb-2-58051-01-19-02h02m17s] are [[]] [2024-01-19T02:11:01.471Z] 2024-01-19 02:11:00 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T02:11:01.471Z] 2024-01-19 02:11:00 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T02:11:01.471Z] 2024-01-19 02:11:01 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T02:11:02.031Z] 2024-01-19 02:11:01 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating CR cleanup [2024-01-19T02:11:02.031Z] 2024-01-19 02:11:01 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T02:11:02.031Z] 2024-01-19 02:11:01 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] All CRs in [mysql-ibm-pxb-4-58051-01-19-02h02m17s] are [[]] [2024-01-19T02:11:02.593Z] 2024-01-19 02:11:02 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T02:11:02.593Z] 2024-01-19 02:11:02 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T02:11:04.483Z] 2024-01-19 02:11:04 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T02:11:04.483Z] 2024-01-19 02:11:04 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T02:11:04.740Z] 2024-01-19 02:11:04 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating CR cleanup [2024-01-19T02:11:04.740Z] 2024-01-19 02:11:04 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T02:11:04.996Z] 2024-01-19 02:11:04 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] All CRs in [mysql-ibm-pxb-0-58051-01-19-02h02m17s] are [[]] [2024-01-19T02:11:05.922Z] 2024-01-19 02:11:05 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T02:11:06.178Z] 2024-01-19 02:11:06 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating CR cleanup [2024-01-19T02:11:06.178Z] 2024-01-19 02:11:06 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T02:11:06.433Z] 2024-01-19 02:11:06 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] All CRs in [mysql-ibm-pxb-3-58051-01-19-02h02m17s] are [[]] [2024-01-19T02:11:06.999Z] 2024-01-19 02:11:06 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Validating CR cleanup [2024-01-19T02:11:06.999Z] 2024-01-19 02:11:06 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T02:11:06.999Z] 2024-01-19 02:11:06 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] All CRs in [mysql-ibm-pxb-1-58051-01-19-02h02m17s] are [[]] [2024-01-19T02:11:06.999Z] INFO[2024-01-19 02:11:06] Verifying : Description : Errors generated while validating restores with replace existing resources - [2024-01-19T02:11:06.999Z]  [2024-01-19T02:11:06.999Z] INFO[2024-01-19 02:11:06] Actual:0, Expected: 0 [2024-01-19T02:11:07.259Z] 2024-01-19 02:11:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Destroying apps [2024-01-19T02:11:07.259Z] 2024-01-19 02:11:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T02:11:07.519Z] 2024-01-19 02:11:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T02:11:07.519Z] 2024-01-19 02:11:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] start destroying mysql-ibm app [2024-01-19T02:11:07.519Z] 2024-01-19 02:11:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T02:11:07.519Z] 2024-01-19 02:11:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T02:11:07.519Z] 2024-01-19 02:11:07 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T02:11:07.519Z] INFO[2024-01-19 02:11:07] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T02:11:07.519Z] INFO[2024-01-19 02:11:07] Actual:true, Expected: true [2024-01-19T02:11:07.519Z] 2024-01-19 02:11:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T02:11:08.080Z] 2024-01-19 02:11:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T02:11:08.080Z] 2024-01-19 02:11:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] start destroying mysql-ibm app [2024-01-19T02:11:08.080Z] 2024-01-19 02:11:07 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T02:11:08.080Z] 2024-01-19 02:11:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T02:11:08.080Z] 2024-01-19 02:11:08 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T02:11:08.080Z] INFO[2024-01-19 02:11:08] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T02:11:08.080Z] INFO[2024-01-19 02:11:08] Actual:true, Expected: true [2024-01-19T02:11:08.080Z] 2024-01-19 02:11:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T02:11:08.647Z] 2024-01-19 02:11:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T02:11:08.647Z] 2024-01-19 02:11:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] start destroying mysql-ibm app [2024-01-19T02:11:08.647Z] 2024-01-19 02:11:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T02:11:08.903Z] 2024-01-19 02:11:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T02:11:08.903Z] 2024-01-19 02:11:08 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T02:11:08.903Z] INFO[2024-01-19 02:11:08] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T02:11:08.903Z] INFO[2024-01-19 02:11:08] Actual:true, Expected: true [2024-01-19T02:11:08.903Z] 2024-01-19 02:11:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T02:11:09.158Z] 2024-01-19 02:11:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T02:11:09.158Z] 2024-01-19 02:11:08 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] start destroying mysql-ibm app [2024-01-19T02:11:09.414Z] 2024-01-19 02:11:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T02:11:09.414Z] 2024-01-19 02:11:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T02:11:09.414Z] 2024-01-19 02:11:09 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T02:11:09.414Z] INFO[2024-01-19 02:11:09] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T02:11:09.414Z] INFO[2024-01-19 02:11:09] Actual:true, Expected: true [2024-01-19T02:11:09.414Z] 2024-01-19 02:11:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T02:11:09.673Z] 2024-01-19 02:11:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T02:11:09.673Z] 2024-01-19 02:11:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] start destroying mysql-ibm app [2024-01-19T02:11:09.929Z] 2024-01-19 02:11:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T02:11:09.929Z] 2024-01-19 02:11:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T02:11:09.930Z] 2024-01-19 02:11:09 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T02:11:09.930Z] INFO[2024-01-19 02:11:09] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T02:11:09.930Z] INFO[2024-01-19 02:11:09] Actual:true, Expected: true [2024-01-19T02:11:09.930Z] 2024-01-19 02:11:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Deleting created restores [2024-01-19T02:11:09.930Z] INFO[2024-01-19 02:11:09] Verifying : Description : Getting the backup driver [2024-01-19T02:11:09.930Z] INFO[2024-01-19 02:11:09] Actual:true, Expected: true [2024-01-19T02:11:09.930Z] INFO[2024-01-19 02:11:09] Verifying : Description : Getting the backup driver [2024-01-19T02:11:09.930Z] INFO[2024-01-19 02:11:09] Actual:true, Expected: true [2024-01-19T02:11:09.930Z] INFO[2024-01-19 02:11:09] Verifying : Description : Getting the backup driver [2024-01-19T02:11:09.930Z] INFO[2024-01-19 02:11:09] Verifying : Description : Getting the backup driver [2024-01-19T02:11:09.930Z] INFO[2024-01-19 02:11:09] Actual:true, Expected: true [2024-01-19T02:11:09.930Z] INFO[2024-01-19 02:11:09] Verifying : Description : Getting the backup driver [2024-01-19T02:11:09.930Z] INFO[2024-01-19 02:11:09] Actual:true, Expected: true [2024-01-19T02:11:09.930Z] INFO[2024-01-19 02:11:09] Verifying : Description : Getting the backup driver [2024-01-19T02:11:09.930Z] INFO[2024-01-19 02:11:09] Actual:true, Expected: true [2024-01-19T02:11:09.930Z] INFO[2024-01-19 02:11:09] Verifying : Description : Getting the backup driver [2024-01-19T02:11:09.930Z] INFO[2024-01-19 02:11:09] Actual:true, Expected: true [2024-01-19T02:11:09.930Z] INFO[2024-01-19 02:11:09] Verifying : Description : Getting the backup driver [2024-01-19T02:11:09.930Z] INFO[2024-01-19 02:11:09] Actual:true, Expected: true [2024-01-19T02:11:09.930Z] INFO[2024-01-19 02:11:09] Verifying : Description : Getting the backup driver [2024-01-19T02:11:09.930Z] INFO[2024-01-19 02:11:09] Actual:true, Expected: true [2024-01-19T02:11:09.930Z] INFO[2024-01-19 02:11:09] Verifying : Description : Getting the backup driver [2024-01-19T02:11:09.930Z] INFO[2024-01-19 02:11:09] Actual:true, Expected: true [2024-01-19T02:11:09.930Z] INFO[2024-01-19 02:11:09] Actual:true, Expected: true [2024-01-19T02:11:10.187Z] INFO[2024-01-19 02:11:09] Verifying : Description : Deleting Restore test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-cfhmj [2024-01-19T02:11:10.187Z] INFO[2024-01-19 02:11:09] Actual:true, Expected: true [2024-01-19T02:11:10.187Z] INFO[2024-01-19 02:11:09] Verifying : Description : Deleting Restore test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-demhz [2024-01-19T02:11:10.187Z] INFO[2024-01-19 02:11:09] Actual:true, Expected: true [2024-01-19T02:11:10.187Z] INFO[2024-01-19 02:11:09] Verifying : Description : Deleting Restore test-restore-recent-backup-mysql-ibm-pxb-4-58051-01-19-02h02m17s-azjsc [2024-01-19T02:11:10.187Z] INFO[2024-01-19 02:11:09] Actual:true, Expected: true [2024-01-19T02:11:10.187Z] INFO[2024-01-19 02:11:09] Verifying : Description : Deleting Restore test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-aulqy [2024-01-19T02:11:10.187Z] INFO[2024-01-19 02:11:09] Actual:true, Expected: true [2024-01-19T02:11:10.188Z] INFO[2024-01-19 02:11:09] Verifying : Description : Deleting Restore test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-edflu [2024-01-19T02:11:10.188Z] INFO[2024-01-19 02:11:09] Actual:true, Expected: true [2024-01-19T02:11:10.188Z] INFO[2024-01-19 02:11:09] Verifying : Description : Deleting Restore test-restore-recent-backup-mysql-ibm-pxb-0-58051-01-19-02h02m17s-akbki [2024-01-19T02:11:10.188Z] INFO[2024-01-19 02:11:09] Actual:true, Expected: true [2024-01-19T02:11:10.188Z] INFO[2024-01-19 02:11:09] Verifying : Description : Deleting Restore test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-qdqei [2024-01-19T02:11:10.188Z] INFO[2024-01-19 02:11:09] Actual:true, Expected: true [2024-01-19T02:11:10.188Z] INFO[2024-01-19 02:11:09] Verifying : Description : Deleting Restore test-restore-recent-backup-mysql-ibm-pxb-2-58051-01-19-02h02m17s-ikbki [2024-01-19T02:11:10.188Z] INFO[2024-01-19 02:11:09] Actual:true, Expected: true [2024-01-19T02:11:10.188Z] INFO[2024-01-19 02:11:09] Verifying : Description : Deleting Restore test-restore-recent-backup-mysql-ibm-pxb-3-58051-01-19-02h02m17s-gwqmc [2024-01-19T02:11:10.188Z] INFO[2024-01-19 02:11:09] Actual:true, Expected: true [2024-01-19T02:11:10.188Z] INFO[2024-01-19 02:11:09] Verifying : Description : Deleting Restore test-restore-recent-backup-mysql-ibm-pxb-1-58051-01-19-02h02m17s-hlbln [2024-01-19T02:11:10.188Z] INFO[2024-01-19 02:11:09] Actual:true, Expected: true [2024-01-19T02:11:10.188Z] 2024-01-19 02:11:09 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Cleaning backup locations in map [map[323eb0bb-6666-4413-bdb1-982c76513ae6:autogenerated-backup-location-exgca]], cloud credential [cred-aws-zlihj], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T02:11:10.188Z] INFO[2024-01-19 02:11:10] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-exgca] [2024-01-19T02:11:10.188Z] INFO[2024-01-19 02:11:10] Actual:true, Expected: true [2024-01-19T02:11:10.188Z] 2024-01-19 02:11:10 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-exgca] is present [2024-01-19T02:11:10.188Z] 2024/01/19 02:11:10 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-exgca is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T02:11:42.318Z] 2024-01-19 02:11:40 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-exgca] is present [2024-01-19T02:11:42.318Z] 2024/01/19 02:11:40 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-exgca is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T02:12:14.363Z] 2024-01-19 02:12:10 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-exgca] is present [2024-01-19T02:12:14.363Z] 2024/01/19 02:12:10 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-exgca is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T02:12:40.891Z] 2024-01-19 02:12:40 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 aws-global-aws-px-backup-system-test-4cac76dc-bl autogenerated-backup-location-dctzc autogenerated-backup-location-1705628002] [2024-01-19T02:12:40.891Z] INFO[2024-01-19 02:12:40] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-exgca [2024-01-19T02:12:40.891Z] INFO[2024-01-19 02:12:40] Actual:true, Expected: true [2024-01-19T02:12:40.891Z] 2024-01-19 02:12:40 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cred-aws-zlihj] is present [2024-01-19T02:12:40.891Z] INFO[2024-01-19 02:12:40] Verifying : Description : Verifying if cloud cred [cred-aws-zlihj] is present [2024-01-19T02:12:40.891Z] INFO[2024-01-19 02:12:40] Actual:true, Expected: true [2024-01-19T02:12:40.891Z] INFO[2024-01-19 02:12:40] Verifying : Description : Verifying deletion of cloud cred [cred-aws-zlihj] [2024-01-19T02:12:40.891Z] INFO[2024-01-19 02:12:40] Actual:true, Expected: true [2024-01-19T02:12:40.891Z] INFO[2024-01-19 02:12:40] Verifying : Description : Deleting cloud cred cred-aws-zlihj [2024-01-19T02:12:40.891Z] INFO[2024-01-19 02:12:40] Actual:true, Expected: true [2024-01-19T02:12:40.891Z] INFO[2024-01-19 02:12:40] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T02:12:40.891Z] INFO[2024-01-19 02:12:40] Actual:true, Expected: true [2024-01-19T02:12:40.891Z] INFO[2024-01-19 02:12:40] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T02:12:40.891Z] ERRO[2024-01-19 02:12:40] Actual:rpc error: code = FailedPrecondition desc = cluster [destination-cluster] in use by backup [tp-backup-schedule-1705605709-41dbde6-interval-2024-01-19-021211], cannot be deleted, Expected:  [2024-01-19T02:12:41.453Z] INFO[2024-01-19 02:12:41] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-zbbvt] [2024-01-19T02:12:41.453Z] ERRO[2024-01-19 02:12:41] Actual:rpc error: code = Unknown desc = cluster [destination-cluster] has reference to cloud credential [ibm-destination-config-cloud-cred-zbbvt:a271ea5b-dc9d-40f1-a82a-651539c7e486], Expected:  [2024-01-19T02:12:41.453Z] INFO[2024-01-19 02:12:41] Verifying : Description : Deleting cluster source-cluster [2024-01-19T02:12:41.453Z] INFO[2024-01-19 02:12:41] Actual:true, Expected: true [2024-01-19T02:12:42.015Z] INFO[2024-01-19 02:12:41] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-pmzso] [2024-01-19T02:12:42.015Z] INFO[2024-01-19 02:12:41] Actual:true, Expected: true [2024-01-19T02:12:42.015Z] INFO[2024-01-19 02:12:41] --------Test End------ [2024-01-19T02:12:42.015Z] INFO[2024-01-19 02:12:41] #Test: [2024-01-19T02:12:42.015Z] INFO[2024-01-19 02:12:41] #Description: [2024-01-19T02:12:42.015Z] INFO[2024-01-19 02:12:41] ------------------------ [2024-01-19T02:12:42.015Z] 2024-01-19 02:12:41 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-58051-01-19-02h02m17s] [2024-01-19T02:12:42.273Z] 2024-01-19 02:12:42 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-1-58051-01-19-02h02m17s] [2024-01-19T02:12:42.273Z] 2024-01-19 02:12:42 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-2-58051-01-19-02h02m17s] [2024-01-19T02:12:42.530Z] 2024-01-19 02:12:42 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-3-58051-01-19-02h02m17s] [2024-01-19T02:12:42.785Z] 2024-01-19 02:12:42 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-4-58051-01-19-02h02m17s] [2024-01-19T02:12:43.043Z] 2024-01-19 02:12:42 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T02:12:43.043Z] 2024-01-19 02:12:42 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T02:12:43.299Z] 2024-01-19 02:12:43 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-58051-01-19-02h02m17s] [2024-01-19T02:12:43.299Z] 2024-01-19 02:12:43 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-1-58051-01-19-02h02m17s] [2024-01-19T02:12:43.554Z] 2024-01-19 02:12:43 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-2-58051-01-19-02h02m17s] [2024-01-19T02:12:43.809Z] 2024-01-19 02:12:43 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-3-58051-01-19-02h02m17s] [2024-01-19T02:12:44.067Z] 2024-01-19 02:12:43 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-4-58051-01-19-02h02m17s] [2024-01-19T02:12:44.067Z] 2024-01-19 02:12:43 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {MultipleInPlaceRestoreSameTime} Issue multiple in-place restores at the same time [2024-01-19T02:12:44.067Z] 2024-01-19 02:12:43 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T02:12:44.067Z] 2024/01/19 02:12:43 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T02:12:49.312Z] 2024/01/19 02:12:48 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T02:12:54.557Z] 2024/01/19 02:12:53 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T02:12:59.800Z] 2024/01/19 02:12:58 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T02:13:05.040Z] 2024/01/19 02:13:03 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T02:13:09.214Z] 2024/01/19 02:13:08 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T02:13:14.456Z] 2024/01/19 02:13:13 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T02:13:19.704Z] 2024/01/19 02:13:19 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T02:13:24.950Z] 2024/01/19 02:13:24 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.128.20. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.20 10.240.128.20]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T02:13:24.950Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T02:13:24.950Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T02:13:24.950Z] {dial tcp 10.240.128.20:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T02:13:29.113Z] 2024-01-19 02:13:29 +0000:[WARNING] [{MultipleInPlaceRestoreSameTime}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.20. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.20 10.240.128.20]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T02:13:29.113Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T02:13:29.113Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T02:13:29.113Z] {dial tcp 10.240.128.20:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T02:13:29.113Z] 2024-01-19 02:13:29 +0000:[ERROR] [{MultipleInPlaceRestoreSameTime}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.20. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.20 10.240.128.20]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T02:13:29.113Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T02:13:29.113Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T02:13:29.113Z] {dial tcp 10.240.128.20:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T02:13:29.113Z] 2024-01-19 02:13:29 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T02:13:29.113Z] 2024-01-19 02:13:29 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] Switching context to [/tmp/source-config] [2024-01-19T02:13:29.113Z] 2024-01-19 02:13:29 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T02:13:29.369Z] 2024-01-19 02:13:29 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T02:13:29.369Z] 2024-01-19 02:13:29 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T02:13:29.369Z] 2024-01-19 02:13:29 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T02:13:29.626Z] 2024-01-19 02:13:29 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T02:13:29.626Z] 2024-01-19 02:13:29 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T02:13:29.881Z] 2024-01-19 02:13:29 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T02:13:30.442Z] 2024-01-19 02:13:30 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T02:13:30.697Z] 2024-01-19 02:13:30 +0000:[INFO] [{MultipleInPlaceRestoreSameTime}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T02:13:30.953Z] 2024/01/19 02:13:30 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:13:40.901Z] 2024/01/19 02:13:40 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:13:53.072Z] 2024/01/19 02:13:50 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:14:01.179Z] 2024/01/19 02:14:00 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:14:11.127Z] 2024/01/19 02:14:10 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:14:21.070Z] 2024/01/19 02:14:21 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:14:33.249Z] 2024/01/19 02:14:31 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:14:41.324Z] 2024/01/19 02:14:41 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:14:46.562Z] INFO[2024-01-19 02:14:45] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T02:14:46.562Z] INFO[2024-01-19 02:14:45] sched stats: tick interval: 38445 < 1.2s (100%) [2024-01-19T02:14:46.562Z] INFO[2024-01-19 02:14:45] sched stats: tick process durations: 38444 < 500ms (100%) [2024-01-19T02:14:51.804Z] 2024/01/19 02:14:51 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:15:01.749Z] 2024/01/19 02:15:01 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:15:11.706Z] 2024/01/19 02:15:11 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:15:21.650Z] 2024/01/19 02:15:21 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:15:31.594Z] 2024/01/19 02:15:31 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:15:41.532Z] 2024/01/19 02:15:41 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:15:51.481Z] 2024/01/19 02:15:51 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:16:01.453Z] 2024/01/19 02:16:01 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:16:12.169Z] 2024/01/19 02:16:11 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:16:22.111Z] 2024/01/19 02:16:21 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:16:32.058Z] 2024/01/19 02:16:31 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:16:42.000Z] 2024/01/19 02:16:41 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:16:51.956Z] 2024/01/19 02:16:51 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:17:01.909Z] 2024/01/19 02:17:01 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:17:11.850Z] 2024/01/19 02:17:11 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:17:21.808Z] 2024/01/19 02:17:21 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:17:31.754Z] 2024/01/19 02:17:31 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:17:41.700Z] 2024/01/19 02:17:41 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:17:51.650Z] 2024/01/19 02:17:51 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:18:03.818Z] 2024/01/19 02:18:01 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:18:11.913Z] 2024/01/19 02:18:11 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:18:21.855Z] 2024/01/19 02:18:21 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:18:31.801Z] 2024-01-19 02:18:31 +0000:[ERROR] [{MultipleInPlaceRestoreSameTime}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.802Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.802Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.802Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.802Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.802Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.802Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.802Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.802Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.802Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.802Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.802Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.802Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.802Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T02:18:31.802Z] [2024-01-19T02:18:31.802Z] ------------------------------ [2024-01-19T02:18:31.802Z] • Failure [974.292 seconds] [2024-01-19T02:18:31.802Z] {MultipleInPlaceRestoreSameTime} [2024-01-19T02:18:31.802Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:1849 [2024-01-19T02:18:31.802Z] Issue multiple in-place restores at the same time [] [2024-01-19T02:18:31.802Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:1885 [2024-01-19T02:18:31.802Z] [2024-01-19T02:18:31.802Z] Unexpected error: [2024-01-19T02:18:31.802Z] <*errors.errorString | 0xc001891e20>: { [2024-01-19T02:18:31.803Z] s: "failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{No... [2024-01-19T02:18:31.803Z] [2024-01-19T02:18:31.803Z] Gomega truncated this representation as it exceeds 'format.MaxLength'. [2024-01-19T02:18:31.803Z] Consider having the object provide a custom 'GomegaStringer' representation [2024-01-19T02:18:31.803Z] or adjust the parameters in Gomega's 'format' package. [2024-01-19T02:18:31.803Z] [2024-01-19T02:18:31.803Z] Learn more here: https://onsi.github.io/gomega/#adjusting-output [2024-01-19T02:18:31.803Z] [2024-01-19T02:18:31.803Z] failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.803Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.804Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.804Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.804Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.804Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.804Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.804Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.804Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.804Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.804Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.804Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.804Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.804Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.804Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.804Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.804Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.804Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.804Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.804Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.804Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.805Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.805Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.805Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.805Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.805Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.805Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.805Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.805Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:18:31.805Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T02:18:31.805Z] occurred [2024-01-19T02:18:31.805Z] [2024-01-19T02:18:31.805Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T02:18:31.805Z] [2024-01-19T02:18:31.805Z] Full Stack Trace [2024-01-19T02:18:31.805Z] github.com/portworx/torpedo/pkg/log.FailOnError({0x88a5060, 0xc001891e20}, {0x7f74bfd?, 0x6cc0320?}, {0x0?, 0x6cc0320?, 0xc000e64cd0?}) [2024-01-19T02:18:31.805Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 +0x290 [2024-01-19T02:18:31.805Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest.func1() [2024-01-19T02:18:31.805Z] /go/src/github.com/portworx/torpedo/tests/common.go:7283 +0x32 [2024-01-19T02:18:31.805Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest({0xc000e44e80, 0x5, 0x8}) [2024-01-19T02:18:31.805Z] /go/src/github.com/portworx/torpedo/tests/common.go:7299 +0x778 [2024-01-19T02:18:31.805Z] github.com/portworx/torpedo/tests/backup.glob..func65.3() [2024-01-19T02:18:31.805Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:2065 +0x22e [2024-01-19T02:18:31.805Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T02:18:31.805Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T02:18:31.805Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T02:18:31.805Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T02:18:31.805Z] created by testing.(*T).Run [2024-01-19T02:18:31.805Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T02:18:31.805Z] ------------------------------ [2024-01-19T02:18:31.805Z] STEP: schedule applications [2024-01-19T02:18:31.805Z] STEP: schedule applications [2024-01-19T02:18:31.805Z] STEP: schedule applications [2024-01-19T02:18:31.805Z] STEP: schedule applications [2024-01-19T02:18:31.805Z] STEP: schedule applications [2024-01-19T02:18:31.805Z] STEP: Validate applications [2024-01-19T02:18:31.805Z] STEP: validate applications [2024-01-19T02:18:31.805Z] STEP: validate mysql-ibm app's volumes [2024-01-19T02:18:31.805Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T02:18:31.805Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T02:18:31.805Z] STEP: get mysql-ibm app's volume: pvc-e05c8f60-e1c6-4309-914d-535f4f735636 inspected by the volume driver [2024-01-19T02:18:31.805Z] STEP: wait for mysql-ibm app to start running [2024-01-19T02:18:31.805Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T02:18:31.805Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T02:18:31.805Z] STEP: Validate Px pod restart count [2024-01-19T02:18:31.805Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T02:18:31.806Z] STEP: validate mysql-ibm app's volumes [2024-01-19T02:18:31.806Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T02:18:31.806Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T02:18:31.806Z] STEP: get mysql-ibm app's volume: pvc-b354e5fc-cf44-4fdc-998e-c48e6f53ac75 inspected by the volume driver [2024-01-19T02:18:31.806Z] STEP: wait for mysql-ibm app to start running [2024-01-19T02:18:31.806Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T02:18:31.806Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T02:18:31.806Z] STEP: Validate Px pod restart count [2024-01-19T02:18:31.806Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T02:18:31.806Z] STEP: validate mysql-ibm app's volumes [2024-01-19T02:18:31.806Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T02:18:31.806Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T02:18:31.806Z] STEP: get mysql-ibm app's volume: pvc-54d25d3d-581b-45da-b799-b682a5ff95a1 inspected by the volume driver [2024-01-19T02:18:31.806Z] STEP: wait for mysql-ibm app to start running [2024-01-19T02:18:31.806Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T02:18:31.806Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T02:18:31.806Z] STEP: Validate Px pod restart count [2024-01-19T02:18:31.806Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T02:18:31.806Z] STEP: validate mysql-ibm app's volumes [2024-01-19T02:18:31.806Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T02:18:31.806Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T02:18:31.806Z] STEP: get mysql-ibm app's volume: pvc-8392530a-d166-4d47-a3b0-d43928fa9c8b inspected by the volume driver [2024-01-19T02:18:31.806Z] STEP: wait for mysql-ibm app to start running [2024-01-19T02:18:31.806Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T02:18:31.806Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T02:18:31.806Z] STEP: Validate Px pod restart count [2024-01-19T02:18:31.806Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T02:18:31.806Z] STEP: validate mysql-ibm app's volumes [2024-01-19T02:18:31.806Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T02:18:31.806Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T02:18:31.806Z] STEP: get mysql-ibm app's volume: pvc-d8cc3133-1781-4ccc-945c-d44343d98501 inspected by the volume driver [2024-01-19T02:18:31.806Z] STEP: wait for mysql-ibm app to start running [2024-01-19T02:18:31.806Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T02:18:31.806Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T02:18:31.806Z] STEP: Validate Px pod restart count [2024-01-19T02:18:31.806Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T02:18:31.806Z] STEP: Register cluster for backup [2024-01-19T02:18:31.806Z] STEP: Creating backup location and cloud setting [2024-01-19T02:18:31.806Z] STEP: Taking backup of application for different combination of restores [2024-01-19T02:18:31.806Z] STEP: Issuing multiple in-place restore at the same time [2024-01-19T02:18:31.806Z] STEP: Validating multiple in-place restore at the same time [2024-01-19T02:18:31.806Z] STEP: validate mysql-ibm app's volumes [2024-01-19T02:18:31.806Z] STEP: wait for mysql-ibm app to start running [2024-01-19T02:18:31.806Z] STEP: validate mysql-ibm app's volumes [2024-01-19T02:18:31.806Z] STEP: wait for mysql-ibm app to start running [2024-01-19T02:18:31.806Z] STEP: validate mysql-ibm app's volumes [2024-01-19T02:18:31.806Z] STEP: wait for mysql-ibm app to start running [2024-01-19T02:18:31.806Z] STEP: validate mysql-ibm app's volumes [2024-01-19T02:18:31.806Z] STEP: wait for mysql-ibm app to start running [2024-01-19T02:18:31.806Z] STEP: validate mysql-ibm app's volumes [2024-01-19T02:18:31.806Z] STEP: wait for mysql-ibm app to start running [2024-01-19T02:18:31.807Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T02:18:31.807Z] STEP: Validate Px pod restart count [2024-01-19T02:18:31.807Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T02:18:31.807Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T02:18:31.807Z] STEP: Validate Px pod restart count [2024-01-19T02:18:31.807Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T02:18:31.807Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T02:18:31.807Z] STEP: Validate Px pod restart count [2024-01-19T02:18:31.807Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T02:18:31.807Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T02:18:31.807Z] STEP: Validate Px pod restart count [2024-01-19T02:18:31.807Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T02:18:31.807Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T02:18:31.807Z] STEP: Validate Px pod restart count [2024-01-19T02:18:31.807Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T02:18:31.807Z] STEP: Issuing multiple in-place restore at the same time with replace existing resources [2024-01-19T02:18:31.807Z] STEP: Validating multiple in-place restore with replace existing resources [2024-01-19T02:18:31.807Z] STEP: validate mysql-ibm app's volumes [2024-01-19T02:18:31.807Z] STEP: wait for mysql-ibm app to start running [2024-01-19T02:18:31.807Z] STEP: validate mysql-ibm app's volumes [2024-01-19T02:18:31.807Z] STEP: wait for mysql-ibm app to start running [2024-01-19T02:18:31.807Z] STEP: validate mysql-ibm app's volumes [2024-01-19T02:18:31.807Z] STEP: wait for mysql-ibm app to start running [2024-01-19T02:18:31.807Z] STEP: validate mysql-ibm app's volumes [2024-01-19T02:18:31.807Z] STEP: wait for mysql-ibm app to start running [2024-01-19T02:18:31.807Z] STEP: validate mysql-ibm app's volumes [2024-01-19T02:18:31.807Z] STEP: wait for mysql-ibm app to start running [2024-01-19T02:18:31.807Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T02:18:31.807Z] STEP: Validate Px pod restart count [2024-01-19T02:18:31.807Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T02:18:31.807Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T02:18:31.807Z] STEP: Validate Px pod restart count [2024-01-19T02:18:31.807Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T02:18:31.807Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T02:18:31.807Z] STEP: Validate Px pod restart count [2024-01-19T02:18:31.807Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T02:18:31.807Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T02:18:31.807Z] STEP: Validate Px pod restart count [2024-01-19T02:18:31.807Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T02:18:31.807Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T02:18:31.807Z] STEP: Validate Px pod restart count [2024-01-19T02:18:31.807Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T02:18:31.807Z] STEP: destroy apps [2024-01-19T02:18:31.807Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T02:18:31.807Z] STEP: start destroying mysql-ibm app [2024-01-19T02:18:31.807Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T02:18:31.807Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T02:18:31.807Z] STEP: start destroying mysql-ibm app [2024-01-19T02:18:31.807Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T02:18:31.807Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T02:18:31.807Z] STEP: start destroying mysql-ibm app [2024-01-19T02:18:31.807Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T02:18:31.807Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T02:18:31.807Z] STEP: start destroying mysql-ibm app [2024-01-19T02:18:31.807Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T02:18:31.807Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T02:18:31.807Z] STEP: start destroying mysql-ibm app [2024-01-19T02:18:31.808Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T02:18:31.808Z] 2024-01-19 02:18:31 +0000:[INFO] [{SwapShareBackup}] Log Dir: /testresults//SwapShareBackup.log [2024-01-19T02:18:31.808Z] INFO[2024-01-19 02:18:31] --------Test Start------ [2024-01-19T02:18:31.808Z] INFO[2024-01-19 02:18:31] #Test: SwapShareBackup [2024-01-19T02:18:31.808Z] INFO[2024-01-19 02:18:31] #Description: Share backup with same name between two users [2024-01-19T02:18:31.808Z] INFO[2024-01-19 02:18:31] ------------------------ [2024-01-19T02:18:31.808Z] 2024-01-19 02:18:31 +0000:[INFO] [{SwapShareBackup}] Deploy applications [2024-01-19T02:18:31.808Z] 2024-01-19 02:18:31 +0000:[INFO] [{SwapShareBackup}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T02:18:31.808Z] 2024-01-19 02:18:31 +0000:[INFO] [{SwapShareBackup}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T02:18:31.808Z] 2024-01-19 02:18:31 +0000:[INFO] [{SwapShareBackup}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T02:18:32.064Z] 2024-01-19 02:18:31 +0000:[INFO] [{SwapShareBackup}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T02:18:32.064Z] 2024-01-19 02:18:31 +0000:[INFO] [{SwapShareBackup}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T02:18:32.064Z] 2024-01-19 02:18:32 +0000:[INFO] [{SwapShareBackup}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T02:18:32.065Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T02:18:32.322Z] 2024-01-19 02:18:32 +0000:[INFO] [{SwapShareBackup}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T02:18:32.323Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T02:18:32.323Z] 2024-01-19 02:18:32 +0000:[INFO] [{SwapShareBackup}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T02:18:32.323Z] 2024-01-19 02:18:32 +0000:[INFO] [{SwapShareBackup}] Validate applications [2024-01-19T02:18:32.323Z] 2024-01-19 02:18:32 +0000:[INFO] [{SwapShareBackup}] Validate applications [2024-01-19T02:18:32.323Z] 2024-01-19 02:18:32 +0000:[INFO] [{SwapShareBackup}] Validating mysql-ibm app [2024-01-19T02:18:32.323Z] 2024-01-19 02:18:32 +0000:[INFO] [{SwapShareBackup}] Validating mysql-ibm app's volumes [2024-01-19T02:18:32.323Z] 2024-01-19 02:18:32 +0000:[INFO] [{SwapShareBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:18:32.323Z] 2024/01/19 02:18:32 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T02:18:42.266Z] 2024/01/19 02:18:42 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T02:18:54.436Z] 2024-01-19 02:18:52 +0000:[INFO] [{SwapShareBackup}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-82940-01-19-02h18m31s [2024-01-19T02:18:54.436Z] 2024-01-19 02:18:52 +0000:[INFO] [{SwapShareBackup}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T02:18:54.436Z] 2024-01-19 02:18:52 +0000:[WARNING] [{SwapShareBackup}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T02:18:54.436Z] 2024-01-19 02:18:52 +0000:[INFO] [{SwapShareBackup}] wait for mysql-ibm app to start running [2024-01-19T02:18:54.436Z] 2024-01-19 02:18:52 +0000:[INFO] [{SwapShareBackup}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T02:18:54.436Z] 2024/01/19 02:18:52 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:18:54.436Z] pod name:mysql-7d766cd65c-9lhr5 namespace:mysql-ibm-pxb-0-82940-01-19-02h18m31s running:false ready:false node:10.240.64.8 [2024-01-19T02:18:54.436Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:19:04.378Z] 2024/01/19 02:19:02 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:19:04.378Z] pod name:mysql-7d766cd65c-9lhr5 namespace:mysql-ibm-pxb-0-82940-01-19-02h18m31s running:true ready:false node:10.240.64.8 [2024-01-19T02:19:04.378Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:19:14.319Z] 2024/01/19 02:19:12 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:19:14.319Z] pod name:mysql-7d766cd65c-9lhr5 namespace:mysql-ibm-pxb-0-82940-01-19-02h18m31s running:true ready:false node:10.240.64.8 [2024-01-19T02:19:14.319Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:19:24.317Z] 2024/01/19 02:19:22 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:19:24.317Z] pod name:mysql-7d766cd65c-9lhr5 namespace:mysql-ibm-pxb-0-82940-01-19-02h18m31s running:true ready:false node:10.240.64.8 [2024-01-19T02:19:24.317Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:19:34.310Z] 2024/01/19 02:19:32 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:19:34.310Z] pod name:mysql-7d766cd65c-9lhr5 namespace:mysql-ibm-pxb-0-82940-01-19-02h18m31s running:true ready:false node:10.240.64.8 [2024-01-19T02:19:34.310Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:19:44.269Z] 2024/01/19 02:19:42 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:19:44.270Z] pod name:mysql-7d766cd65c-9lhr5 namespace:mysql-ibm-pxb-0-82940-01-19-02h18m31s running:true ready:false node:10.240.64.8 [2024-01-19T02:19:44.270Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:19:54.242Z] 2024/01/19 02:19:53 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:19:54.242Z] pod name:mysql-7d766cd65c-9lhr5 namespace:mysql-ibm-pxb-0-82940-01-19-02h18m31s running:true ready:false node:10.240.64.8 [2024-01-19T02:19:54.242Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:20:04.186Z] 2024-01-19 02:20:03 +0000:[INFO] [{SwapShareBackup}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T02:20:04.186Z] 2024-01-19 02:20:03 +0000:[INFO] [{SwapShareBackup}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T02:20:04.186Z] 2024-01-19 02:20:03 +0000:[INFO] [{SwapShareBackup}] validate if mysql-ibm app's volumes are setup [2024-01-19T02:20:04.186Z] 2024-01-19 02:20:03 +0000:[WARNING] [{SwapShareBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:20:04.186Z] 2024-01-19 02:20:03 +0000:[INFO] [{SwapShareBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:20:04.186Z] 2024-01-19 02:20:03 +0000:[INFO] [{SwapShareBackup}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T02:20:04.186Z] 2024-01-19 02:20:03 +0000:[WARNING] [{SwapShareBackup}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T02:20:04.186Z] 2024-01-19 02:20:03 +0000:[INFO] [{SwapShareBackup}] Creating 2 users [2024-01-19T02:20:07.453Z] 2024-01-19 02:20:06 +0000:[INFO] [{SwapShareBackup}] [backup.AddUser:#728] - Response for user [testautouser1705630806] creation - [2024-01-19T02:20:07.453Z] 2024-01-19 02:20:06 +0000:[INFO] [{SwapShareBackup}] [backup.AddUser:#733] - User [testautouser1705630806] created [2024-01-19T02:20:09.967Z] 2024-01-19 02:20:09 +0000:[INFO] [{SwapShareBackup}] [backup.AddUser:#728] - Response for user [testautouser1705630809] creation - [2024-01-19T02:20:09.967Z] 2024-01-19 02:20:09 +0000:[INFO] [{SwapShareBackup}] [backup.AddUser:#733] - User [testautouser1705630809] created [2024-01-19T02:20:09.967Z] 2024-01-19 02:20:09 +0000:[INFO] [{SwapShareBackup}] [backup.glob..func97.2.2:#3993] - List of users are [testautouser1705630806 testautouser1705630809] [2024-01-19T02:20:09.967Z] 2024-01-19 02:20:09 +0000:[INFO] [{SwapShareBackup}] Creating cloud credentials and backup location for testautouser1705630806 and testautouser1705630809 [2024-01-19T02:20:10.223Z] 2024-01-19 02:20:10 +0000:[INFO] [{SwapShareBackup}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705630806 - cdf5be4f-94f1-4bef-a589-9dd552f9f51a [2024-01-19T02:20:11.589Z] 2024-01-19 02:20:11 +0000:[INFO] [{SwapShareBackup}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-1705630809] for org [default] with [aws] as provider [2024-01-19T02:20:11.589Z] 2024-01-19 02:20:11 +0000:[INFO] [{SwapShareBackup}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T02:20:11.589Z] 2024-01-19 02:20:11 +0000:[INFO] [{SwapShareBackup}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-1705630809] is present [2024-01-19T02:20:26.448Z] 2024-01-19 02:20:25 +0000:[INFO] [{SwapShareBackup}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705630809 - fece3bb5-c396-49ac-87bc-1327ed3700fb [2024-01-19T02:20:26.448Z] 2024-01-19 02:20:25 +0000:[INFO] [{SwapShareBackup}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-1705630825] for org [default] with [aws] as provider [2024-01-19T02:20:26.448Z] 2024-01-19 02:20:25 +0000:[INFO] [{SwapShareBackup}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T02:20:26.448Z] 2024-01-19 02:20:26 +0000:[INFO] [{SwapShareBackup}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-1705630825] is present [2024-01-19T02:20:41.318Z] 2024-01-19 02:20:38 +0000:[INFO] [{SwapShareBackup}] Registering Source and Destination clusters as user : testautouser1705630806 and verifying the status [2024-01-19T02:20:41.318Z] INFO[2024-01-19 02:20:38] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T02:20:41.318Z] INFO[2024-01-19 02:20:38] Actual:true, Expected: true [2024-01-19T02:20:41.319Z] 2024-01-19 02:20:38 +0000:[INFO] [{SwapShareBackup}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T02:20:41.319Z] 2024-01-19 02:20:38 +0000:[INFO] [{SwapShareBackup}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T02:20:41.319Z] 2024-01-19 02:20:38 +0000:[INFO] [{SwapShareBackup}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T02:20:41.319Z] 2024-01-19 02:20:38 +0000:[INFO] [{SwapShareBackup}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T02:20:41.319Z] 2024-01-19 02:20:38 +0000:[INFO] [{SwapShareBackup}] Create cluster [source-cluster] in org [default] [2024-01-19T02:20:41.319Z] 2024-01-19 02:20:38 +0000:[INFO] [{SwapShareBackup}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T02:20:41.319Z] 2024-01-19 02:20:38 +0000:[INFO] [{SwapShareBackup}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T02:20:41.319Z] 2024-01-19 02:20:38 +0000:[INFO] [{SwapShareBackup}] Create cluster [destination-cluster] in org [default] [2024-01-19T02:20:41.319Z] 2024-01-19 02:20:38 +0000:[INFO] [{SwapShareBackup}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T02:20:41.319Z] 2024-01-19 02:20:38 +0000:[INFO] [{SwapShareBackup}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T02:20:41.319Z] 2024-01-19 02:20:38 +0000:[INFO] [{SwapShareBackup}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-fwvrl] for IBM [2024-01-19T02:20:41.319Z] 2024-01-19 02:20:38 +0000:[INFO] [{SwapShareBackup}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-fwvrl] for org [default] with [ibm] as provider [2024-01-19T02:20:41.319Z] 2024-01-19 02:20:38 +0000:[INFO] [{SwapShareBackup}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T02:20:41.319Z] 2024-01-19 02:20:38 +0000:[INFO] [{SwapShareBackup}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T02:20:41.319Z] 2024-01-19 02:20:39 +0000:[INFO] [{SwapShareBackup}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-fwvrl] is present [2024-01-19T02:20:41.319Z] 2024-01-19 02:20:39 +0000:[INFO] [{SwapShareBackup}] Create cluster [source-cluster] in org [default] [2024-01-19T02:20:41.319Z] 2024-01-19 02:20:39 +0000:[INFO] [{SwapShareBackup}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-rvkhl] for IBM [2024-01-19T02:20:41.319Z] 2024-01-19 02:20:39 +0000:[INFO] [{SwapShareBackup}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-rvkhl] for org [default] with [ibm] as provider [2024-01-19T02:20:41.319Z] 2024-01-19 02:20:39 +0000:[INFO] [{SwapShareBackup}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T02:20:41.319Z] 2024-01-19 02:20:39 +0000:[INFO] [{SwapShareBackup}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T02:20:41.319Z] 2024-01-19 02:20:39 +0000:[INFO] [{SwapShareBackup}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-rvkhl] is present [2024-01-19T02:20:41.319Z] 2024-01-19 02:20:39 +0000:[INFO] [{SwapShareBackup}] Create cluster [destination-cluster] in org [default] [2024-01-19T02:20:41.319Z] INFO[2024-01-19 02:20:40] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T02:20:41.319Z] INFO[2024-01-19 02:20:40] Actual:Online, Expected: Online [2024-01-19T02:20:41.320Z] INFO[2024-01-19 02:20:40] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T02:20:41.320Z] INFO[2024-01-19 02:20:40] Actual:true, Expected: true [2024-01-19T02:20:41.320Z] 2024-01-19 02:20:40 +0000:[INFO] [{SwapShareBackup}] Taking backup of applications as user : testautouser1705630806 [2024-01-19T02:20:41.320Z] 2024-01-19 02:20:40 +0000:[INFO] [{SwapShareBackup}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T02:20:41.320Z] 2024-01-19 02:20:40 +0000:[INFO] [{SwapShareBackup}] Volume snapshot classes found in the cluster - [2024-01-19T02:20:41.320Z] 2024-01-19 02:20:40 +0000:[INFO] [{SwapShareBackup}] ibmc-vpcblock-snapshot [2024-01-19T02:20:41.320Z] 2024-01-19 02:20:40 +0000:[INFO] [{SwapShareBackup}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T02:20:41.320Z] 2024/01/19 02:20:41 DoRetryWithTimeout - Error: {backup status for [backup1-82940] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T02:21:13.405Z] 2024/01/19 02:21:11 DoRetryWithTimeout - Error: {backup status for [backup1-82940] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:21:45.442Z] 2024/01/19 02:21:41 DoRetryWithTimeout - Error: {backup status for [backup1-82940] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:22:11.946Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] [backup.CreateBackup:#313] - Backup [backup1-82940] created successfully [2024-01-19T02:22:11.946Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] Validating backup [backup1-82940] in org [default] [2024-01-19T02:22:11.946Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [backup1-82940] [2024-01-19T02:22:11.946Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82940-01-19-02h18m31s] in backup [backup1-82940] [2024-01-19T02:22:11.947Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82940-01-19-02h18m31s] in backup [backup1-82940] [2024-01-19T02:22:11.947Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82940-01-19-02h18m31s] [2024-01-19T02:22:11.947Z] 2024-01-19 02:22:11 +0000:[WARNING] [{SwapShareBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:22:11.947Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:22:11.947Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82940-01-19-02h18m31s] are [[mysql-data]] [2024-01-19T02:22:11.947Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T02:22:11.947Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] Validating CR cleanup [2024-01-19T02:22:11.947Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T02:22:11.947Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] All CRs in [mysql-ibm-pxb-0-82940-01-19-02h18m31s] are [[]] [2024-01-19T02:22:11.947Z] INFO[2024-01-19 02:22:11] Verifying : Description : Creation and Validation of backup [backup1-82940] [2024-01-19T02:22:11.947Z] INFO[2024-01-19 02:22:11] Actual:true, Expected: true [2024-01-19T02:22:11.947Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] [backup.glob..func97.2.5:#4046] - Backup uid of backup: backup1-82940 is 22076972-eabe-4d16-9690-64ebbdefcf9b for user testautouser1705630806 [2024-01-19T02:22:11.947Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] Registering Source and Destination clusters as user : testautouser1705630809 and verifying the status [2024-01-19T02:22:11.947Z] INFO[2024-01-19 02:22:11] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T02:22:11.947Z] INFO[2024-01-19 02:22:11] Actual:true, Expected: true [2024-01-19T02:22:11.947Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T02:22:11.947Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T02:22:11.947Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T02:22:11.947Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T02:22:11.947Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] Create cluster [source-cluster] in org [default] [2024-01-19T02:22:11.947Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T02:22:11.948Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T02:22:11.948Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] Create cluster [destination-cluster] in org [default] [2024-01-19T02:22:11.948Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T02:22:11.948Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T02:22:11.948Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-upfjy] for IBM [2024-01-19T02:22:11.948Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-upfjy] for org [default] with [ibm] as provider [2024-01-19T02:22:11.948Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T02:22:11.948Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T02:22:11.948Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-upfjy] is present [2024-01-19T02:22:11.948Z] 2024-01-19 02:22:11 +0000:[INFO] [{SwapShareBackup}] Create cluster [source-cluster] in org [default] [2024-01-19T02:22:12.511Z] 2024-01-19 02:22:12 +0000:[INFO] [{SwapShareBackup}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-gkjfw] for IBM [2024-01-19T02:22:12.511Z] 2024-01-19 02:22:12 +0000:[INFO] [{SwapShareBackup}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-gkjfw] for org [default] with [ibm] as provider [2024-01-19T02:22:12.511Z] 2024-01-19 02:22:12 +0000:[INFO] [{SwapShareBackup}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T02:22:12.511Z] 2024-01-19 02:22:12 +0000:[INFO] [{SwapShareBackup}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T02:22:12.511Z] 2024-01-19 02:22:12 +0000:[INFO] [{SwapShareBackup}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-gkjfw] is present [2024-01-19T02:22:12.511Z] 2024-01-19 02:22:12 +0000:[INFO] [{SwapShareBackup}] Create cluster [destination-cluster] in org [default] [2024-01-19T02:22:13.072Z] INFO[2024-01-19 02:22:13] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T02:22:13.072Z] INFO[2024-01-19 02:22:13] Actual:Online, Expected: Online [2024-01-19T02:22:13.072Z] INFO[2024-01-19 02:22:13] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T02:22:13.072Z] INFO[2024-01-19 02:22:13] Actual:true, Expected: true [2024-01-19T02:22:13.072Z] 2024-01-19 02:22:13 +0000:[INFO] [{SwapShareBackup}] Taking backup of applications as user : testautouser1705630809 [2024-01-19T02:22:13.329Z] 2024-01-19 02:22:13 +0000:[INFO] [{SwapShareBackup}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T02:22:13.329Z] 2024-01-19 02:22:13 +0000:[INFO] [{SwapShareBackup}] Volume snapshot classes found in the cluster - [2024-01-19T02:22:13.329Z] 2024-01-19 02:22:13 +0000:[INFO] [{SwapShareBackup}] ibmc-vpcblock-snapshot [2024-01-19T02:22:13.329Z] 2024-01-19 02:22:13 +0000:[INFO] [{SwapShareBackup}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T02:22:13.891Z] 2024/01/19 02:22:13 DoRetryWithTimeout - Error: {backup status for [backup1-82940] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T02:22:45.927Z] 2024/01/19 02:22:43 DoRetryWithTimeout - Error: {backup status for [backup1-82940] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:23:17.964Z] 2024-01-19 02:23:13 +0000:[INFO] [{SwapShareBackup}] [backup.CreateBackup:#313] - Backup [backup1-82940] created successfully [2024-01-19T02:23:17.964Z] 2024-01-19 02:23:13 +0000:[INFO] [{SwapShareBackup}] Validating backup [backup1-82940] in org [default] [2024-01-19T02:23:17.964Z] 2024-01-19 02:23:13 +0000:[INFO] [{SwapShareBackup}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [backup1-82940] [2024-01-19T02:23:17.964Z] 2024-01-19 02:23:13 +0000:[INFO] [{SwapShareBackup}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82940-01-19-02h18m31s] in backup [backup1-82940] [2024-01-19T02:23:17.964Z] 2024-01-19 02:23:13 +0000:[INFO] [{SwapShareBackup}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82940-01-19-02h18m31s] in backup [backup1-82940] [2024-01-19T02:23:17.964Z] 2024-01-19 02:23:13 +0000:[INFO] [{SwapShareBackup}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82940-01-19-02h18m31s] [2024-01-19T02:23:17.964Z] 2024-01-19 02:23:13 +0000:[WARNING] [{SwapShareBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:23:17.964Z] 2024-01-19 02:23:13 +0000:[INFO] [{SwapShareBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:23:17.964Z] 2024-01-19 02:23:13 +0000:[INFO] [{SwapShareBackup}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82940-01-19-02h18m31s] are [[mysql-data]] [2024-01-19T02:23:17.964Z] 2024-01-19 02:23:13 +0000:[INFO] [{SwapShareBackup}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T02:23:17.965Z] 2024-01-19 02:23:13 +0000:[INFO] [{SwapShareBackup}] Validating CR cleanup [2024-01-19T02:23:17.965Z] 2024-01-19 02:23:13 +0000:[INFO] [{SwapShareBackup}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T02:23:17.965Z] 2024-01-19 02:23:14 +0000:[INFO] [{SwapShareBackup}] All CRs in [mysql-ibm-pxb-0-82940-01-19-02h18m31s] are [[backup1-82940-fa19ead]] [2024-01-19T02:23:17.965Z] 2024-01-19 02:23:14 +0000:[INFO] [{SwapShareBackup}] CR found for [[backup1-82940-fa19ead]] under [mysql-ibm-pxb-0-82940-01-19-02h18m31s] namespace [2024-01-19T02:23:17.965Z] 2024/01/19 02:23:14 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup1-82940]}, Next try in [5s], timeout [5m0s] [2024-01-19T02:23:19.333Z] 2024-01-19 02:23:19 +0000:[INFO] [{SwapShareBackup}] All CRs in [mysql-ibm-pxb-0-82940-01-19-02h18m31s] are [[]] [2024-01-19T02:23:19.333Z] INFO[2024-01-19 02:23:19] Verifying : Description : Creation and Validation of backup [backup1-82940] [2024-01-19T02:23:19.333Z] INFO[2024-01-19 02:23:19] Actual:true, Expected: true [2024-01-19T02:23:19.333Z] 2024-01-19 02:23:19 +0000:[INFO] [{SwapShareBackup}] [backup.glob..func97.2.5:#4046] - Backup uid of backup: backup1-82940 is fa19eadd-0868-48b7-a737-4023715821d2 for user testautouser1705630809 [2024-01-19T02:23:19.333Z] 2024-01-19 02:23:19 +0000:[INFO] [{SwapShareBackup}] Share backup from testautouser1705630806 to testautouser1705630809 and validate [2024-01-19T02:23:19.333Z] 2024-01-19 02:23:19 +0000:[INFO] [{SwapShareBackup}] [backup.ShareBackup:#715] - Backup UID for backup1-82940 - 22076972-eabe-4d16-9690-64ebbdefcf9b [2024-01-19T02:23:19.604Z] 2024-01-19 02:23:19 +0000:[INFO] [{SwapShareBackup}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705630809 - fece3bb5-c396-49ac-87bc-1327ed3700fb [2024-01-19T02:23:19.860Z] INFO[2024-01-19 02:23:19] Verifying : Description : Validating the shared backup [backup1-82940] is present in user context [testautouser1705630809] [2024-01-19T02:23:19.860Z] INFO[2024-01-19 02:23:19] Actual:2, Expected: 2 [2024-01-19T02:23:19.860Z] 2024-01-19 02:23:19 +0000:[INFO] [{SwapShareBackup}] Restore the shared backup testautouser1705630809 with user context testautouser1705630806 [2024-01-19T02:23:19.860Z] 2024-01-19 02:23:19 +0000:[INFO] [{SwapShareBackup}] [backup.CreateRestoreWithUID:#1054] - Getting the UID of the backup needed to be restored [2024-01-19T02:23:20.422Z] 2024/01/19 02:23:20 DoRetryWithTimeout - Error: {restore status for [tp-restore-1705630999] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T02:23:52.460Z] 2024/01/19 02:23:50 DoRetryWithTimeout - Error: {restore status for [tp-restore-1705630999] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:24:24.510Z] 2024-01-19 02:24:20 +0000:[INFO] [{SwapShareBackup}] [backup.CreateRestoreWithUID:#1078] - Restore [tp-restore-1705630999] created successfully [2024-01-19T02:24:24.510Z] 2024-01-19 02:24:20 +0000:[INFO] [{SwapShareBackup}] Share backup from testautouser1705630809 to testautouser1705630806 and validate [2024-01-19T02:24:24.510Z] 2024-01-19 02:24:20 +0000:[INFO] [{SwapShareBackup}] [backup.ShareBackup:#715] - Backup UID for backup1-82940 - fa19eadd-0868-48b7-a737-4023715821d2 [2024-01-19T02:24:24.510Z] 2024-01-19 02:24:20 +0000:[INFO] [{SwapShareBackup}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705630806 - cdf5be4f-94f1-4bef-a589-9dd552f9f51a [2024-01-19T02:24:24.510Z] INFO[2024-01-19 02:24:20] Verifying : Description : Validating the shared backup [backup1-82940] is present in user context [testautouser1705630806] [2024-01-19T02:24:24.510Z] INFO[2024-01-19 02:24:20] Actual:2, Expected: 2 [2024-01-19T02:24:24.510Z] 2024-01-19 02:24:20 +0000:[INFO] [{SwapShareBackup}] Restore the shared backup testautouser1705630806 with user context testautouser1705630806 [2024-01-19T02:24:24.510Z] 2024-01-19 02:24:20 +0000:[INFO] [{SwapShareBackup}] [backup.CreateRestoreWithUID:#1054] - Getting the UID of the backup needed to be restored [2024-01-19T02:24:24.510Z] 2024/01/19 02:24:21 DoRetryWithTimeout - Error: {restore status for [tp-restore-1705631060] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T02:24:51.019Z] INFO[2024-01-19 02:24:46] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T02:24:51.019Z] INFO[2024-01-19 02:24:46] sched stats: tick interval: 39046 < 1.2s (100%) [2024-01-19T02:24:51.019Z] INFO[2024-01-19 02:24:46] sched stats: tick process durations: 39045 < 500ms (100%) [2024-01-19T02:24:51.587Z] 2024-01-19 02:24:51 +0000:[INFO] [{SwapShareBackup}] [backup.CreateRestoreWithUID:#1078] - Restore [tp-restore-1705631060] created successfully [2024-01-19T02:24:51.587Z] 2024-01-19 02:24:51 +0000:[INFO] [{SwapShareBackup}] Deleting the deployed apps after the testcase [2024-01-19T02:24:51.587Z] 2024-01-19 02:24:51 +0000:[INFO] [{SwapShareBackup}] Destroying apps [2024-01-19T02:24:51.587Z] 2024-01-19 02:24:51 +0000:[INFO] [{SwapShareBackup}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T02:24:51.587Z] 2024-01-19 02:24:51 +0000:[INFO] [{SwapShareBackup}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T02:24:51.587Z] 2024-01-19 02:24:51 +0000:[INFO] [{SwapShareBackup}] start destroying mysql-ibm app [2024-01-19T02:24:51.587Z] 2024-01-19 02:24:51 +0000:[INFO] [{SwapShareBackup}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T02:24:51.587Z] 2024-01-19 02:24:51 +0000:[INFO] [{SwapShareBackup}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T02:24:51.587Z] 2024-01-19 02:24:51 +0000:[WARNING] [{SwapShareBackup}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T02:24:51.587Z] INFO[2024-01-19 02:24:51] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T02:24:51.587Z] INFO[2024-01-19 02:24:51] Actual:true, Expected: true [2024-01-19T02:24:51.587Z] 2024-01-19 02:24:51 +0000:[INFO] [{SwapShareBackup}] Deleting all restores [2024-01-19T02:24:51.843Z] INFO[2024-01-19 02:24:51] Verifying : Description : Getting the backup driver [2024-01-19T02:24:51.843Z] INFO[2024-01-19 02:24:51] Actual:true, Expected: true [2024-01-19T02:24:51.843Z] INFO[2024-01-19 02:24:51] Verifying : Description : Verifying restore deletion - tp-restore-1705631060 [2024-01-19T02:24:51.843Z] INFO[2024-01-19 02:24:51] Actual:true, Expected: true [2024-01-19T02:24:51.843Z] INFO[2024-01-19 02:24:51] Verifying : Description : Getting the backup driver [2024-01-19T02:24:51.843Z] INFO[2024-01-19 02:24:51] Actual:true, Expected: true [2024-01-19T02:24:51.843Z] INFO[2024-01-19 02:24:51] Verifying : Description : Verifying restore deletion - tp-restore-1705630999 [2024-01-19T02:24:51.843Z] INFO[2024-01-19 02:24:51] Actual:true, Expected: true [2024-01-19T02:24:51.843Z] 2024-01-19 02:24:51 +0000:[INFO] [{SwapShareBackup}] Delete backups of users [2024-01-19T02:24:52.098Z] INFO[2024-01-19 02:24:51] Verifying : Description : Deleting backup - backup1-82940 of user testautouser1705630809 [2024-01-19T02:24:52.099Z] INFO[2024-01-19 02:24:51] Actual:true, Expected: true [2024-01-19T02:24:52.356Z] INFO[2024-01-19 02:24:52] Verifying : Description : Deleting backup - backup1-82940 of user testautouser1705630806 [2024-01-19T02:24:52.356Z] INFO[2024-01-19 02:24:52] Actual:true, Expected: true [2024-01-19T02:24:52.356Z] 2024/01/19 02:24:52 DoRetryWithTimeout - Error: {backupObject [backup1-82940] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T02:25:24.386Z] INFO[2024-01-19 02:25:22] Verifying : Description : Verifying backup backup1-82940 deletion for user - testautouser1705630809 [2024-01-19T02:25:24.386Z] INFO[2024-01-19 02:25:22] Actual:true, Expected: true [2024-01-19T02:25:24.386Z] INFO[2024-01-19 02:25:22] Verifying : Description : Verifying backup backup1-82940 deletion for user - testautouser1705630806 [2024-01-19T02:25:24.386Z] INFO[2024-01-19 02:25:22] Actual:true, Expected: true [2024-01-19T02:25:24.386Z] 2024-01-19 02:25:22 +0000:[INFO] [{SwapShareBackup}] [backup.glob..func97.3:#4136] - Generating user context [2024-01-19T02:25:24.386Z] 2024-01-19 02:25:22 +0000:[INFO] [{SwapShareBackup}] [backup.glob..func97.3:#4143] - Deleting registered clusters for non-admin context [2024-01-19T02:25:24.386Z] 2024-01-19 02:25:22 +0000:[INFO] [{SwapShareBackup}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T02:25:24.386Z] INFO[2024-01-19 02:25:22] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T02:25:24.386Z] INFO[2024-01-19 02:25:22] Actual:true, Expected: true [2024-01-19T02:25:24.386Z] INFO[2024-01-19 02:25:22] Verifying : Description : Deleting cluster source-cluster [2024-01-19T02:25:24.386Z] INFO[2024-01-19 02:25:22] Actual:true, Expected: true [2024-01-19T02:25:24.386Z] INFO[2024-01-19 02:25:23] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-fwvrl] [2024-01-19T02:25:24.386Z] INFO[2024-01-19 02:25:23] Actual:true, Expected: true [2024-01-19T02:25:24.386Z] INFO[2024-01-19 02:25:23] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T02:25:24.386Z] INFO[2024-01-19 02:25:23] Actual:true, Expected: true [2024-01-19T02:25:24.386Z] INFO[2024-01-19 02:25:24] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-rvkhl] [2024-01-19T02:25:24.386Z] INFO[2024-01-19 02:25:24] Actual:true, Expected: true [2024-01-19T02:25:24.386Z] 2024-01-19 02:25:24 +0000:[INFO] [{SwapShareBackup}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T02:25:24.386Z] INFO[2024-01-19 02:25:24] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T02:25:24.386Z] INFO[2024-01-19 02:25:24] Actual:true, Expected: true [2024-01-19T02:25:24.644Z] INFO[2024-01-19 02:25:24] Verifying : Description : Deleting cluster source-cluster [2024-01-19T02:25:24.644Z] INFO[2024-01-19 02:25:24] Actual:true, Expected: true [2024-01-19T02:25:26.012Z] INFO[2024-01-19 02:25:25] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-upfjy] [2024-01-19T02:25:26.012Z] INFO[2024-01-19 02:25:25] Actual:true, Expected: true [2024-01-19T02:25:26.267Z] INFO[2024-01-19 02:25:25] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T02:25:26.267Z] INFO[2024-01-19 02:25:25] Actual:true, Expected: true [2024-01-19T02:25:27.197Z] INFO[2024-01-19 02:25:27] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-gkjfw] [2024-01-19T02:25:27.197Z] INFO[2024-01-19 02:25:27] Actual:true, Expected: true [2024-01-19T02:25:27.761Z] 2024-01-19 02:25:27 +0000:[INFO] [{SwapShareBackup}] [backup.getAllBackupLocations:#1603] - The backup locations and their UID are map[421a2232-d30b-4bec-ad8d-b5ddc422ba5a:autogenerated-backup-location-1705630826] [2024-01-19T02:25:27.761Z] INFO[2024-01-19 02:25:27] Verifying : Description : Verifying fetching of all backup locations [2024-01-19T02:25:27.761Z] INFO[2024-01-19 02:25:27] Actual:true, Expected: true [2024-01-19T02:25:29.132Z] INFO[2024-01-19 02:25:28] Verifying : Description : Verifying backup location deletion - autogenerated-backup-location-1705630826 [2024-01-19T02:25:29.132Z] INFO[2024-01-19 02:25:28] Actual:true, Expected: true [2024-01-19T02:25:29.132Z] 2024-01-19 02:25:28 +0000:[INFO] [{SwapShareBackup}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705630826] is present [2024-01-19T02:25:29.132Z] 2024/01/19 02:25:28 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705630826 is not deleted yet}, Next try in [30s], timeout [1h0m0s] [2024-01-19T02:26:01.174Z] 2024-01-19 02:25:59 +0000:[INFO] [{SwapShareBackup}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705630826] is present [2024-01-19T02:26:01.174Z] 2024/01/19 02:25:59 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705630826 is not deleted yet}, Next try in [30s], timeout [1h0m0s] [2024-01-19T02:26:33.195Z] 2024-01-19 02:26:29 +0000:[INFO] [{SwapShareBackup}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [autogenerated-backup-location-1705630811] [2024-01-19T02:26:33.195Z] INFO[2024-01-19 02:26:29] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-1705630826 [2024-01-19T02:26:33.195Z] INFO[2024-01-19 02:26:29] Actual:true, Expected: true [2024-01-19T02:26:33.195Z] 2024-01-19 02:26:29 +0000:[INFO] [{SwapShareBackup}] [backup.getAllBackupLocations:#1603] - The backup locations and their UID are map[421a2232-d30b-4bec-ad8d-b5ddc422ba5a:autogenerated-backup-location-1705630811] [2024-01-19T02:26:33.195Z] INFO[2024-01-19 02:26:29] Verifying : Description : Verifying fetching of all backup locations [2024-01-19T02:26:33.195Z] INFO[2024-01-19 02:26:29] Actual:true, Expected: true [2024-01-19T02:26:33.195Z] INFO[2024-01-19 02:26:29] Verifying : Description : Verifying backup location deletion - autogenerated-backup-location-1705630811 [2024-01-19T02:26:33.195Z] INFO[2024-01-19 02:26:29] Actual:true, Expected: true [2024-01-19T02:26:33.195Z] 2024-01-19 02:26:29 +0000:[INFO] [{SwapShareBackup}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705630811] is present [2024-01-19T02:26:33.195Z] 2024/01/19 02:26:29 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705630811 is not deleted yet}, Next try in [30s], timeout [1h0m0s] [2024-01-19T02:26:59.711Z] 2024-01-19 02:26:59 +0000:[INFO] [{SwapShareBackup}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705630811] is present [2024-01-19T02:26:59.711Z] 2024/01/19 02:26:59 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705630811 is not deleted yet}, Next try in [30s], timeout [1h0m0s] [2024-01-19T02:27:31.750Z] 2024-01-19 02:27:29 +0000:[INFO] [{SwapShareBackup}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [] [2024-01-19T02:27:31.750Z] INFO[2024-01-19 02:27:29] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-1705630811 [2024-01-19T02:27:31.750Z] INFO[2024-01-19 02:27:29] Actual:true, Expected: true [2024-01-19T02:27:31.751Z] 2024-01-19 02:27:29 +0000:[INFO] [{SwapShareBackup}] [backup.getAllCloudCredentials:#1624] - The cloud credentials and their UID are map[071bda14-3299-4483-b630-c14951569f56:ibm-destination-config-cloud-cred-bknif 0c9455a6-d605-405b-845e-993a972b38c3:ibm-source-config-cloud-cred-ywkqe 1697e103-e83a-4c95-9ded-b41dc9c33867:ibm-destination-config-cloud-cred-eoxdv 1a56ee86-114b-4005-8334-14320beea8d7:ibm-source-config-cloud-cred-effqh 2afa9330-932c-49f2-aa69-31baca61b110:ibm-destination-config-cloud-cred-tnykx 400c89ab-c1e4-4fec-8856-9968bf6622b3:ibm-destination-config-cloud-cred-wlqhq 48fd2301-2573-49bd-bbe7-b8b20cc65d31:ibm-source-config-cloud-cred-vrmuf 4f8fa3dc-340f-4d11-be2f-31c9cacee9aa:ibm-source-config-cloud-cred-eymzd 52a9c0ab-362a-4d12-83a9-68083878bd71:ibm-source-config-cloud-cred-xoaah 5c6b1236-8d6c-4087-8034-27705179aa52:ibm-source-config-cloud-cred-gbiga 7c161891-6dbd-430b-9003-60e0555832d6:ibm-source-config-cloud-cred-dldli 7e619e3a-1834-4563-b6b5-5c6356601127:ibm-destination-config-cloud-cred-xejzw 85d80493-e444-450d-8e2e-b57b72a3f791:autogenerated-cred-mjwsl a0238056-dd69-42f2-ae5a-2f9e6cbbda5a:ibm-source-config-cloud-cred-kvjhs a1e6b817-24c9-4a38-9596-199a019ea67e:ibm-destination-config-cloud-cred-zqepo a2e242fb-97cc-40d5-92e8-ab560f98bb36:ibm-source-config-cloud-cred-tgism b446c8d6-6de7-4f2f-93fa-42dd0df7fd49:ibm-destination-config-cloud-cred-teqcf b95c050d-602e-40ce-957a-1c5e63895f3a:autogenerated-cred-1705630825 cd18a11c-b21b-4afb-b6fe-5651dc053d42:ibm-destination-config-cloud-cred-vhryz cf8514df-d3b4-43ab-91ee-246ca3c2399b:ibm-source-config-cloud-cred-lbtlt dde43afd-7d7a-4522-b8b3-cc48b9882884:ibm-destination-config-cloud-cred-exwrq ed4c5648-0804-4ce3-a8df-1a267ae72ee8:ibm-destination-config-cloud-cred-dksbt] [2024-01-19T02:27:31.751Z] INFO[2024-01-19 02:27:29] Verifying : Description : Verifying fetching of all cloud credentials [2024-01-19T02:27:31.751Z] INFO[2024-01-19 02:27:29] Actual:true, Expected: true [2024-01-19T02:27:31.751Z] INFO[2024-01-19 02:27:30] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-eymzd [2024-01-19T02:27:31.751Z] INFO[2024-01-19 02:27:30] Actual:true, Expected: true [2024-01-19T02:27:31.751Z] INFO[2024-01-19 02:27:31] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-wlqhq [2024-01-19T02:27:31.751Z] INFO[2024-01-19 02:27:31] Actual:true, Expected: true [2024-01-19T02:27:32.679Z] INFO[2024-01-19 02:27:32] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-vhryz [2024-01-19T02:27:32.679Z] INFO[2024-01-19 02:27:32] Actual:true, Expected: true [2024-01-19T02:27:34.047Z] INFO[2024-01-19 02:27:33] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-dldli [2024-01-19T02:27:34.047Z] INFO[2024-01-19 02:27:33] Actual:true, Expected: true [2024-01-19T02:27:35.410Z] 2024/01/19 02:27:35 DoRetryWithTimeout - Error: {deleting cloud cred autogenerated-cred-mjwsl failed - err}, Next try in [30s], timeout [5m0s] [2024-01-19T02:28:07.433Z] 2024/01/19 02:28:05 DoRetryWithTimeout - Error: {deleting cloud cred autogenerated-cred-mjwsl failed - err}, Next try in [30s], timeout [5m0s] [2024-01-19T02:28:39.464Z] 2024/01/19 02:28:35 DoRetryWithTimeout - Error: {deleting cloud cred autogenerated-cred-mjwsl failed - err}, Next try in [30s], timeout [5m0s] [2024-01-19T02:29:11.500Z] 2024/01/19 02:29:06 DoRetryWithTimeout - Error: {deleting cloud cred autogenerated-cred-mjwsl failed - err}, Next try in [30s], timeout [5m0s] [2024-01-19T02:29:37.992Z] 2024/01/19 02:29:36 DoRetryWithTimeout - Error: {deleting cloud cred autogenerated-cred-mjwsl failed - err}, Next try in [30s], timeout [5m0s] [2024-01-19T02:30:10.044Z] 2024/01/19 02:30:06 DoRetryWithTimeout - Error: {deleting cloud cred autogenerated-cred-mjwsl failed - err}, Next try in [30s], timeout [5m0s] [2024-01-19T02:30:42.076Z] 2024/01/19 02:30:36 DoRetryWithTimeout - Error: {deleting cloud cred autogenerated-cred-mjwsl failed - err}, Next try in [30s], timeout [5m0s] [2024-01-19T02:31:08.569Z] 2024/01/19 02:31:07 DoRetryWithTimeout - Error: {deleting cloud cred autogenerated-cred-mjwsl failed - err}, Next try in [30s], timeout [5m0s] [2024-01-19T02:31:40.593Z] 2024/01/19 02:31:37 DoRetryWithTimeout - Error: {deleting cloud cred autogenerated-cred-mjwsl failed - err}, Next try in [30s], timeout [5m0s] [2024-01-19T02:32:12.614Z] 2024/01/19 02:32:07 DoRetryWithTimeout - Error: {deleting cloud cred autogenerated-cred-mjwsl failed - err}, Next try in [30s], timeout [5m0s] [2024-01-19T02:32:39.109Z] INFO[2024-01-19 02:32:37] Verifying : Description : Deleting cloud cred autogenerated-cred-mjwsl [2024-01-19T02:32:39.109Z] ERRO[2024-01-19 02:32:37] Actual:timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {deleting cloud cred autogenerated-cred-mjwsl failed - err} [2024-01-19T02:32:39.109Z] {deleting cloud cred autogenerated-cred-mjwsl failed - err} [2024-01-19T02:32:39.109Z] {deleting cloud cred autogenerated-cred-mjwsl failed - err} [2024-01-19T02:32:39.109Z] {deleting cloud cred autogenerated-cred-mjwsl failed - err} [2024-01-19T02:32:39.109Z] {deleting cloud cred autogenerated-cred-mjwsl failed - err} [2024-01-19T02:32:39.109Z] {deleting cloud cred autogenerated-cred-mjwsl failed - err} [2024-01-19T02:32:39.109Z] {deleting cloud cred autogenerated-cred-mjwsl failed - err} [2024-01-19T02:32:39.109Z] {deleting cloud cred autogenerated-cred-mjwsl failed - err} [2024-01-19T02:32:39.109Z] {deleting cloud cred autogenerated-cred-mjwsl failed - err} [2024-01-19T02:32:39.109Z] {deleting cloud cred autogenerated-cred-mjwsl failed - err}, Expected:  [2024-01-19T02:32:39.109Z] INFO[2024-01-19 02:32:38] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-lbtlt [2024-01-19T02:32:39.109Z] INFO[2024-01-19 02:32:38] Actual:true, Expected: true [2024-01-19T02:32:39.109Z] INFO[2024-01-19 02:32:38] Verifying : Description : Deleting cloud cred autogenerated-cred-1705630825 [2024-01-19T02:32:39.109Z] INFO[2024-01-19 02:32:38] Actual:true, Expected: true [2024-01-19T02:32:40.481Z] INFO[2024-01-19 02:32:40] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-gbiga [2024-01-19T02:32:40.481Z] INFO[2024-01-19 02:32:40] Actual:true, Expected: true [2024-01-19T02:32:41.846Z] INFO[2024-01-19 02:32:41] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-kvjhs [2024-01-19T02:32:41.846Z] INFO[2024-01-19 02:32:41] Actual:true, Expected: true [2024-01-19T02:32:43.211Z] INFO[2024-01-19 02:32:43] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-eoxdv [2024-01-19T02:32:43.211Z] INFO[2024-01-19 02:32:43] Actual:true, Expected: true [2024-01-19T02:32:44.574Z] INFO[2024-01-19 02:32:44] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-dksbt [2024-01-19T02:32:44.574Z] INFO[2024-01-19 02:32:44] Actual:true, Expected: true [2024-01-19T02:32:45.937Z] INFO[2024-01-19 02:32:45] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-bknif [2024-01-19T02:32:45.937Z] INFO[2024-01-19 02:32:45] Actual:true, Expected: true [2024-01-19T02:32:47.304Z] INFO[2024-01-19 02:32:47] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-ywkqe [2024-01-19T02:32:47.304Z] INFO[2024-01-19 02:32:47] Actual:true, Expected: true [2024-01-19T02:32:48.669Z] INFO[2024-01-19 02:32:48] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-zqepo [2024-01-19T02:32:48.669Z] INFO[2024-01-19 02:32:48] Actual:true, Expected: true [2024-01-19T02:32:50.567Z] INFO[2024-01-19 02:32:50] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-xoaah [2024-01-19T02:32:50.567Z] INFO[2024-01-19 02:32:50] Actual:true, Expected: true [2024-01-19T02:32:51.496Z] INFO[2024-01-19 02:32:51] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-exwrq [2024-01-19T02:32:51.496Z] INFO[2024-01-19 02:32:51] Actual:true, Expected: true [2024-01-19T02:32:52.871Z] INFO[2024-01-19 02:32:52] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-xejzw [2024-01-19T02:32:52.871Z] INFO[2024-01-19 02:32:52] Actual:true, Expected: true [2024-01-19T02:32:54.763Z] INFO[2024-01-19 02:32:54] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-tgism [2024-01-19T02:32:54.763Z] INFO[2024-01-19 02:32:54] Actual:true, Expected: true [2024-01-19T02:32:55.693Z] INFO[2024-01-19 02:32:55] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-teqcf [2024-01-19T02:32:55.693Z] INFO[2024-01-19 02:32:55] Actual:true, Expected: true [2024-01-19T02:32:57.588Z] INFO[2024-01-19 02:32:57] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-vrmuf [2024-01-19T02:32:57.588Z] INFO[2024-01-19 02:32:57] Actual:true, Expected: true [2024-01-19T02:32:58.520Z] INFO[2024-01-19 02:32:58] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-tnykx [2024-01-19T02:32:58.520Z] INFO[2024-01-19 02:32:58] Actual:true, Expected: true [2024-01-19T02:32:59.887Z] INFO[2024-01-19 02:32:59] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-effqh [2024-01-19T02:32:59.887Z] INFO[2024-01-19 02:32:59] Actual:true, Expected: true [2024-01-19T02:33:00.451Z] 2024-01-19 02:33:00 +0000:[INFO] [{SwapShareBackup}] [backup.getAllCloudCredentials:#1624] - The cloud credentials and their UID are map[85d80493-e444-450d-8e2e-b57b72a3f791:autogenerated-cred-mjwsl b95c050d-602e-40ce-957a-1c5e63895f3a:autogenerated-cred-1705630809] [2024-01-19T02:33:00.451Z] INFO[2024-01-19 02:33:00] Verifying : Description : Verifying fetching of all cloud credentials [2024-01-19T02:33:00.451Z] INFO[2024-01-19 02:33:00] Actual:true, Expected: true [2024-01-19T02:33:01.844Z] INFO[2024-01-19 02:33:01] Verifying : Description : Deleting cloud cred autogenerated-cred-1705630809 [2024-01-19T02:33:01.844Z] INFO[2024-01-19 02:33:01] Actual:true, Expected: true [2024-01-19T02:33:03.208Z] INFO[2024-01-19 02:33:03] Verifying : Description : Deleting cloud cred autogenerated-cred-mjwsl [2024-01-19T02:33:03.208Z] INFO[2024-01-19 02:33:03] Actual:true, Expected: true [2024-01-19T02:33:03.208Z] 2024-01-19 02:33:03 +0000:[INFO] [{SwapShareBackup}] [backup.glob..func97.3:#4195] - Cleaning up users [2024-01-19T02:33:05.720Z] 2024-01-19 02:33:05 +0000:[INFO] [{SwapShareBackup}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705630809 - fece3bb5-c396-49ac-87bc-1327ed3700fb [2024-01-19T02:33:05.720Z] 2024-01-19 02:33:05 +0000:[INFO] [{SwapShareBackup}] [backup.FetchIDOfUser:#496] - Fetching ID of user testautouser1705630806 - cdf5be4f-94f1-4bef-a589-9dd552f9f51a [2024-01-19T02:33:06.290Z] 2024-01-19 02:33:06 +0000:[INFO] [{SwapShareBackup}] [backup.DeleteUser:#755] - Response for user [testautouser1705630809] deletion - [2024-01-19T02:33:06.290Z] 2024-01-19 02:33:06 +0000:[INFO] [{SwapShareBackup}] [backup.DeleteUser:#759] - Deleted User - testautouser1705630809 [2024-01-19T02:33:06.290Z] 2024-01-19 02:33:06 +0000:[INFO] [{SwapShareBackup}] [backup.DeleteUser:#755] - Response for user [testautouser1705630806] deletion - [2024-01-19T02:33:06.290Z] 2024-01-19 02:33:06 +0000:[INFO] [{SwapShareBackup}] [backup.DeleteUser:#759] - Deleted User - testautouser1705630806 [2024-01-19T02:33:06.290Z] INFO[2024-01-19 02:33:06] --------Test End------ [2024-01-19T02:33:06.290Z] INFO[2024-01-19 02:33:06] #Test: [2024-01-19T02:33:06.290Z] INFO[2024-01-19 02:33:06] #Description: [2024-01-19T02:33:06.290Z] INFO[2024-01-19 02:33:06] ------------------------ [2024-01-19T02:33:06.546Z] 2024-01-19 02:33:06 +0000:[INFO] [{SwapShareBackup}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-82940-01-19-02h18m31s] [2024-01-19T02:33:06.805Z] 2024-01-19 02:33:06 +0000:[INFO] [{SwapShareBackup}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T02:33:06.805Z] 2024-01-19 02:33:06 +0000:[INFO] [{SwapShareBackup}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T02:33:07.061Z] 2024-01-19 02:33:06 +0000:[INFO] [{SwapShareBackup}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-82940-01-19-02h18m31s] [2024-01-19T02:33:07.061Z] 2024-01-19 02:33:06 +0000:[INFO] [{SwapShareBackup}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {SwapShareBackup} Share the backup with same name [2024-01-19T02:33:07.061Z] 2024-01-19 02:33:06 +0000:[INFO] [{SwapShareBackup}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T02:33:07.061Z] 2024/01/19 02:33:06 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T02:33:12.308Z] 2024/01/19 02:33:11 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T02:33:17.554Z] 2024/01/19 02:33:16 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T02:33:22.810Z] 2024/01/19 02:33:21 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T02:33:26.980Z] 2024/01/19 02:33:26 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T02:33:32.227Z] 2024/01/19 02:33:31 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T02:33:37.472Z] 2024/01/19 02:33:36 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T02:33:42.745Z] 2024/01/19 02:33:41 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T02:33:47.990Z] 2024/01/19 02:33:46 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.0.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.8 10.240.0.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T02:33:47.990Z] {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T02:33:47.990Z] {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T02:33:47.990Z] {dial tcp 10.240.0.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T02:33:52.158Z] 2024-01-19 02:33:51 +0000:[WARNING] [{SwapShareBackup}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.8 10.240.0.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T02:33:52.158Z] {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T02:33:52.158Z] {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T02:33:52.158Z] {dial tcp 10.240.0.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T02:33:52.159Z] 2024-01-19 02:33:51 +0000:[ERROR] [{SwapShareBackup}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.8 10.240.0.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T02:33:52.159Z] {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T02:33:52.159Z] {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T02:33:52.159Z] {dial tcp 10.240.0.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T02:33:52.159Z] 2024-01-19 02:33:51 +0000:[INFO] [{SwapShareBackup}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T02:33:52.159Z] 2024-01-19 02:33:51 +0000:[INFO] [{SwapShareBackup}] Switching context to [/tmp/source-config] [2024-01-19T02:33:52.159Z] 2024-01-19 02:33:52 +0000:[INFO] [{SwapShareBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T02:33:52.159Z] 2024-01-19 02:33:52 +0000:[INFO] [{SwapShareBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T02:33:52.159Z] 2024-01-19 02:33:52 +0000:[INFO] [{SwapShareBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T02:33:52.418Z] 2024-01-19 02:33:52 +0000:[INFO] [{SwapShareBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T02:33:52.418Z] 2024-01-19 02:33:52 +0000:[INFO] [{SwapShareBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T02:33:52.418Z] 2024-01-19 02:33:52 +0000:[INFO] [{SwapShareBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T02:33:52.981Z] 2024-01-19 02:33:52 +0000:[INFO] [{SwapShareBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T02:33:53.237Z] 2024-01-19 02:33:53 +0000:[INFO] [{SwapShareBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T02:33:53.802Z] 2024-01-19 02:33:53 +0000:[INFO] [{SwapShareBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T02:33:54.059Z] 2024/01/19 02:33:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:34:04.001Z] 2024/01/19 02:34:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:34:13.945Z] 2024/01/19 02:34:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:34:26.113Z] 2024/01/19 02:34:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:34:34.193Z] 2024/01/19 02:34:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:34:44.165Z] 2024/01/19 02:34:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:34:46.682Z] INFO[2024-01-19 02:34:46] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T02:34:46.682Z] INFO[2024-01-19 02:34:46] sched stats: tick interval: 39646 < 1.2s (100%) [2024-01-19T02:34:46.682Z] INFO[2024-01-19 02:34:46] sched stats: tick process durations: 39645 < 500ms (100%) [2024-01-19T02:34:54.770Z] 2024/01/19 02:34:54 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:35:04.716Z] 2024/01/19 02:35:04 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:35:14.667Z] 2024/01/19 02:35:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:35:24.803Z] 2024/01/19 02:35:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:35:34.745Z] 2024/01/19 02:35:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:35:44.685Z] 2024/01/19 02:35:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:35:54.621Z] 2024/01/19 02:35:54 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:36:04.597Z] 2024/01/19 02:36:04 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:36:14.553Z] 2024/01/19 02:36:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:36:26.721Z] 2024/01/19 02:36:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:36:34.812Z] 2024/01/19 02:36:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:36:44.760Z] 2024/01/19 02:36:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:36:54.704Z] 2024/01/19 02:36:54 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:37:04.651Z] 2024/01/19 02:37:04 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:37:14.597Z] 2024/01/19 02:37:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:37:24.536Z] 2024/01/19 02:37:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:37:36.711Z] 2024/01/19 02:37:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:37:44.797Z] 2024/01/19 02:37:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:37:54.741Z] 2024/01/19 02:37:54 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:38:04.684Z] 2024/01/19 02:38:04 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:38:16.856Z] 2024/01/19 02:38:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:38:24.948Z] 2024/01/19 02:38:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:38:34.895Z] 2024/01/19 02:38:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:38:44.838Z] 2024/01/19 02:38:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:38:54.785Z] 2024-01-19 02:38:54 +0000:[ERROR] [{SwapShareBackup}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.785Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.785Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.785Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.785Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.785Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.785Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.785Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.785Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.785Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.785Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.785Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.785Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.785Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.785Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.786Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.786Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.786Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.786Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.786Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.786Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.786Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.786Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.786Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.786Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.786Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.786Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.786Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.786Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.786Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T02:38:54.786Z] • Failure [1223.028 seconds] [2024-01-19T02:38:54.786Z] {SwapShareBackup} [2024-01-19T02:38:54.786Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:3932 [2024-01-19T02:38:54.786Z] Share the backup with same name [] [2024-01-19T02:38:54.786Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:3968 [2024-01-19T02:38:54.786Z] [2024-01-19T02:38:54.786Z] Unexpected error: [2024-01-19T02:38:54.786Z] <*errors.errorString | 0xc000fb0680>: { [2024-01-19T02:38:54.787Z] s: "failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{No... [2024-01-19T02:38:54.787Z] [2024-01-19T02:38:54.787Z] Gomega truncated this representation as it exceeds 'format.MaxLength'. [2024-01-19T02:38:54.787Z] Consider having the object provide a custom 'GomegaStringer' representation [2024-01-19T02:38:54.787Z] or adjust the parameters in Gomega's 'format' package. [2024-01-19T02:38:54.787Z] [2024-01-19T02:38:54.787Z] Learn more here: https://onsi.github.io/gomega/#adjusting-output [2024-01-19T02:38:54.787Z] [2024-01-19T02:38:54.787Z] failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.787Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.788Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.788Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.788Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.788Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.788Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.788Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.788Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.788Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.788Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.788Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.788Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.788Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.788Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.788Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.788Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.788Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.788Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.788Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.788Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.788Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.788Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.788Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.789Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.789Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.789Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.789Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.789Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:38:54.789Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T02:38:54.789Z] occurred [2024-01-19T02:38:54.789Z] [2024-01-19T02:38:54.789Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T02:38:54.789Z] [2024-01-19T02:38:54.789Z] Full Stack Trace [2024-01-19T02:38:54.789Z] github.com/portworx/torpedo/pkg/log.FailOnError({0x88a5060, 0xc000fb0680}, {0x7f74bfd?, 0x6cc0320?}, {0x0?, 0x6cc0320?, 0xc000e002f0?}) [2024-01-19T02:38:54.789Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 +0x290 [2024-01-19T02:38:54.789Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest.func1() [2024-01-19T02:38:54.789Z] /go/src/github.com/portworx/torpedo/tests/common.go:7283 +0x32 [2024-01-19T02:38:54.789Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest({0xc000015250, 0x1, 0x1}) [2024-01-19T02:38:54.789Z] /go/src/github.com/portworx/torpedo/tests/common.go:7299 +0x778 [2024-01-19T02:38:54.789Z] github.com/portworx/torpedo/tests/backup.glob..func97.3() [2024-01-19T02:38:54.789Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:4206 +0x690 [2024-01-19T02:38:54.789Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T02:38:54.789Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T02:38:54.789Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T02:38:54.789Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T02:38:54.789Z] created by testing.(*T).Run [2024-01-19T02:38:54.789Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T02:38:54.789Z] ------------------------------ [2024-01-19T02:38:54.789Z] STEP: schedule applications [2024-01-19T02:38:54.789Z] STEP: Validate applications [2024-01-19T02:38:54.789Z] STEP: validate applications [2024-01-19T02:38:54.789Z] STEP: validate mysql-ibm app's volumes [2024-01-19T02:38:54.789Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T02:38:54.789Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T02:38:54.789Z] STEP: get mysql-ibm app's volume: pvc-3156411e-ffcd-4041-ade2-f766358d6fc9 inspected by the volume driver [2024-01-19T02:38:54.789Z] STEP: wait for mysql-ibm app to start running [2024-01-19T02:38:54.789Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T02:38:54.789Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T02:38:54.789Z] STEP: Validate Px pod restart count [2024-01-19T02:38:54.789Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T02:38:54.789Z] STEP: Create Users [2024-01-19T02:38:54.789Z] STEP: Adding Credentials and Registering Backup Location for testautouser1705630806 and testautouser1705630809 [2024-01-19T02:38:54.789Z] STEP: Register source and destination cluster for backup on testautouser1705630806 [2024-01-19T02:38:54.789Z] STEP: Taking backup of applications as testautouser1705630806 [2024-01-19T02:38:54.789Z] STEP: Register source and destination cluster for backup on testautouser1705630809 [2024-01-19T02:38:54.789Z] STEP: Taking backup of applications as testautouser1705630809 [2024-01-19T02:38:54.789Z] STEP: Share backup with testautouser1705630809 [2024-01-19T02:38:54.789Z] STEP: validate the backup shared backup1-82940 is present in user context testautouser1705630809 [2024-01-19T02:38:54.789Z] STEP: Restore the shared backup backup1-82940 with user context testautouser1705630809 [2024-01-19T02:38:54.789Z] STEP: Share backup with testautouser1705630806 [2024-01-19T02:38:54.789Z] STEP: validate the backup shared backup1-82940 is present in user context testautouser1705630806 [2024-01-19T02:38:54.789Z] STEP: Restore the shared backup backup1-82940 with user context testautouser1705630806 [2024-01-19T02:38:54.790Z] STEP: destroy apps [2024-01-19T02:38:54.790Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T02:38:54.790Z] STEP: start destroying mysql-ibm app [2024-01-19T02:38:54.790Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T02:38:54.790Z] S2024-01-19 02:38:54 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Log Dir: /testresults//ShareAndRemoveBackupLocation.log [2024-01-19T02:38:54.790Z] INFO[2024-01-19 02:38:54] --------Test Start------ [2024-01-19T02:38:54.790Z] INFO[2024-01-19 02:38:54] #Test: ShareAndRemoveBackupLocation [2024-01-19T02:38:54.790Z] INFO[2024-01-19 02:38:54] #Description: Share and remove backup location and add it back and check from other users if they show up [2024-01-19T02:38:54.790Z] INFO[2024-01-19 02:38:54] ------------------------ [2024-01-19T02:38:54.790Z] 2024-01-19 02:38:54 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.glob..func92.1:#2946] - Deploy applications needed for backup [2024-01-19T02:38:54.790Z] 2024-01-19 02:38:54 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T02:38:54.790Z] 2024-01-19 02:38:54 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T02:38:54.790Z] 2024-01-19 02:38:54 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T02:38:55.045Z] 2024-01-19 02:38:54 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T02:38:55.045Z] 2024-01-19 02:38:54 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T02:38:55.046Z] 2024-01-19 02:38:54 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T02:38:55.046Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T02:38:55.046Z] 2024-01-19 02:38:54 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T02:38:55.047Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T02:38:55.047Z] 2024-01-19 02:38:54 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T02:38:55.303Z] 2024-01-19 02:38:55 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.glob..func92.2.1:#2964] - Validate applications [2024-01-19T02:38:55.304Z] 2024-01-19 02:38:55 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validate applications [2024-01-19T02:38:55.304Z] 2024-01-19 02:38:55 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating mysql-ibm app [2024-01-19T02:38:55.304Z] 2024-01-19 02:38:55 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating mysql-ibm app's volumes [2024-01-19T02:38:55.304Z] 2024-01-19 02:38:55 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:38:55.304Z] 2024/01/19 02:38:55 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T02:39:05.246Z] 2024/01/19 02:39:05 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T02:39:17.415Z] 2024-01-19 02:39:15 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-82949-01-19-02h38m54s [2024-01-19T02:39:17.415Z] 2024-01-19 02:39:15 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T02:39:17.415Z] 2024-01-19 02:39:15 +0000:[WARNING] [{ShareAndRemoveBackupLocation}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T02:39:17.415Z] 2024-01-19 02:39:15 +0000:[INFO] [{ShareAndRemoveBackupLocation}] wait for mysql-ibm app to start running [2024-01-19T02:39:17.415Z] 2024-01-19 02:39:15 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T02:39:17.415Z] 2024/01/19 02:39:15 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:39:17.415Z] pod name:mysql-7d766cd65c-w7xv6 namespace:mysql-ibm-pxb-0-82949-01-19-02h38m54s running:false ready:false node:10.240.64.8 [2024-01-19T02:39:17.415Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:39:27.357Z] 2024/01/19 02:39:25 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:39:27.357Z] pod name:mysql-7d766cd65c-w7xv6 namespace:mysql-ibm-pxb-0-82949-01-19-02h38m54s running:true ready:false node:10.240.64.8 [2024-01-19T02:39:27.357Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:39:37.298Z] 2024/01/19 02:39:35 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:39:37.298Z] pod name:mysql-7d766cd65c-w7xv6 namespace:mysql-ibm-pxb-0-82949-01-19-02h38m54s running:true ready:false node:10.240.64.8 [2024-01-19T02:39:37.299Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:39:47.245Z] 2024/01/19 02:39:45 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:39:47.245Z] pod name:mysql-7d766cd65c-w7xv6 namespace:mysql-ibm-pxb-0-82949-01-19-02h38m54s running:true ready:false node:10.240.64.8 [2024-01-19T02:39:47.245Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:39:57.199Z] 2024/01/19 02:39:55 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:39:57.199Z] pod name:mysql-7d766cd65c-w7xv6 namespace:mysql-ibm-pxb-0-82949-01-19-02h38m54s running:true ready:false node:10.240.64.8 [2024-01-19T02:39:57.199Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:40:07.151Z] 2024/01/19 02:40:05 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:40:07.151Z] pod name:mysql-7d766cd65c-w7xv6 namespace:mysql-ibm-pxb-0-82949-01-19-02h38m54s running:true ready:false node:10.240.64.8 [2024-01-19T02:40:07.151Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:40:17.096Z] 2024/01/19 02:40:16 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:40:17.096Z] pod name:mysql-7d766cd65c-w7xv6 namespace:mysql-ibm-pxb-0-82949-01-19-02h38m54s running:true ready:false node:10.240.64.8 [2024-01-19T02:40:17.096Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:40:27.059Z] 2024/01/19 02:40:26 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:40:27.059Z] pod name:mysql-7d766cd65c-w7xv6 namespace:mysql-ibm-pxb-0-82949-01-19-02h38m54s running:true ready:false node:10.240.64.8 [2024-01-19T02:40:27.059Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:40:36.999Z] 2024-01-19 02:40:36 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T02:40:36.999Z] 2024-01-19 02:40:36 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T02:40:37.000Z] 2024-01-19 02:40:36 +0000:[INFO] [{ShareAndRemoveBackupLocation}] validate if mysql-ibm app's volumes are setup [2024-01-19T02:40:37.000Z] 2024-01-19 02:40:36 +0000:[WARNING] [{ShareAndRemoveBackupLocation}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:40:37.000Z] 2024-01-19 02:40:36 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:40:37.000Z] 2024-01-19 02:40:36 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T02:40:37.000Z] 2024-01-19 02:40:36 +0000:[WARNING] [{ShareAndRemoveBackupLocation}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T02:40:37.000Z] 2024-01-19 02:40:36 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Creating 3 users [2024-01-19T02:40:37.000Z] 2024-01-19 02:40:36 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Creating 3 users [2024-01-19T02:40:37.561Z] 2024-01-19 02:40:37 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.AddUser:#728] - Response for user [testuser3-1705632036] creation - [2024-01-19T02:40:37.561Z] 2024-01-19 02:40:37 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.AddUser:#733] - User [testuser3-1705632036] created [2024-01-19T02:40:37.561Z] INFO[2024-01-19 02:40:37] Verifying : Description : Creating user - testuser3-1705632036 [2024-01-19T02:40:37.561Z] INFO[2024-01-19 02:40:37] Actual:true, Expected: true [2024-01-19T02:40:37.561Z] 2024-01-19 02:40:37 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.AddUser:#728] - Response for user [testuser2-1705632036] creation - [2024-01-19T02:40:37.561Z] 2024-01-19 02:40:37 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.AddUser:#733] - User [testuser2-1705632036] created [2024-01-19T02:40:37.561Z] INFO[2024-01-19 02:40:37] Verifying : Description : Creating user - testuser2-1705632036 [2024-01-19T02:40:37.561Z] INFO[2024-01-19 02:40:37] Actual:true, Expected: true [2024-01-19T02:40:37.816Z] 2024-01-19 02:40:37 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.AddUser:#728] - Response for user [testuser1-1705632036] creation - [2024-01-19T02:40:37.816Z] 2024-01-19 02:40:37 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.AddUser:#733] - User [testuser1-1705632036] created [2024-01-19T02:40:37.816Z] INFO[2024-01-19 02:40:37] Verifying : Description : Creating user - testuser1-1705632036 [2024-01-19T02:40:37.816Z] INFO[2024-01-19 02:40:37] Actual:true, Expected: true [2024-01-19T02:40:37.816Z] 2024-01-19 02:40:37 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.glob..func92.2.2:#2971] - Created 3 users and users list is [testuser3-1705632036 testuser2-1705632036 testuser1-1705632036] [2024-01-19T02:40:37.816Z] 2024-01-19 02:40:37 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Creating backup location and cloud setting [2024-01-19T02:40:37.816Z] 2024-01-19 02:40:37 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [cred-aws-1705632037] for org [default] with [aws] as provider [2024-01-19T02:40:37.816Z] 2024-01-19 02:40:37 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T02:40:37.816Z] 2024-01-19 02:40:37 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cred-aws-1705632037] is present [2024-01-19T02:40:37.816Z] INFO[2024-01-19 02:40:37] Verifying : Description : Verifying creation of cloud credential named [cred-aws-1705632037] for org [default] with [aws] as provider [2024-01-19T02:40:37.816Z] INFO[2024-01-19 02:40:37] Actual:true, Expected: true [2024-01-19T02:40:41.084Z] INFO[2024-01-19 02:40:40] Verifying : Description : Creating backup location autogenerated-backup-location-1705632037 [2024-01-19T02:40:41.084Z] INFO[2024-01-19 02:40:40] Actual:true, Expected: true [2024-01-19T02:40:41.084Z] 2024-01-19 02:40:40 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Register source and destination cluster for backup [2024-01-19T02:40:41.084Z] INFO[2024-01-19 02:40:40] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T02:40:41.084Z] INFO[2024-01-19 02:40:40] Actual:true, Expected: true [2024-01-19T02:40:41.084Z] 2024-01-19 02:40:40 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T02:40:41.084Z] 2024-01-19 02:40:40 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T02:40:41.085Z] 2024-01-19 02:40:40 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T02:40:41.085Z] 2024-01-19 02:40:40 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T02:40:41.085Z] 2024-01-19 02:40:40 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Create cluster [source-cluster] in org [default] [2024-01-19T02:40:41.085Z] 2024-01-19 02:40:40 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T02:40:41.085Z] 2024-01-19 02:40:40 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T02:40:41.085Z] 2024-01-19 02:40:40 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Create cluster [destination-cluster] in org [default] [2024-01-19T02:40:41.085Z] 2024-01-19 02:40:40 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T02:40:41.085Z] 2024-01-19 02:40:40 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T02:40:41.085Z] 2024-01-19 02:40:40 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-dczfv] for IBM [2024-01-19T02:40:41.085Z] 2024-01-19 02:40:40 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-dczfv] for org [default] with [ibm] as provider [2024-01-19T02:40:41.085Z] 2024-01-19 02:40:40 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T02:40:41.085Z] 2024-01-19 02:40:40 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T02:40:41.342Z] 2024-01-19 02:40:41 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-dczfv] is present [2024-01-19T02:40:41.342Z] 2024-01-19 02:40:41 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Create cluster [source-cluster] in org [default] [2024-01-19T02:40:41.909Z] 2024-01-19 02:40:41 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-gyeyl] for IBM [2024-01-19T02:40:41.909Z] 2024-01-19 02:40:41 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-gyeyl] for org [default] with [ibm] as provider [2024-01-19T02:40:41.909Z] 2024-01-19 02:40:41 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T02:40:41.909Z] 2024-01-19 02:40:41 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T02:40:41.909Z] 2024-01-19 02:40:41 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-gyeyl] is present [2024-01-19T02:40:41.909Z] 2024-01-19 02:40:41 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Create cluster [destination-cluster] in org [default] [2024-01-19T02:40:41.909Z] INFO[2024-01-19 02:40:41] Verifying : Description : Creating source and destination cluster [2024-01-19T02:40:41.909Z] INFO[2024-01-19 02:40:41] Actual:true, Expected: true [2024-01-19T02:40:41.909Z] INFO[2024-01-19 02:40:41] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T02:40:41.909Z] INFO[2024-01-19 02:40:41] Actual:Online, Expected: Online [2024-01-19T02:40:41.909Z] INFO[2024-01-19 02:40:41] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T02:40:41.910Z] INFO[2024-01-19 02:40:41] Actual:true, Expected: true [2024-01-19T02:40:41.910Z] INFO[2024-01-19 02:40:41] Verifying : Description : Verifying if [destination-cluster] cluster is online [2024-01-19T02:40:41.910Z] INFO[2024-01-19 02:40:41] Actual:Online, Expected: Online [2024-01-19T02:40:41.910Z] 2024-01-19 02:40:41 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Taking backup of application for each user [2024-01-19T02:40:54.081Z] 2024-01-19 02:40:51 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T02:40:54.081Z] 2024-01-19 02:40:51 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Volume snapshot classes found in the cluster - [2024-01-19T02:40:54.081Z] 2024-01-19 02:40:51 +0000:[INFO] [{ShareAndRemoveBackupLocation}] ibmc-vpcblock-snapshot [2024-01-19T02:40:54.081Z] 2024-01-19 02:40:51 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T02:40:54.081Z] 2024/01/19 02:40:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705632051] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T02:41:02.171Z] 2024-01-19 02:41:01 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T02:41:02.171Z] 2024-01-19 02:41:01 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Volume snapshot classes found in the cluster - [2024-01-19T02:41:02.171Z] 2024-01-19 02:41:01 +0000:[INFO] [{ShareAndRemoveBackupLocation}] ibmc-vpcblock-snapshot [2024-01-19T02:41:02.171Z] 2024-01-19 02:41:01 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T02:41:02.427Z] 2024/01/19 02:41:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705632061] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T02:41:12.369Z] 2024-01-19 02:41:11 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T02:41:12.369Z] 2024-01-19 02:41:11 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Volume snapshot classes found in the cluster - [2024-01-19T02:41:12.369Z] 2024-01-19 02:41:11 +0000:[INFO] [{ShareAndRemoveBackupLocation}] ibmc-vpcblock-snapshot [2024-01-19T02:41:12.369Z] 2024-01-19 02:41:11 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T02:41:12.369Z] 2024/01/19 02:41:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705632071] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T02:41:22.320Z] 2024/01/19 02:41:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705632051] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:41:32.259Z] 2024/01/19 02:41:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705632061] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:41:44.483Z] 2024/01/19 02:41:42 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705632071] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:41:52.564Z] 2024/01/19 02:41:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705632051] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:42:02.506Z] 2024/01/19 02:42:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705632061] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:42:12.446Z] 2024/01/19 02:42:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705632071] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:42:22.391Z] 2024/01/19 02:42:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705632051] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:42:32.339Z] 2024/01/19 02:42:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705632061] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:42:44.590Z] 2024/01/19 02:42:42 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705632071] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:42:52.674Z] 2024/01/19 02:42:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705632051] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:43:02.612Z] 2024/01/19 02:43:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705632061] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:43:12.557Z] 2024/01/19 02:43:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705632071] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:43:22.496Z] 2024-01-19 02:43:22 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.CreateBackup:#313] - Backup [tp-backup-1705632051] created successfully [2024-01-19T02:43:22.497Z] 2024-01-19 02:43:22 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating backup [tp-backup-1705632051] in org [default] [2024-01-19T02:43:22.497Z] 2024-01-19 02:43:22 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705632051] [2024-01-19T02:43:22.497Z] 2024-01-19 02:43:22 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82949-01-19-02h38m54s] in backup [tp-backup-1705632051] [2024-01-19T02:43:22.497Z] 2024-01-19 02:43:22 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82949-01-19-02h38m54s] in backup [tp-backup-1705632051] [2024-01-19T02:43:22.497Z] 2024-01-19 02:43:22 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82949-01-19-02h38m54s] [2024-01-19T02:43:22.497Z] 2024-01-19 02:43:22 +0000:[WARNING] [{ShareAndRemoveBackupLocation}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:43:22.497Z] 2024-01-19 02:43:22 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:43:22.497Z] 2024-01-19 02:43:22 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82949-01-19-02h38m54s] are [[mysql-data]] [2024-01-19T02:43:22.497Z] 2024-01-19 02:43:22 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T02:43:22.497Z] 2024-01-19 02:43:22 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating CR cleanup [2024-01-19T02:43:22.754Z] 2024-01-19 02:43:22 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T02:43:22.754Z] 2024-01-19 02:43:22 +0000:[INFO] [{ShareAndRemoveBackupLocation}] All CRs in [mysql-ibm-pxb-0-82949-01-19-02h38m54s] are [[]] [2024-01-19T02:43:22.754Z] INFO[2024-01-19 02:43:22] Verifying : Description : Creation and Validation of backup [tp-backup-1705632051] of applications [mysql-ibm-pxb-0-82949-01-19-02h38m54s] [2024-01-19T02:43:22.754Z] INFO[2024-01-19 02:43:22] Actual:true, Expected: true [2024-01-19T02:43:32.693Z] 2024-01-19 02:43:32 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.CreateBackup:#313] - Backup [tp-backup-1705632061] created successfully [2024-01-19T02:43:32.693Z] 2024-01-19 02:43:32 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating backup [tp-backup-1705632061] in org [default] [2024-01-19T02:43:32.693Z] 2024-01-19 02:43:32 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705632061] [2024-01-19T02:43:32.693Z] 2024-01-19 02:43:32 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82949-01-19-02h38m54s] in backup [tp-backup-1705632061] [2024-01-19T02:43:32.693Z] 2024-01-19 02:43:32 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82949-01-19-02h38m54s] in backup [tp-backup-1705632061] [2024-01-19T02:43:32.693Z] 2024-01-19 02:43:32 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82949-01-19-02h38m54s] [2024-01-19T02:43:32.693Z] 2024-01-19 02:43:32 +0000:[WARNING] [{ShareAndRemoveBackupLocation}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:43:32.693Z] 2024-01-19 02:43:32 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:43:32.693Z] 2024-01-19 02:43:32 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82949-01-19-02h38m54s] are [[mysql-data]] [2024-01-19T02:43:32.693Z] 2024-01-19 02:43:32 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T02:43:32.693Z] 2024-01-19 02:43:32 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating CR cleanup [2024-01-19T02:43:32.693Z] 2024-01-19 02:43:32 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T02:43:32.693Z] 2024-01-19 02:43:32 +0000:[INFO] [{ShareAndRemoveBackupLocation}] All CRs in [mysql-ibm-pxb-0-82949-01-19-02h38m54s] are [[]] [2024-01-19T02:43:32.693Z] INFO[2024-01-19 02:43:32] Verifying : Description : Creation and Validation of backup [tp-backup-1705632061] of applications [mysql-ibm-pxb-0-82949-01-19-02h38m54s] [2024-01-19T02:43:32.693Z] INFO[2024-01-19 02:43:32] Actual:true, Expected: true [2024-01-19T02:43:42.633Z] 2024-01-19 02:43:42 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.CreateBackup:#313] - Backup [tp-backup-1705632071] created successfully [2024-01-19T02:43:42.633Z] 2024-01-19 02:43:42 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating backup [tp-backup-1705632071] in org [default] [2024-01-19T02:43:42.633Z] 2024-01-19 02:43:42 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705632071] [2024-01-19T02:43:42.633Z] 2024-01-19 02:43:42 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82949-01-19-02h38m54s] in backup [tp-backup-1705632071] [2024-01-19T02:43:42.633Z] 2024-01-19 02:43:42 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82949-01-19-02h38m54s] in backup [tp-backup-1705632071] [2024-01-19T02:43:42.633Z] 2024-01-19 02:43:42 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82949-01-19-02h38m54s] [2024-01-19T02:43:42.633Z] 2024-01-19 02:43:42 +0000:[WARNING] [{ShareAndRemoveBackupLocation}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:43:42.633Z] 2024-01-19 02:43:42 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:43:42.634Z] 2024-01-19 02:43:42 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82949-01-19-02h38m54s] are [[mysql-data]] [2024-01-19T02:43:42.634Z] 2024-01-19 02:43:42 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T02:43:42.634Z] 2024-01-19 02:43:42 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating CR cleanup [2024-01-19T02:43:42.634Z] 2024-01-19 02:43:42 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T02:43:42.634Z] 2024-01-19 02:43:42 +0000:[INFO] [{ShareAndRemoveBackupLocation}] All CRs in [mysql-ibm-pxb-0-82949-01-19-02h38m54s] are [[]] [2024-01-19T02:43:42.634Z] INFO[2024-01-19 02:43:42] Verifying : Description : Creation and Validation of backup [tp-backup-1705632071] of applications [mysql-ibm-pxb-0-82949-01-19-02h38m54s] [2024-01-19T02:43:42.634Z] INFO[2024-01-19 02:43:42] Actual:true, Expected: true [2024-01-19T02:43:42.634Z] 2024-01-19 02:43:42 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.glob..func92.2.5:#3023] - List of backups - [tp-backup-1705632051 tp-backup-1705632061 tp-backup-1705632071] [2024-01-19T02:43:42.634Z] 2024-01-19 02:43:42 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Share backup with users with different access level [2024-01-19T02:43:42.634Z] 2024-01-19 02:43:42 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Sharing backups with users with different access level [2024-01-19T02:43:42.892Z] 2024-01-19 02:43:42 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705632051 - c85ddb5b-4d03-483e-b99e-dadfa2cb5968 [2024-01-19T02:43:42.892Z] 2024-01-19 02:43:42 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser3-1705632036 - 0b984e4c-a844-42ce-9bb4-3095e8fa6ba0 [2024-01-19T02:43:43.149Z] 2024-01-19 02:43:43 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705632061 - d752415a-1fc8-4e9c-b656-d6e1b2e7661d [2024-01-19T02:43:43.405Z] 2024-01-19 02:43:43 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2-1705632036 - ac8d329f-f0ed-408b-9159-41ca84c52969 [2024-01-19T02:43:43.661Z] 2024-01-19 02:43:43 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705632071 - 5895005a-2e8c-4d44-a541-58e135e4cadd [2024-01-19T02:43:44.589Z] 2024-01-19 02:43:44 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705632036 - 198d1d9c-80c1-4c57-8811-53c7619fccba [2024-01-19T02:43:44.847Z] INFO[2024-01-19 02:43:44] Verifying : Description : Sharing backup [tp-backup-1705632051 tp-backup-1705632061 tp-backup-1705632071] with users [testuser3-1705632036 testuser2-1705632036 testuser1-1705632036] [2024-01-19T02:43:44.847Z] INFO[2024-01-19 02:43:44] Actual:true, Expected: true [2024-01-19T02:43:44.847Z] 2024-01-19 02:43:44 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Removing backup location after sharing backup with all the users [2024-01-19T02:43:46.234Z] INFO[2024-01-19 02:43:46] Verifying : Description : Deleting backup location autogenerated-backup-location-1705632037 [2024-01-19T02:43:46.234Z] INFO[2024-01-19 02:43:46] Actual:true, Expected: true [2024-01-19T02:43:46.234Z] 2024-01-19 02:43:46 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705632037] is present [2024-01-19T02:43:46.234Z] 2024/01/19 02:43:46 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705632037 is not deleted yet}, Next try in [30s], timeout [1h0m0s] [2024-01-19T02:44:18.294Z] 2024-01-19 02:44:16 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705632037] is present [2024-01-19T02:44:18.294Z] 2024/01/19 02:44:16 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705632037 is not deleted yet}, Next try in [30s], timeout [1h0m0s] [2024-01-19T02:44:50.399Z] 2024-01-19 02:44:46 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705632037] is present [2024-01-19T02:44:50.399Z] 2024/01/19 02:44:46 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705632037 is not deleted yet}, Next try in [30s], timeout [1h0m0s] [2024-01-19T02:44:50.399Z] INFO[2024-01-19 02:44:47] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T02:44:50.399Z] INFO[2024-01-19 02:44:47] sched stats: tick interval: 40247 < 1.2s (100%) [2024-01-19T02:44:50.399Z] INFO[2024-01-19 02:44:47] sched stats: tick process durations: 40246 < 500ms (100%) [2024-01-19T02:45:16.905Z] 2024-01-19 02:45:16 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705632037] is present [2024-01-19T02:45:16.905Z] 2024/01/19 02:45:16 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705632037 is not deleted yet}, Next try in [30s], timeout [1h0m0s] [2024-01-19T02:45:48.937Z] 2024-01-19 02:45:46 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 aws-global-aws-px-backup-system-test-4cac76dc-bl autogenerated-backup-location-1705628002] [2024-01-19T02:45:48.937Z] INFO[2024-01-19 02:45:46] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-1705632037 [2024-01-19T02:45:48.937Z] INFO[2024-01-19 02:45:46] Actual:true, Expected: true [2024-01-19T02:45:48.937Z] 2024-01-19 02:45:46 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Adding new backup location to the cluster [2024-01-19T02:45:49.196Z] INFO[2024-01-19 02:45:49] Verifying : Description : Creating new backup location new-aws-1705632346-bl [2024-01-19T02:45:49.196Z] INFO[2024-01-19 02:45:49] Actual:true, Expected: true [2024-01-19T02:45:49.196Z] 2024-01-19 02:45:49 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Taking backup of application for each user again with new backup location [2024-01-19T02:46:01.371Z] 2024-01-19 02:45:59 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T02:46:01.371Z] 2024-01-19 02:45:59 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Volume snapshot classes found in the cluster - [2024-01-19T02:46:01.371Z] 2024-01-19 02:45:59 +0000:[INFO] [{ShareAndRemoveBackupLocation}] ibmc-vpcblock-snapshot [2024-01-19T02:46:01.371Z] 2024-01-19 02:45:59 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T02:46:01.371Z] 2024/01/19 02:45:59 DoRetryWithTimeout - Error: {backup status for [new-tp-backup-1705632359] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T02:46:09.472Z] 2024-01-19 02:46:09 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T02:46:09.472Z] 2024-01-19 02:46:09 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Volume snapshot classes found in the cluster - [2024-01-19T02:46:09.472Z] 2024-01-19 02:46:09 +0000:[INFO] [{ShareAndRemoveBackupLocation}] ibmc-vpcblock-snapshot [2024-01-19T02:46:09.472Z] 2024-01-19 02:46:09 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T02:46:09.728Z] 2024/01/19 02:46:09 DoRetryWithTimeout - Error: {backup status for [new-tp-backup-1705632369] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T02:46:19.680Z] 2024-01-19 02:46:19 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T02:46:19.680Z] 2024-01-19 02:46:19 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Volume snapshot classes found in the cluster - [2024-01-19T02:46:19.680Z] 2024-01-19 02:46:19 +0000:[INFO] [{ShareAndRemoveBackupLocation}] ibmc-vpcblock-snapshot [2024-01-19T02:46:19.680Z] 2024-01-19 02:46:19 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T02:46:19.680Z] 2024/01/19 02:46:19 DoRetryWithTimeout - Error: {backup status for [new-tp-backup-1705632379] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T02:46:29.617Z] 2024/01/19 02:46:29 DoRetryWithTimeout - Error: {backup status for [new-tp-backup-1705632359] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:46:41.047Z] 2024/01/19 02:46:39 DoRetryWithTimeout - Error: {backup status for [new-tp-backup-1705632369] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:46:51.024Z] 2024/01/19 02:46:49 DoRetryWithTimeout - Error: {backup status for [new-tp-backup-1705632379] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:47:00.966Z] 2024/01/19 02:46:59 DoRetryWithTimeout - Error: {backup status for [new-tp-backup-1705632359] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:47:10.909Z] 2024/01/19 02:47:09 DoRetryWithTimeout - Error: {backup status for [new-tp-backup-1705632369] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:47:20.863Z] 2024/01/19 02:47:19 DoRetryWithTimeout - Error: {backup status for [new-tp-backup-1705632379] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:47:30.812Z] 2024/01/19 02:47:29 DoRetryWithTimeout - Error: {backup status for [new-tp-backup-1705632359] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:47:40.770Z] 2024/01/19 02:47:39 DoRetryWithTimeout - Error: {backup status for [new-tp-backup-1705632369] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:47:50.726Z] 2024/01/19 02:47:49 DoRetryWithTimeout - Error: {backup status for [new-tp-backup-1705632379] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:48:00.682Z] 2024/01/19 02:47:59 DoRetryWithTimeout - Error: {backup status for [new-tp-backup-1705632359] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:48:10.622Z] 2024/01/19 02:48:09 DoRetryWithTimeout - Error: {backup status for [new-tp-backup-1705632369] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:48:20.580Z] 2024/01/19 02:48:19 DoRetryWithTimeout - Error: {backup status for [new-tp-backup-1705632379] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:48:30.519Z] 2024-01-19 02:48:29 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.CreateBackup:#313] - Backup [new-tp-backup-1705632359] created successfully [2024-01-19T02:48:30.519Z] 2024-01-19 02:48:29 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating backup [new-tp-backup-1705632359] in org [default] [2024-01-19T02:48:30.519Z] 2024-01-19 02:48:29 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [new-tp-backup-1705632359] [2024-01-19T02:48:30.519Z] 2024-01-19 02:48:29 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82949-01-19-02h38m54s] in backup [new-tp-backup-1705632359] [2024-01-19T02:48:30.519Z] 2024-01-19 02:48:29 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82949-01-19-02h38m54s] in backup [new-tp-backup-1705632359] [2024-01-19T02:48:30.520Z] 2024-01-19 02:48:29 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82949-01-19-02h38m54s] [2024-01-19T02:48:30.520Z] 2024-01-19 02:48:30 +0000:[WARNING] [{ShareAndRemoveBackupLocation}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:48:30.520Z] 2024-01-19 02:48:30 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:48:30.520Z] 2024-01-19 02:48:30 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82949-01-19-02h38m54s] are [[mysql-data]] [2024-01-19T02:48:30.520Z] 2024-01-19 02:48:30 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T02:48:30.520Z] 2024-01-19 02:48:30 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating CR cleanup [2024-01-19T02:48:30.520Z] 2024-01-19 02:48:30 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T02:48:30.520Z] 2024-01-19 02:48:30 +0000:[INFO] [{ShareAndRemoveBackupLocation}] All CRs in [mysql-ibm-pxb-0-82949-01-19-02h38m54s] are [[]] [2024-01-19T02:48:30.520Z] INFO[2024-01-19 02:48:30] Verifying : Description : Creation and Validation of backup [new-tp-backup-1705632359] of applications [mysql-ibm-pxb-0-82949-01-19-02h38m54s] [2024-01-19T02:48:30.520Z] INFO[2024-01-19 02:48:30] Actual:true, Expected: true [2024-01-19T02:48:40.460Z] 2024-01-19 02:48:39 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.CreateBackup:#313] - Backup [new-tp-backup-1705632369] created successfully [2024-01-19T02:48:40.460Z] 2024-01-19 02:48:39 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating backup [new-tp-backup-1705632369] in org [default] [2024-01-19T02:48:40.460Z] 2024-01-19 02:48:39 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [new-tp-backup-1705632369] [2024-01-19T02:48:40.460Z] 2024-01-19 02:48:39 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82949-01-19-02h38m54s] in backup [new-tp-backup-1705632369] [2024-01-19T02:48:40.460Z] 2024-01-19 02:48:39 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82949-01-19-02h38m54s] in backup [new-tp-backup-1705632369] [2024-01-19T02:48:40.460Z] 2024-01-19 02:48:39 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82949-01-19-02h38m54s] [2024-01-19T02:48:40.460Z] 2024-01-19 02:48:39 +0000:[WARNING] [{ShareAndRemoveBackupLocation}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:48:40.460Z] 2024-01-19 02:48:39 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:48:40.460Z] 2024-01-19 02:48:39 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82949-01-19-02h38m54s] are [[mysql-data]] [2024-01-19T02:48:40.460Z] 2024-01-19 02:48:39 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T02:48:40.460Z] 2024-01-19 02:48:39 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating CR cleanup [2024-01-19T02:48:40.460Z] 2024-01-19 02:48:39 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T02:48:40.460Z] 2024-01-19 02:48:40 +0000:[INFO] [{ShareAndRemoveBackupLocation}] All CRs in [mysql-ibm-pxb-0-82949-01-19-02h38m54s] are [[]] [2024-01-19T02:48:40.460Z] INFO[2024-01-19 02:48:40] Verifying : Description : Creation and Validation of backup [new-tp-backup-1705632369] of applications [mysql-ibm-pxb-0-82949-01-19-02h38m54s] [2024-01-19T02:48:40.460Z] INFO[2024-01-19 02:48:40] Actual:true, Expected: true [2024-01-19T02:48:50.404Z] 2024-01-19 02:48:49 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.CreateBackup:#313] - Backup [new-tp-backup-1705632379] created successfully [2024-01-19T02:48:50.404Z] 2024-01-19 02:48:49 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating backup [new-tp-backup-1705632379] in org [default] [2024-01-19T02:48:50.404Z] 2024-01-19 02:48:49 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [new-tp-backup-1705632379] [2024-01-19T02:48:50.404Z] 2024-01-19 02:48:49 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82949-01-19-02h38m54s] in backup [new-tp-backup-1705632379] [2024-01-19T02:48:50.404Z] 2024-01-19 02:48:49 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82949-01-19-02h38m54s] in backup [new-tp-backup-1705632379] [2024-01-19T02:48:50.404Z] 2024-01-19 02:48:49 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82949-01-19-02h38m54s] [2024-01-19T02:48:50.404Z] 2024-01-19 02:48:49 +0000:[WARNING] [{ShareAndRemoveBackupLocation}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T02:48:50.404Z] 2024-01-19 02:48:49 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:48:50.404Z] 2024-01-19 02:48:49 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82949-01-19-02h38m54s] are [[mysql-data]] [2024-01-19T02:48:50.404Z] 2024-01-19 02:48:49 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T02:48:50.404Z] 2024-01-19 02:48:49 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating CR cleanup [2024-01-19T02:48:50.404Z] 2024-01-19 02:48:49 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T02:48:50.404Z] 2024-01-19 02:48:49 +0000:[INFO] [{ShareAndRemoveBackupLocation}] All CRs in [mysql-ibm-pxb-0-82949-01-19-02h38m54s] are [[]] [2024-01-19T02:48:50.404Z] INFO[2024-01-19 02:48:49] Verifying : Description : Creation and Validation of backup [new-tp-backup-1705632379] of applications [mysql-ibm-pxb-0-82949-01-19-02h38m54s] [2024-01-19T02:48:50.404Z] INFO[2024-01-19 02:48:49] Actual:true, Expected: true [2024-01-19T02:48:50.404Z] 2024-01-19 02:48:49 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.glob..func92.2.9:#3081] - List of new backups - [new-tp-backup-1705632359 new-tp-backup-1705632369 new-tp-backup-1705632379] [2024-01-19T02:48:50.404Z] 2024-01-19 02:48:49 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Share backup with users again with different access level [2024-01-19T02:48:50.404Z] 2024-01-19 02:48:49 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Sharing backups with users with different access level [2024-01-19T02:48:50.404Z] 2024-01-19 02:48:49 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ShareBackup:#715] - Backup UID for new-tp-backup-1705632359 - 86aa4dd1-683e-4d88-89a4-4400fa495b8e [2024-01-19T02:48:50.404Z] 2024-01-19 02:48:50 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser3-1705632036 - 0b984e4c-a844-42ce-9bb4-3095e8fa6ba0 [2024-01-19T02:48:50.405Z] 2024-01-19 02:48:50 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ShareBackup:#715] - Backup UID for new-tp-backup-1705632369 - c672cd57-de65-4386-80de-a1e98aaec19a [2024-01-19T02:48:50.660Z] 2024-01-19 02:48:50 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2-1705632036 - ac8d329f-f0ed-408b-9159-41ca84c52969 [2024-01-19T02:48:50.918Z] 2024-01-19 02:48:50 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.ShareBackup:#715] - Backup UID for new-tp-backup-1705632379 - 301e2330-2647-4336-bfed-d9fcf1957ea0 [2024-01-19T02:48:51.856Z] 2024-01-19 02:48:51 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705632036 - 198d1d9c-80c1-4c57-8811-53c7619fccba [2024-01-19T02:48:52.115Z] INFO[2024-01-19 02:48:51] Verifying : Description : Sharing backup [new-tp-backup-1705632359 new-tp-backup-1705632369 new-tp-backup-1705632379] with users [testuser3-1705632036 testuser2-1705632036 testuser1-1705632036] [2024-01-19T02:48:52.115Z] INFO[2024-01-19 02:48:51] Actual:true, Expected: true [2024-01-19T02:48:52.115Z] 2024-01-19 02:48:51 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.glob..func92.2.10:#3088] - The user/access/backup/context mapping is map[{testuser1-1705632036 3 0xc001852570}:new-tp-backup-1705632379 {testuser2-1705632036 2 0xc001958450}:new-tp-backup-1705632369 {testuser3-1705632036 1 0xc00145e720}:new-tp-backup-1705632359] [2024-01-19T02:48:52.115Z] 2024-01-19 02:48:51 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validate if the users with different access level can restore/delete backup [2024-01-19T02:48:52.115Z] 2024-01-19 02:48:51 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.glob..func92.2.11:#3099] - Removing the restores which will be deleted while validating FullAccess [2024-01-19T02:48:53.482Z] INFO[2024-01-19 02:48:53] Verifying : Description : Fetching px-central-admin ctx [2024-01-19T02:48:53.482Z] INFO[2024-01-19 02:48:53] Actual:true, Expected: true [2024-01-19T02:48:54.047Z] INFO[2024-01-19 02:48:53] Verifying : Description : Fetching testuser1-1705632036 user ctx [2024-01-19T02:48:54.047Z] INFO[2024-01-19 02:48:53] Actual:true, Expected: true [2024-01-19T02:48:54.047Z] 2024-01-19 02:48:53 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Registering Source and Destination clusters from user context [2024-01-19T02:48:54.047Z] INFO[2024-01-19 02:48:53] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T02:48:54.047Z] INFO[2024-01-19 02:48:53] Actual:true, Expected: true [2024-01-19T02:48:54.047Z] 2024-01-19 02:48:53 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T02:48:54.047Z] 2024-01-19 02:48:53 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T02:48:54.047Z] 2024-01-19 02:48:53 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T02:48:54.047Z] 2024-01-19 02:48:53 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T02:48:54.047Z] 2024-01-19 02:48:53 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Create cluster [source-cluster] in org [default] [2024-01-19T02:48:54.047Z] 2024-01-19 02:48:53 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T02:48:54.048Z] 2024-01-19 02:48:53 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T02:48:54.048Z] 2024-01-19 02:48:53 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Create cluster [destination-cluster] in org [default] [2024-01-19T02:48:54.048Z] 2024-01-19 02:48:53 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T02:48:54.048Z] 2024-01-19 02:48:53 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T02:48:54.048Z] 2024-01-19 02:48:53 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-czxyh] for IBM [2024-01-19T02:48:54.048Z] 2024-01-19 02:48:53 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-czxyh] for org [default] with [ibm] as provider [2024-01-19T02:48:54.048Z] 2024-01-19 02:48:53 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T02:48:54.048Z] 2024-01-19 02:48:53 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T02:48:54.048Z] 2024-01-19 02:48:53 +0000:[WARNING] [{ShareAndRemoveBackupLocation}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-czxyh] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user]] [2024-01-19T02:48:54.048Z] 2024-01-19 02:48:53 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user] [2024-01-19T02:48:55.417Z] 2024-01-19 02:48:55 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-czxyh from admin context and sharing with all the users [2024-01-19T02:48:55.417Z] 2024-01-19 02:48:55 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-czxyh] for org [default] with [ibm] as provider [2024-01-19T02:48:55.417Z] 2024-01-19 02:48:55 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T02:48:55.417Z] 2024-01-19 02:48:55 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T02:48:55.675Z] 2024-01-19 02:48:55 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-czxyh] is present [2024-01-19T02:48:55.675Z] 2024-01-19 02:48:55 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Create cluster [source-cluster] in org [default] [2024-01-19T02:48:56.237Z] 2024-01-19 02:48:56 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-dnttl] for IBM [2024-01-19T02:48:56.237Z] 2024-01-19 02:48:56 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-dnttl] for org [default] with [ibm] as provider [2024-01-19T02:48:56.237Z] 2024-01-19 02:48:56 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T02:48:56.237Z] 2024-01-19 02:48:56 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T02:48:56.237Z] 2024-01-19 02:48:56 +0000:[WARNING] [{ShareAndRemoveBackupLocation}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-dnttl] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user]] [2024-01-19T02:48:56.237Z] 2024-01-19 02:48:56 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user] [2024-01-19T02:48:56.801Z] 2024-01-19 02:48:56 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-dnttl from admin context and sharing with all the users [2024-01-19T02:48:56.801Z] 2024-01-19 02:48:56 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-dnttl] for org [default] with [ibm] as provider [2024-01-19T02:48:56.801Z] 2024-01-19 02:48:56 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T02:48:56.801Z] 2024-01-19 02:48:56 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T02:48:57.059Z] 2024-01-19 02:48:56 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-dnttl] is present [2024-01-19T02:48:57.059Z] 2024-01-19 02:48:56 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Create cluster [destination-cluster] in org [default] [2024-01-19T02:48:57.620Z] INFO[2024-01-19 02:48:57] Verifying : Description : Creating source and destination cluster [2024-01-19T02:48:57.620Z] INFO[2024-01-19 02:48:57] Actual:true, Expected: true [2024-01-19T02:48:57.620Z] 2024-01-19 02:48:57 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Validating if user [testuser1-1705632036] with access [FullAccess] can restore and delete backup new-tp-backup-1705632379 or not [2024-01-19T02:48:57.875Z] 2024/01/19 02:48:57 DoRetryWithTimeout - Error: {restore status for [testuser1-1705632036-tp-restore-1705632531] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T02:49:29.908Z] 2024/01/19 02:49:27 DoRetryWithTimeout - Error: {restore status for [testuser1-1705632036-tp-restore-1705632531] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:50:01.933Z] 2024/01/19 02:49:57 DoRetryWithTimeout - Error: {restore status for [testuser1-1705632036-tp-restore-1705632531] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T02:50:28.480Z] INFO[2024-01-19 02:50:27] Verifying : Description : Verifying that restore is possible [2024-01-19T02:50:28.481Z] ERRO[2024-01-19 02:50:27] restore status for [testuser1-1705632036-tp-restore-1705632531] expected was [[PartialSuccess Success]] but got [Failed] because of [Error starting Application Restore for volumes: failed to restore pvc mysql-data: max retries done, volumesnapshot [mysql-ibm-pxb-0-82949-01-19-02h38m54s/restore-vs-72502f9d3c87-a925393aea3a] status is not updated] [2024-01-19T02:50:28.481Z] 2024-01-19 02:50:28 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Destroying apps [2024-01-19T02:50:28.481Z] 2024-01-19 02:50:28 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T02:50:28.481Z] 2024-01-19 02:50:28 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T02:50:28.481Z] 2024-01-19 02:50:28 +0000:[INFO] [{ShareAndRemoveBackupLocation}] start destroying mysql-ibm app [2024-01-19T02:50:28.481Z] 2024-01-19 02:50:28 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T02:50:28.481Z] 2024-01-19 02:50:28 +0000:[INFO] [{ShareAndRemoveBackupLocation}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T02:50:28.481Z] 2024-01-19 02:50:28 +0000:[WARNING] [{ShareAndRemoveBackupLocation}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T02:50:28.481Z] INFO[2024-01-19 02:50:28] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T02:50:28.481Z] INFO[2024-01-19 02:50:28] Actual:true, Expected: true [2024-01-19T02:50:28.481Z] INFO[2024-01-19 02:50:28] Verifying : Description : Getting backup UID for backup new-tp-backup-1705632369 [2024-01-19T02:50:28.481Z] INFO[2024-01-19 02:50:28] Actual:true, Expected: true [2024-01-19T02:50:28.481Z] INFO[2024-01-19 02:50:28] Verifying : Description : Getting backup UID for backup new-tp-backup-1705632359 [2024-01-19T02:50:28.481Z] INFO[2024-01-19 02:50:28] Actual:true, Expected: true [2024-01-19T02:50:28.481Z] INFO[2024-01-19 02:50:28] Verifying : Description : Deleting backup new-tp-backup-1705632369 [2024-01-19T02:50:28.481Z] INFO[2024-01-19 02:50:28] Actual:true, Expected: true [2024-01-19T02:50:28.481Z] INFO[2024-01-19 02:50:28] Verifying : Description : Deleting backup new-tp-backup-1705632359 [2024-01-19T02:50:28.481Z] INFO[2024-01-19 02:50:28] Actual:true, Expected: true [2024-01-19T02:50:28.481Z] 2024-01-19 02:50:28 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.glob..func92.3:#3128] - Generating user context [2024-01-19T02:50:29.410Z] 2024-01-19 02:50:29 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.glob..func92.3:#3134] - Deleting restore created by users [2024-01-19T02:50:29.410Z] INFO[2024-01-19 02:50:29] Verifying : Description : Getting the backup driver [2024-01-19T02:50:29.410Z] INFO[2024-01-19 02:50:29] Actual:true, Expected: true [2024-01-19T02:50:29.410Z] INFO[2024-01-19 02:50:29] Verifying : Description : Deleting Restore testuser1-1705632036-tp-restore-1705632531 [2024-01-19T02:50:29.410Z] INFO[2024-01-19 02:50:29] Actual:true, Expected: true [2024-01-19T02:50:29.410Z] 2024-01-19 02:50:29 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.glob..func92.3:#3139] - Deleting registered clusters for non-admin context [2024-01-19T02:50:29.410Z] 2024-01-19 02:50:29 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T02:50:29.410Z] INFO[2024-01-19 02:50:29] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T02:50:29.410Z] INFO[2024-01-19 02:50:29] Actual:true, Expected: true [2024-01-19T02:50:29.410Z] 2024-01-19 02:50:29 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T02:50:29.410Z] INFO[2024-01-19 02:50:29] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T02:50:29.410Z] INFO[2024-01-19 02:50:29] Actual:true, Expected: true [2024-01-19T02:50:29.410Z] 2024-01-19 02:50:29 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T02:50:29.410Z] INFO[2024-01-19 02:50:29] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T02:50:29.410Z] INFO[2024-01-19 02:50:29] Actual:true, Expected: true [2024-01-19T02:50:29.666Z] INFO[2024-01-19 02:50:29] Verifying : Description : Deleting cluster source-cluster [2024-01-19T02:50:29.666Z] INFO[2024-01-19 02:50:29] Actual:true, Expected: true [2024-01-19T02:50:30.594Z] INFO[2024-01-19 02:50:30] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-czxyh] [2024-01-19T02:50:30.594Z] INFO[2024-01-19 02:50:30] Actual:true, Expected: true [2024-01-19T02:50:30.850Z] INFO[2024-01-19 02:50:30] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T02:50:30.850Z] INFO[2024-01-19 02:50:30] Actual:true, Expected: true [2024-01-19T02:50:32.216Z] INFO[2024-01-19 02:50:31] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-dnttl] [2024-01-19T02:50:32.216Z] INFO[2024-01-19 02:50:31] Actual:true, Expected: true [2024-01-19T02:50:32.216Z] 2024-01-19 02:50:31 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.glob..func92.3:#3143] - Cleaning up users [2024-01-19T02:50:35.481Z] 2024-01-19 02:50:35 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705632036 - 198d1d9c-80c1-4c57-8811-53c7619fccba [2024-01-19T02:50:35.481Z] 2024-01-19 02:50:35 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2-1705632036 - ac8d329f-f0ed-408b-9159-41ca84c52969 [2024-01-19T02:50:35.738Z] 2024-01-19 02:50:35 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser3-1705632036 - 0b984e4c-a844-42ce-9bb4-3095e8fa6ba0 [2024-01-19T02:50:36.674Z] 2024-01-19 02:50:36 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.DeleteUser:#755] - Response for user [testuser1-1705632036] deletion - [2024-01-19T02:50:36.674Z] 2024-01-19 02:50:36 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.DeleteUser:#759] - Deleted User - testuser1-1705632036 [2024-01-19T02:50:36.674Z] INFO[2024-01-19 02:50:36] Verifying : Description : Deleting user testuser1-1705632036 [2024-01-19T02:50:36.674Z] INFO[2024-01-19 02:50:36] Actual:true, Expected: true [2024-01-19T02:50:36.674Z] 2024-01-19 02:50:36 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.DeleteUser:#755] - Response for user [testuser2-1705632036] deletion - [2024-01-19T02:50:36.674Z] 2024-01-19 02:50:36 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.DeleteUser:#759] - Deleted User - testuser2-1705632036 [2024-01-19T02:50:36.674Z] INFO[2024-01-19 02:50:36] Verifying : Description : Deleting user testuser2-1705632036 [2024-01-19T02:50:36.674Z] INFO[2024-01-19 02:50:36] Actual:true, Expected: true [2024-01-19T02:50:36.930Z] 2024-01-19 02:50:36 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.DeleteUser:#755] - Response for user [testuser3-1705632036] deletion - [2024-01-19T02:50:36.930Z] 2024-01-19 02:50:36 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.DeleteUser:#759] - Deleted User - testuser3-1705632036 [2024-01-19T02:50:36.930Z] INFO[2024-01-19 02:50:36] Verifying : Description : Deleting user testuser3-1705632036 [2024-01-19T02:50:36.930Z] INFO[2024-01-19 02:50:36] Actual:true, Expected: true [2024-01-19T02:50:36.930Z] 2024-01-19 02:50:36 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Cleaning backup locations in map [map[b14d5399-e9a1-4e2e-bbce-241b6494586a:new-aws-1705632346-bl]], cloud credential [cred-aws-1705632037], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T02:50:36.930Z] INFO[2024-01-19 02:50:36] Verifying : Description : Verifying deletion of backup location [new-aws-1705632346-bl] [2024-01-19T02:50:36.930Z] INFO[2024-01-19 02:50:36] Actual:true, Expected: true [2024-01-19T02:50:36.930Z] 2024-01-19 02:50:36 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.IsBackupLocationPresent:#2442] - Backup location [new-aws-1705632346-bl] is present [2024-01-19T02:50:37.187Z] 2024/01/19 02:50:36 DoRetryWithTimeout - Error: {backup location new-aws-1705632346-bl is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T02:51:09.228Z] 2024-01-19 02:51:07 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.IsBackupLocationPresent:#2442] - Backup location [new-aws-1705632346-bl] is present [2024-01-19T02:51:09.228Z] 2024/01/19 02:51:07 DoRetryWithTimeout - Error: {backup location new-aws-1705632346-bl is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T02:51:41.243Z] 2024-01-19 02:51:37 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.IsBackupLocationPresent:#2442] - Backup location [new-aws-1705632346-bl] is present [2024-01-19T02:51:41.243Z] 2024/01/19 02:51:37 DoRetryWithTimeout - Error: {backup location new-aws-1705632346-bl is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T02:52:07.743Z] 2024-01-19 02:52:07 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.IsBackupLocationPresent:#2442] - Backup location [new-aws-1705632346-bl] is present [2024-01-19T02:52:07.743Z] 2024/01/19 02:52:07 DoRetryWithTimeout - Error: {backup location new-aws-1705632346-bl is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T02:52:39.767Z] 2024-01-19 02:52:37 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 aws-global-aws-px-backup-system-test-4cac76dc-bl autogenerated-backup-location-1705628002] [2024-01-19T02:52:39.767Z] INFO[2024-01-19 02:52:37] Verifying : Description : Verifying backup location deletion status new-aws-1705632346-bl [2024-01-19T02:52:39.767Z] INFO[2024-01-19 02:52:37] Actual:true, Expected: true [2024-01-19T02:52:39.767Z] 2024-01-19 02:52:37 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cred-aws-1705632037] is present [2024-01-19T02:52:39.767Z] INFO[2024-01-19 02:52:37] Verifying : Description : Verifying if cloud cred [cred-aws-1705632037] is present [2024-01-19T02:52:39.767Z] INFO[2024-01-19 02:52:37] Actual:true, Expected: true [2024-01-19T02:52:39.767Z] INFO[2024-01-19 02:52:37] Verifying : Description : Verifying deletion of cloud cred [cred-aws-1705632037] [2024-01-19T02:52:39.767Z] INFO[2024-01-19 02:52:37] Actual:true, Expected: true [2024-01-19T02:52:39.767Z] INFO[2024-01-19 02:52:37] Verifying : Description : Deleting cloud cred cred-aws-1705632037 [2024-01-19T02:52:39.767Z] INFO[2024-01-19 02:52:37] Actual:true, Expected: true [2024-01-19T02:52:39.767Z] INFO[2024-01-19 02:52:37] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T02:52:39.767Z] INFO[2024-01-19 02:52:37] Actual:true, Expected: true [2024-01-19T02:52:39.768Z] INFO[2024-01-19 02:52:37] Verifying : Description : Deleting cluster source-cluster [2024-01-19T02:52:39.768Z] INFO[2024-01-19 02:52:37] Actual:true, Expected: true [2024-01-19T02:52:39.768Z] INFO[2024-01-19 02:52:38] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-dczfv] [2024-01-19T02:52:39.768Z] INFO[2024-01-19 02:52:38] Actual:true, Expected: true [2024-01-19T02:52:39.768Z] INFO[2024-01-19 02:52:38] --------Test End------ [2024-01-19T02:52:39.768Z] INFO[2024-01-19 02:52:38] #Test: [2024-01-19T02:52:39.768Z] INFO[2024-01-19 02:52:38] #Description: [2024-01-19T02:52:39.768Z] INFO[2024-01-19 02:52:38] ------------------------ [2024-01-19T02:52:39.768Z] 2024-01-19 02:52:38 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.EndPxBackupTorpedoTest:#7261] - >>>> FAILED TEST: {ShareAndRemoveBackupLocation} Share and remove backup location and add it back and check from other users if they show up [2024-01-19T02:52:39.768Z] 2024-01-19 02:52:38 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-82949-01-19-02h38m54s] [2024-01-19T02:52:39.768Z] 2024-01-19 02:52:38 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T02:52:39.768Z] 2024-01-19 02:52:38 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T02:52:39.768Z] 2024-01-19 02:52:38 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-82949-01-19-02h38m54s] [2024-01-19T02:52:39.768Z] 2024-01-19 02:52:38 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {ShareAndRemoveBackupLocation} Share and remove backup location and add it back and check from other users if they show up [2024-01-19T02:52:39.768Z] 2024-01-19 02:52:38 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T02:52:39.768Z] 2024/01/19 02:52:38 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T02:52:43.936Z] 2024/01/19 02:52:43 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T02:52:49.174Z] 2024/01/19 02:52:48 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T02:52:54.418Z] 2024/01/19 02:52:53 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T02:52:58.582Z] 2024/01/19 02:52:58 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T02:53:03.824Z] 2024/01/19 02:53:03 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T02:53:09.068Z] 2024/01/19 02:53:08 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T02:53:14.309Z] 2024/01/19 02:53:13 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T02:53:18.475Z] 2024/01/19 02:53:18 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.0.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.7 10.240.0.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T02:53:18.475Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T02:53:18.475Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T02:53:18.475Z] {dial tcp 10.240.0.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T02:53:23.739Z] 2024-01-19 02:53:23 +0000:[WARNING] [{ShareAndRemoveBackupLocation}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.7 10.240.0.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T02:53:23.739Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T02:53:23.739Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T02:53:23.739Z] {dial tcp 10.240.0.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T02:53:23.740Z] 2024-01-19 02:53:23 +0000:[ERROR] [{ShareAndRemoveBackupLocation}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.7 10.240.0.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T02:53:23.740Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T02:53:23.740Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T02:53:23.740Z] {dial tcp 10.240.0.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T02:53:23.740Z] 2024-01-19 02:53:23 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T02:53:23.740Z] 2024-01-19 02:53:23 +0000:[INFO] [{ShareAndRemoveBackupLocation}] Switching context to [/tmp/source-config] [2024-01-19T02:53:23.740Z] 2024-01-19 02:53:23 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T02:53:23.740Z] 2024-01-19 02:53:23 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T02:53:23.740Z] 2024-01-19 02:53:23 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T02:53:23.740Z] 2024-01-19 02:53:23 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T02:53:23.740Z] 2024-01-19 02:53:23 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T02:53:23.995Z] 2024-01-19 02:53:23 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T02:53:24.250Z] 2024-01-19 02:53:24 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T02:53:24.811Z] 2024-01-19 02:53:24 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T02:53:25.067Z] 2024-01-19 02:53:24 +0000:[INFO] [{ShareAndRemoveBackupLocation}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T02:53:25.322Z] 2024/01/19 02:53:25 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:53:35.256Z] 2024/01/19 02:53:35 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:53:47.442Z] 2024/01/19 02:53:45 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:53:55.533Z] 2024/01/19 02:53:55 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:54:05.471Z] 2024/01/19 02:54:05 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:54:17.039Z] 2024/01/19 02:54:15 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:54:26.979Z] 2024/01/19 02:54:25 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:54:36.913Z] 2024/01/19 02:54:35 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:54:46.848Z] 2024/01/19 02:54:45 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:54:48.734Z] INFO[2024-01-19 02:54:48] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T02:54:48.734Z] INFO[2024-01-19 02:54:48] sched stats: tick interval: 40848 < 1.2s (100%) [2024-01-19T02:54:48.734Z] INFO[2024-01-19 02:54:48] sched stats: tick process durations: 40847 < 500ms (100%) [2024-01-19T02:54:56.832Z] 2024/01/19 02:54:55 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:55:06.790Z] 2024/01/19 02:55:05 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:55:16.781Z] 2024/01/19 02:55:15 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:55:26.723Z] 2024/01/19 02:55:25 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:55:36.706Z] 2024/01/19 02:55:35 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:55:46.664Z] 2024/01/19 02:55:45 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:55:56.617Z] 2024/01/19 02:55:55 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:56:06.587Z] 2024/01/19 02:56:05 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:56:16.581Z] 2024/01/19 02:56:15 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:56:26.530Z] 2024/01/19 02:56:25 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:56:36.476Z] 2024/01/19 02:56:35 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:56:46.472Z] 2024/01/19 02:56:45 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:56:56.413Z] 2024/01/19 02:56:55 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:57:06.355Z] 2024/01/19 02:57:05 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:57:16.302Z] 2024/01/19 02:57:15 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:57:26.244Z] 2024/01/19 02:57:26 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:57:36.194Z] 2024/01/19 02:57:36 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:57:46.140Z] 2024/01/19 02:57:46 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:57:58.307Z] 2024/01/19 02:57:56 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:58:06.395Z] 2024/01/19 02:58:06 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:58:16.339Z] 2024/01/19 02:58:16 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T02:58:26.277Z] 2024-01-19 02:58:26 +0000:[ERROR] [{ShareAndRemoveBackupLocation}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.277Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.277Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.277Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.277Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.277Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.277Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.277Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.277Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.277Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.277Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.277Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.277Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.278Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.278Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.278Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.278Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.278Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.278Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.278Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.278Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.278Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.278Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.278Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.278Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.278Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.278Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.278Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.278Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T02:58:26.278Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T02:58:26.278Z] [2024-01-19T02:58:26.278Z] ------------------------------ [2024-01-19T02:58:26.278Z] • Failure [1171.491 seconds] [2024-01-19T02:58:26.278Z] {ShareAndRemoveBackupLocation} [2024-01-19T02:58:26.278Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:2918 [2024-01-19T02:58:26.278Z] Share and remove backup location and add it back and check from other users if they show up [It] [2024-01-19T02:58:26.278Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:2959 [2024-01-19T02:58:26.278Z] [2024-01-19T02:58:26.278Z] Unexpected error: [2024-01-19T02:58:26.278Z] <*errors.errorString | 0xc0015c58b0>: { [2024-01-19T02:58:26.279Z] s: "error: restore status for [testuser1-1705632036-tp-restore-1705632531] expected was [[PartialSuccess Success]] but got [Failed] because of [Error starting Application Restore for volumes: failed to restore pvc mysql-data: max retries done, volumesnapshot [mysql-ibm-pxb-0-82949-01-19-02h38m54s/restore-vs-72502f9d3c87-a925393aea3a] status is not updated], Description: Verifying that restore is possible", [2024-01-19T02:58:26.279Z] } [2024-01-19T02:58:26.279Z] error: restore status for [testuser1-1705632036-tp-restore-1705632531] expected was [[PartialSuccess Success]] but got [Failed] because of [Error starting Application Restore for volumes: failed to restore pvc mysql-data: max retries done, volumesnapshot [mysql-ibm-pxb-0-82949-01-19-02h38m54s/restore-vs-72502f9d3c87-a925393aea3a] status is not updated], Description: Verifying that restore is possible [2024-01-19T02:58:26.279Z] occurred [2024-01-19T02:58:26.279Z] [2024-01-19T02:58:26.279Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-19T02:58:26.279Z] [2024-01-19T02:58:26.279Z] Full Stack Trace [2024-01-19T02:58:26.279Z] github.com/portworx/torpedo/pkg/aetosutil.(*Dashboard).VerifyFatal(0x88eb700?, {0x702af20?, 0xc0015c57c0}, {0x0?, 0x0}, {0x7f3c0f3, 0x22}) [2024-01-19T02:58:26.279Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 +0x143 [2024-01-19T02:58:26.279Z] github.com/portworx/torpedo/tests/backup.ValidateSharedBackupWithUsers({0xc00115e390, 0x14}, 0x3, {0xc00057b8a8, 0x18}, {0xc002088de0, 0x2a}) [2024-01-19T02:58:26.279Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_helper.go:1430 +0x3e7 [2024-01-19T02:58:26.279Z] github.com/portworx/torpedo/tests/backup.glob..func92.2.11() [2024-01-19T02:58:26.279Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:3103 +0xd9 [2024-01-19T02:58:26.279Z] github.com/portworx/torpedo/tests/backup.glob..func92.2() [2024-01-19T02:58:26.279Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:3091 +0xd8e [2024-01-19T02:58:26.279Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T02:58:26.279Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T02:58:26.279Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T02:58:26.279Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T02:58:26.279Z] created by testing.(*T).Run [2024-01-19T02:58:26.279Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T02:58:26.279Z] ------------------------------ [2024-01-19T02:58:26.279Z] STEP: schedule applications [2024-01-19T02:58:26.279Z] STEP: Validate applications [2024-01-19T02:58:26.279Z] STEP: validate applications [2024-01-19T02:58:26.279Z] STEP: validate mysql-ibm app's volumes [2024-01-19T02:58:26.279Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T02:58:26.279Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T02:58:26.279Z] STEP: get mysql-ibm app's volume: pvc-aca58b28-73d1-4bba-8304-5e8dddda4495 inspected by the volume driver [2024-01-19T02:58:26.279Z] STEP: wait for mysql-ibm app to start running [2024-01-19T02:58:26.279Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T02:58:26.279Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T02:58:26.279Z] STEP: Validate Px pod restart count [2024-01-19T02:58:26.279Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T02:58:26.279Z] STEP: Create multiple Users [2024-01-19T02:58:26.279Z] STEP: Creating backup location and cloud setting [2024-01-19T02:58:26.279Z] STEP: Register source and destination cluster for backup [2024-01-19T02:58:26.279Z] STEP: Taking backups of application for each user [2024-01-19T02:58:26.279Z] STEP: Share backup with users with different access level [2024-01-19T02:58:26.279Z] STEP: Removing backup location after sharing backup with all the users [2024-01-19T02:58:26.279Z] STEP: Adding new backup location to the cluster [2024-01-19T02:58:26.279Z] STEP: Taking backups of application for each user again with new backup location [2024-01-19T02:58:26.279Z] STEP: Share backup with users again with different access level [2024-01-19T02:58:26.279Z] STEP: Validate if the users with different access level can restore/delete backup [2024-01-19T02:58:26.279Z] STEP: destroy apps [2024-01-19T02:58:26.279Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T02:58:26.279Z] STEP: start destroying mysql-ibm app [2024-01-19T02:58:26.279Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T02:58:26.280Z] S2024-01-19 02:58:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Log Dir: /testresults//ManualAndScheduledBackupUsingNamespaceAndResourceLabel.log [2024-01-19T02:58:26.280Z] INFO[2024-01-19 02:58:26] --------Test Start------ [2024-01-19T02:58:26.280Z] INFO[2024-01-19 02:58:26] #Test: ManualAndScheduledBackupUsingNamespaceAndResourceLabel [2024-01-19T02:58:26.280Z] INFO[2024-01-19 02:58:26] #Description: Manual, schedule backup and restore, using namespace label and resource label [2024-01-19T02:58:26.280Z] INFO[2024-01-19 02:58:26] ------------------------ [2024-01-19T02:58:26.280Z] 2024-01-19 02:58:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Deploy applications [2024-01-19T02:58:26.280Z] 2024-01-19 02:58:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T02:58:26.280Z] 2024-01-19 02:58:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T02:58:26.280Z] 2024-01-19 02:58:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T02:58:26.538Z] 2024-01-19 02:58:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T02:58:26.538Z] 2024-01-19 02:58:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T02:58:26.538Z] 2024-01-19 02:58:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T02:58:26.539Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T02:58:26.539Z] 2024-01-19 02:58:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T02:58:26.540Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T02:58:26.540Z] 2024-01-19 02:58:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T02:58:26.540Z] 2024-01-19 02:58:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T02:58:26.540Z] 2024-01-19 02:58:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T02:58:26.798Z] 2024-01-19 02:58:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T02:58:26.798Z] 2024-01-19 02:58:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T02:58:26.798Z] 2024-01-19 02:58:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T02:58:26.798Z] 2024-01-19 02:58:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T02:58:26.799Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T02:58:27.056Z] 2024-01-19 02:58:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T02:58:27.057Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T02:58:27.057Z] 2024-01-19 02:58:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T02:58:27.057Z] 2024-01-19 02:58:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T02:58:27.057Z] 2024-01-19 02:58:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T02:58:27.057Z] 2024-01-19 02:58:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T02:58:27.057Z] 2024-01-19 02:58:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T02:58:27.057Z] 2024-01-19 02:58:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T02:58:27.057Z] 2024-01-19 02:58:27 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T02:58:27.058Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T02:58:27.314Z] 2024-01-19 02:58:27 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T02:58:27.314Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T02:58:27.314Z] 2024-01-19 02:58:27 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T02:58:27.314Z] 2024-01-19 02:58:27 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Created namespaces : [mysql-ibm-pxb-0-84850-01-19-02h58m26s mysql-ibm-pxb-1-84850-01-19-02h58m26s mysql-ibm-pxb-2-84850-01-19-02h58m26s] [2024-01-19T02:58:27.314Z] 2024-01-19 02:58:27 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Validate applications [2024-01-19T02:58:27.314Z] 2024-01-19 02:58:27 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Validate applications [2024-01-19T02:58:27.314Z] 2024-01-19 02:58:27 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Validating mysql-ibm app [2024-01-19T02:58:27.315Z] 2024-01-19 02:58:27 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Validating mysql-ibm app's volumes [2024-01-19T02:58:27.315Z] 2024-01-19 02:58:27 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T02:58:27.315Z] 2024/01/19 02:58:27 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T02:58:37.262Z] 2024/01/19 02:58:37 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T02:58:49.432Z] 2024-01-19 02:58:47 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-84850-01-19-02h58m26s [2024-01-19T02:58:49.432Z] 2024-01-19 02:58:47 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T02:58:49.432Z] 2024-01-19 02:58:47 +0000:[WARNING] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T02:58:49.432Z] 2024-01-19 02:58:47 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] wait for mysql-ibm app to start running [2024-01-19T02:58:49.432Z] 2024-01-19 02:58:47 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T02:58:49.432Z] 2024/01/19 02:58:47 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:58:49.432Z] pod name:mysql-7d766cd65c-jflp6 namespace:mysql-ibm-pxb-0-84850-01-19-02h58m26s running:false ready:false node:10.240.64.27 [2024-01-19T02:58:49.432Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:58:59.384Z] 2024/01/19 02:58:57 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:58:59.384Z] pod name:mysql-7d766cd65c-jflp6 namespace:mysql-ibm-pxb-0-84850-01-19-02h58m26s running:true ready:false node:10.240.64.27 [2024-01-19T02:58:59.384Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:59:09.332Z] 2024/01/19 02:59:07 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:59:09.332Z] pod name:mysql-7d766cd65c-jflp6 namespace:mysql-ibm-pxb-0-84850-01-19-02h58m26s running:true ready:false node:10.240.64.27 [2024-01-19T02:59:09.332Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:59:19.278Z] 2024/01/19 02:59:17 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:59:19.278Z] pod name:mysql-7d766cd65c-jflp6 namespace:mysql-ibm-pxb-0-84850-01-19-02h58m26s running:true ready:false node:10.240.64.27 [2024-01-19T02:59:19.278Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:59:29.234Z] 2024/01/19 02:59:27 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:59:29.234Z] pod name:mysql-7d766cd65c-jflp6 namespace:mysql-ibm-pxb-0-84850-01-19-02h58m26s running:true ready:false node:10.240.64.27 [2024-01-19T02:59:29.234Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:59:39.186Z] 2024/01/19 02:59:38 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:59:39.186Z] pod name:mysql-7d766cd65c-jflp6 namespace:mysql-ibm-pxb-0-84850-01-19-02h58m26s running:true ready:false node:10.240.64.27 [2024-01-19T02:59:39.186Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:59:49.150Z] 2024/01/19 02:59:48 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:59:49.150Z] pod name:mysql-7d766cd65c-jflp6 namespace:mysql-ibm-pxb-0-84850-01-19-02h58m26s running:true ready:false node:10.240.64.27 [2024-01-19T02:59:49.150Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T02:59:59.114Z] 2024/01/19 02:59:58 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T02:59:59.114Z] pod name:mysql-7d766cd65c-jflp6 namespace:mysql-ibm-pxb-0-84850-01-19-02h58m26s running:true ready:false node:10.240.64.27 [2024-01-19T02:59:59.114Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:00:09.058Z] 2024-01-19 03:00:08 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T03:00:09.058Z] 2024-01-19 03:00:08 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T03:00:09.058Z] 2024-01-19 03:00:08 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] validate if mysql-ibm app's volumes are setup [2024-01-19T03:00:09.058Z] 2024-01-19 03:00:08 +0000:[WARNING] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:00:09.058Z] 2024-01-19 03:00:08 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:00:09.058Z] 2024-01-19 03:00:08 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T03:00:09.058Z] 2024-01-19 03:00:08 +0000:[WARNING] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T03:00:09.058Z] 2024-01-19 03:00:08 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Validating mysql-ibm app [2024-01-19T03:00:09.058Z] 2024-01-19 03:00:08 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Validating mysql-ibm app's volumes [2024-01-19T03:00:09.058Z] 2024-01-19 03:00:08 +0000:[WARNING] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:00:09.058Z] 2024-01-19 03:00:08 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:00:09.058Z] 2024-01-19 03:00:08 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-1-84850-01-19-02h58m26s [2024-01-19T03:00:09.059Z] 2024-01-19 03:00:08 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T03:00:09.059Z] 2024-01-19 03:00:08 +0000:[WARNING] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T03:00:09.059Z] 2024-01-19 03:00:08 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] wait for mysql-ibm app to start running [2024-01-19T03:00:09.059Z] 2024-01-19 03:00:08 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T03:00:09.316Z] 2024-01-19 03:00:09 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T03:00:09.572Z] 2024-01-19 03:00:09 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T03:00:10.503Z] 2024-01-19 03:00:10 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] validate if mysql-ibm app's volumes are setup [2024-01-19T03:00:10.503Z] 2024-01-19 03:00:10 +0000:[WARNING] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:00:10.503Z] 2024-01-19 03:00:10 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:00:10.503Z] 2024-01-19 03:00:10 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T03:00:10.503Z] 2024-01-19 03:00:10 +0000:[WARNING] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T03:00:10.503Z] 2024-01-19 03:00:10 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Validating mysql-ibm app [2024-01-19T03:00:10.503Z] 2024-01-19 03:00:10 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Validating mysql-ibm app's volumes [2024-01-19T03:00:10.503Z] 2024-01-19 03:00:10 +0000:[WARNING] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:00:10.503Z] 2024-01-19 03:00:10 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:00:10.761Z] 2024-01-19 03:00:10 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-2-84850-01-19-02h58m26s [2024-01-19T03:00:10.761Z] 2024-01-19 03:00:10 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T03:00:10.761Z] 2024-01-19 03:00:10 +0000:[WARNING] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T03:00:10.761Z] 2024-01-19 03:00:10 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] wait for mysql-ibm app to start running [2024-01-19T03:00:10.761Z] 2024-01-19 03:00:10 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T03:00:11.017Z] 2024-01-19 03:00:10 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T03:00:11.579Z] 2024-01-19 03:00:11 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T03:00:12.519Z] 2024-01-19 03:00:12 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] validate if mysql-ibm app's volumes are setup [2024-01-19T03:00:12.519Z] 2024-01-19 03:00:12 +0000:[WARNING] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:00:12.519Z] 2024-01-19 03:00:12 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:00:12.519Z] 2024-01-19 03:00:12 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T03:00:12.519Z] 2024-01-19 03:00:12 +0000:[WARNING] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T03:00:12.519Z] 2024-01-19 03:00:12 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Generate and add labels to namespaces [2024-01-19T03:00:12.776Z] INFO[2024-01-19 03:00:12] Verifying : Description : Adding labels [map[0-bea88fde-30c2-4636-8091-31ff5dd72c03:bea88fde-30c2-4636-8091-31ff5dd72c03]] to namespaces [[mysql-ibm-pxb-0-84850-01-19-02h58m26s mysql-ibm-pxb-1-84850-01-19-02h58m26s mysql-ibm-pxb-2-84850-01-19-02h58m26s]] [2024-01-19T03:00:12.776Z] INFO[2024-01-19 03:00:12] Actual:true, Expected: true [2024-01-19T03:00:12.776Z] 2024-01-19 03:00:12 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Adding labels to resources [2024-01-19T03:00:12.776Z] INFO[2024-01-19 03:00:12] Verifying : Description : Fetching pvc list from namespace [mysql-ibm-pxb-0-84850-01-19-02h58m26s] [2024-01-19T03:00:12.776Z] INFO[2024-01-19 03:00:12] Actual:true, Expected: true [2024-01-19T03:00:13.032Z] INFO[2024-01-19 03:00:12] Verifying : Description : Fetching pvc pointer &PersistentVolumeClaim{ObjectMeta:{mysql-data mysql-ibm-pxb-0-84850-01-19-02h58m26s 3d460a06-3366-4263-b3dd-53df850770d3 1575188 0 2024-01-19 02:58:26 +0000 UTC map[] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-19 02:58:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-19 02:58:37 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{2147483648 0} {} 2Gi BinarySI},},},VolumeName:pvc-3d460a06-3366-4263-b3dd-53df850770d3,Selector:nil,StorageClassName:*mysql-sc,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{10737418240 0} {} 10Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},} for pvc {{ } {mysql-data mysql-ibm-pxb-0-84850-01-19-02h58m26s 3d460a06-3366-4263-b3dd-53df850770d3 1575188 0 2024-01-19 02:58:26 +0000 UTC map[] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-19 02:58:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-19 02:58:37 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{2147483648 0} {} 2Gi BinarySI}]} pvc-3d460a06-3366-4263-b3dd-53df850770d3 0xc00047e940 0xc00047e950 nil nil} {Bound [ReadWriteOnce] map[storage:{{10737418240 0} {} 10Gi BinarySI}] [] map[] }} [2024-01-19T03:00:13.032Z] INFO[2024-01-19 03:00:12] Actual:true, Expected: true [2024-01-19T03:00:13.032Z] 2024-01-19 03:00:12 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.AddLabelToResource:#5030] - Adding label [edffc0c3-8249-4e9f-a79b-744c405b60af=d095e726-c3cc-4da8-b72f-6be96ae30ea7] to PVC mysql-data [2024-01-19T03:00:13.288Z] INFO[2024-01-19 03:00:13] Verifying : Description : Adding labels edffc0c3-8249-4e9f-a79b-744c405b60af=d095e726-c3cc-4da8-b72f-6be96ae30ea7 to resource &PersistentVolumeClaim{ObjectMeta:{mysql-data mysql-ibm-pxb-0-84850-01-19-02h58m26s 3d460a06-3366-4263-b3dd-53df850770d3 1575188 0 2024-01-19 02:58:26 +0000 UTC map[edffc0c3-8249-4e9f-a79b-744c405b60af:d095e726-c3cc-4da8-b72f-6be96ae30ea7] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-19 02:58:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-19 02:58:37 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{2147483648 0} {} 2Gi BinarySI},},},VolumeName:pvc-3d460a06-3366-4263-b3dd-53df850770d3,Selector:nil,StorageClassName:*mysql-sc,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{10737418240 0} {} 10Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},} [2024-01-19T03:00:13.288Z] INFO[2024-01-19 03:00:13] Actual:true, Expected: true [2024-01-19T03:00:13.288Z] INFO[2024-01-19 03:00:13] Verifying : Description : Fetching configmap list from namespace [mysql-ibm-pxb-0-84850-01-19-02h58m26s] [2024-01-19T03:00:13.288Z] INFO[2024-01-19 03:00:13] Actual:true, Expected: true [2024-01-19T03:00:13.544Z] INFO[2024-01-19 03:00:13] Verifying : Description : Fetching configmap pointer &ConfigMap{ObjectMeta:{kube-****-ca.crt mysql-ibm-pxb-0-84850-01-19-02h58m26s 881960ad-356f-4001-8596-7dc43989b632 1574831 0 2024-01-19 02:58:26 +0000 UTC map[] map[kubernetes.io/description:Contains a CA bundle that can be used to verify the kube-apiserver when using internal endpoints such as the internal service IP or kubernetes.default.svc. No other usage is guaranteed across distributions of Kubernetes clusters.] [] [] [{kube-controller-manager Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:data":{".":{},"f:ca.crt":{}},"f:metadata":{"f:annotations":{".":{},"f:kubernetes.io/description":{}}}} }]},Data:map[string]string{ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T03:00:13.544Z] MIIDCjCCAfKgAwIBAgIUWkn6rgBaC+VWV1XdJjGA3DwoRRYwDQYJKoZIhvcNAQEL [2024-01-19T03:00:13.544Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T03:00:13.544Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T03:00:13.544Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T03:00:13.544Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T03:00:13.544Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T03:00:13.544Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T03:00:13.544Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T03:00:13.544Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T03:00:13.544Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T03:00:13.544Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAHbD [2024-01-19T03:00:13.544Z] 4nWTgt/8sat7SbbHxZ+JcMGRAIaeLAjMJxkxyqD8xUq6YQcgbjDVHo1Qbh3PXB3T [2024-01-19T03:00:13.544Z] 5BWUsRbDoKi0eRJx/5cyI3846SYEAOpuUfJrg9X2CH7ZoYhxeJ42dqt3MdoMJEM1 [2024-01-19T03:00:13.544Z] k3e2rRqAKuAxwNXugHryxQM+1ehOJjrEuT5DVK6qu8+JYRcT1hBZb6hrebKPp5eG [2024-01-19T03:00:13.544Z] JvBzHP4YOATIAXhfH0Uj5Y+JhwoOdpXmOFpz+6gITPD4aNzzJZe/0ADSFfOyUJaR [2024-01-19T03:00:13.544Z] g5NnJnkeGqMM3SLsur25r+m3y0i3t+RdocZ/4mHTEnF4KWVWvZy4GjENiEBcXEol [2024-01-19T03:00:13.544Z] o4SZVaP4AF7XQJ5vdA0= [2024-01-19T03:00:13.544Z] -----END CERTIFICATE----- [2024-01-19T03:00:13.544Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:13.544Z] MIIDCjCCAfKgAwIBAgIUXhIIFCc/NxZJn6hNEFoZuPQ8N5UwDQYJKoZIhvcNAQEL [2024-01-19T03:00:13.545Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T03:00:13.545Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T03:00:13.545Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T03:00:13.545Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T03:00:13.545Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T03:00:13.545Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T03:00:13.545Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T03:00:13.545Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T03:00:13.545Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T03:00:13.545Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAKG3 [2024-01-19T03:00:13.545Z] nVfv4IkgHVOxgbrNIoYhjvBvVJcLZub0C9yidexLA3A61p1a7b/OpidG5zShPhH2 [2024-01-19T03:00:13.545Z] HRdQcqgp0HIxaS+/oLD3eElOnnC/EKD359K6n3kN66sMgvNbHxNFM84Au0TXqmbT [2024-01-19T03:00:13.546Z] w8afrKjrfe/VglpHC2GfmgHMF4YH+ZF48C1RZA6z2oXj6Aj+uww0PjiClE9qUYqK [2024-01-19T03:00:13.546Z] R9uRa32prExihsi+Bto7n2Axo92qUiK2KrkEdQitnQfH5z/PojS78vbyIKp1O86N [2024-01-19T03:00:13.546Z] qRImvCYY17CPxP+JUQ7dxWDMF4NTgSnabkp+cd5i/pyoAb0guQFlQQUnHAZa4QNC [2024-01-19T03:00:13.546Z] 7fgTo1IUwWyDmwCnN5U= [2024-01-19T03:00:13.546Z] -----END CERTIFICATE----- [2024-01-19T03:00:13.546Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:13.546Z] MIIDCjCCAfKgAwIBAgIUJPzO3O+wmUt4gHCkl3ojsvwZL6owDQYJKoZIhvcNAQEL [2024-01-19T03:00:13.546Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T03:00:13.546Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T03:00:13.546Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T03:00:13.546Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T03:00:13.546Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T03:00:13.546Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T03:00:13.546Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T03:00:13.546Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T03:00:13.546Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T03:00:13.546Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAGVt [2024-01-19T03:00:13.546Z] g9qX+r8kyGYsvFavmnDxgMK1Zkz80irQJ2BJlDHdK1hp5aqpRbT2z2YgLog5ZMIZ [2024-01-19T03:00:13.546Z] uI36uPCOXo8+Gol1jT7D6vsgMWm0Tk9jCNbO4jPkv+wm2q1B9UZo94UGiS0K+G0S [2024-01-19T03:00:13.546Z] nccl4jmSq1gWpBZIVetdDFNjpipKjhEOVQM7x33APhACxSAWRp9iEr6V/CzMOgkf [2024-01-19T03:00:13.546Z] RHyz5D7ljh4a2XVH7++eKsj2QPfKyiKtIG72tCccQ9zGnzveiXXb4Ai8r+DyKH4Z [2024-01-19T03:00:13.546Z] zDt5Hb4ZFl72E3Rl4bIh/VQBVsr9u/SFR1p/Emyq+kGXsn+9YMiM9tK4mLUtjdox [2024-01-19T03:00:13.546Z] LMWUWTelNRGWAiaTVz8= [2024-01-19T03:00:13.546Z] -----END CERTIFICATE----- [2024-01-19T03:00:13.546Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:13.546Z] MIIDCjCCAfKgAwIBAgIUBxbEoan/HnRZ+2deRj+2RvGMrWQwDQYJKoZIhvcNAQEL [2024-01-19T03:00:13.546Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T03:00:13.546Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T03:00:13.546Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T03:00:13.546Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T03:00:13.546Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T03:00:13.546Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T03:00:13.546Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T03:00:13.546Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T03:00:13.546Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T03:00:13.546Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAB65 [2024-01-19T03:00:13.546Z] nHU+98oTP0U2J0/TsL5GFeBTH7YyUXUNRJvPWxkA8riB3swoScFKbavO6XJ/hrI2 [2024-01-19T03:00:13.546Z] ywApOz77Iyq58OLa5aIC5aaPRf3YE8LbJQYIHmtza21LXhtlKVSrLu4W58PauI8Z [2024-01-19T03:00:13.546Z] 7iaL9QPUihHFK7wrZ2ZpabN2rRJvTGWWKlhmgFTjI+71gPrXWmYBHIKmTv55V5m1 [2024-01-19T03:00:13.546Z] LxORTR/gBXeH9gmM7wz1cbaYNazqs8R1LS5SW76p/y9RLjTzU6N39VYefMMGwbsQ [2024-01-19T03:00:13.546Z] DlD11wvMBxTBp3hQYc3pdVLkVEQt8trVIwEUts7zfwUTCu/z8gH7H5XePpEVNana [2024-01-19T03:00:13.546Z] JhlAcXHCO7nN/ddJmoc= [2024-01-19T03:00:13.546Z] -----END CERTIFICATE----- [2024-01-19T03:00:13.546Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:13.546Z] MIIDGDCCAgCgAwIBAgIUEvLaarpXlv1cVvG6nMuxh2DNCKQwDQYJKoZIhvcNAQEL [2024-01-19T03:00:13.546Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T03:00:13.546Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T03:00:13.546Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T03:00:13.546Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T03:00:13.546Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T03:00:13.546Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T03:00:13.546Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T03:00:13.546Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T03:00:13.546Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T03:00:13.546Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T03:00:13.546Z] 9w0BAQsFAAOCAQEAOPakzVUmAqHhmQViZZHSQeRh/xYQBrtn6dnEo6X55SXNWcPU [2024-01-19T03:00:13.547Z] 4EEW3DvwBf75uqxKeGoXHTBpqNHG2x1+IncvIKJmi5OWM3aR/vEyJoBOchPBl+A4 [2024-01-19T03:00:13.547Z] 0xZO1vKiS2djxz8XHMkR70cxD8hv/zDJeIQbgBroF4Wa7rim5+y7jgO0O7Rs1zhT [2024-01-19T03:00:13.547Z] LDDStJ8aUKwcVRmOv7NFQxbQHXNx07kaUE2SQJfhryzZmotoBiymmQT2v0H4/4q0 [2024-01-19T03:00:13.547Z] 5dylRhSCWfwN5+It40/C3b1pbC2d9Iv/eO+B0wqg13dLF3xuA1JblEgnqA9mdNnp [2024-01-19T03:00:13.547Z] P6EgKTQBMiO6nPFyYIPqg1eWYkb1eXu9aEkmbw== [2024-01-19T03:00:13.547Z] -----END CERTIFICATE----- [2024-01-19T03:00:13.547Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:13.547Z] MIIDGDCCAgCgAwIBAgIUWf/qoBANFWF/a/yZr7wfLBMfItAwDQYJKoZIhvcNAQEL [2024-01-19T03:00:13.547Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T03:00:13.547Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T03:00:13.547Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T03:00:13.547Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T03:00:13.547Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T03:00:13.547Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T03:00:13.547Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T03:00:13.547Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T03:00:13.547Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T03:00:13.547Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T03:00:13.547Z] 9w0BAQsFAAOCAQEAdCguadZU6WJAuuAtWzvYgftPKkDKNBeGVgbqRwWRCB+QqVFl [2024-01-19T03:00:13.547Z] +3ZOQTStpnXJsY3Ig31hmL88BkBPQ47ENEzs7Czhuz59m+dcBhh/buUoIERSnfUY [2024-01-19T03:00:13.547Z] lWIfqxQevwHYZ3Kjx0l3qliYjKBQJo5xDPnukGb4yUnmrCdZ3fxMzlnr0Ais9rUH [2024-01-19T03:00:13.547Z] MjiDsjP/oUa8naQUTZO4DTRZqeEldzNXnxF6YH+3moTkwRLt9xzirNwECmJ9qxrC [2024-01-19T03:00:13.547Z] eCA7zORxbFSftYLnQiGWUS1dnxBuLOvJHjYPC6EkPPrvqnx2fXcTE1V8GizVRrA6 [2024-01-19T03:00:13.547Z] NPxt+e4qZgU/Nk/Jkpsb7h1FVcKtS8CvmtbdqQ== [2024-01-19T03:00:13.547Z] -----END CERTIFICATE----- [2024-01-19T03:00:13.547Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:13.547Z] MIIDGDCCAgCgAwIBAgIUI6D0WS4tl52N+w2lwAS+6V0DPyUwDQYJKoZIhvcNAQEL [2024-01-19T03:00:13.547Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T03:00:13.547Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T03:00:13.547Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T03:00:13.547Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T03:00:13.547Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T03:00:13.547Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T03:00:13.547Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T03:00:13.547Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T03:00:13.547Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T03:00:13.547Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T03:00:13.547Z] 9w0BAQsFAAOCAQEAAS4yElh6TMyP18ipzRA0Wenukevjs/vtkAFOsv1u5GNeDIrv [2024-01-19T03:00:13.547Z] lNi6/VjvtS+DoFNzQnTf3QoO6GTuoWHLQWt6Z/V5AwIsnOXduSXLugMwEv3j9l50 [2024-01-19T03:00:13.547Z] aDkA7vJ3OH5cGqIvIzxQN7IUOEvZNhY67GFP7+zAzVNRpJ1FHYbhQGPuFD//b9G3 [2024-01-19T03:00:13.547Z] hr4E3+iUEc+F/iBA+caA8wRx1zR383Jz6wuoFt7n7HpWNcgMvoh91JlUAK7yUp0B [2024-01-19T03:00:13.547Z] ekuG6q7HtVsk6X+aB1RyWfrSJjih75W7b+p+/bV4vNmQS1wSGSOaSIuVRvMhmpwf [2024-01-19T03:00:13.547Z] FpT+4rUpiCxklD69/MET/1cWxqvao/ppf6phzg== [2024-01-19T03:00:13.547Z] -----END CERTIFICATE----- [2024-01-19T03:00:13.547Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:13.547Z] MIIDGDCCAgCgAwIBAgIUIVCCgYV9enDck18n89YsA5b2IdgwDQYJKoZIhvcNAQEL [2024-01-19T03:00:13.547Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T03:00:13.547Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T03:00:13.547Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T03:00:13.547Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T03:00:13.547Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T03:00:13.547Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T03:00:13.547Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T03:00:13.547Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T03:00:13.547Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T03:00:13.547Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T03:00:13.547Z] 9w0BAQsFAAOCAQEAa3eFm1ivFf5Vom6a/7E1gDdjl7cjUZ5cqg332Ky1NdzEOiwW [2024-01-19T03:00:13.547Z] mx3yEk35qCd8LAoFMvPvPIuMx1NDS8kwy3FmIOjKnZapHA31ZUpn/gT/seH4GFIU [2024-01-19T03:00:13.548Z] oZw+rODV5h6jmtCVugAn48u/LxOXbY417JWwaeDx5uw+NzOSyebe+td2PbtUjf5X [2024-01-19T03:00:13.548Z] 6xcwYUmaUGsIAgfIMM12hzXOI2/q1qz5dAqwSgnR7dKBReOXnUoFpNOMpxlJ70p/ [2024-01-19T03:00:13.548Z] WUU1aWIWZqKrNI7Pg1T5jljq7+KbT3+A6+ux0FLExr2c1tw3NndXS2Y9fO55m5/q [2024-01-19T03:00:13.548Z] 0EcrTdKx4+QjbWcja3WdX2pGDk0XGNgx+EXn6Q== [2024-01-19T03:00:13.548Z] -----END CERTIFICATE----- [2024-01-19T03:00:13.548Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:13.548Z] MIIFATCCAumgAwIBAgIRAKc9ZKBASymy5TLOEp57N98wDQYJKoZIhvcNAQELBQAw [2024-01-19T03:00:13.548Z] GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDMyMzIyNTM0NloXDTM2 [2024-01-19T03:00:13.548Z] MDMyMzIyNTM0NlowGjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMIICIjANBgkq [2024-01-19T03:00:13.548Z] hkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA+pYHvQw5iU3v2b3iNuYNKYgsWD6KU7aJ [2024-01-19T03:00:13.548Z] diddtZQxSWYzUI3U0I1UsRPTxnhTifs/M9NW4ZlV13ZfB7APwC8oqKOIiwo7IwlP [2024-01-19T03:00:13.548Z] xg0VKgyz+kT8RJfYr66PPIYP0fpTeu42LpMJ+CKo9sbpgVNDZN2z/qiXrRNX/VtG [2024-01-19T03:00:13.548Z] TkPV7a44fZ5bHHVruAxvDnylpQxJobtCBWlJSsbIRGFHMc2z88eUz9NmIOWUKGGj [2024-01-19T03:00:13.548Z] EmP76x8OfRHpIpuxRSCjn0+i9+hR2siIOpcMOGd+40uVJxbRRP5ZXnUFa2fF5FWd [2024-01-19T03:00:13.548Z] O0u0RPI8HON0ovhrwPJY+4eWKkQzyC611oLPYGQ4EbifRsTsCxUZqyUuStGyp8oa [2024-01-19T03:00:13.548Z] aoSKfF6X0+KzGgwwnrjRTUpIl19A92KR0Noo6h622OX+4sZiO/JQdkuX5w/HupK0 [2024-01-19T03:00:13.548Z] A0M0WSMCvU6GOhjGotmh2VTEJwHHY4+TUk0iQYRtv1crONklyZoAQPD76hCrC8Cr [2024-01-19T03:00:13.548Z] IbgsZLfTMC8TWUoMbyUDgvgYkHKMoPm0VGVVuwpRKJxv7+2wXO+pivrrUl2Q9fPe [2024-01-19T03:00:13.548Z] Kk055nJLMV9yPUdig8othUKrRfSxli946AEV1eEOhxddfEwBE3Lt2xn0hhiIedbb [2024-01-19T03:00:13.548Z] Ftf/5kEWFZkXyUmMJK8Ra76Kus2ABueUVEcZ48hrRr1Hf1N9n59VbTUaXgeiZA50 [2024-01-19T03:00:13.548Z] qXf2bymE6F8CAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMB [2024-01-19T03:00:13.548Z] Af8wHQYDVR0OBBYEFMEmdKSKRKDm+iAo2FwjmkWIGHngMA0GCSqGSIb3DQEBCwUA [2024-01-19T03:00:13.548Z] A4ICAQBCPw74M9X/Xx04K1VAES3ypgQYH5bf9FXVDrwhRFSVckria/7dMzoF5wln [2024-01-19T03:00:13.548Z] uq9NGsjkkkDg17AohcQdr8alH4LvPdxpKr3BjpvEcmbqF8xH+MbbeUEnmbSfLI8H [2024-01-19T03:00:13.548Z] sefuhXF9AF/9iYvpVNC8FmJ0OhiVv13VgMQw0CRKkbtjZBf8xaEhq/YqxWVsgOjm [2024-01-19T03:00:13.548Z] dm5CAQ2X0aX7502x8wYRgMnZhA5goC1zVWBVAi8yhhmlhhoDUfg17cXkmaJC5pDd [2024-01-19T03:00:13.548Z] oenZ9NVhW8eDb03MFCrWNvIh89DDeCGWuWfDltDq0n3owyL0IeSn7RfpSclpxVmV [2024-01-19T03:00:13.548Z] /53jkYjwIgxIG7Gsv0LKMbsf6QdBcTjhvfZyMIpBRkTe3zuHd2feKzY9lEkbRvRQ [2024-01-19T03:00:13.548Z] zbh4Ps5YBnG6CKJPTbe2hfi3nhnw/MyEmF3zb0hzvLWNrR9XW3ibb2oL3424XOwc [2024-01-19T03:00:13.548Z] VjrTSCLzO9Rv6s5wi03qoWvKAQQAElqTYRHhynJ3w6wuvKYF5zcZF3MDnrVGLbh1 [2024-01-19T03:00:13.548Z] Q9ePRFBCiXOQ6wPLoUhrrbZ8LpFUFYDXHMtYM7P9sc9IAWoONXREJaO08zgFtMp4 [2024-01-19T03:00:13.548Z] 8iyIYUyQAbsvx8oD2M8kRvrIRSrRJSl6L957b4AFiLIQ/GgV2curs0jje7Edx34c [2024-01-19T03:00:13.548Z] idWw1VrejtwclobqNMVtG3EiPUIpJGpbMcJgbiLSmKkrvQtGng== [2024-01-19T03:00:13.548Z] -----END CERTIFICATE----- [2024-01-19T03:00:13.548Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} from namespace mysql-ibm-pxb-0-84850-01-19-02h58m26s [2024-01-19T03:00:13.548Z] INFO[2024-01-19 03:00:13] Actual:true, Expected: true [2024-01-19T03:00:13.548Z] 2024-01-19 03:00:13 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.AddLabelToResource:#5038] - Adding label [edffc0c3-8249-4e9f-a79b-744c405b60af=d095e726-c3cc-4da8-b72f-6be96ae30ea7] to ConfigMap kube-****-ca.crt [2024-01-19T03:00:13.804Z] INFO[2024-01-19 03:00:13] Verifying : Description : Adding labels edffc0c3-8249-4e9f-a79b-744c405b60af=d095e726-c3cc-4da8-b72f-6be96ae30ea7 to resource &ConfigMap{ObjectMeta:{kube-****-ca.crt mysql-ibm-pxb-0-84850-01-19-02h58m26s 881960ad-356f-4001-8596-7dc43989b632 1574831 0 2024-01-19 02:58:26 +0000 UTC map[edffc0c3-8249-4e9f-a79b-744c405b60af:d095e726-c3cc-4da8-b72f-6be96ae30ea7] map[kubernetes.io/description:Contains a CA bundle that can be used to verify the kube-apiserver when using internal endpoints such as the internal service IP or kubernetes.default.svc. No other usage is guaranteed across distributions of Kubernetes clusters.] [] [] [{kube-controller-manager Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:data":{".":{},"f:ca.crt":{}},"f:metadata":{"f:annotations":{".":{},"f:kubernetes.io/description":{}}}} }]},Data:map[string]string{ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T03:00:13.804Z] MIIDCjCCAfKgAwIBAgIUWkn6rgBaC+VWV1XdJjGA3DwoRRYwDQYJKoZIhvcNAQEL [2024-01-19T03:00:13.804Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T03:00:13.804Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T03:00:13.804Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T03:00:13.804Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T03:00:13.804Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T03:00:13.804Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T03:00:13.804Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T03:00:13.804Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T03:00:13.804Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T03:00:13.804Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAHbD [2024-01-19T03:00:13.804Z] 4nWTgt/8sat7SbbHxZ+JcMGRAIaeLAjMJxkxyqD8xUq6YQcgbjDVHo1Qbh3PXB3T [2024-01-19T03:00:13.804Z] 5BWUsRbDoKi0eRJx/5cyI3846SYEAOpuUfJrg9X2CH7ZoYhxeJ42dqt3MdoMJEM1 [2024-01-19T03:00:13.804Z] k3e2rRqAKuAxwNXugHryxQM+1ehOJjrEuT5DVK6qu8+JYRcT1hBZb6hrebKPp5eG [2024-01-19T03:00:13.804Z] JvBzHP4YOATIAXhfH0Uj5Y+JhwoOdpXmOFpz+6gITPD4aNzzJZe/0ADSFfOyUJaR [2024-01-19T03:00:13.804Z] g5NnJnkeGqMM3SLsur25r+m3y0i3t+RdocZ/4mHTEnF4KWVWvZy4GjENiEBcXEol [2024-01-19T03:00:13.804Z] o4SZVaP4AF7XQJ5vdA0= [2024-01-19T03:00:13.805Z] -----END CERTIFICATE----- [2024-01-19T03:00:13.805Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:13.805Z] MIIDCjCCAfKgAwIBAgIUXhIIFCc/NxZJn6hNEFoZuPQ8N5UwDQYJKoZIhvcNAQEL [2024-01-19T03:00:13.805Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T03:00:13.805Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T03:00:13.805Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T03:00:13.805Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T03:00:13.805Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T03:00:13.805Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T03:00:13.805Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T03:00:13.805Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T03:00:13.805Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T03:00:13.805Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAKG3 [2024-01-19T03:00:13.805Z] nVfv4IkgHVOxgbrNIoYhjvBvVJcLZub0C9yidexLA3A61p1a7b/OpidG5zShPhH2 [2024-01-19T03:00:13.805Z] HRdQcqgp0HIxaS+/oLD3eElOnnC/EKD359K6n3kN66sMgvNbHxNFM84Au0TXqmbT [2024-01-19T03:00:13.805Z] w8afrKjrfe/VglpHC2GfmgHMF4YH+ZF48C1RZA6z2oXj6Aj+uww0PjiClE9qUYqK [2024-01-19T03:00:13.805Z] R9uRa32prExihsi+Bto7n2Axo92qUiK2KrkEdQitnQfH5z/PojS78vbyIKp1O86N [2024-01-19T03:00:13.805Z] qRImvCYY17CPxP+JUQ7dxWDMF4NTgSnabkp+cd5i/pyoAb0guQFlQQUnHAZa4QNC [2024-01-19T03:00:13.805Z] 7fgTo1IUwWyDmwCnN5U= [2024-01-19T03:00:13.805Z] -----END CERTIFICATE----- [2024-01-19T03:00:13.805Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:13.805Z] MIIDCjCCAfKgAwIBAgIUJPzO3O+wmUt4gHCkl3ojsvwZL6owDQYJKoZIhvcNAQEL [2024-01-19T03:00:13.805Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T03:00:13.805Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T03:00:13.805Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T03:00:13.805Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T03:00:13.805Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T03:00:13.805Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T03:00:13.805Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T03:00:13.805Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T03:00:13.805Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T03:00:13.805Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAGVt [2024-01-19T03:00:13.805Z] g9qX+r8kyGYsvFavmnDxgMK1Zkz80irQJ2BJlDHdK1hp5aqpRbT2z2YgLog5ZMIZ [2024-01-19T03:00:13.805Z] uI36uPCOXo8+Gol1jT7D6vsgMWm0Tk9jCNbO4jPkv+wm2q1B9UZo94UGiS0K+G0S [2024-01-19T03:00:13.805Z] nccl4jmSq1gWpBZIVetdDFNjpipKjhEOVQM7x33APhACxSAWRp9iEr6V/CzMOgkf [2024-01-19T03:00:13.805Z] RHyz5D7ljh4a2XVH7++eKsj2QPfKyiKtIG72tCccQ9zGnzveiXXb4Ai8r+DyKH4Z [2024-01-19T03:00:13.805Z] zDt5Hb4ZFl72E3Rl4bIh/VQBVsr9u/SFR1p/Emyq+kGXsn+9YMiM9tK4mLUtjdox [2024-01-19T03:00:13.805Z] LMWUWTelNRGWAiaTVz8= [2024-01-19T03:00:13.805Z] -----END CERTIFICATE----- [2024-01-19T03:00:13.805Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:13.805Z] MIIDCjCCAfKgAwIBAgIUBxbEoan/HnRZ+2deRj+2RvGMrWQwDQYJKoZIhvcNAQEL [2024-01-19T03:00:13.805Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T03:00:13.805Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T03:00:13.805Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T03:00:13.805Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T03:00:13.805Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T03:00:13.805Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T03:00:13.805Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T03:00:13.805Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T03:00:13.805Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T03:00:13.805Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAB65 [2024-01-19T03:00:13.805Z] nHU+98oTP0U2J0/TsL5GFeBTH7YyUXUNRJvPWxkA8riB3swoScFKbavO6XJ/hrI2 [2024-01-19T03:00:13.805Z] ywApOz77Iyq58OLa5aIC5aaPRf3YE8LbJQYIHmtza21LXhtlKVSrLu4W58PauI8Z [2024-01-19T03:00:13.805Z] 7iaL9QPUihHFK7wrZ2ZpabN2rRJvTGWWKlhmgFTjI+71gPrXWmYBHIKmTv55V5m1 [2024-01-19T03:00:13.805Z] LxORTR/gBXeH9gmM7wz1cbaYNazqs8R1LS5SW76p/y9RLjTzU6N39VYefMMGwbsQ [2024-01-19T03:00:13.805Z] DlD11wvMBxTBp3hQYc3pdVLkVEQt8trVIwEUts7zfwUTCu/z8gH7H5XePpEVNana [2024-01-19T03:00:13.805Z] JhlAcXHCO7nN/ddJmoc= [2024-01-19T03:00:13.805Z] -----END CERTIFICATE----- [2024-01-19T03:00:13.805Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:13.805Z] MIIDGDCCAgCgAwIBAgIUEvLaarpXlv1cVvG6nMuxh2DNCKQwDQYJKoZIhvcNAQEL [2024-01-19T03:00:13.805Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T03:00:13.805Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T03:00:13.805Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T03:00:13.806Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T03:00:13.806Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T03:00:13.806Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T03:00:13.806Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T03:00:13.806Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T03:00:13.806Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T03:00:13.806Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T03:00:13.806Z] 9w0BAQsFAAOCAQEAOPakzVUmAqHhmQViZZHSQeRh/xYQBrtn6dnEo6X55SXNWcPU [2024-01-19T03:00:13.806Z] 4EEW3DvwBf75uqxKeGoXHTBpqNHG2x1+IncvIKJmi5OWM3aR/vEyJoBOchPBl+A4 [2024-01-19T03:00:13.806Z] 0xZO1vKiS2djxz8XHMkR70cxD8hv/zDJeIQbgBroF4Wa7rim5+y7jgO0O7Rs1zhT [2024-01-19T03:00:13.806Z] LDDStJ8aUKwcVRmOv7NFQxbQHXNx07kaUE2SQJfhryzZmotoBiymmQT2v0H4/4q0 [2024-01-19T03:00:13.806Z] 5dylRhSCWfwN5+It40/C3b1pbC2d9Iv/eO+B0wqg13dLF3xuA1JblEgnqA9mdNnp [2024-01-19T03:00:13.806Z] P6EgKTQBMiO6nPFyYIPqg1eWYkb1eXu9aEkmbw== [2024-01-19T03:00:13.806Z] -----END CERTIFICATE----- [2024-01-19T03:00:13.806Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:13.806Z] MIIDGDCCAgCgAwIBAgIUWf/qoBANFWF/a/yZr7wfLBMfItAwDQYJKoZIhvcNAQEL [2024-01-19T03:00:13.806Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T03:00:13.806Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T03:00:13.806Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T03:00:13.806Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T03:00:13.806Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T03:00:13.806Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T03:00:13.806Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T03:00:13.806Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T03:00:13.806Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T03:00:13.806Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T03:00:13.806Z] 9w0BAQsFAAOCAQEAdCguadZU6WJAuuAtWzvYgftPKkDKNBeGVgbqRwWRCB+QqVFl [2024-01-19T03:00:13.806Z] +3ZOQTStpnXJsY3Ig31hmL88BkBPQ47ENEzs7Czhuz59m+dcBhh/buUoIERSnfUY [2024-01-19T03:00:13.806Z] lWIfqxQevwHYZ3Kjx0l3qliYjKBQJo5xDPnukGb4yUnmrCdZ3fxMzlnr0Ais9rUH [2024-01-19T03:00:13.806Z] MjiDsjP/oUa8naQUTZO4DTRZqeEldzNXnxF6YH+3moTkwRLt9xzirNwECmJ9qxrC [2024-01-19T03:00:13.806Z] eCA7zORxbFSftYLnQiGWUS1dnxBuLOvJHjYPC6EkPPrvqnx2fXcTE1V8GizVRrA6 [2024-01-19T03:00:13.806Z] NPxt+e4qZgU/Nk/Jkpsb7h1FVcKtS8CvmtbdqQ== [2024-01-19T03:00:13.806Z] -----END CERTIFICATE----- [2024-01-19T03:00:13.806Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:13.806Z] MIIDGDCCAgCgAwIBAgIUI6D0WS4tl52N+w2lwAS+6V0DPyUwDQYJKoZIhvcNAQEL [2024-01-19T03:00:13.806Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T03:00:13.806Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T03:00:13.806Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T03:00:13.806Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T03:00:13.806Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T03:00:13.806Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T03:00:13.806Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T03:00:13.806Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T03:00:13.806Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T03:00:13.806Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T03:00:13.806Z] 9w0BAQsFAAOCAQEAAS4yElh6TMyP18ipzRA0Wenukevjs/vtkAFOsv1u5GNeDIrv [2024-01-19T03:00:13.806Z] lNi6/VjvtS+DoFNzQnTf3QoO6GTuoWHLQWt6Z/V5AwIsnOXduSXLugMwEv3j9l50 [2024-01-19T03:00:13.806Z] aDkA7vJ3OH5cGqIvIzxQN7IUOEvZNhY67GFP7+zAzVNRpJ1FHYbhQGPuFD//b9G3 [2024-01-19T03:00:13.806Z] hr4E3+iUEc+F/iBA+caA8wRx1zR383Jz6wuoFt7n7HpWNcgMvoh91JlUAK7yUp0B [2024-01-19T03:00:13.806Z] ekuG6q7HtVsk6X+aB1RyWfrSJjih75W7b+p+/bV4vNmQS1wSGSOaSIuVRvMhmpwf [2024-01-19T03:00:13.806Z] FpT+4rUpiCxklD69/MET/1cWxqvao/ppf6phzg== [2024-01-19T03:00:13.806Z] -----END CERTIFICATE----- [2024-01-19T03:00:13.806Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:13.806Z] MIIDGDCCAgCgAwIBAgIUIVCCgYV9enDck18n89YsA5b2IdgwDQYJKoZIhvcNAQEL [2024-01-19T03:00:13.806Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T03:00:13.806Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T03:00:13.806Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T03:00:13.806Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T03:00:13.806Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T03:00:13.806Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T03:00:13.806Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T03:00:13.806Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T03:00:13.806Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T03:00:13.806Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T03:00:13.807Z] 9w0BAQsFAAOCAQEAa3eFm1ivFf5Vom6a/7E1gDdjl7cjUZ5cqg332Ky1NdzEOiwW [2024-01-19T03:00:13.807Z] mx3yEk35qCd8LAoFMvPvPIuMx1NDS8kwy3FmIOjKnZapHA31ZUpn/gT/seH4GFIU [2024-01-19T03:00:13.807Z] oZw+rODV5h6jmtCVugAn48u/LxOXbY417JWwaeDx5uw+NzOSyebe+td2PbtUjf5X [2024-01-19T03:00:13.807Z] 6xcwYUmaUGsIAgfIMM12hzXOI2/q1qz5dAqwSgnR7dKBReOXnUoFpNOMpxlJ70p/ [2024-01-19T03:00:13.807Z] WUU1aWIWZqKrNI7Pg1T5jljq7+KbT3+A6+ux0FLExr2c1tw3NndXS2Y9fO55m5/q [2024-01-19T03:00:13.807Z] 0EcrTdKx4+QjbWcja3WdX2pGDk0XGNgx+EXn6Q== [2024-01-19T03:00:13.807Z] -----END CERTIFICATE----- [2024-01-19T03:00:13.807Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:13.807Z] MIIFATCCAumgAwIBAgIRAKc9ZKBASymy5TLOEp57N98wDQYJKoZIhvcNAQELBQAw [2024-01-19T03:00:13.807Z] GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDMyMzIyNTM0NloXDTM2 [2024-01-19T03:00:13.807Z] MDMyMzIyNTM0NlowGjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMIICIjANBgkq [2024-01-19T03:00:13.807Z] hkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA+pYHvQw5iU3v2b3iNuYNKYgsWD6KU7aJ [2024-01-19T03:00:13.807Z] diddtZQxSWYzUI3U0I1UsRPTxnhTifs/M9NW4ZlV13ZfB7APwC8oqKOIiwo7IwlP [2024-01-19T03:00:13.807Z] xg0VKgyz+kT8RJfYr66PPIYP0fpTeu42LpMJ+CKo9sbpgVNDZN2z/qiXrRNX/VtG [2024-01-19T03:00:13.807Z] TkPV7a44fZ5bHHVruAxvDnylpQxJobtCBWlJSsbIRGFHMc2z88eUz9NmIOWUKGGj [2024-01-19T03:00:13.807Z] EmP76x8OfRHpIpuxRSCjn0+i9+hR2siIOpcMOGd+40uVJxbRRP5ZXnUFa2fF5FWd [2024-01-19T03:00:13.807Z] O0u0RPI8HON0ovhrwPJY+4eWKkQzyC611oLPYGQ4EbifRsTsCxUZqyUuStGyp8oa [2024-01-19T03:00:13.807Z] aoSKfF6X0+KzGgwwnrjRTUpIl19A92KR0Noo6h622OX+4sZiO/JQdkuX5w/HupK0 [2024-01-19T03:00:13.807Z] A0M0WSMCvU6GOhjGotmh2VTEJwHHY4+TUk0iQYRtv1crONklyZoAQPD76hCrC8Cr [2024-01-19T03:00:13.807Z] IbgsZLfTMC8TWUoMbyUDgvgYkHKMoPm0VGVVuwpRKJxv7+2wXO+pivrrUl2Q9fPe [2024-01-19T03:00:13.807Z] Kk055nJLMV9yPUdig8othUKrRfSxli946AEV1eEOhxddfEwBE3Lt2xn0hhiIedbb [2024-01-19T03:00:13.807Z] Ftf/5kEWFZkXyUmMJK8Ra76Kus2ABueUVEcZ48hrRr1Hf1N9n59VbTUaXgeiZA50 [2024-01-19T03:00:13.807Z] qXf2bymE6F8CAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMB [2024-01-19T03:00:13.807Z] Af8wHQYDVR0OBBYEFMEmdKSKRKDm+iAo2FwjmkWIGHngMA0GCSqGSIb3DQEBCwUA [2024-01-19T03:00:13.807Z] A4ICAQBCPw74M9X/Xx04K1VAES3ypgQYH5bf9FXVDrwhRFSVckria/7dMzoF5wln [2024-01-19T03:00:13.807Z] uq9NGsjkkkDg17AohcQdr8alH4LvPdxpKr3BjpvEcmbqF8xH+MbbeUEnmbSfLI8H [2024-01-19T03:00:13.807Z] sefuhXF9AF/9iYvpVNC8FmJ0OhiVv13VgMQw0CRKkbtjZBf8xaEhq/YqxWVsgOjm [2024-01-19T03:00:13.807Z] dm5CAQ2X0aX7502x8wYRgMnZhA5goC1zVWBVAi8yhhmlhhoDUfg17cXkmaJC5pDd [2024-01-19T03:00:13.807Z] oenZ9NVhW8eDb03MFCrWNvIh89DDeCGWuWfDltDq0n3owyL0IeSn7RfpSclpxVmV [2024-01-19T03:00:13.807Z] /53jkYjwIgxIG7Gsv0LKMbsf6QdBcTjhvfZyMIpBRkTe3zuHd2feKzY9lEkbRvRQ [2024-01-19T03:00:13.807Z] zbh4Ps5YBnG6CKJPTbe2hfi3nhnw/MyEmF3zb0hzvLWNrR9XW3ibb2oL3424XOwc [2024-01-19T03:00:13.807Z] VjrTSCLzO9Rv6s5wi03qoWvKAQQAElqTYRHhynJ3w6wuvKYF5zcZF3MDnrVGLbh1 [2024-01-19T03:00:13.807Z] Q9ePRFBCiXOQ6wPLoUhrrbZ8LpFUFYDXHMtYM7P9sc9IAWoONXREJaO08zgFtMp4 [2024-01-19T03:00:13.807Z] 8iyIYUyQAbsvx8oD2M8kRvrIRSrRJSl6L957b4AFiLIQ/GgV2curs0jje7Edx34c [2024-01-19T03:00:13.807Z] idWw1VrejtwclobqNMVtG3EiPUIpJGpbMcJgbiLSmKkrvQtGng== [2024-01-19T03:00:13.807Z] -----END CERTIFICATE----- [2024-01-19T03:00:13.807Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} [2024-01-19T03:00:13.807Z] INFO[2024-01-19 03:00:13] Actual:true, Expected: true [2024-01-19T03:00:14.064Z] INFO[2024-01-19 03:00:13] Verifying : Description : Fetching configmap pointer &ConfigMap{ObjectMeta:{openshift-service-ca.crt mysql-ibm-pxb-0-84850-01-19-02h58m26s b5a763cd-2b07-469a-9954-578448072ede 1574829 0 2024-01-19 02:58:26 +0000 UTC map[] map[service.beta.openshift.io/inject-cabundle:true] [] [] [{kube-controller-manager Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:data":{},"f:metadata":{"f:annotations":{".":{},"f:service.beta.openshift.io/inject-cabundle":{}}}} } {service-ca-operator Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:data":{"f:service-ca.crt":{}}} }]},Data:map[string]string{service-ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T03:00:14.064Z] MIIDUTCCAjmgAwIBAgIILvRjno8BjvIwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE [2024-01-19T03:00:14.064Z] Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwNTM5NjA4ODAe [2024-01-19T03:00:14.064Z] Fw0yNDAxMTYwOTA4MDhaFw0yNjAzMTYwOTA4MDlaMDYxNDAyBgNVBAMMK29wZW5z [2024-01-19T03:00:14.064Z] aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDUzOTYwODgwggEiMA0GCSqG [2024-01-19T03:00:14.064Z] SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVr8xmLKsMoemVtfnlzzil6B8CGocEhNRo [2024-01-19T03:00:14.064Z] fkzkD8aSOYgIj5zcZo/M0O7nXDXeg9dqx9/h2qrGk+SS2ikU2Xs4Q5gaQDh9U1lB [2024-01-19T03:00:14.064Z] y8paoOEvw06CmJ/1Qqur5A539/OnimRSTdejTu4lwaDwsJiyoGfhvrBJAdjagtYA [2024-01-19T03:00:14.064Z] 0sNMrCgEAT4NWHS5EDx5dF7nMDHmQ0h0gHYei6vUEtiPyLwcTH1T7uVSj22fCEXX [2024-01-19T03:00:14.064Z] LjgN3OSiquJ+Hf4+csPpwgxAEkfjdBldhnImYkunJbg/x4c90uFL1pygZusrbFAf [2024-01-19T03:00:14.064Z] cERW8GngZuL5JY8MQoIstMu+1qS390ouQDDbEcvWjxC8YFNFHywrAgMBAAGjYzBh [2024-01-19T03:00:14.064Z] MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQS5vRk [2024-01-19T03:00:14.064Z] jkBPISX2qYmk1VEQZIXSLzAfBgNVHSMEGDAWgBQS5vRkjkBPISX2qYmk1VEQZIXS [2024-01-19T03:00:14.064Z] LzANBgkqhkiG9w0BAQsFAAOCAQEAZ/qc2vFGJZgDJCEUaPYuufuLRPSF68ixV9fk [2024-01-19T03:00:14.064Z] e5tBkRN8atCsEBygC/RUcwULlhsdgUMGMcREgNC/FYlwkOsAj3gP8KmcXAZ4gHCa [2024-01-19T03:00:14.064Z] T6PIbIcj1hCX0mdX0tvtaj7PWvRUz9CHh6iFzmPsXK5xjaBA9vZbkUAa7odguUit [2024-01-19T03:00:14.064Z] vz1QnKoV1OaZyccDLeQL50khM1XEBn8L+zeY3k85uo2X514HZW4LDPTQ94cpyNnb [2024-01-19T03:00:14.064Z] zqI6utp7JoPJexsHMWNFKGb//HgjluNg7TsywXIeebk+/GZUEDfNKid0oiTpQvzR [2024-01-19T03:00:14.064Z] TFFF+5KO/sa4+qIh1ERm140JxeaurUMaJzn7cFQbekEgt/N+iA== [2024-01-19T03:00:14.064Z] -----END CERTIFICATE----- [2024-01-19T03:00:14.064Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} from namespace mysql-ibm-pxb-0-84850-01-19-02h58m26s [2024-01-19T03:00:14.064Z] INFO[2024-01-19 03:00:13] Actual:true, Expected: true [2024-01-19T03:00:14.064Z] 2024-01-19 03:00:13 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.AddLabelToResource:#5038] - Adding label [edffc0c3-8249-4e9f-a79b-744c405b60af=d095e726-c3cc-4da8-b72f-6be96ae30ea7] to ConfigMap openshift-service-ca.crt [2024-01-19T03:00:14.320Z] INFO[2024-01-19 03:00:14] Verifying : Description : Adding labels edffc0c3-8249-4e9f-a79b-744c405b60af=d095e726-c3cc-4da8-b72f-6be96ae30ea7 to resource &ConfigMap{ObjectMeta:{openshift-service-ca.crt mysql-ibm-pxb-0-84850-01-19-02h58m26s b5a763cd-2b07-469a-9954-578448072ede 1574829 0 2024-01-19 02:58:26 +0000 UTC map[edffc0c3-8249-4e9f-a79b-744c405b60af:d095e726-c3cc-4da8-b72f-6be96ae30ea7] map[service.beta.openshift.io/inject-cabundle:true] [] [] [{kube-controller-manager Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:data":{},"f:metadata":{"f:annotations":{".":{},"f:service.beta.openshift.io/inject-cabundle":{}}}} } {service-ca-operator Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:data":{"f:service-ca.crt":{}}} }]},Data:map[string]string{service-ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T03:00:14.320Z] MIIDUTCCAjmgAwIBAgIILvRjno8BjvIwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE [2024-01-19T03:00:14.320Z] Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwNTM5NjA4ODAe [2024-01-19T03:00:14.320Z] Fw0yNDAxMTYwOTA4MDhaFw0yNjAzMTYwOTA4MDlaMDYxNDAyBgNVBAMMK29wZW5z [2024-01-19T03:00:14.320Z] aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDUzOTYwODgwggEiMA0GCSqG [2024-01-19T03:00:14.320Z] SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVr8xmLKsMoemVtfnlzzil6B8CGocEhNRo [2024-01-19T03:00:14.320Z] fkzkD8aSOYgIj5zcZo/M0O7nXDXeg9dqx9/h2qrGk+SS2ikU2Xs4Q5gaQDh9U1lB [2024-01-19T03:00:14.320Z] y8paoOEvw06CmJ/1Qqur5A539/OnimRSTdejTu4lwaDwsJiyoGfhvrBJAdjagtYA [2024-01-19T03:00:14.320Z] 0sNMrCgEAT4NWHS5EDx5dF7nMDHmQ0h0gHYei6vUEtiPyLwcTH1T7uVSj22fCEXX [2024-01-19T03:00:14.320Z] LjgN3OSiquJ+Hf4+csPpwgxAEkfjdBldhnImYkunJbg/x4c90uFL1pygZusrbFAf [2024-01-19T03:00:14.320Z] cERW8GngZuL5JY8MQoIstMu+1qS390ouQDDbEcvWjxC8YFNFHywrAgMBAAGjYzBh [2024-01-19T03:00:14.320Z] MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQS5vRk [2024-01-19T03:00:14.320Z] jkBPISX2qYmk1VEQZIXSLzAfBgNVHSMEGDAWgBQS5vRkjkBPISX2qYmk1VEQZIXS [2024-01-19T03:00:14.320Z] LzANBgkqhkiG9w0BAQsFAAOCAQEAZ/qc2vFGJZgDJCEUaPYuufuLRPSF68ixV9fk [2024-01-19T03:00:14.320Z] e5tBkRN8atCsEBygC/RUcwULlhsdgUMGMcREgNC/FYlwkOsAj3gP8KmcXAZ4gHCa [2024-01-19T03:00:14.320Z] T6PIbIcj1hCX0mdX0tvtaj7PWvRUz9CHh6iFzmPsXK5xjaBA9vZbkUAa7odguUit [2024-01-19T03:00:14.320Z] vz1QnKoV1OaZyccDLeQL50khM1XEBn8L+zeY3k85uo2X514HZW4LDPTQ94cpyNnb [2024-01-19T03:00:14.320Z] zqI6utp7JoPJexsHMWNFKGb//HgjluNg7TsywXIeebk+/GZUEDfNKid0oiTpQvzR [2024-01-19T03:00:14.320Z] TFFF+5KO/sa4+qIh1ERm140JxeaurUMaJzn7cFQbekEgt/N+iA== [2024-01-19T03:00:14.321Z] -----END CERTIFICATE----- [2024-01-19T03:00:14.321Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} [2024-01-19T03:00:14.321Z] INFO[2024-01-19 03:00:14] Actual:true, Expected: true [2024-01-19T03:00:14.321Z] INFO[2024-01-19 03:00:14] Verifying : Description : Fetching pvc list from namespace [mysql-ibm-pxb-1-84850-01-19-02h58m26s] [2024-01-19T03:00:14.321Z] INFO[2024-01-19 03:00:14] Actual:true, Expected: true [2024-01-19T03:00:14.577Z] INFO[2024-01-19 03:00:14] Verifying : Description : Fetching pvc pointer &PersistentVolumeClaim{ObjectMeta:{mysql-data mysql-ibm-pxb-1-84850-01-19-02h58m26s 6122a103-2fe7-4e98-a869-50439b87761b 1575195 0 2024-01-19 02:58:26 +0000 UTC map[] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-19 02:58:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-19 02:58:37 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{2147483648 0} {} 2Gi BinarySI},},},VolumeName:pvc-6122a103-2fe7-4e98-a869-50439b87761b,Selector:nil,StorageClassName:*mysql-sc,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{10737418240 0} {} 10Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},} for pvc {{ } {mysql-data mysql-ibm-pxb-1-84850-01-19-02h58m26s 6122a103-2fe7-4e98-a869-50439b87761b 1575195 0 2024-01-19 02:58:26 +0000 UTC map[] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-19 02:58:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-19 02:58:37 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{2147483648 0} {} 2Gi BinarySI}]} pvc-6122a103-2fe7-4e98-a869-50439b87761b 0xc0017f9970 0xc0017f9980 nil nil} {Bound [ReadWriteOnce] map[storage:{{10737418240 0} {} 10Gi BinarySI}] [] map[] }} [2024-01-19T03:00:14.577Z] INFO[2024-01-19 03:00:14] Actual:true, Expected: true [2024-01-19T03:00:14.577Z] 2024-01-19 03:00:14 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.AddLabelToResource:#5030] - Adding label [edffc0c3-8249-4e9f-a79b-744c405b60af=d095e726-c3cc-4da8-b72f-6be96ae30ea7] to PVC mysql-data [2024-01-19T03:00:14.843Z] INFO[2024-01-19 03:00:14] Verifying : Description : Adding labels edffc0c3-8249-4e9f-a79b-744c405b60af=d095e726-c3cc-4da8-b72f-6be96ae30ea7 to resource &PersistentVolumeClaim{ObjectMeta:{mysql-data mysql-ibm-pxb-1-84850-01-19-02h58m26s 6122a103-2fe7-4e98-a869-50439b87761b 1575195 0 2024-01-19 02:58:26 +0000 UTC map[edffc0c3-8249-4e9f-a79b-744c405b60af:d095e726-c3cc-4da8-b72f-6be96ae30ea7] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-19 02:58:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-19 02:58:37 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{2147483648 0} {} 2Gi BinarySI},},},VolumeName:pvc-6122a103-2fe7-4e98-a869-50439b87761b,Selector:nil,StorageClassName:*mysql-sc,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{10737418240 0} {} 10Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},} [2024-01-19T03:00:14.843Z] INFO[2024-01-19 03:00:14] Actual:true, Expected: true [2024-01-19T03:00:15.099Z] INFO[2024-01-19 03:00:14] Verifying : Description : Fetching configmap list from namespace [mysql-ibm-pxb-1-84850-01-19-02h58m26s] [2024-01-19T03:00:15.099Z] INFO[2024-01-19 03:00:14] Actual:true, Expected: true [2024-01-19T03:00:15.099Z] INFO[2024-01-19 03:00:15] Verifying : Description : Fetching configmap pointer &ConfigMap{ObjectMeta:{kube-****-ca.crt mysql-ibm-pxb-1-84850-01-19-02h58m26s e3d8a1e7-870e-4497-9241-30e09d1e9c35 1574885 0 2024-01-19 02:58:26 +0000 UTC map[] map[kubernetes.io/description:Contains a CA bundle that can be used to verify the kube-apiserver when using internal endpoints such as the internal service IP or kubernetes.default.svc. No other usage is guaranteed across distributions of Kubernetes clusters.] [] [] [{kube-controller-manager Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:data":{".":{},"f:ca.crt":{}},"f:metadata":{"f:annotations":{".":{},"f:kubernetes.io/description":{}}}} }]},Data:map[string]string{ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T03:00:15.099Z] MIIDCjCCAfKgAwIBAgIUWkn6rgBaC+VWV1XdJjGA3DwoRRYwDQYJKoZIhvcNAQEL [2024-01-19T03:00:15.099Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T03:00:15.099Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T03:00:15.099Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T03:00:15.099Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T03:00:15.099Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T03:00:15.100Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T03:00:15.100Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T03:00:15.100Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T03:00:15.100Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T03:00:15.100Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAHbD [2024-01-19T03:00:15.100Z] 4nWTgt/8sat7SbbHxZ+JcMGRAIaeLAjMJxkxyqD8xUq6YQcgbjDVHo1Qbh3PXB3T [2024-01-19T03:00:15.100Z] 5BWUsRbDoKi0eRJx/5cyI3846SYEAOpuUfJrg9X2CH7ZoYhxeJ42dqt3MdoMJEM1 [2024-01-19T03:00:15.100Z] k3e2rRqAKuAxwNXugHryxQM+1ehOJjrEuT5DVK6qu8+JYRcT1hBZb6hrebKPp5eG [2024-01-19T03:00:15.100Z] JvBzHP4YOATIAXhfH0Uj5Y+JhwoOdpXmOFpz+6gITPD4aNzzJZe/0ADSFfOyUJaR [2024-01-19T03:00:15.100Z] g5NnJnkeGqMM3SLsur25r+m3y0i3t+RdocZ/4mHTEnF4KWVWvZy4GjENiEBcXEol [2024-01-19T03:00:15.100Z] o4SZVaP4AF7XQJ5vdA0= [2024-01-19T03:00:15.100Z] -----END CERTIFICATE----- [2024-01-19T03:00:15.100Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:15.100Z] MIIDCjCCAfKgAwIBAgIUXhIIFCc/NxZJn6hNEFoZuPQ8N5UwDQYJKoZIhvcNAQEL [2024-01-19T03:00:15.100Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T03:00:15.100Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T03:00:15.100Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T03:00:15.100Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T03:00:15.100Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T03:00:15.100Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T03:00:15.100Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T03:00:15.100Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T03:00:15.100Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T03:00:15.100Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAKG3 [2024-01-19T03:00:15.100Z] nVfv4IkgHVOxgbrNIoYhjvBvVJcLZub0C9yidexLA3A61p1a7b/OpidG5zShPhH2 [2024-01-19T03:00:15.100Z] HRdQcqgp0HIxaS+/oLD3eElOnnC/EKD359K6n3kN66sMgvNbHxNFM84Au0TXqmbT [2024-01-19T03:00:15.100Z] w8afrKjrfe/VglpHC2GfmgHMF4YH+ZF48C1RZA6z2oXj6Aj+uww0PjiClE9qUYqK [2024-01-19T03:00:15.100Z] R9uRa32prExihsi+Bto7n2Axo92qUiK2KrkEdQitnQfH5z/PojS78vbyIKp1O86N [2024-01-19T03:00:15.100Z] qRImvCYY17CPxP+JUQ7dxWDMF4NTgSnabkp+cd5i/pyoAb0guQFlQQUnHAZa4QNC [2024-01-19T03:00:15.100Z] 7fgTo1IUwWyDmwCnN5U= [2024-01-19T03:00:15.100Z] -----END CERTIFICATE----- [2024-01-19T03:00:15.100Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:15.100Z] MIIDCjCCAfKgAwIBAgIUJPzO3O+wmUt4gHCkl3ojsvwZL6owDQYJKoZIhvcNAQEL [2024-01-19T03:00:15.100Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T03:00:15.100Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T03:00:15.100Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T03:00:15.100Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T03:00:15.100Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T03:00:15.100Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T03:00:15.100Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T03:00:15.100Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T03:00:15.100Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T03:00:15.100Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAGVt [2024-01-19T03:00:15.100Z] g9qX+r8kyGYsvFavmnDxgMK1Zkz80irQJ2BJlDHdK1hp5aqpRbT2z2YgLog5ZMIZ [2024-01-19T03:00:15.100Z] uI36uPCOXo8+Gol1jT7D6vsgMWm0Tk9jCNbO4jPkv+wm2q1B9UZo94UGiS0K+G0S [2024-01-19T03:00:15.100Z] nccl4jmSq1gWpBZIVetdDFNjpipKjhEOVQM7x33APhACxSAWRp9iEr6V/CzMOgkf [2024-01-19T03:00:15.100Z] RHyz5D7ljh4a2XVH7++eKsj2QPfKyiKtIG72tCccQ9zGnzveiXXb4Ai8r+DyKH4Z [2024-01-19T03:00:15.100Z] zDt5Hb4ZFl72E3Rl4bIh/VQBVsr9u/SFR1p/Emyq+kGXsn+9YMiM9tK4mLUtjdox [2024-01-19T03:00:15.100Z] LMWUWTelNRGWAiaTVz8= [2024-01-19T03:00:15.100Z] -----END CERTIFICATE----- [2024-01-19T03:00:15.100Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:15.100Z] MIIDCjCCAfKgAwIBAgIUBxbEoan/HnRZ+2deRj+2RvGMrWQwDQYJKoZIhvcNAQEL [2024-01-19T03:00:15.100Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T03:00:15.100Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T03:00:15.100Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T03:00:15.100Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T03:00:15.100Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T03:00:15.100Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T03:00:15.100Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T03:00:15.100Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T03:00:15.100Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T03:00:15.100Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAB65 [2024-01-19T03:00:15.101Z] nHU+98oTP0U2J0/TsL5GFeBTH7YyUXUNRJvPWxkA8riB3swoScFKbavO6XJ/hrI2 [2024-01-19T03:00:15.101Z] ywApOz77Iyq58OLa5aIC5aaPRf3YE8LbJQYIHmtza21LXhtlKVSrLu4W58PauI8Z [2024-01-19T03:00:15.101Z] 7iaL9QPUihHFK7wrZ2ZpabN2rRJvTGWWKlhmgFTjI+71gPrXWmYBHIKmTv55V5m1 [2024-01-19T03:00:15.101Z] LxORTR/gBXeH9gmM7wz1cbaYNazqs8R1LS5SW76p/y9RLjTzU6N39VYefMMGwbsQ [2024-01-19T03:00:15.101Z] DlD11wvMBxTBp3hQYc3pdVLkVEQt8trVIwEUts7zfwUTCu/z8gH7H5XePpEVNana [2024-01-19T03:00:15.101Z] JhlAcXHCO7nN/ddJmoc= [2024-01-19T03:00:15.101Z] -----END CERTIFICATE----- [2024-01-19T03:00:15.101Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:15.101Z] MIIDGDCCAgCgAwIBAgIUEvLaarpXlv1cVvG6nMuxh2DNCKQwDQYJKoZIhvcNAQEL [2024-01-19T03:00:15.101Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T03:00:15.101Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T03:00:15.101Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T03:00:15.101Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T03:00:15.101Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T03:00:15.101Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T03:00:15.101Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T03:00:15.101Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T03:00:15.101Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T03:00:15.101Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T03:00:15.101Z] 9w0BAQsFAAOCAQEAOPakzVUmAqHhmQViZZHSQeRh/xYQBrtn6dnEo6X55SXNWcPU [2024-01-19T03:00:15.101Z] 4EEW3DvwBf75uqxKeGoXHTBpqNHG2x1+IncvIKJmi5OWM3aR/vEyJoBOchPBl+A4 [2024-01-19T03:00:15.101Z] 0xZO1vKiS2djxz8XHMkR70cxD8hv/zDJeIQbgBroF4Wa7rim5+y7jgO0O7Rs1zhT [2024-01-19T03:00:15.101Z] LDDStJ8aUKwcVRmOv7NFQxbQHXNx07kaUE2SQJfhryzZmotoBiymmQT2v0H4/4q0 [2024-01-19T03:00:15.101Z] 5dylRhSCWfwN5+It40/C3b1pbC2d9Iv/eO+B0wqg13dLF3xuA1JblEgnqA9mdNnp [2024-01-19T03:00:15.101Z] P6EgKTQBMiO6nPFyYIPqg1eWYkb1eXu9aEkmbw== [2024-01-19T03:00:15.101Z] -----END CERTIFICATE----- [2024-01-19T03:00:15.101Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:15.101Z] MIIDGDCCAgCgAwIBAgIUWf/qoBANFWF/a/yZr7wfLBMfItAwDQYJKoZIhvcNAQEL [2024-01-19T03:00:15.101Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T03:00:15.101Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T03:00:15.101Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T03:00:15.101Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T03:00:15.101Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T03:00:15.101Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T03:00:15.101Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T03:00:15.101Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T03:00:15.101Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T03:00:15.101Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T03:00:15.101Z] 9w0BAQsFAAOCAQEAdCguadZU6WJAuuAtWzvYgftPKkDKNBeGVgbqRwWRCB+QqVFl [2024-01-19T03:00:15.101Z] +3ZOQTStpnXJsY3Ig31hmL88BkBPQ47ENEzs7Czhuz59m+dcBhh/buUoIERSnfUY [2024-01-19T03:00:15.101Z] lWIfqxQevwHYZ3Kjx0l3qliYjKBQJo5xDPnukGb4yUnmrCdZ3fxMzlnr0Ais9rUH [2024-01-19T03:00:15.101Z] MjiDsjP/oUa8naQUTZO4DTRZqeEldzNXnxF6YH+3moTkwRLt9xzirNwECmJ9qxrC [2024-01-19T03:00:15.101Z] eCA7zORxbFSftYLnQiGWUS1dnxBuLOvJHjYPC6EkPPrvqnx2fXcTE1V8GizVRrA6 [2024-01-19T03:00:15.101Z] NPxt+e4qZgU/Nk/Jkpsb7h1FVcKtS8CvmtbdqQ== [2024-01-19T03:00:15.101Z] -----END CERTIFICATE----- [2024-01-19T03:00:15.101Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:15.101Z] MIIDGDCCAgCgAwIBAgIUI6D0WS4tl52N+w2lwAS+6V0DPyUwDQYJKoZIhvcNAQEL [2024-01-19T03:00:15.101Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T03:00:15.101Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T03:00:15.101Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T03:00:15.101Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T03:00:15.101Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T03:00:15.101Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T03:00:15.101Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T03:00:15.101Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T03:00:15.101Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T03:00:15.101Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T03:00:15.101Z] 9w0BAQsFAAOCAQEAAS4yElh6TMyP18ipzRA0Wenukevjs/vtkAFOsv1u5GNeDIrv [2024-01-19T03:00:15.101Z] lNi6/VjvtS+DoFNzQnTf3QoO6GTuoWHLQWt6Z/V5AwIsnOXduSXLugMwEv3j9l50 [2024-01-19T03:00:15.101Z] aDkA7vJ3OH5cGqIvIzxQN7IUOEvZNhY67GFP7+zAzVNRpJ1FHYbhQGPuFD//b9G3 [2024-01-19T03:00:15.101Z] hr4E3+iUEc+F/iBA+caA8wRx1zR383Jz6wuoFt7n7HpWNcgMvoh91JlUAK7yUp0B [2024-01-19T03:00:15.101Z] ekuG6q7HtVsk6X+aB1RyWfrSJjih75W7b+p+/bV4vNmQS1wSGSOaSIuVRvMhmpwf [2024-01-19T03:00:15.101Z] FpT+4rUpiCxklD69/MET/1cWxqvao/ppf6phzg== [2024-01-19T03:00:15.102Z] -----END CERTIFICATE----- [2024-01-19T03:00:15.102Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:15.102Z] MIIDGDCCAgCgAwIBAgIUIVCCgYV9enDck18n89YsA5b2IdgwDQYJKoZIhvcNAQEL [2024-01-19T03:00:15.102Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T03:00:15.102Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T03:00:15.102Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T03:00:15.102Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T03:00:15.102Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T03:00:15.102Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T03:00:15.102Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T03:00:15.102Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T03:00:15.102Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T03:00:15.102Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T03:00:15.102Z] 9w0BAQsFAAOCAQEAa3eFm1ivFf5Vom6a/7E1gDdjl7cjUZ5cqg332Ky1NdzEOiwW [2024-01-19T03:00:15.102Z] mx3yEk35qCd8LAoFMvPvPIuMx1NDS8kwy3FmIOjKnZapHA31ZUpn/gT/seH4GFIU [2024-01-19T03:00:15.102Z] oZw+rODV5h6jmtCVugAn48u/LxOXbY417JWwaeDx5uw+NzOSyebe+td2PbtUjf5X [2024-01-19T03:00:15.102Z] 6xcwYUmaUGsIAgfIMM12hzXOI2/q1qz5dAqwSgnR7dKBReOXnUoFpNOMpxlJ70p/ [2024-01-19T03:00:15.102Z] WUU1aWIWZqKrNI7Pg1T5jljq7+KbT3+A6+ux0FLExr2c1tw3NndXS2Y9fO55m5/q [2024-01-19T03:00:15.102Z] 0EcrTdKx4+QjbWcja3WdX2pGDk0XGNgx+EXn6Q== [2024-01-19T03:00:15.102Z] -----END CERTIFICATE----- [2024-01-19T03:00:15.102Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:15.102Z] MIIFATCCAumgAwIBAgIRAKc9ZKBASymy5TLOEp57N98wDQYJKoZIhvcNAQELBQAw [2024-01-19T03:00:15.102Z] GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDMyMzIyNTM0NloXDTM2 [2024-01-19T03:00:15.102Z] MDMyMzIyNTM0NlowGjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMIICIjANBgkq [2024-01-19T03:00:15.102Z] hkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA+pYHvQw5iU3v2b3iNuYNKYgsWD6KU7aJ [2024-01-19T03:00:15.102Z] diddtZQxSWYzUI3U0I1UsRPTxnhTifs/M9NW4ZlV13ZfB7APwC8oqKOIiwo7IwlP [2024-01-19T03:00:15.102Z] xg0VKgyz+kT8RJfYr66PPIYP0fpTeu42LpMJ+CKo9sbpgVNDZN2z/qiXrRNX/VtG [2024-01-19T03:00:15.102Z] TkPV7a44fZ5bHHVruAxvDnylpQxJobtCBWlJSsbIRGFHMc2z88eUz9NmIOWUKGGj [2024-01-19T03:00:15.102Z] EmP76x8OfRHpIpuxRSCjn0+i9+hR2siIOpcMOGd+40uVJxbRRP5ZXnUFa2fF5FWd [2024-01-19T03:00:15.102Z] O0u0RPI8HON0ovhrwPJY+4eWKkQzyC611oLPYGQ4EbifRsTsCxUZqyUuStGyp8oa [2024-01-19T03:00:15.102Z] aoSKfF6X0+KzGgwwnrjRTUpIl19A92KR0Noo6h622OX+4sZiO/JQdkuX5w/HupK0 [2024-01-19T03:00:15.102Z] A0M0WSMCvU6GOhjGotmh2VTEJwHHY4+TUk0iQYRtv1crONklyZoAQPD76hCrC8Cr [2024-01-19T03:00:15.102Z] IbgsZLfTMC8TWUoMbyUDgvgYkHKMoPm0VGVVuwpRKJxv7+2wXO+pivrrUl2Q9fPe [2024-01-19T03:00:15.102Z] Kk055nJLMV9yPUdig8othUKrRfSxli946AEV1eEOhxddfEwBE3Lt2xn0hhiIedbb [2024-01-19T03:00:15.102Z] Ftf/5kEWFZkXyUmMJK8Ra76Kus2ABueUVEcZ48hrRr1Hf1N9n59VbTUaXgeiZA50 [2024-01-19T03:00:15.102Z] qXf2bymE6F8CAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMB [2024-01-19T03:00:15.102Z] Af8wHQYDVR0OBBYEFMEmdKSKRKDm+iAo2FwjmkWIGHngMA0GCSqGSIb3DQEBCwUA [2024-01-19T03:00:15.102Z] A4ICAQBCPw74M9X/Xx04K1VAES3ypgQYH5bf9FXVDrwhRFSVckria/7dMzoF5wln [2024-01-19T03:00:15.102Z] uq9NGsjkkkDg17AohcQdr8alH4LvPdxpKr3BjpvEcmbqF8xH+MbbeUEnmbSfLI8H [2024-01-19T03:00:15.102Z] sefuhXF9AF/9iYvpVNC8FmJ0OhiVv13VgMQw0CRKkbtjZBf8xaEhq/YqxWVsgOjm [2024-01-19T03:00:15.102Z] dm5CAQ2X0aX7502x8wYRgMnZhA5goC1zVWBVAi8yhhmlhhoDUfg17cXkmaJC5pDd [2024-01-19T03:00:15.102Z] oenZ9NVhW8eDb03MFCrWNvIh89DDeCGWuWfDltDq0n3owyL0IeSn7RfpSclpxVmV [2024-01-19T03:00:15.102Z] /53jkYjwIgxIG7Gsv0LKMbsf6QdBcTjhvfZyMIpBRkTe3zuHd2feKzY9lEkbRvRQ [2024-01-19T03:00:15.102Z] zbh4Ps5YBnG6CKJPTbe2hfi3nhnw/MyEmF3zb0hzvLWNrR9XW3ibb2oL3424XOwc [2024-01-19T03:00:15.102Z] VjrTSCLzO9Rv6s5wi03qoWvKAQQAElqTYRHhynJ3w6wuvKYF5zcZF3MDnrVGLbh1 [2024-01-19T03:00:15.102Z] Q9ePRFBCiXOQ6wPLoUhrrbZ8LpFUFYDXHMtYM7P9sc9IAWoONXREJaO08zgFtMp4 [2024-01-19T03:00:15.102Z] 8iyIYUyQAbsvx8oD2M8kRvrIRSrRJSl6L957b4AFiLIQ/GgV2curs0jje7Edx34c [2024-01-19T03:00:15.102Z] idWw1VrejtwclobqNMVtG3EiPUIpJGpbMcJgbiLSmKkrvQtGng== [2024-01-19T03:00:15.102Z] -----END CERTIFICATE----- [2024-01-19T03:00:15.102Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} from namespace mysql-ibm-pxb-1-84850-01-19-02h58m26s [2024-01-19T03:00:15.102Z] INFO[2024-01-19 03:00:15] Actual:true, Expected: true [2024-01-19T03:00:15.102Z] 2024-01-19 03:00:15 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.AddLabelToResource:#5038] - Adding label [edffc0c3-8249-4e9f-a79b-744c405b60af=d095e726-c3cc-4da8-b72f-6be96ae30ea7] to ConfigMap kube-****-ca.crt [2024-01-19T03:00:15.359Z] INFO[2024-01-19 03:00:15] Verifying : Description : Adding labels edffc0c3-8249-4e9f-a79b-744c405b60af=d095e726-c3cc-4da8-b72f-6be96ae30ea7 to resource &ConfigMap{ObjectMeta:{kube-****-ca.crt mysql-ibm-pxb-1-84850-01-19-02h58m26s e3d8a1e7-870e-4497-9241-30e09d1e9c35 1574885 0 2024-01-19 02:58:26 +0000 UTC map[edffc0c3-8249-4e9f-a79b-744c405b60af:d095e726-c3cc-4da8-b72f-6be96ae30ea7] map[kubernetes.io/description:Contains a CA bundle that can be used to verify the kube-apiserver when using internal endpoints such as the internal service IP or kubernetes.default.svc. No other usage is guaranteed across distributions of Kubernetes clusters.] [] [] [{kube-controller-manager Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:data":{".":{},"f:ca.crt":{}},"f:metadata":{"f:annotations":{".":{},"f:kubernetes.io/description":{}}}} }]},Data:map[string]string{ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T03:00:15.359Z] MIIDCjCCAfKgAwIBAgIUWkn6rgBaC+VWV1XdJjGA3DwoRRYwDQYJKoZIhvcNAQEL [2024-01-19T03:00:15.359Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T03:00:15.360Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T03:00:15.360Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T03:00:15.360Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T03:00:15.360Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T03:00:15.360Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T03:00:15.360Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T03:00:15.360Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T03:00:15.360Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T03:00:15.360Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAHbD [2024-01-19T03:00:15.360Z] 4nWTgt/8sat7SbbHxZ+JcMGRAIaeLAjMJxkxyqD8xUq6YQcgbjDVHo1Qbh3PXB3T [2024-01-19T03:00:15.360Z] 5BWUsRbDoKi0eRJx/5cyI3846SYEAOpuUfJrg9X2CH7ZoYhxeJ42dqt3MdoMJEM1 [2024-01-19T03:00:15.360Z] k3e2rRqAKuAxwNXugHryxQM+1ehOJjrEuT5DVK6qu8+JYRcT1hBZb6hrebKPp5eG [2024-01-19T03:00:15.360Z] JvBzHP4YOATIAXhfH0Uj5Y+JhwoOdpXmOFpz+6gITPD4aNzzJZe/0ADSFfOyUJaR [2024-01-19T03:00:15.360Z] g5NnJnkeGqMM3SLsur25r+m3y0i3t+RdocZ/4mHTEnF4KWVWvZy4GjENiEBcXEol [2024-01-19T03:00:15.360Z] o4SZVaP4AF7XQJ5vdA0= [2024-01-19T03:00:15.360Z] -----END CERTIFICATE----- [2024-01-19T03:00:15.360Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:15.360Z] MIIDCjCCAfKgAwIBAgIUXhIIFCc/NxZJn6hNEFoZuPQ8N5UwDQYJKoZIhvcNAQEL [2024-01-19T03:00:15.360Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T03:00:15.360Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T03:00:15.360Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T03:00:15.360Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T03:00:15.360Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T03:00:15.360Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T03:00:15.360Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T03:00:15.360Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T03:00:15.360Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T03:00:15.360Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAKG3 [2024-01-19T03:00:15.360Z] nVfv4IkgHVOxgbrNIoYhjvBvVJcLZub0C9yidexLA3A61p1a7b/OpidG5zShPhH2 [2024-01-19T03:00:15.360Z] HRdQcqgp0HIxaS+/oLD3eElOnnC/EKD359K6n3kN66sMgvNbHxNFM84Au0TXqmbT [2024-01-19T03:00:15.360Z] w8afrKjrfe/VglpHC2GfmgHMF4YH+ZF48C1RZA6z2oXj6Aj+uww0PjiClE9qUYqK [2024-01-19T03:00:15.360Z] R9uRa32prExihsi+Bto7n2Axo92qUiK2KrkEdQitnQfH5z/PojS78vbyIKp1O86N [2024-01-19T03:00:15.360Z] qRImvCYY17CPxP+JUQ7dxWDMF4NTgSnabkp+cd5i/pyoAb0guQFlQQUnHAZa4QNC [2024-01-19T03:00:15.360Z] 7fgTo1IUwWyDmwCnN5U= [2024-01-19T03:00:15.360Z] -----END CERTIFICATE----- [2024-01-19T03:00:15.360Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:15.360Z] MIIDCjCCAfKgAwIBAgIUJPzO3O+wmUt4gHCkl3ojsvwZL6owDQYJKoZIhvcNAQEL [2024-01-19T03:00:15.360Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T03:00:15.360Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T03:00:15.360Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T03:00:15.360Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T03:00:15.360Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T03:00:15.360Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T03:00:15.360Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T03:00:15.360Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T03:00:15.360Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T03:00:15.360Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAGVt [2024-01-19T03:00:15.360Z] g9qX+r8kyGYsvFavmnDxgMK1Zkz80irQJ2BJlDHdK1hp5aqpRbT2z2YgLog5ZMIZ [2024-01-19T03:00:15.360Z] uI36uPCOXo8+Gol1jT7D6vsgMWm0Tk9jCNbO4jPkv+wm2q1B9UZo94UGiS0K+G0S [2024-01-19T03:00:15.360Z] nccl4jmSq1gWpBZIVetdDFNjpipKjhEOVQM7x33APhACxSAWRp9iEr6V/CzMOgkf [2024-01-19T03:00:15.360Z] RHyz5D7ljh4a2XVH7++eKsj2QPfKyiKtIG72tCccQ9zGnzveiXXb4Ai8r+DyKH4Z [2024-01-19T03:00:15.360Z] zDt5Hb4ZFl72E3Rl4bIh/VQBVsr9u/SFR1p/Emyq+kGXsn+9YMiM9tK4mLUtjdox [2024-01-19T03:00:15.360Z] LMWUWTelNRGWAiaTVz8= [2024-01-19T03:00:15.360Z] -----END CERTIFICATE----- [2024-01-19T03:00:15.360Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:15.360Z] MIIDCjCCAfKgAwIBAgIUBxbEoan/HnRZ+2deRj+2RvGMrWQwDQYJKoZIhvcNAQEL [2024-01-19T03:00:15.360Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T03:00:15.360Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T03:00:15.361Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T03:00:15.361Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T03:00:15.361Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T03:00:15.361Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T03:00:15.361Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T03:00:15.361Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T03:00:15.361Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T03:00:15.361Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAB65 [2024-01-19T03:00:15.361Z] nHU+98oTP0U2J0/TsL5GFeBTH7YyUXUNRJvPWxkA8riB3swoScFKbavO6XJ/hrI2 [2024-01-19T03:00:15.361Z] ywApOz77Iyq58OLa5aIC5aaPRf3YE8LbJQYIHmtza21LXhtlKVSrLu4W58PauI8Z [2024-01-19T03:00:15.361Z] 7iaL9QPUihHFK7wrZ2ZpabN2rRJvTGWWKlhmgFTjI+71gPrXWmYBHIKmTv55V5m1 [2024-01-19T03:00:15.361Z] LxORTR/gBXeH9gmM7wz1cbaYNazqs8R1LS5SW76p/y9RLjTzU6N39VYefMMGwbsQ [2024-01-19T03:00:15.361Z] DlD11wvMBxTBp3hQYc3pdVLkVEQt8trVIwEUts7zfwUTCu/z8gH7H5XePpEVNana [2024-01-19T03:00:15.361Z] JhlAcXHCO7nN/ddJmoc= [2024-01-19T03:00:15.361Z] -----END CERTIFICATE----- [2024-01-19T03:00:15.361Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:15.361Z] MIIDGDCCAgCgAwIBAgIUEvLaarpXlv1cVvG6nMuxh2DNCKQwDQYJKoZIhvcNAQEL [2024-01-19T03:00:15.361Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T03:00:15.361Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T03:00:15.361Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T03:00:15.361Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T03:00:15.361Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T03:00:15.361Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T03:00:15.361Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T03:00:15.361Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T03:00:15.361Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T03:00:15.361Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T03:00:15.362Z] 9w0BAQsFAAOCAQEAOPakzVUmAqHhmQViZZHSQeRh/xYQBrtn6dnEo6X55SXNWcPU [2024-01-19T03:00:15.362Z] 4EEW3DvwBf75uqxKeGoXHTBpqNHG2x1+IncvIKJmi5OWM3aR/vEyJoBOchPBl+A4 [2024-01-19T03:00:15.362Z] 0xZO1vKiS2djxz8XHMkR70cxD8hv/zDJeIQbgBroF4Wa7rim5+y7jgO0O7Rs1zhT [2024-01-19T03:00:15.362Z] LDDStJ8aUKwcVRmOv7NFQxbQHXNx07kaUE2SQJfhryzZmotoBiymmQT2v0H4/4q0 [2024-01-19T03:00:15.362Z] 5dylRhSCWfwN5+It40/C3b1pbC2d9Iv/eO+B0wqg13dLF3xuA1JblEgnqA9mdNnp [2024-01-19T03:00:15.362Z] P6EgKTQBMiO6nPFyYIPqg1eWYkb1eXu9aEkmbw== [2024-01-19T03:00:15.362Z] -----END CERTIFICATE----- [2024-01-19T03:00:15.362Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:15.362Z] MIIDGDCCAgCgAwIBAgIUWf/qoBANFWF/a/yZr7wfLBMfItAwDQYJKoZIhvcNAQEL [2024-01-19T03:00:15.362Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T03:00:15.362Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T03:00:15.362Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T03:00:15.362Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T03:00:15.362Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T03:00:15.362Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T03:00:15.362Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T03:00:15.362Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T03:00:15.362Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T03:00:15.362Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T03:00:15.362Z] 9w0BAQsFAAOCAQEAdCguadZU6WJAuuAtWzvYgftPKkDKNBeGVgbqRwWRCB+QqVFl [2024-01-19T03:00:15.362Z] +3ZOQTStpnXJsY3Ig31hmL88BkBPQ47ENEzs7Czhuz59m+dcBhh/buUoIERSnfUY [2024-01-19T03:00:15.362Z] lWIfqxQevwHYZ3Kjx0l3qliYjKBQJo5xDPnukGb4yUnmrCdZ3fxMzlnr0Ais9rUH [2024-01-19T03:00:15.362Z] MjiDsjP/oUa8naQUTZO4DTRZqeEldzNXnxF6YH+3moTkwRLt9xzirNwECmJ9qxrC [2024-01-19T03:00:15.362Z] eCA7zORxbFSftYLnQiGWUS1dnxBuLOvJHjYPC6EkPPrvqnx2fXcTE1V8GizVRrA6 [2024-01-19T03:00:15.362Z] NPxt+e4qZgU/Nk/Jkpsb7h1FVcKtS8CvmtbdqQ== [2024-01-19T03:00:15.362Z] -----END CERTIFICATE----- [2024-01-19T03:00:15.362Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:15.362Z] MIIDGDCCAgCgAwIBAgIUI6D0WS4tl52N+w2lwAS+6V0DPyUwDQYJKoZIhvcNAQEL [2024-01-19T03:00:15.362Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T03:00:15.362Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T03:00:15.362Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T03:00:15.362Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T03:00:15.362Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T03:00:15.362Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T03:00:15.362Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T03:00:15.362Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T03:00:15.362Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T03:00:15.362Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T03:00:15.363Z] 9w0BAQsFAAOCAQEAAS4yElh6TMyP18ipzRA0Wenukevjs/vtkAFOsv1u5GNeDIrv [2024-01-19T03:00:15.363Z] lNi6/VjvtS+DoFNzQnTf3QoO6GTuoWHLQWt6Z/V5AwIsnOXduSXLugMwEv3j9l50 [2024-01-19T03:00:15.363Z] aDkA7vJ3OH5cGqIvIzxQN7IUOEvZNhY67GFP7+zAzVNRpJ1FHYbhQGPuFD//b9G3 [2024-01-19T03:00:15.363Z] hr4E3+iUEc+F/iBA+caA8wRx1zR383Jz6wuoFt7n7HpWNcgMvoh91JlUAK7yUp0B [2024-01-19T03:00:15.363Z] ekuG6q7HtVsk6X+aB1RyWfrSJjih75W7b+p+/bV4vNmQS1wSGSOaSIuVRvMhmpwf [2024-01-19T03:00:15.363Z] FpT+4rUpiCxklD69/MET/1cWxqvao/ppf6phzg== [2024-01-19T03:00:15.363Z] -----END CERTIFICATE----- [2024-01-19T03:00:15.363Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:15.363Z] MIIDGDCCAgCgAwIBAgIUIVCCgYV9enDck18n89YsA5b2IdgwDQYJKoZIhvcNAQEL [2024-01-19T03:00:15.363Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T03:00:15.363Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T03:00:15.363Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T03:00:15.363Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T03:00:15.363Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T03:00:15.363Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T03:00:15.363Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T03:00:15.363Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T03:00:15.363Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T03:00:15.363Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T03:00:15.363Z] 9w0BAQsFAAOCAQEAa3eFm1ivFf5Vom6a/7E1gDdjl7cjUZ5cqg332Ky1NdzEOiwW [2024-01-19T03:00:15.363Z] mx3yEk35qCd8LAoFMvPvPIuMx1NDS8kwy3FmIOjKnZapHA31ZUpn/gT/seH4GFIU [2024-01-19T03:00:15.363Z] oZw+rODV5h6jmtCVugAn48u/LxOXbY417JWwaeDx5uw+NzOSyebe+td2PbtUjf5X [2024-01-19T03:00:15.364Z] 6xcwYUmaUGsIAgfIMM12hzXOI2/q1qz5dAqwSgnR7dKBReOXnUoFpNOMpxlJ70p/ [2024-01-19T03:00:15.364Z] WUU1aWIWZqKrNI7Pg1T5jljq7+KbT3+A6+ux0FLExr2c1tw3NndXS2Y9fO55m5/q [2024-01-19T03:00:15.364Z] 0EcrTdKx4+QjbWcja3WdX2pGDk0XGNgx+EXn6Q== [2024-01-19T03:00:15.364Z] -----END CERTIFICATE----- [2024-01-19T03:00:15.364Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:15.364Z] MIIFATCCAumgAwIBAgIRAKc9ZKBASymy5TLOEp57N98wDQYJKoZIhvcNAQELBQAw [2024-01-19T03:00:15.364Z] GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDMyMzIyNTM0NloXDTM2 [2024-01-19T03:00:15.364Z] MDMyMzIyNTM0NlowGjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMIICIjANBgkq [2024-01-19T03:00:15.364Z] hkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA+pYHvQw5iU3v2b3iNuYNKYgsWD6KU7aJ [2024-01-19T03:00:15.364Z] diddtZQxSWYzUI3U0I1UsRPTxnhTifs/M9NW4ZlV13ZfB7APwC8oqKOIiwo7IwlP [2024-01-19T03:00:15.364Z] xg0VKgyz+kT8RJfYr66PPIYP0fpTeu42LpMJ+CKo9sbpgVNDZN2z/qiXrRNX/VtG [2024-01-19T03:00:15.364Z] TkPV7a44fZ5bHHVruAxvDnylpQxJobtCBWlJSsbIRGFHMc2z88eUz9NmIOWUKGGj [2024-01-19T03:00:15.364Z] EmP76x8OfRHpIpuxRSCjn0+i9+hR2siIOpcMOGd+40uVJxbRRP5ZXnUFa2fF5FWd [2024-01-19T03:00:15.364Z] O0u0RPI8HON0ovhrwPJY+4eWKkQzyC611oLPYGQ4EbifRsTsCxUZqyUuStGyp8oa [2024-01-19T03:00:15.364Z] aoSKfF6X0+KzGgwwnrjRTUpIl19A92KR0Noo6h622OX+4sZiO/JQdkuX5w/HupK0 [2024-01-19T03:00:15.364Z] A0M0WSMCvU6GOhjGotmh2VTEJwHHY4+TUk0iQYRtv1crONklyZoAQPD76hCrC8Cr [2024-01-19T03:00:15.364Z] IbgsZLfTMC8TWUoMbyUDgvgYkHKMoPm0VGVVuwpRKJxv7+2wXO+pivrrUl2Q9fPe [2024-01-19T03:00:15.364Z] Kk055nJLMV9yPUdig8othUKrRfSxli946AEV1eEOhxddfEwBE3Lt2xn0hhiIedbb [2024-01-19T03:00:15.364Z] Ftf/5kEWFZkXyUmMJK8Ra76Kus2ABueUVEcZ48hrRr1Hf1N9n59VbTUaXgeiZA50 [2024-01-19T03:00:15.364Z] qXf2bymE6F8CAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMB [2024-01-19T03:00:15.364Z] Af8wHQYDVR0OBBYEFMEmdKSKRKDm+iAo2FwjmkWIGHngMA0GCSqGSIb3DQEBCwUA [2024-01-19T03:00:15.364Z] A4ICAQBCPw74M9X/Xx04K1VAES3ypgQYH5bf9FXVDrwhRFSVckria/7dMzoF5wln [2024-01-19T03:00:15.364Z] uq9NGsjkkkDg17AohcQdr8alH4LvPdxpKr3BjpvEcmbqF8xH+MbbeUEnmbSfLI8H [2024-01-19T03:00:15.364Z] sefuhXF9AF/9iYvpVNC8FmJ0OhiVv13VgMQw0CRKkbtjZBf8xaEhq/YqxWVsgOjm [2024-01-19T03:00:15.364Z] dm5CAQ2X0aX7502x8wYRgMnZhA5goC1zVWBVAi8yhhmlhhoDUfg17cXkmaJC5pDd [2024-01-19T03:00:15.364Z] oenZ9NVhW8eDb03MFCrWNvIh89DDeCGWuWfDltDq0n3owyL0IeSn7RfpSclpxVmV [2024-01-19T03:00:15.364Z] /53jkYjwIgxIG7Gsv0LKMbsf6QdBcTjhvfZyMIpBRkTe3zuHd2feKzY9lEkbRvRQ [2024-01-19T03:00:15.364Z] zbh4Ps5YBnG6CKJPTbe2hfi3nhnw/MyEmF3zb0hzvLWNrR9XW3ibb2oL3424XOwc [2024-01-19T03:00:15.364Z] VjrTSCLzO9Rv6s5wi03qoWvKAQQAElqTYRHhynJ3w6wuvKYF5zcZF3MDnrVGLbh1 [2024-01-19T03:00:15.364Z] Q9ePRFBCiXOQ6wPLoUhrrbZ8LpFUFYDXHMtYM7P9sc9IAWoONXREJaO08zgFtMp4 [2024-01-19T03:00:15.364Z] 8iyIYUyQAbsvx8oD2M8kRvrIRSrRJSl6L957b4AFiLIQ/GgV2curs0jje7Edx34c [2024-01-19T03:00:15.365Z] idWw1VrejtwclobqNMVtG3EiPUIpJGpbMcJgbiLSmKkrvQtGng== [2024-01-19T03:00:15.365Z] -----END CERTIFICATE----- [2024-01-19T03:00:15.365Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} [2024-01-19T03:00:15.365Z] INFO[2024-01-19 03:00:15] Actual:true, Expected: true [2024-01-19T03:00:15.620Z] INFO[2024-01-19 03:00:15] Verifying : Description : Fetching configmap pointer &ConfigMap{ObjectMeta:{openshift-service-ca.crt mysql-ibm-pxb-1-84850-01-19-02h58m26s eaf9343a-d326-4fb2-b7db-a470fe25e775 1574895 0 2024-01-19 02:58:26 +0000 UTC map[] map[service.beta.openshift.io/inject-cabundle:true] [] [] [{kube-controller-manager Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:data":{},"f:metadata":{"f:annotations":{".":{},"f:service.beta.openshift.io/inject-cabundle":{}}}} } {service-ca-operator Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:data":{"f:service-ca.crt":{}}} }]},Data:map[string]string{service-ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T03:00:15.620Z] MIIDUTCCAjmgAwIBAgIILvRjno8BjvIwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE [2024-01-19T03:00:15.620Z] Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwNTM5NjA4ODAe [2024-01-19T03:00:15.620Z] Fw0yNDAxMTYwOTA4MDhaFw0yNjAzMTYwOTA4MDlaMDYxNDAyBgNVBAMMK29wZW5z [2024-01-19T03:00:15.620Z] aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDUzOTYwODgwggEiMA0GCSqG [2024-01-19T03:00:15.620Z] SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVr8xmLKsMoemVtfnlzzil6B8CGocEhNRo [2024-01-19T03:00:15.620Z] fkzkD8aSOYgIj5zcZo/M0O7nXDXeg9dqx9/h2qrGk+SS2ikU2Xs4Q5gaQDh9U1lB [2024-01-19T03:00:15.620Z] y8paoOEvw06CmJ/1Qqur5A539/OnimRSTdejTu4lwaDwsJiyoGfhvrBJAdjagtYA [2024-01-19T03:00:15.620Z] 0sNMrCgEAT4NWHS5EDx5dF7nMDHmQ0h0gHYei6vUEtiPyLwcTH1T7uVSj22fCEXX [2024-01-19T03:00:15.620Z] LjgN3OSiquJ+Hf4+csPpwgxAEkfjdBldhnImYkunJbg/x4c90uFL1pygZusrbFAf [2024-01-19T03:00:15.620Z] cERW8GngZuL5JY8MQoIstMu+1qS390ouQDDbEcvWjxC8YFNFHywrAgMBAAGjYzBh [2024-01-19T03:00:15.620Z] MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQS5vRk [2024-01-19T03:00:15.620Z] jkBPISX2qYmk1VEQZIXSLzAfBgNVHSMEGDAWgBQS5vRkjkBPISX2qYmk1VEQZIXS [2024-01-19T03:00:15.620Z] LzANBgkqhkiG9w0BAQsFAAOCAQEAZ/qc2vFGJZgDJCEUaPYuufuLRPSF68ixV9fk [2024-01-19T03:00:15.620Z] e5tBkRN8atCsEBygC/RUcwULlhsdgUMGMcREgNC/FYlwkOsAj3gP8KmcXAZ4gHCa [2024-01-19T03:00:15.620Z] T6PIbIcj1hCX0mdX0tvtaj7PWvRUz9CHh6iFzmPsXK5xjaBA9vZbkUAa7odguUit [2024-01-19T03:00:15.620Z] vz1QnKoV1OaZyccDLeQL50khM1XEBn8L+zeY3k85uo2X514HZW4LDPTQ94cpyNnb [2024-01-19T03:00:15.620Z] zqI6utp7JoPJexsHMWNFKGb//HgjluNg7TsywXIeebk+/GZUEDfNKid0oiTpQvzR [2024-01-19T03:00:15.620Z] TFFF+5KO/sa4+qIh1ERm140JxeaurUMaJzn7cFQbekEgt/N+iA== [2024-01-19T03:00:15.620Z] -----END CERTIFICATE----- [2024-01-19T03:00:15.620Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} from namespace mysql-ibm-pxb-1-84850-01-19-02h58m26s [2024-01-19T03:00:15.620Z] INFO[2024-01-19 03:00:15] Actual:true, Expected: true [2024-01-19T03:00:15.621Z] 2024-01-19 03:00:15 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.AddLabelToResource:#5038] - Adding label [edffc0c3-8249-4e9f-a79b-744c405b60af=d095e726-c3cc-4da8-b72f-6be96ae30ea7] to ConfigMap openshift-service-ca.crt [2024-01-19T03:00:15.876Z] INFO[2024-01-19 03:00:15] Verifying : Description : Adding labels edffc0c3-8249-4e9f-a79b-744c405b60af=d095e726-c3cc-4da8-b72f-6be96ae30ea7 to resource &ConfigMap{ObjectMeta:{openshift-service-ca.crt mysql-ibm-pxb-1-84850-01-19-02h58m26s eaf9343a-d326-4fb2-b7db-a470fe25e775 1574895 0 2024-01-19 02:58:26 +0000 UTC map[edffc0c3-8249-4e9f-a79b-744c405b60af:d095e726-c3cc-4da8-b72f-6be96ae30ea7] map[service.beta.openshift.io/inject-cabundle:true] [] [] [{kube-controller-manager Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:data":{},"f:metadata":{"f:annotations":{".":{},"f:service.beta.openshift.io/inject-cabundle":{}}}} } {service-ca-operator Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:data":{"f:service-ca.crt":{}}} }]},Data:map[string]string{service-ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T03:00:15.876Z] MIIDUTCCAjmgAwIBAgIILvRjno8BjvIwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE [2024-01-19T03:00:15.876Z] Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwNTM5NjA4ODAe [2024-01-19T03:00:15.876Z] Fw0yNDAxMTYwOTA4MDhaFw0yNjAzMTYwOTA4MDlaMDYxNDAyBgNVBAMMK29wZW5z [2024-01-19T03:00:15.876Z] aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDUzOTYwODgwggEiMA0GCSqG [2024-01-19T03:00:15.876Z] SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVr8xmLKsMoemVtfnlzzil6B8CGocEhNRo [2024-01-19T03:00:15.876Z] fkzkD8aSOYgIj5zcZo/M0O7nXDXeg9dqx9/h2qrGk+SS2ikU2Xs4Q5gaQDh9U1lB [2024-01-19T03:00:15.876Z] y8paoOEvw06CmJ/1Qqur5A539/OnimRSTdejTu4lwaDwsJiyoGfhvrBJAdjagtYA [2024-01-19T03:00:15.876Z] 0sNMrCgEAT4NWHS5EDx5dF7nMDHmQ0h0gHYei6vUEtiPyLwcTH1T7uVSj22fCEXX [2024-01-19T03:00:15.876Z] LjgN3OSiquJ+Hf4+csPpwgxAEkfjdBldhnImYkunJbg/x4c90uFL1pygZusrbFAf [2024-01-19T03:00:15.876Z] cERW8GngZuL5JY8MQoIstMu+1qS390ouQDDbEcvWjxC8YFNFHywrAgMBAAGjYzBh [2024-01-19T03:00:15.876Z] MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQS5vRk [2024-01-19T03:00:15.876Z] jkBPISX2qYmk1VEQZIXSLzAfBgNVHSMEGDAWgBQS5vRkjkBPISX2qYmk1VEQZIXS [2024-01-19T03:00:15.876Z] LzANBgkqhkiG9w0BAQsFAAOCAQEAZ/qc2vFGJZgDJCEUaPYuufuLRPSF68ixV9fk [2024-01-19T03:00:15.876Z] e5tBkRN8atCsEBygC/RUcwULlhsdgUMGMcREgNC/FYlwkOsAj3gP8KmcXAZ4gHCa [2024-01-19T03:00:15.876Z] T6PIbIcj1hCX0mdX0tvtaj7PWvRUz9CHh6iFzmPsXK5xjaBA9vZbkUAa7odguUit [2024-01-19T03:00:15.876Z] vz1QnKoV1OaZyccDLeQL50khM1XEBn8L+zeY3k85uo2X514HZW4LDPTQ94cpyNnb [2024-01-19T03:00:15.876Z] zqI6utp7JoPJexsHMWNFKGb//HgjluNg7TsywXIeebk+/GZUEDfNKid0oiTpQvzR [2024-01-19T03:00:15.876Z] TFFF+5KO/sa4+qIh1ERm140JxeaurUMaJzn7cFQbekEgt/N+iA== [2024-01-19T03:00:15.876Z] -----END CERTIFICATE----- [2024-01-19T03:00:15.876Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} [2024-01-19T03:00:15.876Z] INFO[2024-01-19 03:00:15] Actual:true, Expected: true [2024-01-19T03:00:16.162Z] INFO[2024-01-19 03:00:15] Verifying : Description : Fetching pvc list from namespace [mysql-ibm-pxb-2-84850-01-19-02h58m26s] [2024-01-19T03:00:16.162Z] INFO[2024-01-19 03:00:15] Actual:true, Expected: true [2024-01-19T03:00:16.163Z] INFO[2024-01-19 03:00:16] Verifying : Description : Fetching pvc pointer &PersistentVolumeClaim{ObjectMeta:{mysql-data mysql-ibm-pxb-2-84850-01-19-02h58m26s fd756974-c524-4313-b0a4-e5c874272060 1575201 0 2024-01-19 02:58:26 +0000 UTC map[] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-19 02:58:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-19 02:58:37 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{2147483648 0} {} 2Gi BinarySI},},},VolumeName:pvc-fd756974-c524-4313-b0a4-e5c874272060,Selector:nil,StorageClassName:*mysql-sc,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{10737418240 0} {} 10Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},} for pvc {{ } {mysql-data mysql-ibm-pxb-2-84850-01-19-02h58m26s fd756974-c524-4313-b0a4-e5c874272060 1575201 0 2024-01-19 02:58:26 +0000 UTC map[] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-19 02:58:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-19 02:58:37 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{2147483648 0} {} 2Gi BinarySI}]} pvc-fd756974-c524-4313-b0a4-e5c874272060 0xc0018a8320 0xc0018a8330 nil nil} {Bound [ReadWriteOnce] map[storage:{{10737418240 0} {} 10Gi BinarySI}] [] map[] }} [2024-01-19T03:00:16.163Z] INFO[2024-01-19 03:00:16] Actual:true, Expected: true [2024-01-19T03:00:16.163Z] 2024-01-19 03:00:16 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.AddLabelToResource:#5030] - Adding label [edffc0c3-8249-4e9f-a79b-744c405b60af=d095e726-c3cc-4da8-b72f-6be96ae30ea7] to PVC mysql-data [2024-01-19T03:00:16.420Z] INFO[2024-01-19 03:00:16] Verifying : Description : Adding labels edffc0c3-8249-4e9f-a79b-744c405b60af=d095e726-c3cc-4da8-b72f-6be96ae30ea7 to resource &PersistentVolumeClaim{ObjectMeta:{mysql-data mysql-ibm-pxb-2-84850-01-19-02h58m26s fd756974-c524-4313-b0a4-e5c874272060 1575201 0 2024-01-19 02:58:26 +0000 UTC map[edffc0c3-8249-4e9f-a79b-744c405b60af:d095e726-c3cc-4da8-b72f-6be96ae30ea7] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-19 02:58:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-19 02:58:37 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{2147483648 0} {} 2Gi BinarySI},},},VolumeName:pvc-fd756974-c524-4313-b0a4-e5c874272060,Selector:nil,StorageClassName:*mysql-sc,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{10737418240 0} {} 10Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},} [2024-01-19T03:00:16.420Z] INFO[2024-01-19 03:00:16] Actual:true, Expected: true [2024-01-19T03:00:16.677Z] INFO[2024-01-19 03:00:16] Verifying : Description : Fetching configmap list from namespace [mysql-ibm-pxb-2-84850-01-19-02h58m26s] [2024-01-19T03:00:16.677Z] INFO[2024-01-19 03:00:16] Actual:true, Expected: true [2024-01-19T03:00:16.934Z] INFO[2024-01-19 03:00:16] Verifying : Description : Fetching configmap pointer &ConfigMap{ObjectMeta:{kube-****-ca.crt mysql-ibm-pxb-2-84850-01-19-02h58m26s 4086550a-4d71-4a21-b2ee-2414ec79fc1e 1574952 0 2024-01-19 02:58:26 +0000 UTC map[] map[kubernetes.io/description:Contains a CA bundle that can be used to verify the kube-apiserver when using internal endpoints such as the internal service IP or kubernetes.default.svc. No other usage is guaranteed across distributions of Kubernetes clusters.] [] [] [{kube-controller-manager Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:data":{".":{},"f:ca.crt":{}},"f:metadata":{"f:annotations":{".":{},"f:kubernetes.io/description":{}}}} }]},Data:map[string]string{ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T03:00:16.934Z] MIIDCjCCAfKgAwIBAgIUWkn6rgBaC+VWV1XdJjGA3DwoRRYwDQYJKoZIhvcNAQEL [2024-01-19T03:00:16.934Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T03:00:16.934Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T03:00:16.934Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T03:00:16.934Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T03:00:16.934Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T03:00:16.934Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T03:00:16.934Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T03:00:16.934Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T03:00:16.934Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T03:00:16.934Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAHbD [2024-01-19T03:00:16.934Z] 4nWTgt/8sat7SbbHxZ+JcMGRAIaeLAjMJxkxyqD8xUq6YQcgbjDVHo1Qbh3PXB3T [2024-01-19T03:00:16.934Z] 5BWUsRbDoKi0eRJx/5cyI3846SYEAOpuUfJrg9X2CH7ZoYhxeJ42dqt3MdoMJEM1 [2024-01-19T03:00:16.934Z] k3e2rRqAKuAxwNXugHryxQM+1ehOJjrEuT5DVK6qu8+JYRcT1hBZb6hrebKPp5eG [2024-01-19T03:00:16.934Z] JvBzHP4YOATIAXhfH0Uj5Y+JhwoOdpXmOFpz+6gITPD4aNzzJZe/0ADSFfOyUJaR [2024-01-19T03:00:16.934Z] g5NnJnkeGqMM3SLsur25r+m3y0i3t+RdocZ/4mHTEnF4KWVWvZy4GjENiEBcXEol [2024-01-19T03:00:16.934Z] o4SZVaP4AF7XQJ5vdA0= [2024-01-19T03:00:16.935Z] -----END CERTIFICATE----- [2024-01-19T03:00:16.935Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:16.935Z] MIIDCjCCAfKgAwIBAgIUXhIIFCc/NxZJn6hNEFoZuPQ8N5UwDQYJKoZIhvcNAQEL [2024-01-19T03:00:16.935Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T03:00:16.935Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T03:00:16.935Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T03:00:16.935Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T03:00:16.935Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T03:00:16.935Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T03:00:16.935Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T03:00:16.935Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T03:00:16.935Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T03:00:16.935Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAKG3 [2024-01-19T03:00:16.935Z] nVfv4IkgHVOxgbrNIoYhjvBvVJcLZub0C9yidexLA3A61p1a7b/OpidG5zShPhH2 [2024-01-19T03:00:16.935Z] HRdQcqgp0HIxaS+/oLD3eElOnnC/EKD359K6n3kN66sMgvNbHxNFM84Au0TXqmbT [2024-01-19T03:00:16.935Z] w8afrKjrfe/VglpHC2GfmgHMF4YH+ZF48C1RZA6z2oXj6Aj+uww0PjiClE9qUYqK [2024-01-19T03:00:16.935Z] R9uRa32prExihsi+Bto7n2Axo92qUiK2KrkEdQitnQfH5z/PojS78vbyIKp1O86N [2024-01-19T03:00:16.935Z] qRImvCYY17CPxP+JUQ7dxWDMF4NTgSnabkp+cd5i/pyoAb0guQFlQQUnHAZa4QNC [2024-01-19T03:00:16.935Z] 7fgTo1IUwWyDmwCnN5U= [2024-01-19T03:00:16.935Z] -----END CERTIFICATE----- [2024-01-19T03:00:16.935Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:16.935Z] MIIDCjCCAfKgAwIBAgIUJPzO3O+wmUt4gHCkl3ojsvwZL6owDQYJKoZIhvcNAQEL [2024-01-19T03:00:16.935Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T03:00:16.935Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T03:00:16.935Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T03:00:16.935Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T03:00:16.935Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T03:00:16.935Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T03:00:16.935Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T03:00:16.935Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T03:00:16.935Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T03:00:16.935Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAGVt [2024-01-19T03:00:16.935Z] g9qX+r8kyGYsvFavmnDxgMK1Zkz80irQJ2BJlDHdK1hp5aqpRbT2z2YgLog5ZMIZ [2024-01-19T03:00:16.935Z] uI36uPCOXo8+Gol1jT7D6vsgMWm0Tk9jCNbO4jPkv+wm2q1B9UZo94UGiS0K+G0S [2024-01-19T03:00:16.935Z] nccl4jmSq1gWpBZIVetdDFNjpipKjhEOVQM7x33APhACxSAWRp9iEr6V/CzMOgkf [2024-01-19T03:00:16.935Z] RHyz5D7ljh4a2XVH7++eKsj2QPfKyiKtIG72tCccQ9zGnzveiXXb4Ai8r+DyKH4Z [2024-01-19T03:00:16.935Z] zDt5Hb4ZFl72E3Rl4bIh/VQBVsr9u/SFR1p/Emyq+kGXsn+9YMiM9tK4mLUtjdox [2024-01-19T03:00:16.935Z] LMWUWTelNRGWAiaTVz8= [2024-01-19T03:00:16.935Z] -----END CERTIFICATE----- [2024-01-19T03:00:16.935Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:16.935Z] MIIDCjCCAfKgAwIBAgIUBxbEoan/HnRZ+2deRj+2RvGMrWQwDQYJKoZIhvcNAQEL [2024-01-19T03:00:16.935Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T03:00:16.935Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T03:00:16.935Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T03:00:16.935Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T03:00:16.935Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T03:00:16.935Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T03:00:16.935Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T03:00:16.935Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T03:00:16.935Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T03:00:16.935Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAB65 [2024-01-19T03:00:16.935Z] nHU+98oTP0U2J0/TsL5GFeBTH7YyUXUNRJvPWxkA8riB3swoScFKbavO6XJ/hrI2 [2024-01-19T03:00:16.935Z] ywApOz77Iyq58OLa5aIC5aaPRf3YE8LbJQYIHmtza21LXhtlKVSrLu4W58PauI8Z [2024-01-19T03:00:16.935Z] 7iaL9QPUihHFK7wrZ2ZpabN2rRJvTGWWKlhmgFTjI+71gPrXWmYBHIKmTv55V5m1 [2024-01-19T03:00:16.935Z] LxORTR/gBXeH9gmM7wz1cbaYNazqs8R1LS5SW76p/y9RLjTzU6N39VYefMMGwbsQ [2024-01-19T03:00:16.935Z] DlD11wvMBxTBp3hQYc3pdVLkVEQt8trVIwEUts7zfwUTCu/z8gH7H5XePpEVNana [2024-01-19T03:00:16.935Z] JhlAcXHCO7nN/ddJmoc= [2024-01-19T03:00:16.935Z] -----END CERTIFICATE----- [2024-01-19T03:00:16.935Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:16.935Z] MIIDGDCCAgCgAwIBAgIUEvLaarpXlv1cVvG6nMuxh2DNCKQwDQYJKoZIhvcNAQEL [2024-01-19T03:00:16.935Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T03:00:16.935Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T03:00:16.935Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T03:00:16.936Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T03:00:16.936Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T03:00:16.936Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T03:00:16.936Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T03:00:16.936Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T03:00:16.936Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T03:00:16.936Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T03:00:16.936Z] 9w0BAQsFAAOCAQEAOPakzVUmAqHhmQViZZHSQeRh/xYQBrtn6dnEo6X55SXNWcPU [2024-01-19T03:00:16.936Z] 4EEW3DvwBf75uqxKeGoXHTBpqNHG2x1+IncvIKJmi5OWM3aR/vEyJoBOchPBl+A4 [2024-01-19T03:00:16.936Z] 0xZO1vKiS2djxz8XHMkR70cxD8hv/zDJeIQbgBroF4Wa7rim5+y7jgO0O7Rs1zhT [2024-01-19T03:00:16.936Z] LDDStJ8aUKwcVRmOv7NFQxbQHXNx07kaUE2SQJfhryzZmotoBiymmQT2v0H4/4q0 [2024-01-19T03:00:16.936Z] 5dylRhSCWfwN5+It40/C3b1pbC2d9Iv/eO+B0wqg13dLF3xuA1JblEgnqA9mdNnp [2024-01-19T03:00:16.936Z] P6EgKTQBMiO6nPFyYIPqg1eWYkb1eXu9aEkmbw== [2024-01-19T03:00:16.936Z] -----END CERTIFICATE----- [2024-01-19T03:00:16.936Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:16.936Z] MIIDGDCCAgCgAwIBAgIUWf/qoBANFWF/a/yZr7wfLBMfItAwDQYJKoZIhvcNAQEL [2024-01-19T03:00:16.936Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T03:00:16.936Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T03:00:16.936Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T03:00:16.936Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T03:00:16.936Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T03:00:16.936Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T03:00:16.936Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T03:00:16.936Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T03:00:16.936Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T03:00:16.936Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T03:00:16.936Z] 9w0BAQsFAAOCAQEAdCguadZU6WJAuuAtWzvYgftPKkDKNBeGVgbqRwWRCB+QqVFl [2024-01-19T03:00:16.936Z] +3ZOQTStpnXJsY3Ig31hmL88BkBPQ47ENEzs7Czhuz59m+dcBhh/buUoIERSnfUY [2024-01-19T03:00:16.936Z] lWIfqxQevwHYZ3Kjx0l3qliYjKBQJo5xDPnukGb4yUnmrCdZ3fxMzlnr0Ais9rUH [2024-01-19T03:00:16.936Z] MjiDsjP/oUa8naQUTZO4DTRZqeEldzNXnxF6YH+3moTkwRLt9xzirNwECmJ9qxrC [2024-01-19T03:00:16.936Z] eCA7zORxbFSftYLnQiGWUS1dnxBuLOvJHjYPC6EkPPrvqnx2fXcTE1V8GizVRrA6 [2024-01-19T03:00:16.936Z] NPxt+e4qZgU/Nk/Jkpsb7h1FVcKtS8CvmtbdqQ== [2024-01-19T03:00:16.936Z] -----END CERTIFICATE----- [2024-01-19T03:00:16.936Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:16.936Z] MIIDGDCCAgCgAwIBAgIUI6D0WS4tl52N+w2lwAS+6V0DPyUwDQYJKoZIhvcNAQEL [2024-01-19T03:00:16.936Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T03:00:16.936Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T03:00:16.936Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T03:00:16.936Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T03:00:16.936Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T03:00:16.936Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T03:00:16.936Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T03:00:16.936Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T03:00:16.936Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T03:00:16.936Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T03:00:16.936Z] 9w0BAQsFAAOCAQEAAS4yElh6TMyP18ipzRA0Wenukevjs/vtkAFOsv1u5GNeDIrv [2024-01-19T03:00:16.936Z] lNi6/VjvtS+DoFNzQnTf3QoO6GTuoWHLQWt6Z/V5AwIsnOXduSXLugMwEv3j9l50 [2024-01-19T03:00:16.936Z] aDkA7vJ3OH5cGqIvIzxQN7IUOEvZNhY67GFP7+zAzVNRpJ1FHYbhQGPuFD//b9G3 [2024-01-19T03:00:16.936Z] hr4E3+iUEc+F/iBA+caA8wRx1zR383Jz6wuoFt7n7HpWNcgMvoh91JlUAK7yUp0B [2024-01-19T03:00:16.936Z] ekuG6q7HtVsk6X+aB1RyWfrSJjih75W7b+p+/bV4vNmQS1wSGSOaSIuVRvMhmpwf [2024-01-19T03:00:16.936Z] FpT+4rUpiCxklD69/MET/1cWxqvao/ppf6phzg== [2024-01-19T03:00:16.936Z] -----END CERTIFICATE----- [2024-01-19T03:00:16.936Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:16.936Z] MIIDGDCCAgCgAwIBAgIUIVCCgYV9enDck18n89YsA5b2IdgwDQYJKoZIhvcNAQEL [2024-01-19T03:00:16.936Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T03:00:16.936Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T03:00:16.936Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T03:00:16.937Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T03:00:16.937Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T03:00:16.937Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T03:00:16.937Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T03:00:16.937Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T03:00:16.937Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T03:00:16.937Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T03:00:16.937Z] 9w0BAQsFAAOCAQEAa3eFm1ivFf5Vom6a/7E1gDdjl7cjUZ5cqg332Ky1NdzEOiwW [2024-01-19T03:00:16.937Z] mx3yEk35qCd8LAoFMvPvPIuMx1NDS8kwy3FmIOjKnZapHA31ZUpn/gT/seH4GFIU [2024-01-19T03:00:16.937Z] oZw+rODV5h6jmtCVugAn48u/LxOXbY417JWwaeDx5uw+NzOSyebe+td2PbtUjf5X [2024-01-19T03:00:16.937Z] 6xcwYUmaUGsIAgfIMM12hzXOI2/q1qz5dAqwSgnR7dKBReOXnUoFpNOMpxlJ70p/ [2024-01-19T03:00:16.937Z] WUU1aWIWZqKrNI7Pg1T5jljq7+KbT3+A6+ux0FLExr2c1tw3NndXS2Y9fO55m5/q [2024-01-19T03:00:16.937Z] 0EcrTdKx4+QjbWcja3WdX2pGDk0XGNgx+EXn6Q== [2024-01-19T03:00:16.937Z] -----END CERTIFICATE----- [2024-01-19T03:00:16.937Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:16.937Z] MIIFATCCAumgAwIBAgIRAKc9ZKBASymy5TLOEp57N98wDQYJKoZIhvcNAQELBQAw [2024-01-19T03:00:16.937Z] GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDMyMzIyNTM0NloXDTM2 [2024-01-19T03:00:16.937Z] MDMyMzIyNTM0NlowGjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMIICIjANBgkq [2024-01-19T03:00:16.937Z] hkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA+pYHvQw5iU3v2b3iNuYNKYgsWD6KU7aJ [2024-01-19T03:00:16.937Z] diddtZQxSWYzUI3U0I1UsRPTxnhTifs/M9NW4ZlV13ZfB7APwC8oqKOIiwo7IwlP [2024-01-19T03:00:16.937Z] xg0VKgyz+kT8RJfYr66PPIYP0fpTeu42LpMJ+CKo9sbpgVNDZN2z/qiXrRNX/VtG [2024-01-19T03:00:16.937Z] TkPV7a44fZ5bHHVruAxvDnylpQxJobtCBWlJSsbIRGFHMc2z88eUz9NmIOWUKGGj [2024-01-19T03:00:16.937Z] EmP76x8OfRHpIpuxRSCjn0+i9+hR2siIOpcMOGd+40uVJxbRRP5ZXnUFa2fF5FWd [2024-01-19T03:00:16.937Z] O0u0RPI8HON0ovhrwPJY+4eWKkQzyC611oLPYGQ4EbifRsTsCxUZqyUuStGyp8oa [2024-01-19T03:00:16.937Z] aoSKfF6X0+KzGgwwnrjRTUpIl19A92KR0Noo6h622OX+4sZiO/JQdkuX5w/HupK0 [2024-01-19T03:00:16.937Z] A0M0WSMCvU6GOhjGotmh2VTEJwHHY4+TUk0iQYRtv1crONklyZoAQPD76hCrC8Cr [2024-01-19T03:00:16.937Z] IbgsZLfTMC8TWUoMbyUDgvgYkHKMoPm0VGVVuwpRKJxv7+2wXO+pivrrUl2Q9fPe [2024-01-19T03:00:16.937Z] Kk055nJLMV9yPUdig8othUKrRfSxli946AEV1eEOhxddfEwBE3Lt2xn0hhiIedbb [2024-01-19T03:00:16.937Z] Ftf/5kEWFZkXyUmMJK8Ra76Kus2ABueUVEcZ48hrRr1Hf1N9n59VbTUaXgeiZA50 [2024-01-19T03:00:16.937Z] qXf2bymE6F8CAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMB [2024-01-19T03:00:16.937Z] Af8wHQYDVR0OBBYEFMEmdKSKRKDm+iAo2FwjmkWIGHngMA0GCSqGSIb3DQEBCwUA [2024-01-19T03:00:16.937Z] A4ICAQBCPw74M9X/Xx04K1VAES3ypgQYH5bf9FXVDrwhRFSVckria/7dMzoF5wln [2024-01-19T03:00:16.937Z] uq9NGsjkkkDg17AohcQdr8alH4LvPdxpKr3BjpvEcmbqF8xH+MbbeUEnmbSfLI8H [2024-01-19T03:00:16.937Z] sefuhXF9AF/9iYvpVNC8FmJ0OhiVv13VgMQw0CRKkbtjZBf8xaEhq/YqxWVsgOjm [2024-01-19T03:00:16.937Z] dm5CAQ2X0aX7502x8wYRgMnZhA5goC1zVWBVAi8yhhmlhhoDUfg17cXkmaJC5pDd [2024-01-19T03:00:16.937Z] oenZ9NVhW8eDb03MFCrWNvIh89DDeCGWuWfDltDq0n3owyL0IeSn7RfpSclpxVmV [2024-01-19T03:00:16.937Z] /53jkYjwIgxIG7Gsv0LKMbsf6QdBcTjhvfZyMIpBRkTe3zuHd2feKzY9lEkbRvRQ [2024-01-19T03:00:16.937Z] zbh4Ps5YBnG6CKJPTbe2hfi3nhnw/MyEmF3zb0hzvLWNrR9XW3ibb2oL3424XOwc [2024-01-19T03:00:16.937Z] VjrTSCLzO9Rv6s5wi03qoWvKAQQAElqTYRHhynJ3w6wuvKYF5zcZF3MDnrVGLbh1 [2024-01-19T03:00:16.937Z] Q9ePRFBCiXOQ6wPLoUhrrbZ8LpFUFYDXHMtYM7P9sc9IAWoONXREJaO08zgFtMp4 [2024-01-19T03:00:16.937Z] 8iyIYUyQAbsvx8oD2M8kRvrIRSrRJSl6L957b4AFiLIQ/GgV2curs0jje7Edx34c [2024-01-19T03:00:16.937Z] idWw1VrejtwclobqNMVtG3EiPUIpJGpbMcJgbiLSmKkrvQtGng== [2024-01-19T03:00:16.937Z] -----END CERTIFICATE----- [2024-01-19T03:00:16.937Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} from namespace mysql-ibm-pxb-2-84850-01-19-02h58m26s [2024-01-19T03:00:16.937Z] INFO[2024-01-19 03:00:16] Actual:true, Expected: true [2024-01-19T03:00:16.937Z] 2024-01-19 03:00:16 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.AddLabelToResource:#5038] - Adding label [edffc0c3-8249-4e9f-a79b-744c405b60af=d095e726-c3cc-4da8-b72f-6be96ae30ea7] to ConfigMap kube-****-ca.crt [2024-01-19T03:00:16.938Z] INFO[2024-01-19 03:00:16] Verifying : Description : Adding labels edffc0c3-8249-4e9f-a79b-744c405b60af=d095e726-c3cc-4da8-b72f-6be96ae30ea7 to resource &ConfigMap{ObjectMeta:{kube-****-ca.crt mysql-ibm-pxb-2-84850-01-19-02h58m26s 4086550a-4d71-4a21-b2ee-2414ec79fc1e 1574952 0 2024-01-19 02:58:26 +0000 UTC map[edffc0c3-8249-4e9f-a79b-744c405b60af:d095e726-c3cc-4da8-b72f-6be96ae30ea7] map[kubernetes.io/description:Contains a CA bundle that can be used to verify the kube-apiserver when using internal endpoints such as the internal service IP or kubernetes.default.svc. No other usage is guaranteed across distributions of Kubernetes clusters.] [] [] [{kube-controller-manager Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:data":{".":{},"f:ca.crt":{}},"f:metadata":{"f:annotations":{".":{},"f:kubernetes.io/description":{}}}} }]},Data:map[string]string{ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T03:00:16.938Z] MIIDCjCCAfKgAwIBAgIUWkn6rgBaC+VWV1XdJjGA3DwoRRYwDQYJKoZIhvcNAQEL [2024-01-19T03:00:16.938Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T03:00:16.938Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T03:00:16.938Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T03:00:16.938Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T03:00:16.938Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T03:00:16.938Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T03:00:16.938Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T03:00:16.938Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T03:00:16.938Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T03:00:16.938Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAHbD [2024-01-19T03:00:16.938Z] 4nWTgt/8sat7SbbHxZ+JcMGRAIaeLAjMJxkxyqD8xUq6YQcgbjDVHo1Qbh3PXB3T [2024-01-19T03:00:16.938Z] 5BWUsRbDoKi0eRJx/5cyI3846SYEAOpuUfJrg9X2CH7ZoYhxeJ42dqt3MdoMJEM1 [2024-01-19T03:00:16.938Z] k3e2rRqAKuAxwNXugHryxQM+1ehOJjrEuT5DVK6qu8+JYRcT1hBZb6hrebKPp5eG [2024-01-19T03:00:16.938Z] JvBzHP4YOATIAXhfH0Uj5Y+JhwoOdpXmOFpz+6gITPD4aNzzJZe/0ADSFfOyUJaR [2024-01-19T03:00:16.938Z] g5NnJnkeGqMM3SLsur25r+m3y0i3t+RdocZ/4mHTEnF4KWVWvZy4GjENiEBcXEol [2024-01-19T03:00:16.938Z] o4SZVaP4AF7XQJ5vdA0= [2024-01-19T03:00:16.938Z] -----END CERTIFICATE----- [2024-01-19T03:00:16.938Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:16.938Z] MIIDCjCCAfKgAwIBAgIUXhIIFCc/NxZJn6hNEFoZuPQ8N5UwDQYJKoZIhvcNAQEL [2024-01-19T03:00:16.938Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T03:00:16.938Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T03:00:16.938Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T03:00:16.938Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T03:00:16.938Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T03:00:16.938Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T03:00:16.938Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T03:00:16.938Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T03:00:16.938Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T03:00:16.938Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAKG3 [2024-01-19T03:00:16.938Z] nVfv4IkgHVOxgbrNIoYhjvBvVJcLZub0C9yidexLA3A61p1a7b/OpidG5zShPhH2 [2024-01-19T03:00:16.938Z] HRdQcqgp0HIxaS+/oLD3eElOnnC/EKD359K6n3kN66sMgvNbHxNFM84Au0TXqmbT [2024-01-19T03:00:16.938Z] w8afrKjrfe/VglpHC2GfmgHMF4YH+ZF48C1RZA6z2oXj6Aj+uww0PjiClE9qUYqK [2024-01-19T03:00:16.938Z] R9uRa32prExihsi+Bto7n2Axo92qUiK2KrkEdQitnQfH5z/PojS78vbyIKp1O86N [2024-01-19T03:00:16.938Z] qRImvCYY17CPxP+JUQ7dxWDMF4NTgSnabkp+cd5i/pyoAb0guQFlQQUnHAZa4QNC [2024-01-19T03:00:16.938Z] 7fgTo1IUwWyDmwCnN5U= [2024-01-19T03:00:16.938Z] -----END CERTIFICATE----- [2024-01-19T03:00:16.938Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:16.938Z] MIIDCjCCAfKgAwIBAgIUJPzO3O+wmUt4gHCkl3ojsvwZL6owDQYJKoZIhvcNAQEL [2024-01-19T03:00:16.938Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T03:00:16.938Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T03:00:16.938Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T03:00:16.938Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T03:00:16.938Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T03:00:16.938Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T03:00:16.938Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T03:00:16.938Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T03:00:16.938Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T03:00:16.938Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAGVt [2024-01-19T03:00:16.938Z] g9qX+r8kyGYsvFavmnDxgMK1Zkz80irQJ2BJlDHdK1hp5aqpRbT2z2YgLog5ZMIZ [2024-01-19T03:00:16.938Z] uI36uPCOXo8+Gol1jT7D6vsgMWm0Tk9jCNbO4jPkv+wm2q1B9UZo94UGiS0K+G0S [2024-01-19T03:00:16.938Z] nccl4jmSq1gWpBZIVetdDFNjpipKjhEOVQM7x33APhACxSAWRp9iEr6V/CzMOgkf [2024-01-19T03:00:16.938Z] RHyz5D7ljh4a2XVH7++eKsj2QPfKyiKtIG72tCccQ9zGnzveiXXb4Ai8r+DyKH4Z [2024-01-19T03:00:16.938Z] zDt5Hb4ZFl72E3Rl4bIh/VQBVsr9u/SFR1p/Emyq+kGXsn+9YMiM9tK4mLUtjdox [2024-01-19T03:00:16.938Z] LMWUWTelNRGWAiaTVz8= [2024-01-19T03:00:16.938Z] -----END CERTIFICATE----- [2024-01-19T03:00:16.938Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:16.938Z] MIIDCjCCAfKgAwIBAgIUBxbEoan/HnRZ+2deRj+2RvGMrWQwDQYJKoZIhvcNAQEL [2024-01-19T03:00:16.938Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T03:00:16.939Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T03:00:16.939Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T03:00:16.939Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T03:00:16.939Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T03:00:16.939Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T03:00:16.939Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T03:00:16.939Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T03:00:16.939Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T03:00:16.939Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAB65 [2024-01-19T03:00:16.939Z] nHU+98oTP0U2J0/TsL5GFeBTH7YyUXUNRJvPWxkA8riB3swoScFKbavO6XJ/hrI2 [2024-01-19T03:00:16.939Z] ywApOz77Iyq58OLa5aIC5aaPRf3YE8LbJQYIHmtza21LXhtlKVSrLu4W58PauI8Z [2024-01-19T03:00:16.939Z] 7iaL9QPUihHFK7wrZ2ZpabN2rRJvTGWWKlhmgFTjI+71gPrXWmYBHIKmTv55V5m1 [2024-01-19T03:00:16.939Z] LxORTR/gBXeH9gmM7wz1cbaYNazqs8R1LS5SW76p/y9RLjTzU6N39VYefMMGwbsQ [2024-01-19T03:00:16.939Z] DlD11wvMBxTBp3hQYc3pdVLkVEQt8trVIwEUts7zfwUTCu/z8gH7H5XePpEVNana [2024-01-19T03:00:16.939Z] JhlAcXHCO7nN/ddJmoc= [2024-01-19T03:00:16.939Z] -----END CERTIFICATE----- [2024-01-19T03:00:16.939Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:16.939Z] MIIDGDCCAgCgAwIBAgIUEvLaarpXlv1cVvG6nMuxh2DNCKQwDQYJKoZIhvcNAQEL [2024-01-19T03:00:16.939Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T03:00:16.939Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T03:00:16.939Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T03:00:16.939Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T03:00:16.939Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T03:00:16.939Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T03:00:16.939Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T03:00:16.939Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T03:00:16.939Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T03:00:16.939Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T03:00:16.939Z] 9w0BAQsFAAOCAQEAOPakzVUmAqHhmQViZZHSQeRh/xYQBrtn6dnEo6X55SXNWcPU [2024-01-19T03:00:16.939Z] 4EEW3DvwBf75uqxKeGoXHTBpqNHG2x1+IncvIKJmi5OWM3aR/vEyJoBOchPBl+A4 [2024-01-19T03:00:16.939Z] 0xZO1vKiS2djxz8XHMkR70cxD8hv/zDJeIQbgBroF4Wa7rim5+y7jgO0O7Rs1zhT [2024-01-19T03:00:16.939Z] LDDStJ8aUKwcVRmOv7NFQxbQHXNx07kaUE2SQJfhryzZmotoBiymmQT2v0H4/4q0 [2024-01-19T03:00:16.939Z] 5dylRhSCWfwN5+It40/C3b1pbC2d9Iv/eO+B0wqg13dLF3xuA1JblEgnqA9mdNnp [2024-01-19T03:00:16.939Z] P6EgKTQBMiO6nPFyYIPqg1eWYkb1eXu9aEkmbw== [2024-01-19T03:00:16.939Z] -----END CERTIFICATE----- [2024-01-19T03:00:16.939Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:16.939Z] MIIDGDCCAgCgAwIBAgIUWf/qoBANFWF/a/yZr7wfLBMfItAwDQYJKoZIhvcNAQEL [2024-01-19T03:00:16.939Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T03:00:16.939Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T03:00:16.939Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T03:00:16.939Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T03:00:16.939Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T03:00:16.939Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T03:00:16.939Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T03:00:16.939Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T03:00:16.939Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T03:00:16.939Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T03:00:16.939Z] 9w0BAQsFAAOCAQEAdCguadZU6WJAuuAtWzvYgftPKkDKNBeGVgbqRwWRCB+QqVFl [2024-01-19T03:00:16.939Z] +3ZOQTStpnXJsY3Ig31hmL88BkBPQ47ENEzs7Czhuz59m+dcBhh/buUoIERSnfUY [2024-01-19T03:00:16.939Z] lWIfqxQevwHYZ3Kjx0l3qliYjKBQJo5xDPnukGb4yUnmrCdZ3fxMzlnr0Ais9rUH [2024-01-19T03:00:16.939Z] MjiDsjP/oUa8naQUTZO4DTRZqeEldzNXnxF6YH+3moTkwRLt9xzirNwECmJ9qxrC [2024-01-19T03:00:16.939Z] eCA7zORxbFSftYLnQiGWUS1dnxBuLOvJHjYPC6EkPPrvqnx2fXcTE1V8GizVRrA6 [2024-01-19T03:00:16.939Z] NPxt+e4qZgU/Nk/Jkpsb7h1FVcKtS8CvmtbdqQ== [2024-01-19T03:00:16.939Z] -----END CERTIFICATE----- [2024-01-19T03:00:16.939Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:16.939Z] MIIDGDCCAgCgAwIBAgIUI6D0WS4tl52N+w2lwAS+6V0DPyUwDQYJKoZIhvcNAQEL [2024-01-19T03:00:16.939Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T03:00:16.939Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T03:00:16.939Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T03:00:16.939Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T03:00:16.939Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T03:00:16.939Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T03:00:16.939Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T03:00:16.939Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T03:00:16.939Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T03:00:16.940Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T03:00:16.940Z] 9w0BAQsFAAOCAQEAAS4yElh6TMyP18ipzRA0Wenukevjs/vtkAFOsv1u5GNeDIrv [2024-01-19T03:00:16.940Z] lNi6/VjvtS+DoFNzQnTf3QoO6GTuoWHLQWt6Z/V5AwIsnOXduSXLugMwEv3j9l50 [2024-01-19T03:00:16.940Z] aDkA7vJ3OH5cGqIvIzxQN7IUOEvZNhY67GFP7+zAzVNRpJ1FHYbhQGPuFD//b9G3 [2024-01-19T03:00:16.940Z] hr4E3+iUEc+F/iBA+caA8wRx1zR383Jz6wuoFt7n7HpWNcgMvoh91JlUAK7yUp0B [2024-01-19T03:00:16.940Z] ekuG6q7HtVsk6X+aB1RyWfrSJjih75W7b+p+/bV4vNmQS1wSGSOaSIuVRvMhmpwf [2024-01-19T03:00:16.940Z] FpT+4rUpiCxklD69/MET/1cWxqvao/ppf6phzg== [2024-01-19T03:00:16.940Z] -----END CERTIFICATE----- [2024-01-19T03:00:16.940Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:16.940Z] MIIDGDCCAgCgAwIBAgIUIVCCgYV9enDck18n89YsA5b2IdgwDQYJKoZIhvcNAQEL [2024-01-19T03:00:16.940Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T03:00:16.940Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T03:00:16.940Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T03:00:16.940Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T03:00:16.940Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T03:00:16.940Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T03:00:16.940Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T03:00:16.940Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T03:00:16.940Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T03:00:16.940Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T03:00:16.940Z] 9w0BAQsFAAOCAQEAa3eFm1ivFf5Vom6a/7E1gDdjl7cjUZ5cqg332Ky1NdzEOiwW [2024-01-19T03:00:16.940Z] mx3yEk35qCd8LAoFMvPvPIuMx1NDS8kwy3FmIOjKnZapHA31ZUpn/gT/seH4GFIU [2024-01-19T03:00:16.940Z] oZw+rODV5h6jmtCVugAn48u/LxOXbY417JWwaeDx5uw+NzOSyebe+td2PbtUjf5X [2024-01-19T03:00:16.940Z] 6xcwYUmaUGsIAgfIMM12hzXOI2/q1qz5dAqwSgnR7dKBReOXnUoFpNOMpxlJ70p/ [2024-01-19T03:00:16.940Z] WUU1aWIWZqKrNI7Pg1T5jljq7+KbT3+A6+ux0FLExr2c1tw3NndXS2Y9fO55m5/q [2024-01-19T03:00:16.940Z] 0EcrTdKx4+QjbWcja3WdX2pGDk0XGNgx+EXn6Q== [2024-01-19T03:00:16.940Z] -----END CERTIFICATE----- [2024-01-19T03:00:16.940Z] -----BEGIN CERTIFICATE----- [2024-01-19T03:00:16.940Z] MIIFATCCAumgAwIBAgIRAKc9ZKBASymy5TLOEp57N98wDQYJKoZIhvcNAQELBQAw [2024-01-19T03:00:16.940Z] GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDMyMzIyNTM0NloXDTM2 [2024-01-19T03:00:16.940Z] MDMyMzIyNTM0NlowGjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMIICIjANBgkq [2024-01-19T03:00:16.940Z] hkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA+pYHvQw5iU3v2b3iNuYNKYgsWD6KU7aJ [2024-01-19T03:00:16.940Z] diddtZQxSWYzUI3U0I1UsRPTxnhTifs/M9NW4ZlV13ZfB7APwC8oqKOIiwo7IwlP [2024-01-19T03:00:16.940Z] xg0VKgyz+kT8RJfYr66PPIYP0fpTeu42LpMJ+CKo9sbpgVNDZN2z/qiXrRNX/VtG [2024-01-19T03:00:16.940Z] TkPV7a44fZ5bHHVruAxvDnylpQxJobtCBWlJSsbIRGFHMc2z88eUz9NmIOWUKGGj [2024-01-19T03:00:16.940Z] EmP76x8OfRHpIpuxRSCjn0+i9+hR2siIOpcMOGd+40uVJxbRRP5ZXnUFa2fF5FWd [2024-01-19T03:00:16.940Z] O0u0RPI8HON0ovhrwPJY+4eWKkQzyC611oLPYGQ4EbifRsTsCxUZqyUuStGyp8oa [2024-01-19T03:00:16.940Z] aoSKfF6X0+KzGgwwnrjRTUpIl19A92KR0Noo6h622OX+4sZiO/JQdkuX5w/HupK0 [2024-01-19T03:00:16.940Z] A0M0WSMCvU6GOhjGotmh2VTEJwHHY4+TUk0iQYRtv1crONklyZoAQPD76hCrC8Cr [2024-01-19T03:00:16.940Z] IbgsZLfTMC8TWUoMbyUDgvgYkHKMoPm0VGVVuwpRKJxv7+2wXO+pivrrUl2Q9fPe [2024-01-19T03:00:16.940Z] Kk055nJLMV9yPUdig8othUKrRfSxli946AEV1eEOhxddfEwBE3Lt2xn0hhiIedbb [2024-01-19T03:00:16.940Z] Ftf/5kEWFZkXyUmMJK8Ra76Kus2ABueUVEcZ48hrRr1Hf1N9n59VbTUaXgeiZA50 [2024-01-19T03:00:16.940Z] qXf2bymE6F8CAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMB [2024-01-19T03:00:16.940Z] Af8wHQYDVR0OBBYEFMEmdKSKRKDm+iAo2FwjmkWIGHngMA0GCSqGSIb3DQEBCwUA [2024-01-19T03:00:16.940Z] A4ICAQBCPw74M9X/Xx04K1VAES3ypgQYH5bf9FXVDrwhRFSVckria/7dMzoF5wln [2024-01-19T03:00:16.940Z] uq9NGsjkkkDg17AohcQdr8alH4LvPdxpKr3BjpvEcmbqF8xH+MbbeUEnmbSfLI8H [2024-01-19T03:00:16.940Z] sefuhXF9AF/9iYvpVNC8FmJ0OhiVv13VgMQw0CRKkbtjZBf8xaEhq/YqxWVsgOjm [2024-01-19T03:00:16.940Z] dm5CAQ2X0aX7502x8wYRgMnZhA5goC1zVWBVAi8yhhmlhhoDUfg17cXkmaJC5pDd [2024-01-19T03:00:16.940Z] oenZ9NVhW8eDb03MFCrWNvIh89DDeCGWuWfDltDq0n3owyL0IeSn7RfpSclpxVmV [2024-01-19T03:00:16.940Z] /53jkYjwIgxIG7Gsv0LKMbsf6QdBcTjhvfZyMIpBRkTe3zuHd2feKzY9lEkbRvRQ [2024-01-19T03:00:16.940Z] zbh4Ps5YBnG6CKJPTbe2hfi3nhnw/MyEmF3zb0hzvLWNrR9XW3ibb2oL3424XOwc [2024-01-19T03:00:16.940Z] VjrTSCLzO9Rv6s5wi03qoWvKAQQAElqTYRHhynJ3w6wuvKYF5zcZF3MDnrVGLbh1 [2024-01-19T03:00:16.940Z] Q9ePRFBCiXOQ6wPLoUhrrbZ8LpFUFYDXHMtYM7P9sc9IAWoONXREJaO08zgFtMp4 [2024-01-19T03:00:16.940Z] 8iyIYUyQAbsvx8oD2M8kRvrIRSrRJSl6L957b4AFiLIQ/GgV2curs0jje7Edx34c [2024-01-19T03:00:16.940Z] idWw1VrejtwclobqNMVtG3EiPUIpJGpbMcJgbiLSmKkrvQtGng== [2024-01-19T03:00:16.940Z] -----END CERTIFICATE----- [2024-01-19T03:00:16.940Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} [2024-01-19T03:00:16.940Z] INFO[2024-01-19 03:00:16] Actual:true, Expected: true [2024-01-19T03:00:17.196Z] INFO[2024-01-19 03:00:17] Verifying : Description : Fetching configmap pointer &ConfigMap{ObjectMeta:{openshift-service-ca.crt mysql-ibm-pxb-2-84850-01-19-02h58m26s 3a6fe959-b1a5-4654-bfa4-e36932a60810 1574960 0 2024-01-19 02:58:26 +0000 UTC map[] map[service.beta.openshift.io/inject-cabundle:true] [] [] [{kube-controller-manager Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:data":{},"f:metadata":{"f:annotations":{".":{},"f:service.beta.openshift.io/inject-cabundle":{}}}} } {service-ca-operator Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:data":{"f:service-ca.crt":{}}} }]},Data:map[string]string{service-ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T03:00:17.196Z] MIIDUTCCAjmgAwIBAgIILvRjno8BjvIwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE [2024-01-19T03:00:17.196Z] Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwNTM5NjA4ODAe [2024-01-19T03:00:17.196Z] Fw0yNDAxMTYwOTA4MDhaFw0yNjAzMTYwOTA4MDlaMDYxNDAyBgNVBAMMK29wZW5z [2024-01-19T03:00:17.196Z] aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDUzOTYwODgwggEiMA0GCSqG [2024-01-19T03:00:17.197Z] SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVr8xmLKsMoemVtfnlzzil6B8CGocEhNRo [2024-01-19T03:00:17.197Z] fkzkD8aSOYgIj5zcZo/M0O7nXDXeg9dqx9/h2qrGk+SS2ikU2Xs4Q5gaQDh9U1lB [2024-01-19T03:00:17.197Z] y8paoOEvw06CmJ/1Qqur5A539/OnimRSTdejTu4lwaDwsJiyoGfhvrBJAdjagtYA [2024-01-19T03:00:17.197Z] 0sNMrCgEAT4NWHS5EDx5dF7nMDHmQ0h0gHYei6vUEtiPyLwcTH1T7uVSj22fCEXX [2024-01-19T03:00:17.197Z] LjgN3OSiquJ+Hf4+csPpwgxAEkfjdBldhnImYkunJbg/x4c90uFL1pygZusrbFAf [2024-01-19T03:00:17.197Z] cERW8GngZuL5JY8MQoIstMu+1qS390ouQDDbEcvWjxC8YFNFHywrAgMBAAGjYzBh [2024-01-19T03:00:17.197Z] MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQS5vRk [2024-01-19T03:00:17.197Z] jkBPISX2qYmk1VEQZIXSLzAfBgNVHSMEGDAWgBQS5vRkjkBPISX2qYmk1VEQZIXS [2024-01-19T03:00:17.197Z] LzANBgkqhkiG9w0BAQsFAAOCAQEAZ/qc2vFGJZgDJCEUaPYuufuLRPSF68ixV9fk [2024-01-19T03:00:17.197Z] e5tBkRN8atCsEBygC/RUcwULlhsdgUMGMcREgNC/FYlwkOsAj3gP8KmcXAZ4gHCa [2024-01-19T03:00:17.197Z] T6PIbIcj1hCX0mdX0tvtaj7PWvRUz9CHh6iFzmPsXK5xjaBA9vZbkUAa7odguUit [2024-01-19T03:00:17.197Z] vz1QnKoV1OaZyccDLeQL50khM1XEBn8L+zeY3k85uo2X514HZW4LDPTQ94cpyNnb [2024-01-19T03:00:17.197Z] zqI6utp7JoPJexsHMWNFKGb//HgjluNg7TsywXIeebk+/GZUEDfNKid0oiTpQvzR [2024-01-19T03:00:17.197Z] TFFF+5KO/sa4+qIh1ERm140JxeaurUMaJzn7cFQbekEgt/N+iA== [2024-01-19T03:00:17.197Z] -----END CERTIFICATE----- [2024-01-19T03:00:17.197Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} from namespace mysql-ibm-pxb-2-84850-01-19-02h58m26s [2024-01-19T03:00:17.197Z] INFO[2024-01-19 03:00:17] Actual:true, Expected: true [2024-01-19T03:00:17.197Z] 2024-01-19 03:00:17 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.AddLabelToResource:#5038] - Adding label [edffc0c3-8249-4e9f-a79b-744c405b60af=d095e726-c3cc-4da8-b72f-6be96ae30ea7] to ConfigMap openshift-service-ca.crt [2024-01-19T03:00:17.480Z] INFO[2024-01-19 03:00:17] Verifying : Description : Adding labels edffc0c3-8249-4e9f-a79b-744c405b60af=d095e726-c3cc-4da8-b72f-6be96ae30ea7 to resource &ConfigMap{ObjectMeta:{openshift-service-ca.crt mysql-ibm-pxb-2-84850-01-19-02h58m26s 3a6fe959-b1a5-4654-bfa4-e36932a60810 1574960 0 2024-01-19 02:58:26 +0000 UTC map[edffc0c3-8249-4e9f-a79b-744c405b60af:d095e726-c3cc-4da8-b72f-6be96ae30ea7] map[service.beta.openshift.io/inject-cabundle:true] [] [] [{kube-controller-manager Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:data":{},"f:metadata":{"f:annotations":{".":{},"f:service.beta.openshift.io/inject-cabundle":{}}}} } {service-ca-operator Update v1 2024-01-19 02:58:26 +0000 UTC FieldsV1 {"f:data":{"f:service-ca.crt":{}}} }]},Data:map[string]string{service-ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T03:00:17.480Z] MIIDUTCCAjmgAwIBAgIILvRjno8BjvIwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE [2024-01-19T03:00:17.480Z] Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwNTM5NjA4ODAe [2024-01-19T03:00:17.480Z] Fw0yNDAxMTYwOTA4MDhaFw0yNjAzMTYwOTA4MDlaMDYxNDAyBgNVBAMMK29wZW5z [2024-01-19T03:00:17.480Z] aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDUzOTYwODgwggEiMA0GCSqG [2024-01-19T03:00:17.480Z] SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVr8xmLKsMoemVtfnlzzil6B8CGocEhNRo [2024-01-19T03:00:17.480Z] fkzkD8aSOYgIj5zcZo/M0O7nXDXeg9dqx9/h2qrGk+SS2ikU2Xs4Q5gaQDh9U1lB [2024-01-19T03:00:17.480Z] y8paoOEvw06CmJ/1Qqur5A539/OnimRSTdejTu4lwaDwsJiyoGfhvrBJAdjagtYA [2024-01-19T03:00:17.480Z] 0sNMrCgEAT4NWHS5EDx5dF7nMDHmQ0h0gHYei6vUEtiPyLwcTH1T7uVSj22fCEXX [2024-01-19T03:00:17.480Z] LjgN3OSiquJ+Hf4+csPpwgxAEkfjdBldhnImYkunJbg/x4c90uFL1pygZusrbFAf [2024-01-19T03:00:17.480Z] cERW8GngZuL5JY8MQoIstMu+1qS390ouQDDbEcvWjxC8YFNFHywrAgMBAAGjYzBh [2024-01-19T03:00:17.480Z] MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQS5vRk [2024-01-19T03:00:17.480Z] jkBPISX2qYmk1VEQZIXSLzAfBgNVHSMEGDAWgBQS5vRkjkBPISX2qYmk1VEQZIXS [2024-01-19T03:00:17.480Z] LzANBgkqhkiG9w0BAQsFAAOCAQEAZ/qc2vFGJZgDJCEUaPYuufuLRPSF68ixV9fk [2024-01-19T03:00:17.480Z] e5tBkRN8atCsEBygC/RUcwULlhsdgUMGMcREgNC/FYlwkOsAj3gP8KmcXAZ4gHCa [2024-01-19T03:00:17.480Z] T6PIbIcj1hCX0mdX0tvtaj7PWvRUz9CHh6iFzmPsXK5xjaBA9vZbkUAa7odguUit [2024-01-19T03:00:17.480Z] vz1QnKoV1OaZyccDLeQL50khM1XEBn8L+zeY3k85uo2X514HZW4LDPTQ94cpyNnb [2024-01-19T03:00:17.480Z] zqI6utp7JoPJexsHMWNFKGb//HgjluNg7TsywXIeebk+/GZUEDfNKid0oiTpQvzR [2024-01-19T03:00:17.480Z] TFFF+5KO/sa4+qIh1ERm140JxeaurUMaJzn7cFQbekEgt/N+iA== [2024-01-19T03:00:17.480Z] -----END CERTIFICATE----- [2024-01-19T03:00:17.480Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} [2024-01-19T03:00:17.480Z] INFO[2024-01-19 03:00:17] Actual:true, Expected: true [2024-01-19T03:00:17.480Z] 2024-01-19 03:00:17 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Generating namespace label string from label map [2024-01-19T03:00:17.480Z] 2024-01-19 03:00:17 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [backup.glob..func20.2.4:#515] - Generated labels [0-bea88fde-30c2-4636-8091-31ff5dd72c03=bea88fde-30c2-4636-8091-31ff5dd72c03] [2024-01-19T03:00:17.480Z] 2024-01-19 03:00:17 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Creating cloud credentials and registering backup location [2024-01-19T03:00:18.847Z] 2024-01-19 03:00:18 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-1705633218] for org [default] with [aws] as provider [2024-01-19T03:00:18.847Z] 2024-01-19 03:00:18 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T03:00:18.847Z] 2024-01-19 03:00:18 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-1705633218] is present [2024-01-19T03:00:18.847Z] INFO[2024-01-19 03:00:18] Verifying : Description : Creating cloud credentials autogenerated-cred-1705633218 with provider aws [2024-01-19T03:00:18.847Z] INFO[2024-01-19 03:00:18] Actual:true, Expected: true [2024-01-19T03:00:18.847Z] 2024-01-19 03:00:18 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Created Cloud Credentials with name - autogenerated-cred-1705633218 [2024-01-19T03:00:23.017Z] INFO[2024-01-19 03:00:22] Verifying : Description : Creating backup location autogenerated-backup-location-1705633218 [2024-01-19T03:00:23.017Z] INFO[2024-01-19 03:00:22] Actual:true, Expected: true [2024-01-19T03:00:23.017Z] 2024-01-19 03:00:22 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Adding source and destination clusters with px-central-admin ctx [2024-01-19T03:00:23.017Z] INFO[2024-01-19 03:00:22] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T03:00:23.017Z] INFO[2024-01-19 03:00:22] Actual:true, Expected: true [2024-01-19T03:00:23.017Z] 2024-01-19 03:00:22 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T03:00:23.017Z] 2024-01-19 03:00:22 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T03:00:23.017Z] 2024-01-19 03:00:22 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T03:00:23.017Z] 2024-01-19 03:00:22 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T03:00:23.017Z] 2024-01-19 03:00:22 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Create cluster [source-cluster] in org [default] [2024-01-19T03:00:23.017Z] 2024-01-19 03:00:22 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T03:00:23.017Z] 2024-01-19 03:00:22 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T03:00:23.017Z] 2024-01-19 03:00:22 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Create cluster [destination-cluster] in org [default] [2024-01-19T03:00:23.017Z] 2024-01-19 03:00:22 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T03:00:23.018Z] 2024-01-19 03:00:22 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T03:00:23.018Z] 2024-01-19 03:00:22 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-diplq] for IBM [2024-01-19T03:00:23.018Z] 2024-01-19 03:00:22 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-diplq] for org [default] with [ibm] as provider [2024-01-19T03:00:23.018Z] 2024-01-19 03:00:22 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T03:00:23.018Z] 2024-01-19 03:00:22 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T03:00:23.018Z] 2024-01-19 03:00:22 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-diplq] is present [2024-01-19T03:00:23.018Z] 2024-01-19 03:00:22 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Create cluster [source-cluster] in org [default] [2024-01-19T03:00:23.288Z] 2024-01-19 03:00:23 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-hmmgh] for IBM [2024-01-19T03:00:23.288Z] 2024-01-19 03:00:23 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-hmmgh] for org [default] with [ibm] as provider [2024-01-19T03:00:23.288Z] 2024-01-19 03:00:23 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T03:00:23.288Z] 2024-01-19 03:00:23 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T03:00:23.544Z] 2024-01-19 03:00:23 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-hmmgh] is present [2024-01-19T03:00:23.544Z] 2024-01-19 03:00:23 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Create cluster [destination-cluster] in org [default] [2024-01-19T03:00:24.105Z] INFO[2024-01-19 03:00:23] Verifying : Description : Verifying addition of source [source-cluster] and destination [destination-cluster] clusters [2024-01-19T03:00:24.105Z] INFO[2024-01-19 03:00:23] Actual:true, Expected: true [2024-01-19T03:00:24.105Z] INFO[2024-01-19 03:00:23] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T03:00:24.105Z] INFO[2024-01-19 03:00:23] Actual:Online, Expected: Online [2024-01-19T03:00:24.105Z] INFO[2024-01-19 03:00:23] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T03:00:24.105Z] INFO[2024-01-19 03:00:23] Actual:true, Expected: true [2024-01-19T03:00:24.105Z] 2024-01-19 03:00:23 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] UID of [source-cluster] cluster is 1d6e317c-652c-4538-a3d4-3aa5c45b943e [2024-01-19T03:00:24.105Z] INFO[2024-01-19 03:00:23] Verifying : Description : Verifying if [destination-cluster] cluster is online [2024-01-19T03:00:24.105Z] INFO[2024-01-19 03:00:23] Actual:Online, Expected: Online [2024-01-19T03:00:24.105Z] 2024-01-19 03:00:23 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Taking manual backup of applications with namespace label [2024-01-19T03:00:24.362Z] 2024-01-19 03:00:24 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T03:00:24.362Z] 2024-01-19 03:00:24 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Volume snapshot classes found in the cluster - [2024-01-19T03:00:24.362Z] 2024-01-19 03:00:24 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] ibmc-vpcblock-snapshot [2024-01-19T03:00:24.362Z] 2024-01-19 03:00:24 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T03:00:24.923Z] 2024/01/19 03:00:24 DoRetryWithTimeout - Error: {backup status for [backup-1705633224] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T03:00:56.966Z] 2024/01/19 03:00:54 DoRetryWithTimeout - Error: {backup status for [backup-1705633224] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:01:28.999Z] 2024/01/19 03:01:24 DoRetryWithTimeout - Error: {backup status for [backup-1705633224] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:01:55.495Z] 2024-01-19 03:01:54 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [backup.CreateBackupWithNamespaceLabel:#3030] - Successfully created backup [backup-1705633224] with namespace label [0-bea88fde-30c2-4636-8091-31ff5dd72c03=bea88fde-30c2-4636-8091-31ff5dd72c03] [2024-01-19T03:01:55.495Z] 2024-01-19 03:01:54 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Validating backup [backup-1705633224] in org [default] [2024-01-19T03:01:55.495Z] 2024-01-19 03:01:54 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [backup-1705633224] [2024-01-19T03:01:55.495Z] 2024-01-19 03:01:54 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84850-01-19-02h58m26s] in backup [backup-1705633224] [2024-01-19T03:01:55.495Z] 2024-01-19 03:01:54 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84850-01-19-02h58m26s] in backup [backup-1705633224] [2024-01-19T03:01:55.495Z] 2024-01-19 03:01:54 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84850-01-19-02h58m26s] [2024-01-19T03:01:55.495Z] 2024-01-19 03:01:54 +0000:[WARNING] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:01:55.495Z] 2024-01-19 03:01:54 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:01:55.495Z] 2024-01-19 03:01:54 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-84850-01-19-02h58m26s] are [[mysql-data]] [2024-01-19T03:01:55.495Z] 2024-01-19 03:01:54 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T03:01:55.495Z] 2024-01-19 03:01:54 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84850-01-19-02h58m26s] in backup [backup-1705633224] [2024-01-19T03:01:55.496Z] 2024-01-19 03:01:54 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84850-01-19-02h58m26s] in backup [backup-1705633224] [2024-01-19T03:01:55.496Z] 2024-01-19 03:01:54 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84850-01-19-02h58m26s] [2024-01-19T03:01:55.496Z] 2024-01-19 03:01:54 +0000:[WARNING] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:01:55.496Z] 2024-01-19 03:01:54 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:01:55.496Z] 2024-01-19 03:01:54 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-84850-01-19-02h58m26s] are [[mysql-data]] [2024-01-19T03:01:55.496Z] 2024-01-19 03:01:54 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T03:01:55.496Z] 2024-01-19 03:01:54 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84850-01-19-02h58m26s] in backup [backup-1705633224] [2024-01-19T03:01:55.496Z] 2024-01-19 03:01:54 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84850-01-19-02h58m26s] in backup [backup-1705633224] [2024-01-19T03:01:55.496Z] 2024-01-19 03:01:54 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84850-01-19-02h58m26s] [2024-01-19T03:01:55.496Z] 2024-01-19 03:01:54 +0000:[WARNING] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:01:55.496Z] 2024-01-19 03:01:54 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:01:55.496Z] 2024-01-19 03:01:54 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-2-84850-01-19-02h58m26s] are [[mysql-data]] [2024-01-19T03:01:55.496Z] 2024-01-19 03:01:54 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T03:01:55.496Z] 2024-01-19 03:01:54 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Validating CR cleanup [2024-01-19T03:01:55.496Z] 2024-01-19 03:01:55 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T03:01:55.496Z] 2024-01-19 03:01:55 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] All CRs in [kube-system] are [[backup-1705633224-c969c5d tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133]] [2024-01-19T03:01:55.496Z] 2024-01-19 03:01:55 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] CR found for [[backup-1705633224-c969c5d tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133]] under [kube-system] namespace [2024-01-19T03:01:55.497Z] 2024/01/19 03:01:55 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [backup-1705633224]}, Next try in [5s], timeout [5m0s] [2024-01-19T03:02:00.744Z] 2024-01-19 03:02:00 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133]] [2024-01-19T03:02:00.744Z] INFO[2024-01-19 03:02:00] Verifying : Description : Verifying backup [backup-1705633224] creation with labels [0-bea88fde-30c2-4636-8091-31ff5dd72c03=bea88fde-30c2-4636-8091-31ff5dd72c03] [2024-01-19T03:02:00.744Z] INFO[2024-01-19 03:02:00] Actual:true, Expected: true [2024-01-19T03:02:00.744Z] 2024-01-19 03:02:00 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Restoring manual backup [2024-01-19T03:02:01.000Z] 2024/01/19 03:02:00 DoRetryWithTimeout - Error: {restore status for [tp-restore-backup-1705633224] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T03:02:33.028Z] 2024-01-19 03:02:30 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [backup.CreateRestore:#936] - Restore [tp-restore-backup-1705633224] created successfully [2024-01-19T03:02:33.028Z] 2024-01-19 03:02:30 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Switching cluster context to cluster [source-cluster] [2024-01-19T03:02:33.028Z] 2024-01-19 03:02:30 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Switching context to [/tmp/source-config] [2024-01-19T03:02:33.028Z] 2024-01-19 03:02:30 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T03:02:33.028Z] 2024-01-19 03:02:30 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T03:02:33.028Z] 2024-01-19 03:02:31 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T03:02:33.028Z] 2024-01-19 03:02:31 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T03:02:33.028Z] 2024-01-19 03:02:31 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T03:02:33.028Z] 2024-01-19 03:02:31 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T03:02:33.028Z] 2024-01-19 03:02:31 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T03:02:33.028Z] 2024-01-19 03:02:32 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T03:02:33.028Z] 2024-01-19 03:02:32 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T03:02:33.028Z] 2024/01/19 03:02:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:02:42.974Z] 2024/01/19 03:02:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:02:53.713Z] 2024/01/19 03:02:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:03:03.660Z] 2024/01/19 03:03:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:03:13.610Z] 2024/01/19 03:03:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:03:23.607Z] 2024/01/19 03:03:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:03:33.549Z] 2024/01/19 03:03:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:03:43.499Z] 2024/01/19 03:03:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:03:53.440Z] 2024/01/19 03:03:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:04:03.382Z] 2024/01/19 03:04:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:04:13.341Z] 2024/01/19 03:04:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:04:23.319Z] 2024/01/19 03:04:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:04:33.274Z] 2024/01/19 03:04:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:04:43.227Z] 2024/01/19 03:04:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:04:49.761Z] INFO[2024-01-19 03:04:48] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T03:04:49.761Z] INFO[2024-01-19 03:04:48] sched stats: tick interval: 41448 < 1.2s (100%) [2024-01-19T03:04:49.761Z] INFO[2024-01-19 03:04:48] sched stats: tick process durations: 41447 < 500ms (100%) [2024-01-19T03:04:53.968Z] 2024/01/19 03:04:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:05:03.908Z] 2024/01/19 03:05:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:05:13.855Z] 2024/01/19 03:05:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:05:23.794Z] 2024/01/19 03:05:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:05:33.737Z] 2024/01/19 03:05:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:05:43.673Z] 2024/01/19 03:05:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:05:53.618Z] 2024/01/19 03:05:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:06:03.557Z] 2024/01/19 03:06:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:06:13.503Z] 2024/01/19 03:06:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:06:23.445Z] 2024/01/19 03:06:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:06:33.404Z] 2024/01/19 03:06:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:06:45.583Z] 2024/01/19 03:06:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:06:53.674Z] 2024/01/19 03:06:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:07:03.627Z] 2024/01/19 03:07:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:07:13.573Z] 2024/01/19 03:07:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:07:25.744Z] 2024/01/19 03:07:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:07:33.829Z] INFO[2024-01-19 03:07:33] Verifying : Description : Verifying restoration of backup tp-restore-backup-1705633224 [2024-01-19T03:07:33.829Z] ERRO[2024-01-19 03:07:33] failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.830Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.830Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.830Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.830Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.830Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.830Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.830Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.830Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:07:33.830Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T03:07:33.830Z] INFO[2024-01-19 03:07:33] Verifying : Description : Verification of deleting backup schedule - [2024-01-19T03:07:33.830Z] ERRO[2024-01-19 03:07:33] backup schedule with name '' not found for org 'default' [2024-01-19T03:07:34.760Z] INFO[2024-01-19 03:07:34] Verifying : Description : Deleting backup schedule policies [] [2024-01-19T03:07:34.760Z] ERRO[2024-01-19 03:07:34] Failed to delete schedule policy with error [Unable to find schedule policy Uid] [2024-01-19T03:07:34.760Z] 2024-01-19 03:07:34 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Deleting labels from namespaces - [mysql-ibm-pxb-0-84850-01-19-02h58m26s mysql-ibm-pxb-1-84850-01-19-02h58m26s mysql-ibm-pxb-2-84850-01-19-02h58m26s] [2024-01-19T03:07:35.690Z] INFO[2024-01-19 03:07:35] Verifying : Description : Deleting labels [map[0-bea88fde-30c2-4636-8091-31ff5dd72c03:bea88fde-30c2-4636-8091-31ff5dd72c03]] from namespaces [[mysql-ibm-pxb-0-84850-01-19-02h58m26s mysql-ibm-pxb-1-84850-01-19-02h58m26s mysql-ibm-pxb-2-84850-01-19-02h58m26s]] [2024-01-19T03:07:35.690Z] INFO[2024-01-19 03:07:35] Actual:true, Expected: true [2024-01-19T03:07:35.690Z] 2024-01-19 03:07:35 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Deleting deployed namespaces - [mysql-ibm-pxb-0-84850-01-19-02h58m26s mysql-ibm-pxb-1-84850-01-19-02h58m26s mysql-ibm-pxb-2-84850-01-19-02h58m26s] [2024-01-19T03:07:35.690Z] 2024-01-19 03:07:35 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Destroying apps [2024-01-19T03:07:35.690Z] 2024-01-19 03:07:35 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T03:07:36.253Z] 2024-01-19 03:07:35 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T03:07:36.253Z] 2024-01-19 03:07:35 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] start destroying mysql-ibm app [2024-01-19T03:07:36.253Z] 2024-01-19 03:07:36 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T03:07:36.253Z] 2024-01-19 03:07:36 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:07:36.253Z] 2024-01-19 03:07:36 +0000:[WARNING] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T03:07:36.254Z] INFO[2024-01-19 03:07:36] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T03:07:36.254Z] INFO[2024-01-19 03:07:36] Actual:true, Expected: true [2024-01-19T03:07:36.254Z] 2024-01-19 03:07:36 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T03:07:36.817Z] 2024-01-19 03:07:36 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T03:07:36.817Z] 2024-01-19 03:07:36 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] start destroying mysql-ibm app [2024-01-19T03:07:36.817Z] 2024-01-19 03:07:36 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T03:07:37.077Z] 2024-01-19 03:07:36 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:07:37.077Z] 2024-01-19 03:07:36 +0000:[WARNING] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T03:07:37.078Z] INFO[2024-01-19 03:07:36] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T03:07:37.078Z] INFO[2024-01-19 03:07:36] Actual:true, Expected: true [2024-01-19T03:07:37.078Z] 2024-01-19 03:07:36 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T03:07:37.334Z] 2024-01-19 03:07:37 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T03:07:37.334Z] 2024-01-19 03:07:37 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] start destroying mysql-ibm app [2024-01-19T03:07:37.590Z] 2024-01-19 03:07:37 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T03:07:37.590Z] 2024-01-19 03:07:37 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:07:37.590Z] 2024-01-19 03:07:37 +0000:[WARNING] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T03:07:37.590Z] INFO[2024-01-19 03:07:37] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T03:07:37.590Z] INFO[2024-01-19 03:07:37] Actual:true, Expected: true [2024-01-19T03:07:37.590Z] 2024-01-19 03:07:37 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Cleaning backup locations in map [map[7bd3597a-08ca-4975-9ac2-2509ef709597:autogenerated-backup-location-1705633218]], cloud credential [autogenerated-cred-1705633218], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T03:07:37.590Z] INFO[2024-01-19 03:07:37] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-1705633218] [2024-01-19T03:07:37.590Z] INFO[2024-01-19 03:07:37] Actual:true, Expected: true [2024-01-19T03:07:37.850Z] 2024-01-19 03:07:37 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705633218] is present [2024-01-19T03:07:37.851Z] 2024/01/19 03:07:37 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705633218 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T03:08:09.881Z] 2024-01-19 03:08:07 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705633218] is present [2024-01-19T03:08:09.881Z] 2024/01/19 03:08:07 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705633218 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T03:08:41.917Z] 2024-01-19 03:08:37 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705633218] is present [2024-01-19T03:08:41.917Z] 2024/01/19 03:08:37 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705633218 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T03:09:08.456Z] 2024-01-19 03:09:07 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705633218] is present [2024-01-19T03:09:08.456Z] 2024/01/19 03:09:07 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705633218 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T03:09:40.489Z] 2024-01-19 03:09:38 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 aws-global-aws-px-backup-system-test-4cac76dc-bl autogenerated-backup-location-1705628002] [2024-01-19T03:09:40.489Z] INFO[2024-01-19 03:09:38] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-1705633218 [2024-01-19T03:09:40.489Z] INFO[2024-01-19 03:09:38] Actual:true, Expected: true [2024-01-19T03:09:40.489Z] 2024-01-19 03:09:38 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-1705633218] is present [2024-01-19T03:09:40.489Z] INFO[2024-01-19 03:09:38] Verifying : Description : Verifying if cloud cred [autogenerated-cred-1705633218] is present [2024-01-19T03:09:40.489Z] INFO[2024-01-19 03:09:38] Actual:true, Expected: true [2024-01-19T03:09:40.489Z] INFO[2024-01-19 03:09:38] Verifying : Description : Verifying deletion of cloud cred [autogenerated-cred-1705633218] [2024-01-19T03:09:40.489Z] INFO[2024-01-19 03:09:38] Actual:true, Expected: true [2024-01-19T03:09:40.489Z] INFO[2024-01-19 03:09:38] Verifying : Description : Deleting cloud cred autogenerated-cred-1705633218 [2024-01-19T03:09:40.489Z] INFO[2024-01-19 03:09:38] Actual:true, Expected: true [2024-01-19T03:09:40.489Z] INFO[2024-01-19 03:09:38] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T03:09:40.489Z] INFO[2024-01-19 03:09:38] Actual:true, Expected: true [2024-01-19T03:09:40.489Z] INFO[2024-01-19 03:09:38] Verifying : Description : Deleting cluster source-cluster [2024-01-19T03:09:40.489Z] INFO[2024-01-19 03:09:38] Actual:true, Expected: true [2024-01-19T03:09:40.489Z] INFO[2024-01-19 03:09:38] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-diplq] [2024-01-19T03:09:40.489Z] INFO[2024-01-19 03:09:38] Actual:true, Expected: true [2024-01-19T03:09:40.489Z] INFO[2024-01-19 03:09:38] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T03:09:40.489Z] INFO[2024-01-19 03:09:38] Actual:true, Expected: true [2024-01-19T03:09:40.489Z] INFO[2024-01-19 03:09:39] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-hmmgh] [2024-01-19T03:09:40.489Z] INFO[2024-01-19 03:09:39] Actual:true, Expected: true [2024-01-19T03:09:40.489Z] INFO[2024-01-19 03:09:39] --------Test End------ [2024-01-19T03:09:40.489Z] INFO[2024-01-19 03:09:39] #Test: [2024-01-19T03:09:40.490Z] INFO[2024-01-19 03:09:39] #Description: [2024-01-19T03:09:40.490Z] INFO[2024-01-19 03:09:39] ------------------------ [2024-01-19T03:09:40.490Z] 2024-01-19 03:09:39 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.EndPxBackupTorpedoTest:#7261] - >>>> FAILED TEST: {ManualAndScheduledBackupUsingNamespaceAndResourceLabel} Manual, schedule backup and restore using namespace label and resource label [2024-01-19T03:09:40.490Z] 2024-01-19 03:09:39 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-84850-01-19-02h58m26s] [2024-01-19T03:09:40.490Z] 2024-01-19 03:09:39 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-1-84850-01-19-02h58m26s] [2024-01-19T03:09:40.490Z] 2024-01-19 03:09:39 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-2-84850-01-19-02h58m26s] [2024-01-19T03:09:40.490Z] 2024-01-19 03:09:40 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T03:09:40.490Z] 2024-01-19 03:09:40 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T03:09:40.490Z] 2024-01-19 03:09:40 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-84850-01-19-02h58m26s] [2024-01-19T03:09:40.490Z] 2024-01-19 03:09:40 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-1-84850-01-19-02h58m26s] [2024-01-19T03:09:40.747Z] 2024-01-19 03:09:40 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-2-84850-01-19-02h58m26s] [2024-01-19T03:09:41.003Z] 2024-01-19 03:09:40 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {ManualAndScheduledBackupUsingNamespaceAndResourceLabel} Manual, schedule backup and restore using namespace label and resource label [2024-01-19T03:09:41.003Z] 2024-01-19 03:09:40 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T03:09:41.003Z] 2024/01/19 03:09:40 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:09:46.269Z] 2024/01/19 03:09:45 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:09:51.517Z] 2024/01/19 03:09:50 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:09:56.767Z] 2024/01/19 03:09:55 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:10:00.934Z] 2024/01/19 03:10:00 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:10:06.178Z] 2024/01/19 03:10:05 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:10:11.417Z] 2024/01/19 03:10:10 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:10:16.663Z] 2024/01/19 03:10:15 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:10:21.905Z] 2024/01/19 03:10:20 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.64.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.8 10.240.64.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T03:10:21.905Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T03:10:21.905Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T03:10:21.905Z] {dial tcp 10.240.64.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T03:10:26.069Z] 2024-01-19 03:10:25 +0000:[WARNING] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.64.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.8 10.240.64.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T03:10:26.069Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T03:10:26.069Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T03:10:26.069Z] {dial tcp 10.240.64.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T03:10:26.069Z] 2024-01-19 03:10:25 +0000:[ERROR] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.64.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.8 10.240.64.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T03:10:26.069Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T03:10:26.069Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T03:10:26.070Z] {dial tcp 10.240.64.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T03:10:26.070Z] 2024-01-19 03:10:25 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T03:10:26.070Z] 2024-01-19 03:10:25 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] Switching context to [/tmp/source-config] [2024-01-19T03:10:26.070Z] 2024-01-19 03:10:25 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T03:10:26.070Z] 2024-01-19 03:10:25 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T03:10:26.070Z] 2024-01-19 03:10:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T03:10:26.337Z] 2024-01-19 03:10:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T03:10:26.337Z] 2024-01-19 03:10:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T03:10:26.337Z] 2024-01-19 03:10:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T03:10:26.901Z] 2024-01-19 03:10:26 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T03:10:27.156Z] 2024-01-19 03:10:27 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T03:10:27.716Z] 2024-01-19 03:10:27 +0000:[INFO] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T03:10:27.716Z] 2024/01/19 03:10:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:10:39.892Z] 2024/01/19 03:10:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:10:47.969Z] 2024/01/19 03:10:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:10:57.910Z] 2024/01/19 03:10:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:11:07.854Z] 2024/01/19 03:11:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:11:17.840Z] 2024/01/19 03:11:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:11:30.010Z] 2024/01/19 03:11:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:11:38.101Z] 2024/01/19 03:11:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:11:49.226Z] 2024/01/19 03:11:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:11:59.175Z] 2024/01/19 03:11:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:12:09.118Z] 2024/01/19 03:12:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:12:19.056Z] 2024/01/19 03:12:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:12:28.996Z] 2024/01/19 03:12:28 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:12:38.938Z] 2024/01/19 03:12:38 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:12:48.880Z] 2024/01/19 03:12:48 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:12:58.820Z] 2024/01/19 03:12:58 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:13:08.772Z] 2024/01/19 03:13:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:13:18.724Z] 2024/01/19 03:13:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:13:28.672Z] 2024/01/19 03:13:28 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:13:38.611Z] 2024/01/19 03:13:38 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:13:48.553Z] 2024/01/19 03:13:48 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:13:58.494Z] 2024/01/19 03:13:58 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:14:08.440Z] 2024/01/19 03:14:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:14:18.389Z] 2024/01/19 03:14:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:14:30.557Z] 2024/01/19 03:14:28 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:14:38.672Z] 2024/01/19 03:14:38 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:14:48.610Z] 2024/01/19 03:14:48 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:14:49.973Z] INFO[2024-01-19 03:14:49] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T03:14:49.973Z] INFO[2024-01-19 03:14:49] sched stats: tick interval: 42049 < 1.2s (100%) [2024-01-19T03:14:49.973Z] INFO[2024-01-19 03:14:49] sched stats: tick process durations: 42048 < 500ms (100%) [2024-01-19T03:14:59.912Z] 2024/01/19 03:14:58 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:15:09.854Z] 2024/01/19 03:15:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:15:19.791Z] 2024/01/19 03:15:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:15:29.766Z] 2024-01-19 03:15:28 +0000:[ERROR] [{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.766Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.766Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.766Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.766Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.766Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.767Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.767Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.767Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.767Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.767Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.767Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.767Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.767Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.767Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.767Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.767Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.767Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.767Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.767Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.767Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.767Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.767Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.767Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.767Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.768Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.768Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.768Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.768Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.768Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T03:15:29.768Z] [2024-01-19T03:15:29.768Z] ------------------------------ [2024-01-19T03:15:29.768Z] • Failure [1022.287 seconds] [2024-01-19T03:15:29.768Z] {ManualAndScheduledBackupUsingNamespaceAndResourceLabel} [2024-01-19T03:15:29.768Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_namespace_labelled_test.go:436 [2024-01-19T03:15:29.768Z] Manual, schedule backup and restore using namespace label and resource label [It] [2024-01-19T03:15:29.768Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_namespace_labelled_test.go:477 [2024-01-19T03:15:29.768Z] [2024-01-19T03:15:29.768Z] Unexpected error: [2024-01-19T03:15:29.768Z] <*errors.errorString | 0xc001423e80>: { [2024-01-19T03:15:29.769Z] s: "error: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknow... [2024-01-19T03:15:29.769Z] [2024-01-19T03:15:29.769Z] Gomega truncated this representation as it exceeds 'format.MaxLength'. [2024-01-19T03:15:29.769Z] Consider having the object provide a custom 'GomegaStringer' representation [2024-01-19T03:15:29.769Z] or adjust the parameters in Gomega's 'format' package. [2024-01-19T03:15:29.769Z] [2024-01-19T03:15:29.769Z] Learn more here: https://onsi.github.io/gomega/#adjusting-output [2024-01-19T03:15:29.769Z] [2024-01-19T03:15:29.769Z] error: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.769Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.769Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.769Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.769Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.769Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.769Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.769Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.769Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.770Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.770Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.770Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.770Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.770Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.770Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.770Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.770Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.770Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.770Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.770Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.770Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.770Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.770Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.770Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.770Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.770Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.770Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.770Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.770Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:15:29.771Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}], Description: Verifying restoration of backup tp-restore-backup-1705633224 [2024-01-19T03:15:29.771Z] occurred [2024-01-19T03:15:29.771Z] [2024-01-19T03:15:29.771Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-19T03:15:29.771Z] [2024-01-19T03:15:29.771Z] Full Stack Trace [2024-01-19T03:15:29.771Z] github.com/portworx/torpedo/pkg/aetosutil.(*Dashboard).VerifyFatal(0x7f3c0d1?, {0x702af20?, 0xc001423d80}, {0x0?, 0x0}, {0xc000d9f380, 0x3c}) [2024-01-19T03:15:29.771Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 +0x143 [2024-01-19T03:15:29.771Z] github.com/portworx/torpedo/tests/backup.glob..func20.2.8() [2024-01-19T03:15:29.771Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_namespace_labelled_test.go:568 +0x3ff [2024-01-19T03:15:29.771Z] github.com/portworx/torpedo/tests/backup.glob..func20.2() [2024-01-19T03:15:29.771Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_namespace_labelled_test.go:561 +0x809 [2024-01-19T03:15:29.771Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T03:15:29.771Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T03:15:29.771Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T03:15:29.771Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T03:15:29.771Z] created by testing.(*T).Run [2024-01-19T03:15:29.771Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T03:15:29.771Z] ------------------------------ [2024-01-19T03:15:29.771Z] STEP: schedule applications [2024-01-19T03:15:29.771Z] STEP: schedule applications [2024-01-19T03:15:29.771Z] STEP: schedule applications [2024-01-19T03:15:29.771Z] STEP: Validate applications [2024-01-19T03:15:29.771Z] STEP: validate applications [2024-01-19T03:15:29.771Z] STEP: validate mysql-ibm app's volumes [2024-01-19T03:15:29.771Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T03:15:29.771Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T03:15:29.771Z] STEP: get mysql-ibm app's volume: pvc-3d460a06-3366-4263-b3dd-53df850770d3 inspected by the volume driver [2024-01-19T03:15:29.771Z] STEP: wait for mysql-ibm app to start running [2024-01-19T03:15:29.771Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T03:15:29.771Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T03:15:29.771Z] STEP: Validate Px pod restart count [2024-01-19T03:15:29.771Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T03:15:29.771Z] STEP: validate mysql-ibm app's volumes [2024-01-19T03:15:29.771Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T03:15:29.771Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T03:15:29.771Z] STEP: get mysql-ibm app's volume: pvc-6122a103-2fe7-4e98-a869-50439b87761b inspected by the volume driver [2024-01-19T03:15:29.771Z] STEP: wait for mysql-ibm app to start running [2024-01-19T03:15:29.771Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T03:15:29.771Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T03:15:29.771Z] STEP: Validate Px pod restart count [2024-01-19T03:15:29.771Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T03:15:29.771Z] STEP: validate mysql-ibm app's volumes [2024-01-19T03:15:29.771Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T03:15:29.771Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T03:15:29.771Z] STEP: get mysql-ibm app's volume: pvc-fd756974-c524-4313-b0a4-e5c874272060 inspected by the volume driver [2024-01-19T03:15:29.771Z] STEP: wait for mysql-ibm app to start running [2024-01-19T03:15:29.771Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T03:15:29.771Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T03:15:29.771Z] STEP: Validate Px pod restart count [2024-01-19T03:15:29.771Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T03:15:29.771Z] STEP: Generate and add labels to namespaces [2024-01-19T03:15:29.771Z] STEP: Adding labels to resources [2024-01-19T03:15:29.771Z] STEP: Generating namespace label string from label map [2024-01-19T03:15:29.771Z] STEP: Creating cloud credentials and backup location [2024-01-19T03:15:29.771Z] STEP: Add source and destination clusters with px-central-admin ctx [2024-01-19T03:15:29.771Z] STEP: Taking manual backup of applications with namespace label [2024-01-19T03:15:29.771Z] STEP: Restoring manual backup [2024-01-19T03:15:29.771Z] STEP: destroy apps [2024-01-19T03:15:29.771Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T03:15:29.771Z] STEP: start destroying mysql-ibm app [2024-01-19T03:15:29.771Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:15:29.772Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T03:15:29.772Z] STEP: start destroying mysql-ibm app [2024-01-19T03:15:29.772Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:15:29.772Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T03:15:29.772Z] STEP: start destroying mysql-ibm app [2024-01-19T03:15:29.772Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:15:29.772Z] 2024-01-19 03:15:28 +0000:[INFO] [{ShareBackupAndEdit}] Log Dir: /testresults//ShareBackupAndEdit.log [2024-01-19T03:15:29.772Z] INFO[2024-01-19 03:15:28] --------Test Start------ [2024-01-19T03:15:29.772Z] INFO[2024-01-19 03:15:28] #Test: ShareBackupAndEdit [2024-01-19T03:15:29.772Z] INFO[2024-01-19 03:15:28] #Description: Share backup with restore and full access mode and edit the shared backup [2024-01-19T03:15:29.772Z] INFO[2024-01-19 03:15:28] ------------------------ [2024-01-19T03:15:29.772Z] 2024-01-19 03:15:28 +0000:[INFO] [{ShareBackupAndEdit}] Deploy applications [2024-01-19T03:15:29.772Z] 2024-01-19 03:15:28 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T03:15:29.772Z] 2024-01-19 03:15:28 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T03:15:29.772Z] 2024-01-19 03:15:28 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T03:15:29.772Z] 2024-01-19 03:15:28 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T03:15:29.772Z] 2024-01-19 03:15:28 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T03:15:29.772Z] 2024-01-19 03:15:28 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T03:15:29.773Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T03:15:29.773Z] 2024-01-19 03:15:28 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T03:15:29.773Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T03:15:29.773Z] 2024-01-19 03:15:28 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T03:15:29.773Z] 2024-01-19 03:15:28 +0000:[INFO] [{ShareBackupAndEdit}] Validate applications and get their labels [2024-01-19T03:15:29.773Z] 2024-01-19 03:15:28 +0000:[INFO] [{ShareBackupAndEdit}] Validate applications [2024-01-19T03:15:29.773Z] 2024-01-19 03:15:28 +0000:[INFO] [{ShareBackupAndEdit}] Validating mysql-ibm app [2024-01-19T03:15:29.773Z] 2024-01-19 03:15:28 +0000:[INFO] [{ShareBackupAndEdit}] Validating mysql-ibm app's volumes [2024-01-19T03:15:29.773Z] 2024-01-19 03:15:28 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:15:29.773Z] 2024/01/19 03:15:28 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T03:15:39.717Z] 2024/01/19 03:15:38 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T03:15:49.663Z] 2024-01-19 03:15:48 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-82950-01-19-03h15m28s [2024-01-19T03:15:49.663Z] 2024-01-19 03:15:48 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T03:15:49.663Z] 2024-01-19 03:15:48 +0000:[WARNING] [{ShareBackupAndEdit}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T03:15:49.663Z] 2024-01-19 03:15:48 +0000:[INFO] [{ShareBackupAndEdit}] wait for mysql-ibm app to start running [2024-01-19T03:15:49.663Z] 2024-01-19 03:15:48 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T03:15:49.663Z] 2024/01/19 03:15:49 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:15:49.663Z] pod name:mysql-7d766cd65c-zndbd namespace:mysql-ibm-pxb-0-82950-01-19-03h15m28s running:false ready:false node:10.240.64.8 [2024-01-19T03:15:49.663Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:15:59.633Z] 2024/01/19 03:15:59 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:15:59.633Z] pod name:mysql-7d766cd65c-zndbd namespace:mysql-ibm-pxb-0-82950-01-19-03h15m28s running:true ready:false node:10.240.64.8 [2024-01-19T03:15:59.633Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:16:09.579Z] 2024/01/19 03:16:09 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:16:09.579Z] pod name:mysql-7d766cd65c-zndbd namespace:mysql-ibm-pxb-0-82950-01-19-03h15m28s running:true ready:false node:10.240.64.8 [2024-01-19T03:16:09.579Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:16:19.533Z] 2024/01/19 03:16:19 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:16:19.533Z] pod name:mysql-7d766cd65c-zndbd namespace:mysql-ibm-pxb-0-82950-01-19-03h15m28s running:true ready:false node:10.240.64.8 [2024-01-19T03:16:19.533Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:16:31.715Z] 2024/01/19 03:16:29 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:16:31.715Z] pod name:mysql-7d766cd65c-zndbd namespace:mysql-ibm-pxb-0-82950-01-19-03h15m28s running:true ready:false node:10.240.64.8 [2024-01-19T03:16:31.715Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:16:39.799Z] 2024/01/19 03:16:39 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:16:39.799Z] pod name:mysql-7d766cd65c-zndbd namespace:mysql-ibm-pxb-0-82950-01-19-03h15m28s running:true ready:false node:10.240.64.8 [2024-01-19T03:16:39.799Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:16:51.968Z] 2024/01/19 03:16:49 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:16:51.968Z] pod name:mysql-7d766cd65c-zndbd namespace:mysql-ibm-pxb-0-82950-01-19-03h15m28s running:true ready:false node:10.240.64.8 [2024-01-19T03:16:51.968Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:17:00.054Z] 2024-01-19 03:16:59 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T03:17:00.054Z] 2024-01-19 03:16:59 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T03:17:00.311Z] 2024-01-19 03:17:00 +0000:[INFO] [{ShareBackupAndEdit}] validate if mysql-ibm app's volumes are setup [2024-01-19T03:17:00.311Z] 2024-01-19 03:17:00 +0000:[WARNING] [{ShareBackupAndEdit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:17:00.311Z] 2024-01-19 03:17:00 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:17:00.311Z] 2024-01-19 03:17:00 +0000:[INFO] [{ShareBackupAndEdit}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T03:17:00.311Z] 2024-01-19 03:17:00 +0000:[WARNING] [{ShareBackupAndEdit}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T03:17:00.311Z] 2024-01-19 03:17:00 +0000:[INFO] [{ShareBackupAndEdit}] [backup.glob..func86.2.1:#1825] - Create list of pod selector for the apps deployed [2024-01-19T03:17:00.311Z] 2024-01-19 03:17:00 +0000:[INFO] [{ShareBackupAndEdit}] Creating 2 users [2024-01-19T03:17:00.569Z] 2024-01-19 03:17:00 +0000:[INFO] [{ShareBackupAndEdit}] [backup.AddUser:#728] - Response for user [testuser2] creation - [2024-01-19T03:17:00.569Z] 2024-01-19 03:17:00 +0000:[INFO] [{ShareBackupAndEdit}] [backup.AddUser:#733] - User [testuser2] created [2024-01-19T03:17:00.569Z] 2024-01-19 03:17:00 +0000:[INFO] [{ShareBackupAndEdit}] [backup.AddUser:#728] - Response for user [testuser1] creation - [2024-01-19T03:17:00.569Z] 2024-01-19 03:17:00 +0000:[INFO] [{ShareBackupAndEdit}] [backup.AddUser:#733] - User [testuser1] created [2024-01-19T03:17:00.569Z] 2024-01-19 03:17:00 +0000:[INFO] [{ShareBackupAndEdit}] Creating cloud credentials and backup location [2024-01-19T03:17:01.933Z] 2024-01-19 03:17:01 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-1705634221] for org [default] with [aws] as provider [2024-01-19T03:17:01.933Z] 2024-01-19 03:17:01 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T03:17:01.933Z] 2024-01-19 03:17:01 +0000:[INFO] [{ShareBackupAndEdit}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-1705634221] is present [2024-01-19T03:17:01.933Z] INFO[2024-01-19 03:17:01] Verifying : Description : Verifying creation of cloud credential named [autogenerated-cred-1705634221] for org [default] with [aws] as provider [2024-01-19T03:17:01.933Z] INFO[2024-01-19 03:17:01] Actual:true, Expected: true [2024-01-19T03:17:01.933Z] 2024-01-19 03:17:01 +0000:[INFO] [{ShareBackupAndEdit}] Created Cloud Credentials with name - autogenerated-cred-1705634221 [2024-01-19T03:17:05.201Z] INFO[2024-01-19 03:17:04] Verifying : Description : Creating backup location autogenerated-backup-location-1705634221 [2024-01-19T03:17:05.201Z] INFO[2024-01-19 03:17:04] Actual:true, Expected: true [2024-01-19T03:17:05.201Z] 2024-01-19 03:17:04 +0000:[INFO] [{ShareBackupAndEdit}] Created Backup Location with name - autogenerated-backup-location-1705634221 [2024-01-19T03:17:05.201Z] 2024-01-19 03:17:04 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-1705634224] for org [default] with [aws] as provider [2024-01-19T03:17:05.201Z] 2024-01-19 03:17:04 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T03:17:05.201Z] 2024-01-19 03:17:05 +0000:[INFO] [{ShareBackupAndEdit}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-1705634224] is present [2024-01-19T03:17:05.201Z] INFO[2024-01-19 03:17:05] Verifying : Description : Verifying creation of cloud credential named [autogenerated-cred-1705634224] for org [default] with [aws] as provider [2024-01-19T03:17:05.201Z] INFO[2024-01-19 03:17:05] Actual:true, Expected: true [2024-01-19T03:17:05.202Z] 2024-01-19 03:17:05 +0000:[INFO] [{ShareBackupAndEdit}] Created Cloud Credentials with name - autogenerated-cred-1705634224 [2024-01-19T03:17:05.202Z] 2024-01-19 03:17:05 +0000:[INFO] [{ShareBackupAndEdit}] Registering Source and Destination clusters and verifying the status [2024-01-19T03:17:05.458Z] INFO[2024-01-19 03:17:05] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T03:17:05.458Z] INFO[2024-01-19 03:17:05] Actual:true, Expected: true [2024-01-19T03:17:05.458Z] 2024-01-19 03:17:05 +0000:[INFO] [{ShareBackupAndEdit}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T03:17:05.458Z] 2024-01-19 03:17:05 +0000:[INFO] [{ShareBackupAndEdit}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T03:17:05.458Z] 2024-01-19 03:17:05 +0000:[INFO] [{ShareBackupAndEdit}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T03:17:05.458Z] 2024-01-19 03:17:05 +0000:[INFO] [{ShareBackupAndEdit}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T03:17:05.458Z] 2024-01-19 03:17:05 +0000:[INFO] [{ShareBackupAndEdit}] Create cluster [source-cluster] in org [default] [2024-01-19T03:17:05.458Z] 2024-01-19 03:17:05 +0000:[INFO] [{ShareBackupAndEdit}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T03:17:05.458Z] 2024-01-19 03:17:05 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T03:17:05.458Z] 2024-01-19 03:17:05 +0000:[INFO] [{ShareBackupAndEdit}] Create cluster [destination-cluster] in org [default] [2024-01-19T03:17:05.459Z] 2024-01-19 03:17:05 +0000:[INFO] [{ShareBackupAndEdit}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T03:17:05.459Z] 2024-01-19 03:17:05 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T03:17:05.459Z] 2024-01-19 03:17:05 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-lkeso] for IBM [2024-01-19T03:17:05.459Z] 2024-01-19 03:17:05 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-lkeso] for org [default] with [ibm] as provider [2024-01-19T03:17:05.459Z] 2024-01-19 03:17:05 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T03:17:05.459Z] 2024-01-19 03:17:05 +0000:[INFO] [{ShareBackupAndEdit}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T03:17:05.714Z] 2024-01-19 03:17:05 +0000:[INFO] [{ShareBackupAndEdit}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-lkeso] is present [2024-01-19T03:17:05.714Z] 2024-01-19 03:17:05 +0000:[INFO] [{ShareBackupAndEdit}] Create cluster [source-cluster] in org [default] [2024-01-19T03:17:06.277Z] 2024-01-19 03:17:06 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-mgglf] for IBM [2024-01-19T03:17:06.277Z] 2024-01-19 03:17:06 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-mgglf] for org [default] with [ibm] as provider [2024-01-19T03:17:06.277Z] 2024-01-19 03:17:06 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T03:17:06.277Z] 2024-01-19 03:17:06 +0000:[INFO] [{ShareBackupAndEdit}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T03:17:06.277Z] 2024-01-19 03:17:06 +0000:[INFO] [{ShareBackupAndEdit}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-mgglf] is present [2024-01-19T03:17:06.277Z] 2024-01-19 03:17:06 +0000:[INFO] [{ShareBackupAndEdit}] Create cluster [destination-cluster] in org [default] [2024-01-19T03:17:06.844Z] INFO[2024-01-19 03:17:06] Verifying : Description : Creating source and destination cluster [2024-01-19T03:17:06.844Z] INFO[2024-01-19 03:17:06] Actual:true, Expected: true [2024-01-19T03:17:06.844Z] INFO[2024-01-19 03:17:06] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T03:17:06.844Z] INFO[2024-01-19 03:17:06] Actual:Online, Expected: Online [2024-01-19T03:17:06.844Z] INFO[2024-01-19 03:17:06] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T03:17:06.844Z] INFO[2024-01-19 03:17:06] Actual:true, Expected: true [2024-01-19T03:17:06.844Z] 2024-01-19 03:17:06 +0000:[INFO] [{ShareBackupAndEdit}] Taking backup of applications [2024-01-19T03:17:07.103Z] 2024-01-19 03:17:06 +0000:[INFO] [{ShareBackupAndEdit}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T03:17:07.103Z] 2024-01-19 03:17:06 +0000:[INFO] [{ShareBackupAndEdit}] Volume snapshot classes found in the cluster - [2024-01-19T03:17:07.103Z] 2024-01-19 03:17:06 +0000:[INFO] [{ShareBackupAndEdit}] ibmc-vpcblock-snapshot [2024-01-19T03:17:07.103Z] 2024-01-19 03:17:06 +0000:[INFO] [{ShareBackupAndEdit}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T03:17:07.666Z] 2024/01/19 03:17:07 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705634226] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T03:17:39.699Z] 2024/01/19 03:17:37 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705634226] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:18:11.737Z] 2024/01/19 03:18:07 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705634226] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:18:38.235Z] 2024-01-19 03:18:37 +0000:[INFO] [{ShareBackupAndEdit}] [backup.CreateBackup:#313] - Backup [tp-backup-1705634226] created successfully [2024-01-19T03:18:38.235Z] 2024-01-19 03:18:37 +0000:[INFO] [{ShareBackupAndEdit}] Validating backup [tp-backup-1705634226] in org [default] [2024-01-19T03:18:38.235Z] 2024-01-19 03:18:37 +0000:[INFO] [{ShareBackupAndEdit}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705634226] [2024-01-19T03:18:38.235Z] 2024-01-19 03:18:37 +0000:[INFO] [{ShareBackupAndEdit}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82950-01-19-03h15m28s] in backup [tp-backup-1705634226] [2024-01-19T03:18:38.235Z] 2024-01-19 03:18:37 +0000:[INFO] [{ShareBackupAndEdit}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82950-01-19-03h15m28s] in backup [tp-backup-1705634226] [2024-01-19T03:18:38.235Z] 2024-01-19 03:18:37 +0000:[INFO] [{ShareBackupAndEdit}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82950-01-19-03h15m28s] [2024-01-19T03:18:38.235Z] 2024-01-19 03:18:37 +0000:[WARNING] [{ShareBackupAndEdit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:18:38.235Z] 2024-01-19 03:18:37 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:18:38.235Z] 2024-01-19 03:18:37 +0000:[INFO] [{ShareBackupAndEdit}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82950-01-19-03h15m28s] are [[mysql-data]] [2024-01-19T03:18:38.235Z] 2024-01-19 03:18:37 +0000:[INFO] [{ShareBackupAndEdit}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T03:18:38.235Z] 2024-01-19 03:18:37 +0000:[INFO] [{ShareBackupAndEdit}] Validating CR cleanup [2024-01-19T03:18:38.235Z] 2024-01-19 03:18:37 +0000:[INFO] [{ShareBackupAndEdit}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T03:18:38.235Z] 2024-01-19 03:18:37 +0000:[INFO] [{ShareBackupAndEdit}] All CRs in [mysql-ibm-pxb-0-82950-01-19-03h15m28s] are [[]] [2024-01-19T03:18:38.235Z] INFO[2024-01-19 03:18:37] Verifying : Description : Creation and Validation of backup [tp-backup-1705634226] [2024-01-19T03:18:38.235Z] INFO[2024-01-19 03:18:37] Actual:true, Expected: true [2024-01-19T03:18:38.235Z] 2024-01-19 03:18:37 +0000:[INFO] [{ShareBackupAndEdit}] Share backup with user restore mode and validate [2024-01-19T03:18:38.235Z] 2024-01-19 03:18:37 +0000:[INFO] [{ShareBackupAndEdit}] [backup.glob..func86.2.6:#1896] - Sharing backup with user - testuser2 [2024-01-19T03:18:38.235Z] 2024-01-19 03:18:37 +0000:[INFO] [{ShareBackupAndEdit}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705634226 - 42d91acd-b74e-4453-8114-229b746cd5c6 [2024-01-19T03:18:38.492Z] 2024-01-19 03:18:38 +0000:[INFO] [{ShareBackupAndEdit}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2 - 2625a3c6-313c-49e7-8ad3-539bf007b341 [2024-01-19T03:18:38.492Z] 2024-01-19 03:18:38 +0000:[INFO] [{ShareBackupAndEdit}] Update the backup with another cred [2024-01-19T03:18:38.749Z] INFO[2024-01-19 03:18:38] Verifying : Description : Updating backup tp-backup-1705634226 with new cred autogenerated-cred-1705634224 [2024-01-19T03:18:38.749Z] INFO[2024-01-19 03:18:38] Actual:true, Expected: true [2024-01-19T03:18:38.749Z] 2024-01-19 03:18:38 +0000:[INFO] [{ShareBackupAndEdit}] [backup.glob..func86.2.6:#1913] - The status after updating backup tp-backup-1705634226 with new cred autogenerated-cred-1705634224 is [2024-01-19T03:18:39.005Z] 2024-01-19 03:18:38 +0000:[INFO] [{ShareBackupAndEdit}] Registering Source and Destination clusters from user context [2024-01-19T03:18:39.005Z] INFO[2024-01-19 03:18:38] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T03:18:39.005Z] INFO[2024-01-19 03:18:38] Actual:true, Expected: true [2024-01-19T03:18:39.005Z] 2024-01-19 03:18:38 +0000:[INFO] [{ShareBackupAndEdit}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T03:18:39.005Z] 2024-01-19 03:18:38 +0000:[INFO] [{ShareBackupAndEdit}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T03:18:39.005Z] 2024-01-19 03:18:38 +0000:[INFO] [{ShareBackupAndEdit}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T03:18:39.005Z] 2024-01-19 03:18:38 +0000:[INFO] [{ShareBackupAndEdit}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T03:18:39.006Z] 2024-01-19 03:18:38 +0000:[INFO] [{ShareBackupAndEdit}] Create cluster [source-cluster] in org [default] [2024-01-19T03:18:39.006Z] 2024-01-19 03:18:38 +0000:[INFO] [{ShareBackupAndEdit}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T03:18:39.006Z] 2024-01-19 03:18:38 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T03:18:39.006Z] 2024-01-19 03:18:38 +0000:[INFO] [{ShareBackupAndEdit}] Create cluster [destination-cluster] in org [default] [2024-01-19T03:18:39.006Z] 2024-01-19 03:18:38 +0000:[INFO] [{ShareBackupAndEdit}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T03:18:39.006Z] 2024-01-19 03:18:38 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T03:18:39.006Z] 2024-01-19 03:18:38 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-zfpcr] for IBM [2024-01-19T03:18:39.006Z] 2024-01-19 03:18:38 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-zfpcr] for org [default] with [ibm] as provider [2024-01-19T03:18:39.006Z] 2024-01-19 03:18:38 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T03:18:39.006Z] 2024-01-19 03:18:38 +0000:[INFO] [{ShareBackupAndEdit}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T03:18:39.006Z] 2024-01-19 03:18:38 +0000:[WARNING] [{ShareBackupAndEdit}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-zfpcr] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName2 LastName2 with roles: [px-backup-app.user]] [2024-01-19T03:18:39.006Z] 2024-01-19 03:18:38 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName2 LastName2 with roles: [px-backup-app.user] [2024-01-19T03:18:40.392Z] 2024-01-19 03:18:40 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-zfpcr from admin context and sharing with all the users [2024-01-19T03:18:40.392Z] 2024-01-19 03:18:40 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-zfpcr] for org [default] with [ibm] as provider [2024-01-19T03:18:40.392Z] 2024-01-19 03:18:40 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T03:18:40.392Z] 2024-01-19 03:18:40 +0000:[INFO] [{ShareBackupAndEdit}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T03:18:40.651Z] 2024-01-19 03:18:40 +0000:[INFO] [{ShareBackupAndEdit}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-zfpcr] is present [2024-01-19T03:18:40.651Z] 2024-01-19 03:18:40 +0000:[INFO] [{ShareBackupAndEdit}] Create cluster [source-cluster] in org [default] [2024-01-19T03:18:41.213Z] 2024-01-19 03:18:41 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-gxeby] for IBM [2024-01-19T03:18:41.213Z] 2024-01-19 03:18:41 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-gxeby] for org [default] with [ibm] as provider [2024-01-19T03:18:41.213Z] 2024-01-19 03:18:41 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T03:18:41.213Z] 2024-01-19 03:18:41 +0000:[INFO] [{ShareBackupAndEdit}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T03:18:41.469Z] 2024-01-19 03:18:41 +0000:[WARNING] [{ShareBackupAndEdit}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-gxeby] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName2 LastName2 with roles: [px-backup-app.user]] [2024-01-19T03:18:41.469Z] 2024-01-19 03:18:41 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName2 LastName2 with roles: [px-backup-app.user] [2024-01-19T03:18:41.819Z] 2024-01-19 03:18:41 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-gxeby from admin context and sharing with all the users [2024-01-19T03:18:41.819Z] 2024-01-19 03:18:41 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-gxeby] for org [default] with [ibm] as provider [2024-01-19T03:18:41.819Z] 2024-01-19 03:18:41 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T03:18:41.819Z] 2024-01-19 03:18:41 +0000:[INFO] [{ShareBackupAndEdit}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T03:18:42.084Z] 2024-01-19 03:18:41 +0000:[INFO] [{ShareBackupAndEdit}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-gxeby] is present [2024-01-19T03:18:42.084Z] 2024-01-19 03:18:41 +0000:[INFO] [{ShareBackupAndEdit}] Create cluster [destination-cluster] in org [default] [2024-01-19T03:18:42.671Z] INFO[2024-01-19 03:18:42] Verifying : Description : Creating source and destination cluster [2024-01-19T03:18:42.671Z] INFO[2024-01-19 03:18:42] Actual:true, Expected: true [2024-01-19T03:18:42.949Z] 2024/01/19 03:18:42 DoRetryWithTimeout - Error: {restore status for [tp-restore-1705634322] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T03:19:14.985Z] 2024/01/19 03:19:12 DoRetryWithTimeout - Error: {restore status for [tp-restore-1705634322] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:19:42.828Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] [backup.CreateRestore:#936] - Restore [tp-restore-1705634322] created successfully [2024-01-19T03:19:42.828Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] Switching cluster context to cluster [destination-cluster] [2024-01-19T03:19:42.828Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T03:19:42.829Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-82950-01-19-03h15m28s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-19T03:19:42.829Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-82950-01-19-03h15m28s] [2024-01-19T03:19:42.829Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T03:19:42.829Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T03:19:42.829Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] Validating restore [tp-restore-1705634322] in org [default] [2024-01-19T03:19:42.829Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-restore-1705634322] [2024-01-19T03:19:43.086Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-restore-1705634322] [2024-01-19T03:19:43.086Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-82950-01-19-03h15m28s] in restore [tp-restore-1705634322] [2024-01-19T03:19:43.086Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-82950-01-19-03h15m28s]) was found in restore [tp-restore-1705634322], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82950-01-19-03h15m28s] [2024-01-19T03:19:43.086Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-82950-01-19-03h15m28s]) found in the restore [tp-restore-1705634322] was also present on the cluster/namespace [mysql-ibm-pxb-0-82950-01-19-03h15m28s] [2024-01-19T03:19:43.086Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-82950-01-19-03h15m28s]) was found in restore [tp-restore-1705634322], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82950-01-19-03h15m28s] [2024-01-19T03:19:43.086Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-82950-01-19-03h15m28s]) found in the restore [tp-restore-1705634322] was also present on the cluster/namespace [mysql-ibm-pxb-0-82950-01-19-03h15m28s] [2024-01-19T03:19:43.086Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-82950-01-19-03h15m28s]) was found in restore [tp-restore-1705634322], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82950-01-19-03h15m28s] [2024-01-19T03:19:43.086Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-82950-01-19-03h15m28s]) found in the restore [tp-restore-1705634322] was also present on the cluster/namespace [mysql-ibm-pxb-0-82950-01-19-03h15m28s] [2024-01-19T03:19:43.086Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-82950-01-19-03h15m28s]) was found in restore [tp-restore-1705634322], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82950-01-19-03h15m28s] [2024-01-19T03:19:43.086Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-82950-01-19-03h15m28s]) found in the restore [tp-restore-1705634322] was also present on the cluster/namespace [mysql-ibm-pxb-0-82950-01-19-03h15m28s] [2024-01-19T03:19:43.086Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-82950-01-19-03h15m28s] in restore [tp-restore-1705634322] [2024-01-19T03:19:43.086Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-82950-01-19-03h15m28s] in restore [tp-restore-1705634322] [2024-01-19T03:19:43.086Z] 2024-01-19 03:19:42 +0000:[WARNING] [{ShareBackupAndEdit}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:19:43.087Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:19:43.087Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82950-01-19-03h15m28s] are [[mysql-data]] [2024-01-19T03:19:43.087Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] [backup.ValidateRestore:#2240] - in restore [tp-restore-1705634322], the status of the restored volume [pvc-da687159-cfb4-4c62-b671-a8f9410d4c05] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-19T03:19:43.087Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] Validate applications in restored namespace [mysql-ibm-pxb-0-82950-01-19-03h15m28s] due to restore [tp-restore-1705634322] [2024-01-19T03:19:43.087Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] Validating mysql-ibm app [2024-01-19T03:19:43.087Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] wait for mysql-ibm app to start running [2024-01-19T03:19:43.087Z] 2024-01-19 03:19:42 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T03:19:43.087Z] 2024-01-19 03:19:43 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T03:19:43.343Z] 2024-01-19 03:19:43 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T03:19:43.599Z] 2024-01-19 03:19:43 +0000:[INFO] [{ShareBackupAndEdit}] Switching cluster context back to cluster path [/tmp/destination-config] [2024-01-19T03:19:43.599Z] 2024-01-19 03:19:43 +0000:[INFO] [{ShareBackupAndEdit}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T03:19:43.600Z] 2024-01-19 03:19:43 +0000:[ERROR] [{ShareBackupAndEdit}] [backup.glob..func86.2.6:#1929] - Restoring of backup [tp-backup-1705634226] has failed with name - [tp-restore-1705634322]. Err: ValidateRestore Errors: {in restore [tp-restore-1705634322], said restored volume [pvc-da687159-cfb4-4c62-b671-a8f9410d4c05] cannot be found in the actual cluster [mysql-ibm-pxb-0-82950-01-19-03h15m28s]} [2024-01-19T03:19:43.600Z] 2024-01-19 03:19:43 +0000:[INFO] [{ShareBackupAndEdit}] Deleting the deployed apps after the testcase [2024-01-19T03:19:43.600Z] 2024-01-19 03:19:43 +0000:[INFO] [{ShareBackupAndEdit}] Destroying apps [2024-01-19T03:19:43.600Z] 2024-01-19 03:19:43 +0000:[INFO] [{ShareBackupAndEdit}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T03:19:43.600Z] 2024-01-19 03:19:43 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T03:19:43.600Z] 2024-01-19 03:19:43 +0000:[INFO] [{ShareBackupAndEdit}] start destroying mysql-ibm app [2024-01-19T03:19:43.600Z] 2024-01-19 03:19:43 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T03:19:43.856Z] 2024-01-19 03:19:43 +0000:[INFO] [{ShareBackupAndEdit}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:19:43.856Z] 2024-01-19 03:19:43 +0000:[WARNING] [{ShareBackupAndEdit}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T03:19:43.856Z] INFO[2024-01-19 03:19:43] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T03:19:43.856Z] INFO[2024-01-19 03:19:43] Actual:true, Expected: true [2024-01-19T03:19:43.856Z] 2024-01-19 03:19:43 +0000:[INFO] [{ShareBackupAndEdit}] [backup.glob..func86.3:#1996] - Deleting registered clusters for non-admin context [2024-01-19T03:19:43.856Z] 2024-01-19 03:19:43 +0000:[INFO] [{ShareBackupAndEdit}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T03:19:43.856Z] INFO[2024-01-19 03:19:43] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T03:19:43.856Z] INFO[2024-01-19 03:19:43] Actual:true, Expected: true [2024-01-19T03:19:44.112Z] INFO[2024-01-19 03:19:43] Verifying : Description : Deleting cluster source-cluster [2024-01-19T03:19:44.112Z] INFO[2024-01-19 03:19:43] Actual:true, Expected: true [2024-01-19T03:19:44.374Z] INFO[2024-01-19 03:19:44] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-zfpcr] [2024-01-19T03:19:44.374Z] INFO[2024-01-19 03:19:44] Actual:true, Expected: true [2024-01-19T03:19:44.936Z] INFO[2024-01-19 03:19:44] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T03:19:44.936Z] INFO[2024-01-19 03:19:44] Actual:true, Expected: true [2024-01-19T03:19:45.864Z] INFO[2024-01-19 03:19:45] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-gxeby] [2024-01-19T03:19:45.864Z] INFO[2024-01-19 03:19:45] Actual:true, Expected: true [2024-01-19T03:19:45.864Z] 2024-01-19 03:19:45 +0000:[INFO] [{ShareBackupAndEdit}] [backup.glob..func86.3:#2002] - Cleaning up users [2024-01-19T03:19:47.753Z] 2024-01-19 03:19:47 +0000:[INFO] [{ShareBackupAndEdit}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1 - 05f95f45-0e35-4161-bccc-a9f079f5fe99 [2024-01-19T03:19:48.009Z] 2024-01-19 03:19:47 +0000:[INFO] [{ShareBackupAndEdit}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2 - 2625a3c6-313c-49e7-8ad3-539bf007b341 [2024-01-19T03:19:48.938Z] 2024-01-19 03:19:48 +0000:[INFO] [{ShareBackupAndEdit}] [backup.DeleteUser:#755] - Response for user [testuser1] deletion - [2024-01-19T03:19:48.938Z] 2024-01-19 03:19:48 +0000:[INFO] [{ShareBackupAndEdit}] [backup.DeleteUser:#759] - Deleted User - testuser1 [2024-01-19T03:19:48.938Z] 2024-01-19 03:19:48 +0000:[INFO] [{ShareBackupAndEdit}] [backup.DeleteUser:#755] - Response for user [testuser2] deletion - [2024-01-19T03:19:48.938Z] 2024-01-19 03:19:48 +0000:[INFO] [{ShareBackupAndEdit}] [backup.DeleteUser:#759] - Deleted User - testuser2 [2024-01-19T03:19:50.304Z] 2024-01-19 03:19:50 +0000:[INFO] [{ShareBackupAndEdit}] Cleaning backup locations in map [map[654a1515-d800-4a43-bf14-8b83a0c5ec1e:autogenerated-backup-location-1705634221]], cloud credential [autogenerated-cred-1705634221], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T03:19:50.304Z] INFO[2024-01-19 03:19:50] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-1705634221] [2024-01-19T03:19:50.304Z] ERRO[2024-01-19 03:19:50] Actual:rpc error: code = Unknown desc = restore object [tp-restore-1705634322] has reference to backuplocation [autogenerated-backup-location-1705634221:654a1515-d800-4a43-bf14-8b83a0c5ec1e], Expected:  [2024-01-19T03:19:50.304Z] INFO[2024-01-19 03:19:50] --------Test End------ [2024-01-19T03:19:50.304Z] INFO[2024-01-19 03:19:50] #Test: [2024-01-19T03:19:50.304Z] INFO[2024-01-19 03:19:50] #Description: [2024-01-19T03:19:50.304Z] INFO[2024-01-19 03:19:50] ------------------------ [2024-01-19T03:19:50.304Z] 2024-01-19 03:19:50 +0000:[INFO] [{ShareBackupAndEdit}] [tests.EndPxBackupTorpedoTest:#7261] - >>>> FAILED TEST: {ShareBackupAndEdit} Share the backup and edit [2024-01-19T03:19:50.560Z] 2024-01-19 03:19:50 +0000:[INFO] [{ShareBackupAndEdit}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-82950-01-19-03h15m28s] [2024-01-19T03:19:50.560Z] 2024-01-19 03:19:50 +0000:[INFO] [{ShareBackupAndEdit}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T03:19:50.560Z] 2024-01-19 03:19:50 +0000:[INFO] [{ShareBackupAndEdit}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T03:19:50.822Z] 2024-01-19 03:19:50 +0000:[INFO] [{ShareBackupAndEdit}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-82950-01-19-03h15m28s] [2024-01-19T03:19:51.078Z] 2024-01-19 03:19:50 +0000:[INFO] [{ShareBackupAndEdit}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {ShareBackupAndEdit} Share the backup and edit [2024-01-19T03:19:51.078Z] 2024-01-19 03:19:50 +0000:[INFO] [{ShareBackupAndEdit}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T03:19:51.078Z] 2024/01/19 03:19:50 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:19:56.327Z] 2024/01/19 03:19:55 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:20:01.648Z] 2024/01/19 03:20:00 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:20:06.909Z] 2024/01/19 03:20:05 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:20:11.076Z] 2024/01/19 03:20:10 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:20:16.320Z] 2024/01/19 03:20:15 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:20:21.568Z] 2024/01/19 03:20:20 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:20:26.817Z] 2024/01/19 03:20:25 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:20:30.986Z] 2024/01/19 03:20:30 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.128.20. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.20 10.240.128.20]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T03:20:30.986Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T03:20:30.986Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T03:20:30.986Z] {dial tcp 10.240.128.20:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T03:20:36.233Z] 2024-01-19 03:20:35 +0000:[WARNING] [{ShareBackupAndEdit}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.20. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.20 10.240.128.20]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T03:20:36.233Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T03:20:36.233Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T03:20:36.233Z] {dial tcp 10.240.128.20:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T03:20:36.233Z] 2024-01-19 03:20:35 +0000:[ERROR] [{ShareBackupAndEdit}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.20. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.20 10.240.128.20]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T03:20:36.233Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T03:20:36.233Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T03:20:36.233Z] {dial tcp 10.240.128.20:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T03:20:36.233Z] 2024-01-19 03:20:35 +0000:[INFO] [{ShareBackupAndEdit}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T03:20:36.233Z] 2024-01-19 03:20:35 +0000:[INFO] [{ShareBackupAndEdit}] Switching context to [/tmp/source-config] [2024-01-19T03:20:36.233Z] 2024-01-19 03:20:35 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T03:20:36.233Z] 2024-01-19 03:20:35 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T03:20:36.233Z] 2024-01-19 03:20:35 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T03:20:36.233Z] 2024-01-19 03:20:36 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T03:20:36.233Z] 2024-01-19 03:20:36 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T03:20:36.489Z] 2024-01-19 03:20:36 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T03:20:36.746Z] 2024-01-19 03:20:36 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T03:20:37.313Z] 2024-01-19 03:20:37 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T03:20:37.569Z] 2024-01-19 03:20:37 +0000:[INFO] [{ShareBackupAndEdit}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T03:20:37.826Z] 2024/01/19 03:20:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:20:47.770Z] 2024/01/19 03:20:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:20:59.939Z] 2024/01/19 03:20:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:21:08.034Z] 2024/01/19 03:21:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:21:17.987Z] 2024/01/19 03:21:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:21:27.932Z] 2024/01/19 03:21:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:21:40.099Z] 2024/01/19 03:21:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:21:48.187Z] 2024/01/19 03:21:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:21:58.125Z] 2024/01/19 03:21:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:22:08.062Z] 2024/01/19 03:22:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:22:17.999Z] 2024/01/19 03:22:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:22:30.174Z] 2024/01/19 03:22:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:22:38.255Z] 2024/01/19 03:22:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:22:48.194Z] 2024/01/19 03:22:48 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:22:58.154Z] 2024/01/19 03:22:58 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:23:10.324Z] 2024/01/19 03:23:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:23:18.487Z] 2024/01/19 03:23:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:23:28.428Z] 2024/01/19 03:23:28 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:23:38.366Z] 2024/01/19 03:23:38 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:23:48.309Z] 2024/01/19 03:23:48 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:24:00.484Z] 2024/01/19 03:23:58 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:24:08.566Z] 2024/01/19 03:24:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:24:18.503Z] 2024/01/19 03:24:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:24:28.437Z] 2024/01/19 03:24:28 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:24:40.607Z] 2024/01/19 03:24:38 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:24:48.686Z] 2024/01/19 03:24:48 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:24:50.051Z] INFO[2024-01-19 03:24:49] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T03:24:50.051Z] INFO[2024-01-19 03:24:49] sched stats: tick interval: 42649 < 1.2s (100%) [2024-01-19T03:24:50.051Z] INFO[2024-01-19 03:24:49] sched stats: tick process durations: 42648 < 500ms (100%) [2024-01-19T03:24:59.989Z] 2024/01/19 03:24:58 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:25:09.929Z] 2024/01/19 03:25:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:25:19.878Z] 2024/01/19 03:25:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:25:29.854Z] 2024/01/19 03:25:28 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:25:39.800Z] 2024-01-19 03:25:38 +0000:[ERROR] [{ShareBackupAndEdit}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.800Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.800Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.800Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.800Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.800Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.800Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.800Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.800Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:25:39.801Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T03:25:39.801Z] • Failure [610.138 seconds] [2024-01-19T03:25:39.801Z] {ShareBackupAndEdit} [2024-01-19T03:25:39.801Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:1786 [2024-01-19T03:25:39.801Z] Share the backup and edit [It] [2024-01-19T03:25:39.801Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:1820 [2024-01-19T03:25:39.801Z] [2024-01-19T03:25:39.801Z] Unexpected error: [2024-01-19T03:25:39.801Z] <*errors.errorString | 0xc0012e8950>: { [2024-01-19T03:25:39.801Z] s: "ValidateRestore Errors: {in restore [tp-restore-1705634322], said restored volume [pvc-da687159-cfb4-4c62-b671-a8f9410d4c05] cannot be found in the actual cluster [mysql-ibm-pxb-0-82950-01-19-03h15m28s]}", [2024-01-19T03:25:39.802Z] } [2024-01-19T03:25:39.802Z] ValidateRestore Errors: {in restore [tp-restore-1705634322], said restored volume [pvc-da687159-cfb4-4c62-b671-a8f9410d4c05] cannot be found in the actual cluster [mysql-ibm-pxb-0-82950-01-19-03h15m28s]} [2024-01-19T03:25:39.802Z] occurred [2024-01-19T03:25:39.802Z] [2024-01-19T03:25:39.802Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T03:25:39.802Z] [2024-01-19T03:25:39.802Z] Full Stack Trace [2024-01-19T03:25:39.802Z] github.com/portworx/torpedo/pkg/log.FailOnError({0x88a5060, 0xc0012e8950}, {0x7fb35bb?, 0x7?}, {0xc001485078?, 0x1?, 0x1?}) [2024-01-19T03:25:39.802Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 +0x290 [2024-01-19T03:25:39.802Z] github.com/portworx/torpedo/tests/backup.glob..func86.2.6() [2024-01-19T03:25:39.802Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:1929 +0xab7 [2024-01-19T03:25:39.802Z] github.com/portworx/torpedo/tests/backup.glob..func86.2() [2024-01-19T03:25:39.802Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:1894 +0x693 [2024-01-19T03:25:39.802Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T03:25:39.802Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T03:25:39.802Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T03:25:39.802Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T03:25:39.802Z] created by testing.(*T).Run [2024-01-19T03:25:39.802Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T03:25:39.802Z] ------------------------------ [2024-01-19T03:25:39.802Z] STEP: schedule applications [2024-01-19T03:25:39.802Z] STEP: Validate applications and get their labels [2024-01-19T03:25:39.802Z] STEP: validate applications [2024-01-19T03:25:39.802Z] STEP: validate mysql-ibm app's volumes [2024-01-19T03:25:39.802Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T03:25:39.802Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T03:25:39.802Z] STEP: get mysql-ibm app's volume: pvc-bf614768-3741-4b91-8db1-9e09d3532ce9 inspected by the volume driver [2024-01-19T03:25:39.802Z] STEP: wait for mysql-ibm app to start running [2024-01-19T03:25:39.802Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T03:25:39.802Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T03:25:39.802Z] STEP: Validate Px pod restart count [2024-01-19T03:25:39.802Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T03:25:39.802Z] STEP: Create Users [2024-01-19T03:25:39.802Z] STEP: Adding Credentials and Registering Backup Location [2024-01-19T03:25:39.802Z] STEP: Register source and destination cluster for backup [2024-01-19T03:25:39.802Z] STEP: Taking backup of applications [2024-01-19T03:25:39.802Z] STEP: Share backup with user restore mode and validate [2024-01-19T03:25:39.802Z] STEP: validate mysql-ibm app's volumes [2024-01-19T03:25:39.802Z] STEP: wait for mysql-ibm app to start running [2024-01-19T03:25:39.802Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T03:25:39.802Z] STEP: Validate Px pod restart count [2024-01-19T03:25:39.802Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T03:25:39.802Z] STEP: destroy apps [2024-01-19T03:25:39.802Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T03:25:39.802Z] STEP: start destroying mysql-ibm app [2024-01-19T03:25:39.802Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:25:39.802Z] 2024-01-19 03:25:38 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] Log Dir: /testresults//BackupMultipleNsWithSameLabel.log [2024-01-19T03:25:39.802Z] INFO[2024-01-19 03:25:38] --------Test Start------ [2024-01-19T03:25:39.802Z] INFO[2024-01-19 03:25:38] #Test: BackupMultipleNsWithSameLabel [2024-01-19T03:25:39.802Z] INFO[2024-01-19 03:25:38] #Description: Taking backup and restoring multiple namespace having same labels [2024-01-19T03:25:39.802Z] INFO[2024-01-19 03:25:38] ------------------------ [2024-01-19T03:25:39.802Z] 2024-01-19 03:25:38 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] Deploy applications [2024-01-19T03:25:39.803Z] 2024-01-19 03:25:38 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T03:25:39.803Z] 2024-01-19 03:25:38 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T03:25:39.803Z] 2024-01-19 03:25:38 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T03:25:39.803Z] 2024-01-19 03:25:38 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T03:25:39.803Z] 2024-01-19 03:25:38 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T03:25:39.803Z] 2024-01-19 03:25:38 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T03:25:39.804Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T03:25:39.804Z] 2024-01-19 03:25:38 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T03:25:39.805Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T03:25:39.805Z] 2024-01-19 03:25:38 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T03:25:39.805Z] 2024-01-19 03:25:38 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T03:25:39.805Z] 2024-01-19 03:25:38 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T03:25:39.805Z] 2024-01-19 03:25:38 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T03:25:39.805Z] 2024-01-19 03:25:38 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T03:25:39.805Z] 2024-01-19 03:25:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T03:25:39.805Z] 2024-01-19 03:25:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T03:25:39.806Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T03:25:39.806Z] 2024-01-19 03:25:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T03:25:39.806Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T03:25:39.806Z] 2024-01-19 03:25:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T03:25:39.806Z] 2024-01-19 03:25:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T03:25:39.806Z] 2024-01-19 03:25:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T03:25:39.806Z] 2024-01-19 03:25:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T03:25:39.806Z] 2024-01-19 03:25:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T03:25:39.806Z] 2024-01-19 03:25:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T03:25:39.806Z] 2024-01-19 03:25:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T03:25:39.807Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T03:25:39.807Z] 2024-01-19 03:25:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T03:25:39.807Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T03:25:39.807Z] 2024-01-19 03:25:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T03:25:39.807Z] 2024-01-19 03:25:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T03:25:39.807Z] 2024-01-19 03:25:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T03:25:39.807Z] 2024-01-19 03:25:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T03:25:39.807Z] 2024-01-19 03:25:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T03:25:39.808Z] 2024-01-19 03:25:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T03:25:39.808Z] 2024-01-19 03:25:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T03:25:39.808Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T03:25:39.808Z] 2024-01-19 03:25:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T03:25:39.809Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T03:25:40.065Z] 2024-01-19 03:25:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T03:25:40.065Z] 2024-01-19 03:25:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T03:25:40.065Z] 2024-01-19 03:25:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T03:25:40.065Z] 2024-01-19 03:25:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T03:25:40.065Z] 2024-01-19 03:25:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T03:25:40.065Z] 2024-01-19 03:25:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T03:25:40.065Z] 2024-01-19 03:25:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T03:25:40.066Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T03:25:40.066Z] 2024-01-19 03:25:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T03:25:40.066Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T03:25:40.322Z] 2024-01-19 03:25:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T03:25:40.322Z] 2024-01-19 03:25:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T03:25:40.323Z] 2024-01-19 03:25:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T03:25:40.323Z] 2024-01-19 03:25:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T03:25:40.323Z] 2024-01-19 03:25:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T03:25:40.323Z] 2024-01-19 03:25:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T03:25:40.323Z] 2024-01-19 03:25:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T03:25:40.323Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T03:25:40.324Z] 2024-01-19 03:25:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T03:25:40.324Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T03:25:40.581Z] 2024-01-19 03:25:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T03:25:40.581Z] 2024-01-19 03:25:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T03:25:40.581Z] 2024-01-19 03:25:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T03:25:40.581Z] 2024-01-19 03:25:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T03:25:40.581Z] 2024-01-19 03:25:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T03:25:40.837Z] 2024-01-19 03:25:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T03:25:41.093Z] 2024-01-19 03:25:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T03:25:41.094Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T03:25:41.094Z] 2024-01-19 03:25:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T03:25:41.094Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T03:25:41.094Z] 2024-01-19 03:25:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T03:25:41.094Z] 2024-01-19 03:25:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T03:25:41.094Z] 2024-01-19 03:25:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T03:25:41.095Z] 2024-01-19 03:25:41 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T03:25:41.350Z] 2024-01-19 03:25:41 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T03:25:41.350Z] 2024-01-19 03:25:41 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T03:25:41.605Z] 2024-01-19 03:25:41 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T03:25:41.606Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T03:25:41.606Z] 2024-01-19 03:25:41 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T03:25:41.607Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T03:25:41.607Z] 2024-01-19 03:25:41 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T03:25:41.607Z] 2024-01-19 03:25:41 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T03:25:41.607Z] 2024-01-19 03:25:41 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T03:25:41.862Z] 2024-01-19 03:25:41 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T03:25:41.862Z] 2024-01-19 03:25:41 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T03:25:42.118Z] 2024-01-19 03:25:41 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T03:25:42.118Z] 2024-01-19 03:25:42 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T03:25:42.119Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T03:25:42.119Z] 2024-01-19 03:25:42 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T03:25:42.119Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T03:25:42.376Z] 2024-01-19 03:25:42 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T03:25:42.376Z] 2024-01-19 03:25:42 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T03:25:42.376Z] 2024-01-19 03:25:42 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T03:25:42.376Z] 2024-01-19 03:25:42 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T03:25:42.376Z] 2024-01-19 03:25:42 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T03:25:42.633Z] 2024-01-19 03:25:42 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T03:25:42.889Z] 2024-01-19 03:25:42 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T03:25:42.890Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T03:25:42.890Z] 2024-01-19 03:25:42 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T03:25:42.890Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T03:25:42.890Z] 2024-01-19 03:25:42 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T03:25:42.890Z] 2024-01-19 03:25:42 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] Created namespaces [mysql-ibm-pxb-0-84851-01-19-03h25m38s mysql-ibm-pxb-1-84851-01-19-03h25m38s mysql-ibm-pxb-2-84851-01-19-03h25m38s mysql-ibm-pxb-3-84851-01-19-03h25m38s mysql-ibm-pxb-4-84851-01-19-03h25m38s mysql-ibm-pxb-5-84851-01-19-03h25m38s mysql-ibm-pxb-6-84851-01-19-03h25m38s mysql-ibm-pxb-7-84851-01-19-03h25m38s mysql-ibm-pxb-8-84851-01-19-03h25m38s mysql-ibm-pxb-9-84851-01-19-03h25m38s] [2024-01-19T03:25:42.890Z] 2024-01-19 03:25:42 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] Validate applications [2024-01-19T03:25:42.890Z] 2024-01-19 03:25:42 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] Validate applications [2024-01-19T03:25:42.890Z] 2024-01-19 03:25:42 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] Validating mysql-ibm app [2024-01-19T03:25:42.890Z] 2024-01-19 03:25:42 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] Validating mysql-ibm app's volumes [2024-01-19T03:25:42.890Z] 2024-01-19 03:25:42 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:25:43.147Z] 2024/01/19 03:25:43 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T03:25:53.088Z] 2024-01-19 03:25:53 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-84851-01-19-03h25m38s [2024-01-19T03:25:53.345Z] 2024-01-19 03:25:53 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T03:25:53.345Z] 2024-01-19 03:25:53 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T03:25:53.345Z] 2024-01-19 03:25:53 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] wait for mysql-ibm app to start running [2024-01-19T03:25:53.345Z] 2024-01-19 03:25:53 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T03:25:53.345Z] 2024/01/19 03:25:53 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:25:53.345Z] pod name:mysql-7d766cd65c-wltx9 namespace:mysql-ibm-pxb-0-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:25:53.345Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:26:05.514Z] 2024/01/19 03:26:03 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:26:05.514Z] pod name:mysql-7d766cd65c-wltx9 namespace:mysql-ibm-pxb-0-84851-01-19-03h25m38s running:false ready:false node:10.240.64.27 [2024-01-19T03:26:05.514Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:26:13.622Z] 2024/01/19 03:26:13 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:26:13.622Z] pod name:mysql-7d766cd65c-wltx9 namespace:mysql-ibm-pxb-0-84851-01-19-03h25m38s running:false ready:false node:10.240.64.27 [2024-01-19T03:26:13.622Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:26:23.672Z] 2024/01/19 03:26:23 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:26:23.672Z] pod name:mysql-7d766cd65c-wltx9 namespace:mysql-ibm-pxb-0-84851-01-19-03h25m38s running:true ready:false node:10.240.64.27 [2024-01-19T03:26:23.672Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:26:35.840Z] 2024/01/19 03:26:33 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:26:35.840Z] pod name:mysql-7d766cd65c-wltx9 namespace:mysql-ibm-pxb-0-84851-01-19-03h25m38s running:true ready:false node:10.240.64.27 [2024-01-19T03:26:35.841Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:26:43.931Z] 2024/01/19 03:26:43 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:26:43.931Z] pod name:mysql-7d766cd65c-wltx9 namespace:mysql-ibm-pxb-0-84851-01-19-03h25m38s running:true ready:false node:10.240.64.27 [2024-01-19T03:26:43.931Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:26:56.101Z] 2024/01/19 03:26:53 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:26:56.101Z] pod name:mysql-7d766cd65c-wltx9 namespace:mysql-ibm-pxb-0-84851-01-19-03h25m38s running:true ready:false node:10.240.64.27 [2024-01-19T03:26:56.101Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:27:04.189Z] 2024/01/19 03:27:03 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:27:04.189Z] pod name:mysql-7d766cd65c-wltx9 namespace:mysql-ibm-pxb-0-84851-01-19-03h25m38s running:true ready:false node:10.240.64.27 [2024-01-19T03:27:04.189Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:27:14.131Z] 2024/01/19 03:27:14 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:27:14.131Z] pod name:mysql-7d766cd65c-wltx9 namespace:mysql-ibm-pxb-0-84851-01-19-03h25m38s running:true ready:false node:10.240.64.27 [2024-01-19T03:27:14.131Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:27:26.302Z] 2024-01-19 03:27:24 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T03:27:26.302Z] 2024-01-19 03:27:24 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T03:27:26.303Z] 2024-01-19 03:27:24 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] validate if mysql-ibm app's volumes are setup [2024-01-19T03:27:26.303Z] 2024-01-19 03:27:24 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:27:26.303Z] 2024-01-19 03:27:24 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:27:26.303Z] 2024-01-19 03:27:24 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T03:27:26.303Z] 2024-01-19 03:27:24 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T03:27:26.303Z] 2024-01-19 03:27:24 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] Validating mysql-ibm app [2024-01-19T03:27:26.303Z] 2024-01-19 03:27:24 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] Validating mysql-ibm app's volumes [2024-01-19T03:27:26.303Z] 2024-01-19 03:27:24 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:27:26.303Z] 2024-01-19 03:27:24 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:27:26.303Z] 2024-01-19 03:27:24 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-1-84851-01-19-03h25m38s [2024-01-19T03:27:26.303Z] 2024-01-19 03:27:24 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T03:27:26.303Z] 2024-01-19 03:27:24 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T03:27:26.303Z] 2024-01-19 03:27:24 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] wait for mysql-ibm app to start running [2024-01-19T03:27:26.303Z] 2024-01-19 03:27:24 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T03:27:26.303Z] 2024/01/19 03:27:24 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:27:26.304Z] pod name:mysql-7d766cd65c-5fndp namespace:mysql-ibm-pxb-1-84851-01-19-03h25m38s running:true ready:false node:10.240.64.8 [2024-01-19T03:27:26.304Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:27:36.257Z] 2024-01-19 03:27:34 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T03:27:36.257Z] 2024-01-19 03:27:34 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T03:27:36.257Z] 2024-01-19 03:27:35 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] validate if mysql-ibm app's volumes are setup [2024-01-19T03:27:36.257Z] 2024-01-19 03:27:35 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:27:36.257Z] 2024-01-19 03:27:35 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:27:36.257Z] 2024-01-19 03:27:35 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T03:27:36.257Z] 2024-01-19 03:27:35 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T03:27:36.257Z] 2024-01-19 03:27:35 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] Validating mysql-ibm app [2024-01-19T03:27:36.257Z] 2024-01-19 03:27:35 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] Validating mysql-ibm app's volumes [2024-01-19T03:27:36.258Z] 2024-01-19 03:27:35 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:27:36.258Z] 2024-01-19 03:27:35 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:27:36.258Z] 2024-01-19 03:27:35 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-2-84851-01-19-03h25m38s [2024-01-19T03:27:36.258Z] 2024-01-19 03:27:35 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T03:27:36.258Z] 2024-01-19 03:27:35 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T03:27:36.258Z] 2024-01-19 03:27:35 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] wait for mysql-ibm app to start running [2024-01-19T03:27:36.258Z] 2024-01-19 03:27:35 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T03:27:36.258Z] 2024-01-19 03:27:35 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T03:27:36.258Z] 2024-01-19 03:27:35 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T03:27:37.185Z] 2024-01-19 03:27:36 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] validate if mysql-ibm app's volumes are setup [2024-01-19T03:27:37.185Z] 2024-01-19 03:27:36 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:27:37.185Z] 2024-01-19 03:27:36 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:27:37.185Z] 2024-01-19 03:27:36 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T03:27:37.185Z] 2024-01-19 03:27:36 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T03:27:37.185Z] 2024-01-19 03:27:36 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] Validating mysql-ibm app [2024-01-19T03:27:37.185Z] 2024-01-19 03:27:36 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] Validating mysql-ibm app's volumes [2024-01-19T03:27:37.185Z] 2024-01-19 03:27:36 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:27:37.185Z] 2024-01-19 03:27:36 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:27:37.185Z] 2024-01-19 03:27:36 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-3-84851-01-19-03h25m38s [2024-01-19T03:27:37.185Z] 2024-01-19 03:27:36 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T03:27:37.185Z] 2024-01-19 03:27:37 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T03:27:37.185Z] 2024-01-19 03:27:37 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] wait for mysql-ibm app to start running [2024-01-19T03:27:37.185Z] 2024-01-19 03:27:37 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T03:27:37.752Z] 2024-01-19 03:27:37 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T03:27:38.317Z] 2024-01-19 03:27:37 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T03:27:38.877Z] 2024-01-19 03:27:38 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] validate if mysql-ibm app's volumes are setup [2024-01-19T03:27:38.877Z] 2024-01-19 03:27:38 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:27:38.877Z] 2024-01-19 03:27:38 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:27:38.877Z] 2024-01-19 03:27:38 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T03:27:38.877Z] 2024-01-19 03:27:38 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T03:27:39.134Z] 2024-01-19 03:27:38 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] Validating mysql-ibm app [2024-01-19T03:27:39.134Z] 2024-01-19 03:27:38 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] Validating mysql-ibm app's volumes [2024-01-19T03:27:39.134Z] 2024-01-19 03:27:38 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:27:39.134Z] 2024-01-19 03:27:38 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:27:39.134Z] 2024-01-19 03:27:38 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-4-84851-01-19-03h25m38s [2024-01-19T03:27:39.134Z] 2024-01-19 03:27:38 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T03:27:39.134Z] 2024-01-19 03:27:39 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T03:27:39.134Z] 2024-01-19 03:27:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] wait for mysql-ibm app to start running [2024-01-19T03:27:39.392Z] 2024-01-19 03:27:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T03:27:39.655Z] 2024-01-19 03:27:39 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T03:27:40.217Z] 2024-01-19 03:27:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T03:27:41.149Z] 2024-01-19 03:27:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] validate if mysql-ibm app's volumes are setup [2024-01-19T03:27:41.149Z] 2024-01-19 03:27:40 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:27:41.149Z] 2024-01-19 03:27:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:27:41.149Z] 2024-01-19 03:27:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T03:27:41.149Z] 2024-01-19 03:27:40 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T03:27:41.149Z] 2024-01-19 03:27:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] Validating mysql-ibm app [2024-01-19T03:27:41.149Z] 2024-01-19 03:27:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] Validating mysql-ibm app's volumes [2024-01-19T03:27:41.149Z] 2024-01-19 03:27:40 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:27:41.149Z] 2024-01-19 03:27:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:27:41.149Z] 2024-01-19 03:27:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-5-84851-01-19-03h25m38s [2024-01-19T03:27:41.149Z] 2024-01-19 03:27:40 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T03:27:41.149Z] 2024-01-19 03:27:41 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T03:27:41.149Z] 2024-01-19 03:27:41 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] wait for mysql-ibm app to start running [2024-01-19T03:27:41.149Z] 2024-01-19 03:27:41 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T03:27:41.713Z] 2024/01/19 03:27:41 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:27:41.713Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:27:41.713Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:27:51.663Z] 2024/01/19 03:27:51 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:27:51.663Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:27:51.663Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:28:03.826Z] 2024/01/19 03:28:01 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:28:03.827Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:28:03.827Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:28:11.912Z] 2024/01/19 03:28:11 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:28:11.912Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:28:11.912Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:28:21.849Z] 2024/01/19 03:28:21 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:28:21.849Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:28:21.849Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:28:34.021Z] 2024/01/19 03:28:31 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:28:34.021Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:28:34.021Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:28:42.107Z] 2024/01/19 03:28:41 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:28:42.107Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:28:42.107Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:28:54.278Z] 2024/01/19 03:28:52 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:28:54.278Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:28:54.278Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:29:02.367Z] 2024/01/19 03:29:02 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:29:02.367Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:29:02.367Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:29:13.961Z] 2024/01/19 03:29:12 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:29:13.961Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:29:13.961Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:29:23.915Z] 2024/01/19 03:29:22 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:29:23.915Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:29:23.915Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:29:33.856Z] 2024/01/19 03:29:32 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:29:33.856Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:29:33.856Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:29:43.796Z] 2024/01/19 03:29:42 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:29:43.796Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:29:43.796Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:29:53.741Z] 2024/01/19 03:29:52 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:29:53.741Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:29:53.741Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:30:03.685Z] 2024/01/19 03:30:02 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:30:03.686Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:30:03.686Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:30:13.623Z] 2024/01/19 03:30:12 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:30:13.623Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:30:13.623Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:30:23.632Z] 2024/01/19 03:30:22 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:30:23.632Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:30:23.632Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:30:33.607Z] 2024/01/19 03:30:33 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:30:33.607Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:30:33.607Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:30:43.546Z] 2024/01/19 03:30:43 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:30:43.546Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:30:43.546Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:30:53.490Z] 2024/01/19 03:30:53 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:30:53.490Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:30:53.490Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:31:03.442Z] 2024/01/19 03:31:03 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:31:03.442Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:31:03.442Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:31:15.609Z] 2024/01/19 03:31:13 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:31:15.609Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:31:15.609Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:31:23.702Z] 2024/01/19 03:31:23 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:31:23.702Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:31:23.702Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:31:33.644Z] 2024/01/19 03:31:33 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:31:33.644Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:31:33.644Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:31:45.817Z] 2024/01/19 03:31:43 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:31:45.817Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:31:45.817Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:31:53.899Z] 2024/01/19 03:31:53 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:31:53.899Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:31:53.899Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:32:06.070Z] 2024/01/19 03:32:03 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:32:06.070Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:32:06.070Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:32:14.152Z] 2024/01/19 03:32:13 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:32:14.152Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:32:14.152Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:32:26.322Z] 2024/01/19 03:32:24 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:32:26.322Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:32:26.322Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:32:34.406Z] 2024/01/19 03:32:34 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:32:34.406Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:32:34.406Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:32:44.351Z] 2024/01/19 03:32:44 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:32:44.351Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:32:44.351Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:32:56.551Z] 2024/01/19 03:32:54 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:32:56.551Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:32:56.551Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:33:04.658Z] 2024/01/19 03:33:04 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:33:04.658Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:33:04.658Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:33:14.613Z] 2024/01/19 03:33:14 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:33:14.613Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:33:14.613Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:33:26.781Z] 2024/01/19 03:33:24 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:33:26.781Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:33:26.781Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:33:34.876Z] 2024/01/19 03:33:34 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:33:34.877Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:33:34.877Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:33:44.834Z] 2024/01/19 03:33:44 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:33:44.834Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:33:44.834Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:33:57.073Z] 2024/01/19 03:33:54 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:33:57.073Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:33:57.073Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:34:05.156Z] 2024/01/19 03:34:04 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:34:05.156Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:34:05.156Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:34:10.400Z] Iteration: 62 [2024-01-19T03:34:11.328Z] Logs from iteration: 62 [2024-01-19T03:34:11.890Z] 2024/01/19 03:33:14 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:34:11.890Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:34:11.890Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:34:11.890Z] 2024/01/19 03:33:24 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:34:11.890Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:34:11.890Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:34:11.890Z] 2024/01/19 03:33:34 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:34:11.890Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:34:11.890Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:34:11.890Z] 2024/01/19 03:33:44 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:34:11.890Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:34:11.890Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:34:11.890Z] 2024/01/19 03:33:54 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:34:11.890Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:34:11.890Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:34:11.890Z] 2024/01/19 03:34:04 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:34:11.890Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:34:11.890Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:34:15.160Z] 2024/01/19 03:34:15 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:34:15.160Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:34:15.160Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:34:27.336Z] 2024/01/19 03:34:25 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:34:27.336Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:34:27.336Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:34:35.426Z] 2024/01/19 03:34:35 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:34:35.426Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:34:35.426Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:34:45.393Z] 2024/01/19 03:34:45 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:34:45.393Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:34:45.393Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:34:51.937Z] INFO[2024-01-19 03:34:50] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T03:34:51.937Z] INFO[2024-01-19 03:34:50] sched stats: tick interval: 43250 < 1.2s (100%) [2024-01-19T03:34:51.937Z] INFO[2024-01-19 03:34:50] sched stats: tick process durations: 43249 < 500ms (100%) [2024-01-19T03:34:56.105Z] 2024/01/19 03:34:55 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:34:56.105Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:34:56.105Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:35:06.055Z] 2024/01/19 03:35:05 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:35:06.055Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:35:06.055Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:35:16.017Z] 2024/01/19 03:35:15 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:35:16.017Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:35:16.017Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:35:25.963Z] 2024/01/19 03:35:25 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:35:25.963Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:35:25.963Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:35:35.924Z] 2024/01/19 03:35:35 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:35:35.924Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:35:35.924Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:35:45.870Z] 2024/01/19 03:35:45 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:35:45.870Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:35:45.870Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:35:58.087Z] 2024/01/19 03:35:55 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:35:58.087Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:35:58.087Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:36:06.191Z] 2024/01/19 03:36:05 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:36:06.191Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:36:06.191Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:36:16.138Z] 2024/01/19 03:36:16 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:36:16.138Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:36:16.138Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:36:28.310Z] 2024/01/19 03:36:26 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:36:28.310Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:36:28.310Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:36:36.399Z] 2024/01/19 03:36:36 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:36:36.399Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:36:36.399Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:36:48.573Z] 2024/01/19 03:36:46 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:36:48.573Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:36:48.573Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:36:56.661Z] 2024/01/19 03:36:56 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:36:56.661Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:36:56.661Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:37:06.604Z] 2024/01/19 03:37:06 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:06.604Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:06.604Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:37:18.774Z] 2024/01/19 03:37:16 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:18.774Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:18.774Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:37:26.866Z] 2024/01/19 03:37:26 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:26.866Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:26.866Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:37:39.031Z] 2024/01/19 03:37:36 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:39.031Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:39.031Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:37:48.980Z] 2024-01-19 03:37:47 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [tests.PrintDescribeContext:#1673] - Context mysql-ibm Details: [2024-01-19T03:37:48.980Z] 2024-01-19 03:37:47 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [tests.PrintDescribeContext:#1674] - ------------------------------ [2024-01-19T03:37:48.980Z] StorageClass: mysql-sc [2024-01-19T03:37:48.980Z] ------------------------------ [2024-01-19T03:37:48.980Z] map[billingType:hourly classVersion:1 csi.storage.k8s.io/fstype:ext4 encrypted:false encryptionKey: profile:10iops-tier region: resourceGroup: tags: zone:] [2024-01-19T03:37:48.980Z] ------------------------------ [2024-01-19T03:37:48.980Z] Events: [2024-01-19T03:37:48.980Z] ------------------------------ [2024-01-19T03:37:48.980Z] &EventList{ListMeta:{ 1602207 },Items:[]Event{},} [2024-01-19T03:37:48.980Z] ------------------------------ [2024-01-19T03:37:48.980Z] END StorageClass [2024-01-19T03:37:48.980Z] ------------------------------ [2024-01-19T03:37:48.980Z] ------------------------------ [2024-01-19T03:37:48.980Z] PersistentVolumeClaim: [mysql-ibm-pxb-5-84851-01-19-03h25m38s] mysql-data [2024-01-19T03:37:48.980Z] ------------------------------ [2024-01-19T03:37:48.980Z] Status: {Phase:Bound AccessModes:[ReadWriteOnce] Capacity:map[storage:{i:{value:10737418240 scale:0} d:{Dec:} s:10Gi Format:BinarySI}] Conditions:[] AllocatedResources:map[] ResizeStatus:} [2024-01-19T03:37:48.980Z] ------------------------------ [2024-01-19T03:37:48.980Z] Events: [2024-01-19T03:37:48.980Z] ------------------------------ [2024-01-19T03:37:48.981Z] &EventList{ListMeta:{ 1602207 },Items:[]Event{Event{ObjectMeta:{mysql-data.17aba1c266d88c1a mysql-ibm-pxb-5-84851-01-19-03h25m38s bf5a7f3d-3c90-41be-b9f6-ca8d55f5c933 1593874 0 2024-01-19 03:25:40 +0000 UTC map[] map[] [] [] [{csi-provisioner Update v1 2024-01-19 03:25:40 +0000 UTC FieldsV1 {"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:source":{"f:component":{}},"f:type":{}} }]},InvolvedObject:ObjectReference{Kind:PersistentVolumeClaim,Namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s,Name:mysql-data,UID:c6180d49-7015-4e39-82c0-13ae8a4c5037,APIVersion:v1,ResourceVersion:1593870,FieldPath:,},Reason:Provisioning,Message:External provisioner is provisioning volume for claim "mysql-ibm-pxb-5-84851-01-19-03h25m38s/mysql-data",Source:EventSource{Component:vpc.block.csi.ibm.io_ibm-vpc-block-csi-controller-0_bdd3258f-11f9-47f4-9d18-9367c4bac327,Host:,},FirstTimestamp:2024-01-19 03:25:40 +0000 UTC,LastTimestamp:2024-01-19 03:25:40 +0000 UTC,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:,ReportingInstance:,},Event{ObjectMeta:{mysql-data.17aba1c266f91aeb mysql-ibm-pxb-5-84851-01-19-03h25m38s 4a333366-41b4-47d2-9444-bf4727cda110 1594376 0 2024-01-19 03:25:40 +0000 UTC map[] map[] [] [] [{kube-controller-manager Update v1 2024-01-19 03:25:47 +0000 UTC FieldsV1 {"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:source":{"f:component":{}},"f:type":{}} }]},InvolvedObject:ObjectReference{Kind:PersistentVolumeClaim,Namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s,Name:mysql-data,UID:c6180d49-7015-4e39-82c0-13ae8a4c5037,APIVersion:v1,ResourceVersion:1593870,FieldPath:,},Reason:ExternalProvisioning,Message:waiting for a volume to be created, either by external provisioner "vpc.block.csi.ibm.io" or manually created by system administrator,Source:EventSource{Component:persistentvolume-controller,Host:,},FirstTimestamp:2024-01-19 03:25:40 +0000 UTC,LastTimestamp:2024-01-19 03:25:47 +0000 UTC,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:persistentvolume-controller,ReportingInstance:,},Event{ObjectMeta:{mysql-data.17aba1c4e8f39bd6 mysql-ibm-pxb-5-84851-01-19-03h25m38s 0c3ad4aa-6e9c-4e20-9d88-7b6003824a15 1594457 0 2024-01-19 03:25:50 +0000 UTC map[] map[] [] [] [{csi-provisioner Update v1 2024-01-19 03:25:50 +0000 UTC FieldsV1 {"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:source":{"f:component":{}},"f:type":{}} }]},InvolvedObject:ObjectReference{Kind:PersistentVolumeClaim,Namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s,Name:mysql-data,UID:c6180d49-7015-4e39-82c0-13ae8a4c5037,APIVersion:v1,ResourceVersion:1593870,FieldPath:,},Reason:ProvisioningSucceeded,Message:Successfully provisioned volume pvc-c6180d49-7015-4e39-82c0-13ae8a4c5037,Source:EventSource{Component:vpc.block.csi.ibm.io_ibm-vpc-block-csi-controller-0_bdd3258f-11f9-47f4-9d18-9367c4bac327,Host:,},FirstTimestamp:2024-01-19 03:25:50 +0000 UTC,LastTimestamp:2024-01-19 03:25:50 +0000 UTC,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:,ReportingInstance:,},},} [2024-01-19T03:37:48.981Z] ------------------------------ [2024-01-19T03:37:48.981Z] END PersistentVolumeClaim [2024-01-19T03:37:48.981Z] ------------------------------ [2024-01-19T03:37:48.981Z] ------------------------------ [2024-01-19T03:37:48.981Z] Service: [mysql-ibm-pxb-5-84851-01-19-03h25m38s] mysql [2024-01-19T03:37:48.981Z] ------------------------------ [2024-01-19T03:37:48.981Z] Status: {LoadBalancer:{Ingress:[]} Conditions:[]} [2024-01-19T03:37:48.981Z] ------------------------------ [2024-01-19T03:37:48.981Z] Events: [2024-01-19T03:37:48.981Z] ------------------------------ [2024-01-19T03:37:48.981Z] &EventList{ListMeta:{ 1602207 },Items:[]Event{},} [2024-01-19T03:37:48.981Z] ------------------------------ [2024-01-19T03:37:48.981Z] END Service [2024-01-19T03:37:48.981Z] ------------------------------ [2024-01-19T03:37:48.981Z] ------------------------------ [2024-01-19T03:37:48.981Z] Deployment: [mysql-ibm-pxb-5-84851-01-19-03h25m38s] mysql [2024-01-19T03:37:48.981Z] ------------------------------ [2024-01-19T03:37:48.982Z] Status: {ObservedGeneration:1 Replicas:1 UpdatedReplicas:1 ReadyReplicas:0 AvailableReplicas:0 UnavailableReplicas:1 Conditions:[{Type:Available Status:True LastUpdateTime:2024-01-19 03:25:40 +0000 UTC LastTransitionTime:2024-01-19 03:25:40 +0000 UTC Reason:MinimumReplicasAvailable Message:Deployment has minimum availability.} {Type:Progressing Status:False LastUpdateTime:2024-01-19 03:35:41 +0000 UTC LastTransitionTime:2024-01-19 03:35:41 +0000 UTC Reason:ProgressDeadlineExceeded Message:ReplicaSet "mysql-7d766cd65c" has timed out progressing.}] CollisionCount:} [2024-01-19T03:37:48.982Z] ------------------------------ [2024-01-19T03:37:48.982Z] Events: [2024-01-19T03:37:48.982Z] ------------------------------ [2024-01-19T03:37:48.982Z] &EventList{ListMeta:{ 1602207 },Items:[]Event{Event{ObjectMeta:{mysql.17aba1c26c63d09a mysql-ibm-pxb-5-84851-01-19-03h25m38s 196b9419-3cc6-4736-9106-7dbe7d896de8 1593892 0 2024-01-19 03:25:40 +0000 UTC map[] map[] [] [] [{kube-controller-manager Update v1 2024-01-19 03:25:40 +0000 UTC FieldsV1 {"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:source":{"f:component":{}},"f:type":{}} }]},InvolvedObject:ObjectReference{Kind:Deployment,Namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s,Name:mysql,UID:59c0825a-c9ba-45b5-843d-d084b3958db6,APIVersion:apps/v1,ResourceVersion:1593887,FieldPath:,},Reason:ScalingReplicaSet,Message:Scaled up replica set mysql-7d766cd65c to 1,Source:EventSource{Component:deployment-controller,Host:,},FirstTimestamp:2024-01-19 03:25:40 +0000 UTC,LastTimestamp:2024-01-19 03:25:40 +0000 UTC,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:deployment-controller,ReportingInstance:,},},} [2024-01-19T03:37:48.982Z] ------------------------------ [2024-01-19T03:37:48.982Z] Pod: [mysql-ibm-pxb-5-84851-01-19-03h25m38s] mysql-7d766cd65c-l7fkm [2024-01-19T03:37:48.982Z] ------------------------------ [2024-01-19T03:37:48.982Z] {Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2024-01-19 03:25:40 +0000 UTC Unschedulable 0/9 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/unreachable: }, 2 node(s) exceed max volume count, 6 node(s) had volume node affinity conflict. preemption: 0/9 nodes are available: 2 No preemption victims found for incoming pod, 7 Preemption is not helpful for scheduling..}] [] [] [] BestEffort []} [2024-01-19T03:37:48.982Z] ------------------------------ [2024-01-19T03:37:48.982Z] Events: [2024-01-19T03:37:48.982Z] ------------------------------ [2024-01-19T03:37:48.983Z] &EventList{ListMeta:{ 1602207 },Items:[]Event{Event{ObjectMeta:{mysql-7d766cd65c-l7fkm.17aba1c26dcdcf28 mysql-ibm-pxb-5-84851-01-19-03h25m38s 6fb2983a-4d2b-420a-a4b7-698f8e446211 1593902 0 2024-01-19 03:25:40 +0000 UTC map[] map[] [] [] [{kube-scheduler Update events.k8s.io/v1 2024-01-19 03:25:40 +0000 UTC FieldsV1 {"f:action":{},"f:eventTime":{},"f:note":{},"f:reason":{},"f:regarding":{},"f:reportingController":{},"f:reportingInstance":{},"f:type":{}} }]},InvolvedObject:ObjectReference{Kind:Pod,Namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s,Name:mysql-7d766cd65c-l7fkm,UID:89e1607a-873e-412d-a575-7d9d3dfae73d,APIVersion:v1,ResourceVersion:1593895,FieldPath:,},Reason:FailedScheduling,Message:0/9 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/9 nodes are available: 9 Preemption is not helpful for scheduling..,Source:EventSource{Component:,Host:,},FirstTimestamp:0001-01-01 00:00:00 +0000 UTC,LastTimestamp:0001-01-01 00:00:00 +0000 UTC,Count:0,Type:Warning,EventTime:2024-01-19 03:25:40.327779 +0000 UTC,Series:nil,Action:Scheduling,Related:nil,ReportingController:default-scheduler,ReportingInstance:default-scheduler-kube-scheduler-86bb6cfd6-kpgrw,},Event{ObjectMeta:{mysql-7d766cd65c-l7fkm.17aba1c2c7958355 mysql-ibm-pxb-5-84851-01-19-03h25m38s 757c7072-4732-41f7-9ce5-e87f10127c0e 1596778 0 2024-01-19 03:25:41 +0000 UTC map[] map[] [] [] [{kube-scheduler Update events.k8s.io/v1 2024-01-19 03:29:08 +0000 UTC FieldsV1 {"f:action":{},"f:eventTime":{},"f:note":{},"f:reason":{},"f:regarding":{},"f:reportingController":{},"f:reportingInstance":{},"f:series":{".":{},"f:count":{},"f:lastObservedTime":{}},"f:type":{}} }]},InvolvedObject:ObjectReference{Kind:Pod,Namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s,Name:mysql-7d766cd65c-l7fkm,UID:89e1607a-873e-412d-a575-7d9d3dfae73d,APIVersion:v1,ResourceVersion:1593904,FieldPath:,},Reason:FailedScheduling,Message:0/9 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/9 nodes are available: 9 Preemption is not helpful for scheduling..,Source:EventSource{Component:,Host:,},FirstTimestamp:0001-01-01 00:00:00 +0000 UTC,LastTimestamp:0001-01-01 00:00:00 +0000 UTC,Count:0,Type:Warning,EventTime:2024-01-19 03:25:41.834042 +0000 UTC,Series:&EventSeries{Count:4,LastObservedTime:2024-01-19 03:25:58.858062 +0000 UTC,},Action:Scheduling,Related:nil,ReportingController:default-scheduler,ReportingInstance:default-scheduler-kube-scheduler-86bb6cfd6-kpgrw,},Event{ObjectMeta:{mysql-7d766cd65c-l7fkm.17aba1c94e0b7b32 mysql-ibm-pxb-5-84851-01-19-03h25m38s be112c85-f998-435c-839b-413eaa71a78a 1595255 0 2024-01-19 03:26:09 +0000 UTC map[] map[] [] [] [{kube-scheduler Update events.k8s.io/v1 2024-01-19 03:26:45 +0000 UTC FieldsV1 {"f:action":{},"f:eventTime":{},"f:note":{},"f:reason":{},"f:regarding":{},"f:reportingController":{},"f:reportingInstance":{},"f:series":{".":{},"f:count":{},"f:lastObservedTime":{}},"f:type":{}} }]},InvolvedObject:ObjectReference{Kind:Pod,Namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s,Name:mysql-7d766cd65c-l7fkm,UID:89e1607a-873e-412d-a575-7d9d3dfae73d,APIVersion:v1,ResourceVersion:1594590,FieldPath:,},Reason:FailedScheduling,Message:0/9 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/unreachable: }, 2 node(s) exceed max volume count, 6 node(s) had volume node affinity conflict. preemption: 0/9 nodes are available: 2 No preemption victims found for incoming pod, 7 Preemption is not helpful for scheduling..,Source:EventSource{Component:,Host:,},FirstTimestamp:0001-01-01 00:00:00 +0000 UTC,LastTimestamp:0001-01-01 00:00:00 +0000 UTC,Count:0,Type:Warning,EventTime:2024-01-19 03:26:09.859724 +0000 UTC,Series:&EventSeries{Count:2,LastObservedTime:2024-01-19 03:26:45.339045 +0000 UTC,},Action:Scheduling,Related:nil,ReportingController:default-scheduler,ReportingInstance:default-scheduler-kube-scheduler-86bb6cfd6-kpgrw,},},} [2024-01-19T03:37:48.983Z] ------------------------------ [2024-01-19T03:37:48.983Z] END pod [2024-01-19T03:37:48.983Z] ------------------------------ [2024-01-19T03:37:48.983Z] ------------------------------ [2024-01-19T03:37:48.983Z] END Deployment [2024-01-19T03:37:48.983Z] ------------------------------ [2024-01-19T03:37:48.983Z] ------------------------------ [2024-01-19T03:37:48.983Z] Deployment: [mysql-ibm-pxb-5-84851-01-19-03h25m38s] mysqlslap [2024-01-19T03:37:48.983Z] ------------------------------ [2024-01-19T03:37:48.983Z] Status: {ObservedGeneration:1 Replicas:1 UpdatedReplicas:1 ReadyReplicas:1 AvailableReplicas:1 UnavailableReplicas:0 Conditions:[{Type:Available Status:True LastUpdateTime:2024-01-19 03:25:40 +0000 UTC LastTransitionTime:2024-01-19 03:25:40 +0000 UTC Reason:MinimumReplicasAvailable Message:Deployment has minimum availability.} {Type:Progressing Status:True LastUpdateTime:2024-01-19 03:25:42 +0000 UTC LastTransitionTime:2024-01-19 03:25:40 +0000 UTC Reason:NewReplicaSetAvailable Message:ReplicaSet "mysqlslap-7b87bb5695" has successfully progressed.}] CollisionCount:} [2024-01-19T03:37:48.983Z] ------------------------------ [2024-01-19T03:37:48.983Z] Events: [2024-01-19T03:37:48.983Z] ------------------------------ [2024-01-19T03:37:48.984Z] &EventList{ListMeta:{ 1602207 },Items:[]Event{Event{ObjectMeta:{mysqlslap.17aba1c26df51cb0 mysql-ibm-pxb-5-84851-01-19-03h25m38s fc717b06-a9b1-4cd4-8246-f1d52bc73330 1593898 0 2024-01-19 03:25:40 +0000 UTC map[] map[] [] [] [{kube-controller-manager Update v1 2024-01-19 03:25:40 +0000 UTC FieldsV1 {"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:source":{"f:component":{}},"f:type":{}} }]},InvolvedObject:ObjectReference{Kind:Deployment,Namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s,Name:mysqlslap,UID:5b7e3501-b0cd-42dd-b77f-f2156d1694d3,APIVersion:apps/v1,ResourceVersion:1593891,FieldPath:,},Reason:ScalingReplicaSet,Message:Scaled up replica set mysqlslap-7b87bb5695 to 1,Source:EventSource{Component:deployment-controller,Host:,},FirstTimestamp:2024-01-19 03:25:40 +0000 UTC,LastTimestamp:2024-01-19 03:25:40 +0000 UTC,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:deployment-controller,ReportingInstance:,},},} [2024-01-19T03:37:48.984Z] ------------------------------ [2024-01-19T03:37:48.984Z] Pod: [mysql-ibm-pxb-5-84851-01-19-03h25m38s] mysqlslap-7b87bb5695-528cz [2024-01-19T03:37:48.984Z] ------------------------------ [2024-01-19T03:37:48.984Z] {Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2024-01-19 03:25:40 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2024-01-19 03:25:42 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2024-01-19 03:25:42 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2024-01-19 03:25:40 +0000 UTC }] 10.240.0.7 172.17.84.247 [{172.17.84.247}] 2024-01-19 03:25:40 +0000 UTC [] [{mysqlslap {nil &ContainerStateRunning{StartedAt:2024-01-19 03:25:42 +0000 UTC,} nil} {nil nil nil} true 0 docker.io/adityadani/mysqlslap:latest docker.io/adityadani/mysqlslap@sha256:45640240b52978e1bacf52666af94b21e242218f3f94e2e970815eee68f655e6 cri-o://d98989e7ee0be5ec510f924f12e31b55a7bc6332e170b5ed618b1d869ea55726 0xc001960df9}] BestEffort []} [2024-01-19T03:37:48.984Z] ------------------------------ [2024-01-19T03:37:48.984Z] container: mysqlslap [2024-01-19T03:37:48.984Z] ------------------------------ [2024-01-19T03:37:48.984Z] {mysqlslap {nil &ContainerStateRunning{StartedAt:2024-01-19 03:25:42 +0000 UTC,} nil} {nil nil nil} true 0 docker.io/adityadani/mysqlslap:latest docker.io/adityadani/mysqlslap@sha256:45640240b52978e1bacf52666af94b21e242218f3f94e2e970815eee68f655e6 cri-o://d98989e7ee0be5ec510f924f12e31b55a7bc6332e170b5ed618b1d869ea55726 0xc001960df9} [2024-01-19T03:37:48.984Z] ------------------------------ [2024-01-19T03:37:48.984Z] END container [2024-01-19T03:37:48.984Z] ------------------------------ [2024-01-19T03:37:48.984Z] ------------------------------ [2024-01-19T03:37:48.984Z] Events: [2024-01-19T03:37:48.984Z] ------------------------------ [2024-01-19T03:37:48.985Z] &EventList{ListMeta:{ 1602208 },Items:[]Event{Event{ObjectMeta:{mysqlslap-7b87bb5695-528cz.17aba1c271fc6fc2 mysql-ibm-pxb-5-84851-01-19-03h25m38s e2cd9845-4373-42ad-afb6-9c42406870b9 1593919 0 2024-01-19 03:25:40 +0000 UTC map[] map[] [] [] [{kube-scheduler Update events.k8s.io/v1 2024-01-19 03:25:40 +0000 UTC FieldsV1 {"f:action":{},"f:eventTime":{},"f:note":{},"f:reason":{},"f:regarding":{},"f:reportingController":{},"f:reportingInstance":{},"f:type":{}} }]},InvolvedObject:ObjectReference{Kind:Pod,Namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s,Name:mysqlslap-7b87bb5695-528cz,UID:ed42e0af-33c1-47ce-a13f-de2844fd6eb1,APIVersion:v1,ResourceVersion:1593901,FieldPath:,},Reason:Scheduled,Message:Successfully assigned mysql-ibm-pxb-5-84851-01-19-03h25m38s/mysqlslap-7b87bb5695-528cz to 10.240.0.7,Source:EventSource{Component:,Host:,},FirstTimestamp:0001-01-01 00:00:00 +0000 UTC,LastTimestamp:0001-01-01 00:00:00 +0000 UTC,Count:0,Type:Normal,EventTime:2024-01-19 03:25:40.397938 +0000 UTC,Series:nil,Action:Binding,Related:nil,ReportingController:default-scheduler,ReportingInstance:default-scheduler-kube-scheduler-86bb6cfd6-kpgrw,},Event{ObjectMeta:{mysqlslap-7b87bb5695-528cz.17aba1c2b5ca5e4e mysql-ibm-pxb-5-84851-01-19-03h25m38s 74dda847-dc17-4b70-9777-8063da6f944a 1594091 0 2024-01-19 03:25:41 +0000 UTC map[] map[] [] [] [{multus-daemon Update v1 2024-01-19 03:25:41 +0000 UTC FieldsV1 {"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:source":{"f:component":{}},"f:type":{}} }]},InvolvedObject:ObjectReference{Kind:Pod,Namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s,Name:mysqlslap-7b87bb5695-528cz,UID:ed42e0af-33c1-47ce-a13f-de2844fd6eb1,APIVersion:v1,ResourceVersion:1593923,FieldPath:,},Reason:AddedInterface,Message:Add eth0 [172.17.84.247/32] from k8s-pod-network,Source:EventSource{Component:multus,Host:,},FirstTimestamp:2024-01-19 03:25:41 +0000 UTC,LastTimestamp:2024-01-19 03:25:41 +0000 UTC,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:,ReportingInstance:,},Event{ObjectMeta:{mysqlslap-7b87bb5695-528cz.17aba1c2c6928b8c mysql-ibm-pxb-5-84851-01-19-03h25m38s 1e6f8fb5-ef0c-4513-9a76-c4112efc4ef5 1594148 0 2024-01-19 03:25:41 +0000 UTC map[] map[] [] [] [{kubelet Update v1 2024-01-19 03:25:41 +0000 UTC FieldsV1 {"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:reportingInstance":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}} }]},InvolvedObject:ObjectReference{Kind:Pod,Namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s,Name:mysqlslap-7b87bb5695-528cz,UID:ed42e0af-33c1-47ce-a13f-de2844fd6eb1,APIVersion:v1,ResourceVersion:1593914,FieldPath:spec.containers{mysqlslap},},Reason:Pulled,Message:Container image "adityadani/mysqlslap" already present on machine,Source:EventSource{Component:kubelet,Host:10.240.0.7,},FirstTimestamp:2024-01-19 03:25:41 +0000 UTC,LastTimestamp:2024-01-19 03:25:41 +0000 UTC,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:10.240.0.7,},Event{ObjectMeta:{mysqlslap-7b87bb5695-528cz.17aba1c2d5061157 mysql-ibm-pxb-5-84851-01-19-03h25m38s b92fb606-9c8a-4b1c-a2c5-90e03da09aa7 1594169 0 2024-01-19 03:25:42 +0000 UTC map[] map[] [] [] [{kubelet Update v1 2024-01-19 03:25:42 +0000 UTC FieldsV1 {"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:reportingInstance":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}} }]},InvolvedObject:ObjectReference{Kind:Pod,Namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s,Name:mysqlslap-7b87bb5695-528cz,UID:ed42e0af-33c1-47ce-a13f-de2844fd6eb1,APIVersion:v1,ResourceVersion:1593914,FieldPath:spec.containers{mysqlslap},},Reason:Created,Message:Created container mysqlslap,Source:EventSource{Component:kubelet,Host:10.240.0.7,},FirstTimestamp:2024-01-19 03:25:42 +0000 UTC,LastTimestamp:2024-01-19 03:25:42 +0000 UTC,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:10.240.0.7,},Event{ObjectMeta:{mysqlslap-7b87bb5695-528cz.17aba1c2d6d3587c mysql-ibm-pxb-5-84851-01-19-03h25m38s e21ef2e0-9068-4c17-a230-440965cd841c 1594178 0 2024-01-19 03:25:42 +0000 UTC map[] map[] [] [] [{kubelet Update v1 2024-01-19 03:25:42 +0000 UTC FieldsV1 {"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:reportingInstance":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}} }]},InvolvedObject:ObjectReference{Kind:Pod,Namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s,Name:mysqlslap-7b87bb5695-528cz,UID:ed42e0af-33c1-47ce-a13f-de2844fd6eb1,APIVersion:v1,ResourceVersion:1593914,FieldPath:spec.containers{mysqlslap},},Reason:Started,Message:Started container mysqlslap,Source:EventSource{Component:kubelet,Host:10.240.0.7,},FirstTimestamp:2024-01-19 03:25:42 +0000 UTC,LastTimestamp:2024-01-19 03:25:42 +0000 UTC,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:10.240.0.7,},},} [2024-01-19T03:37:48.985Z] ------------------------------ [2024-01-19T03:37:48.985Z] END pod [2024-01-19T03:37:48.985Z] ------------------------------ [2024-01-19T03:37:48.985Z] ------------------------------ [2024-01-19T03:37:48.985Z] END Deployment [2024-01-19T03:37:48.985Z] ------------------------------ [2024-01-19T03:37:48.985Z] [2024-01-19T03:37:48.986Z] 2024-01-19 03:37:47 +0000:[ERROR] [{BackupMultipleNsWithSameLabel}] [tests.processError:#600] - processError. Err: Failed to validate app: mysql-ibm due to err: Failed to validate Deployment: mysql,Namespace: mysql-ibm-pxb-5-84851-01-19-03h25m38s. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.986Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.986Z] } [2024-01-19T03:37:48.986Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.986Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.986Z] } [2024-01-19T03:37:48.986Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.986Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.986Z] } [2024-01-19T03:37:48.986Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.986Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.986Z] } [2024-01-19T03:37:48.986Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.986Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.986Z] } [2024-01-19T03:37:48.986Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.986Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.986Z] } [2024-01-19T03:37:48.986Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.986Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.986Z] } [2024-01-19T03:37:48.986Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.986Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.986Z] } [2024-01-19T03:37:48.986Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.986Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.986Z] } [2024-01-19T03:37:48.986Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.986Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.986Z] } [2024-01-19T03:37:48.986Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.986Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.986Z] } [2024-01-19T03:37:48.986Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.987Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.987Z] } [2024-01-19T03:37:48.987Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.987Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.987Z] } [2024-01-19T03:37:48.987Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.987Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.987Z] } [2024-01-19T03:37:48.987Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.987Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.987Z] } [2024-01-19T03:37:48.987Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.987Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.987Z] } [2024-01-19T03:37:48.987Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.987Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.987Z] } [2024-01-19T03:37:48.987Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.987Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.987Z] } [2024-01-19T03:37:48.987Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.987Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.987Z] } [2024-01-19T03:37:48.987Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.987Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.987Z] } [2024-01-19T03:37:48.987Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.987Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.987Z] } [2024-01-19T03:37:48.987Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.987Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.987Z] } [2024-01-19T03:37:48.987Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.987Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.987Z] } [2024-01-19T03:37:48.987Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.988Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.988Z] } [2024-01-19T03:37:48.988Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.988Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.988Z] } [2024-01-19T03:37:48.988Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.988Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.988Z] } [2024-01-19T03:37:48.988Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.988Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.988Z] } [2024-01-19T03:37:48.988Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.988Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.988Z] } [2024-01-19T03:37:48.988Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.988Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.988Z] } [2024-01-19T03:37:48.988Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.988Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.988Z] } [2024-01-19T03:37:48.988Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.988Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.988Z] } [2024-01-19T03:37:48.988Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.988Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.988Z] } [2024-01-19T03:37:48.988Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.988Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.988Z] } [2024-01-19T03:37:48.988Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.988Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.988Z] } [2024-01-19T03:37:48.988Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.988Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.988Z] } [2024-01-19T03:37:48.988Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.988Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.988Z] } [2024-01-19T03:37:48.988Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.989Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.989Z] } [2024-01-19T03:37:48.989Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.989Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.989Z] } [2024-01-19T03:37:48.989Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.989Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.989Z] } [2024-01-19T03:37:48.989Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.989Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.989Z] } [2024-01-19T03:37:48.989Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.989Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.989Z] } [2024-01-19T03:37:48.989Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.989Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.989Z] } [2024-01-19T03:37:48.989Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.989Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.989Z] } [2024-01-19T03:37:48.989Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.989Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.989Z] } [2024-01-19T03:37:48.989Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.989Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.989Z] } [2024-01-19T03:37:48.989Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.989Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.989Z] } [2024-01-19T03:37:48.989Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.989Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.989Z] } [2024-01-19T03:37:48.989Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.989Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.989Z] } [2024-01-19T03:37:48.989Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.989Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.989Z] } [2024-01-19T03:37:48.989Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.989Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.989Z] } [2024-01-19T03:37:48.990Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.990Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.990Z] } [2024-01-19T03:37:48.990Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.990Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.990Z] } [2024-01-19T03:37:48.990Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.990Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.990Z] } [2024-01-19T03:37:48.990Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.990Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.990Z] } [2024-01-19T03:37:48.990Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.990Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.990Z] } [2024-01-19T03:37:48.990Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.990Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.990Z] } [2024-01-19T03:37:48.990Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.990Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.990Z] } [2024-01-19T03:37:48.990Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.990Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.990Z] } [2024-01-19T03:37:48.990Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.990Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.990Z] } [2024-01-19T03:37:48.990Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:37:48.990Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:37:48.990Z] } [2024-01-19T03:37:48.990Z] INFO[2024-01-19 03:37:48] Verifying : Description : Fetching px-central-admin ctx [2024-01-19T03:37:48.990Z] INFO[2024-01-19 03:37:48] Actual:true, Expected: true [2024-01-19T03:37:48.990Z] 2024-01-19 03:37:48 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] Deleting labels from namespaces - [mysql-ibm-pxb-0-84851-01-19-03h25m38s mysql-ibm-pxb-1-84851-01-19-03h25m38s mysql-ibm-pxb-2-84851-01-19-03h25m38s mysql-ibm-pxb-3-84851-01-19-03h25m38s mysql-ibm-pxb-4-84851-01-19-03h25m38s mysql-ibm-pxb-5-84851-01-19-03h25m38s mysql-ibm-pxb-6-84851-01-19-03h25m38s mysql-ibm-pxb-7-84851-01-19-03h25m38s mysql-ibm-pxb-8-84851-01-19-03h25m38s mysql-ibm-pxb-9-84851-01-19-03h25m38s] [2024-01-19T03:37:52.257Z] INFO[2024-01-19 03:37:52] Verifying : Description : Deleting labels [map[]] from namespaces [[mysql-ibm-pxb-0-84851-01-19-03h25m38s mysql-ibm-pxb-1-84851-01-19-03h25m38s mysql-ibm-pxb-2-84851-01-19-03h25m38s mysql-ibm-pxb-3-84851-01-19-03h25m38s mysql-ibm-pxb-4-84851-01-19-03h25m38s mysql-ibm-pxb-5-84851-01-19-03h25m38s mysql-ibm-pxb-6-84851-01-19-03h25m38s mysql-ibm-pxb-7-84851-01-19-03h25m38s mysql-ibm-pxb-8-84851-01-19-03h25m38s mysql-ibm-pxb-9-84851-01-19-03h25m38s]] [2024-01-19T03:37:52.257Z] INFO[2024-01-19 03:37:52] Actual:true, Expected: true [2024-01-19T03:37:52.258Z] 2024-01-19 03:37:52 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] Deleting deployed namespaces - [mysql-ibm-pxb-0-84851-01-19-03h25m38s mysql-ibm-pxb-1-84851-01-19-03h25m38s mysql-ibm-pxb-2-84851-01-19-03h25m38s mysql-ibm-pxb-3-84851-01-19-03h25m38s mysql-ibm-pxb-4-84851-01-19-03h25m38s mysql-ibm-pxb-5-84851-01-19-03h25m38s mysql-ibm-pxb-6-84851-01-19-03h25m38s mysql-ibm-pxb-7-84851-01-19-03h25m38s mysql-ibm-pxb-8-84851-01-19-03h25m38s mysql-ibm-pxb-9-84851-01-19-03h25m38s] [2024-01-19T03:37:52.258Z] 2024-01-19 03:37:52 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] Destroying apps [2024-01-19T03:37:52.258Z] 2024-01-19 03:37:52 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T03:37:52.819Z] 2024-01-19 03:37:52 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T03:37:52.819Z] 2024-01-19 03:37:52 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] start destroying mysql-ibm app [2024-01-19T03:37:52.819Z] 2024-01-19 03:37:52 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T03:37:53.077Z] 2024-01-19 03:37:52 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:37:53.077Z] 2024-01-19 03:37:52 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T03:37:53.077Z] INFO[2024-01-19 03:37:52] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T03:37:53.077Z] INFO[2024-01-19 03:37:52] Actual:true, Expected: true [2024-01-19T03:37:53.077Z] 2024-01-19 03:37:52 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T03:37:53.333Z] 2024-01-19 03:37:53 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T03:37:53.333Z] 2024-01-19 03:37:53 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] start destroying mysql-ibm app [2024-01-19T03:37:53.588Z] 2024-01-19 03:37:53 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T03:37:53.588Z] 2024-01-19 03:37:53 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:37:53.588Z] 2024-01-19 03:37:53 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T03:37:53.588Z] INFO[2024-01-19 03:37:53] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T03:37:53.588Z] INFO[2024-01-19 03:37:53] Actual:true, Expected: true [2024-01-19T03:37:53.588Z] 2024-01-19 03:37:53 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T03:37:53.844Z] 2024-01-19 03:37:53 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T03:37:53.844Z] 2024-01-19 03:37:53 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] start destroying mysql-ibm app [2024-01-19T03:37:54.100Z] 2024-01-19 03:37:53 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T03:37:54.357Z] 2024-01-19 03:37:54 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:37:54.357Z] 2024-01-19 03:37:54 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T03:37:54.357Z] INFO[2024-01-19 03:37:54] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T03:37:54.357Z] INFO[2024-01-19 03:37:54] Actual:true, Expected: true [2024-01-19T03:37:54.357Z] 2024-01-19 03:37:54 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T03:37:54.358Z] 2024-01-19 03:37:54 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T03:37:54.358Z] 2024-01-19 03:37:54 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] start destroying mysql-ibm app [2024-01-19T03:37:54.614Z] 2024-01-19 03:37:54 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T03:37:54.614Z] 2024-01-19 03:37:54 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:37:54.614Z] 2024-01-19 03:37:54 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T03:37:54.614Z] INFO[2024-01-19 03:37:54] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T03:37:54.614Z] INFO[2024-01-19 03:37:54] Actual:true, Expected: true [2024-01-19T03:37:54.614Z] 2024-01-19 03:37:54 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T03:37:55.175Z] 2024-01-19 03:37:54 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T03:37:55.175Z] 2024-01-19 03:37:54 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] start destroying mysql-ibm app [2024-01-19T03:37:55.175Z] 2024-01-19 03:37:55 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T03:37:55.431Z] 2024-01-19 03:37:55 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:37:55.431Z] 2024-01-19 03:37:55 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T03:37:55.431Z] INFO[2024-01-19 03:37:55] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T03:37:55.431Z] INFO[2024-01-19 03:37:55] Actual:true, Expected: true [2024-01-19T03:37:55.431Z] 2024-01-19 03:37:55 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T03:37:55.687Z] 2024-01-19 03:37:55 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T03:37:55.687Z] 2024-01-19 03:37:55 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] start destroying mysql-ibm app [2024-01-19T03:37:55.943Z] 2024-01-19 03:37:55 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T03:37:55.943Z] 2024-01-19 03:37:55 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:37:55.943Z] 2024-01-19 03:37:55 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T03:37:55.943Z] INFO[2024-01-19 03:37:55] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T03:37:55.943Z] INFO[2024-01-19 03:37:55] Actual:true, Expected: true [2024-01-19T03:37:55.943Z] 2024-01-19 03:37:55 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T03:37:56.198Z] 2024-01-19 03:37:56 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T03:37:56.198Z] 2024-01-19 03:37:56 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] start destroying mysql-ibm app [2024-01-19T03:37:56.454Z] 2024-01-19 03:37:56 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T03:37:56.454Z] 2024-01-19 03:37:56 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:37:56.454Z] 2024-01-19 03:37:56 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T03:37:56.454Z] INFO[2024-01-19 03:37:56] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T03:37:56.454Z] INFO[2024-01-19 03:37:56] Actual:true, Expected: true [2024-01-19T03:37:56.454Z] 2024-01-19 03:37:56 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T03:37:57.015Z] 2024-01-19 03:37:56 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T03:37:57.015Z] 2024-01-19 03:37:56 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] start destroying mysql-ibm app [2024-01-19T03:37:57.015Z] 2024-01-19 03:37:56 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T03:37:57.271Z] 2024-01-19 03:37:56 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:37:57.271Z] 2024-01-19 03:37:56 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T03:37:57.271Z] INFO[2024-01-19 03:37:56] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T03:37:57.271Z] INFO[2024-01-19 03:37:56] Actual:true, Expected: true [2024-01-19T03:37:57.271Z] 2024-01-19 03:37:56 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T03:37:57.526Z] 2024-01-19 03:37:57 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T03:37:57.526Z] 2024-01-19 03:37:57 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] start destroying mysql-ibm app [2024-01-19T03:37:57.782Z] 2024-01-19 03:37:57 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T03:37:57.782Z] 2024-01-19 03:37:57 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:37:57.782Z] 2024-01-19 03:37:57 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T03:37:57.782Z] INFO[2024-01-19 03:37:57] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T03:37:57.782Z] INFO[2024-01-19 03:37:57] Actual:true, Expected: true [2024-01-19T03:37:57.783Z] 2024-01-19 03:37:57 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T03:37:58.038Z] 2024-01-19 03:37:57 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T03:37:58.038Z] 2024-01-19 03:37:57 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] start destroying mysql-ibm app [2024-01-19T03:37:58.308Z] 2024-01-19 03:37:58 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T03:37:58.308Z] 2024-01-19 03:37:58 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:37:58.308Z] 2024-01-19 03:37:58 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T03:37:58.308Z] INFO[2024-01-19 03:37:58] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T03:37:58.308Z] INFO[2024-01-19 03:37:58] Actual:true, Expected: true [2024-01-19T03:37:58.308Z] 2024-01-19 03:37:58 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T03:37:58.309Z] INFO[2024-01-19 03:37:58] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T03:37:58.309Z] INFO[2024-01-19 03:37:58] Actual:true, Expected: true [2024-01-19T03:37:58.566Z] INFO[2024-01-19 03:37:58] Verifying : Description : Deleting cluster source-cluster [2024-01-19T03:37:58.566Z] INFO[2024-01-19 03:37:58] Actual:true, Expected: true [2024-01-19T03:37:59.932Z] INFO[2024-01-19 03:37:59] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-lkeso] [2024-01-19T03:37:59.932Z] INFO[2024-01-19 03:37:59] Actual:true, Expected: true [2024-01-19T03:37:59.932Z] INFO[2024-01-19 03:37:59] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T03:37:59.932Z] INFO[2024-01-19 03:37:59] Actual:true, Expected: true [2024-01-19T03:38:01.305Z] INFO[2024-01-19 03:38:01] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-mgglf] [2024-01-19T03:38:01.305Z] INFO[2024-01-19 03:38:01] Actual:true, Expected: true [2024-01-19T03:38:01.305Z] INFO[2024-01-19 03:38:01] --------Test End------ [2024-01-19T03:38:01.305Z] INFO[2024-01-19 03:38:01] #Test: [2024-01-19T03:38:01.305Z] INFO[2024-01-19 03:38:01] #Description: [2024-01-19T03:38:01.305Z] INFO[2024-01-19 03:38:01] ------------------------ [2024-01-19T03:38:01.305Z] 2024-01-19 03:38:01 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.EndPxBackupTorpedoTest:#7261] - >>>> FAILED TEST: {BackupMultipleNsWithSameLabel} Taking backup and restoring multiple namespace having same labels [2024-01-19T03:38:01.305Z] 2024-01-19 03:38:01 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-84851-01-19-03h25m38s] [2024-01-19T03:38:01.561Z] 2024-01-19 03:38:01 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-1-84851-01-19-03h25m38s] [2024-01-19T03:38:01.561Z] 2024-01-19 03:38:01 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-2-84851-01-19-03h25m38s] [2024-01-19T03:38:01.816Z] 2024-01-19 03:38:01 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-3-84851-01-19-03h25m38s] [2024-01-19T03:38:02.073Z] 2024-01-19 03:38:01 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-4-84851-01-19-03h25m38s] [2024-01-19T03:38:02.328Z] 2024-01-19 03:38:02 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-5-84851-01-19-03h25m38s] [2024-01-19T03:38:02.583Z] 2024-01-19 03:38:02 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-6-84851-01-19-03h25m38s] [2024-01-19T03:38:02.583Z] 2024-01-19 03:38:02 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-7-84851-01-19-03h25m38s] [2024-01-19T03:38:02.841Z] 2024-01-19 03:38:02 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-8-84851-01-19-03h25m38s] [2024-01-19T03:38:03.097Z] 2024-01-19 03:38:02 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-9-84851-01-19-03h25m38s] [2024-01-19T03:38:03.353Z] 2024-01-19 03:38:03 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T03:38:03.353Z] 2024-01-19 03:38:03 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T03:38:03.609Z] 2024-01-19 03:38:03 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-84851-01-19-03h25m38s] [2024-01-19T03:38:03.609Z] 2024-01-19 03:38:03 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-1-84851-01-19-03h25m38s] [2024-01-19T03:38:03.865Z] 2024-01-19 03:38:03 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-2-84851-01-19-03h25m38s] [2024-01-19T03:38:04.121Z] 2024-01-19 03:38:03 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-3-84851-01-19-03h25m38s] [2024-01-19T03:38:04.378Z] 2024-01-19 03:38:04 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-4-84851-01-19-03h25m38s] [2024-01-19T03:38:04.378Z] 2024-01-19 03:38:04 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-5-84851-01-19-03h25m38s] [2024-01-19T03:38:04.634Z] 2024-01-19 03:38:04 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-6-84851-01-19-03h25m38s] [2024-01-19T03:38:04.890Z] 2024-01-19 03:38:04 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-7-84851-01-19-03h25m38s] [2024-01-19T03:38:05.146Z] 2024-01-19 03:38:04 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-8-84851-01-19-03h25m38s] [2024-01-19T03:38:05.146Z] 2024-01-19 03:38:05 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-9-84851-01-19-03h25m38s] [2024-01-19T03:38:05.402Z] 2024-01-19 03:38:05 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {BackupMultipleNsWithSameLabel} Taking backup and restoring multiple namespace having same labels [2024-01-19T03:38:05.402Z] 2024-01-19 03:38:05 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T03:38:05.402Z] 2024/01/19 03:38:05 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:38:11.618Z] 2024/01/19 03:38:10 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:38:15.783Z] 2024/01/19 03:38:15 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:38:21.033Z] 2024/01/19 03:38:20 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:38:26.280Z] 2024/01/19 03:38:25 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:38:30.448Z] 2024/01/19 03:38:30 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:38:35.694Z] 2024/01/19 03:38:35 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:38:40.944Z] 2024/01/19 03:38:40 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:38:46.187Z] 2024/01/19 03:38:45 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.0.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.8 10.240.0.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T03:38:46.187Z] {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T03:38:46.187Z] {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T03:38:46.187Z] {dial tcp 10.240.0.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T03:38:50.362Z] 2024-01-19 03:38:50 +0000:[WARNING] [{BackupMultipleNsWithSameLabel}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.8 10.240.0.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T03:38:50.362Z] {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T03:38:50.362Z] {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T03:38:50.362Z] {dial tcp 10.240.0.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T03:38:50.362Z] 2024-01-19 03:38:50 +0000:[ERROR] [{BackupMultipleNsWithSameLabel}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.8 10.240.0.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T03:38:50.362Z] {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T03:38:50.362Z] {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T03:38:50.362Z] {dial tcp 10.240.0.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T03:38:50.362Z] 2024-01-19 03:38:50 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T03:38:50.362Z] 2024-01-19 03:38:50 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] Switching context to [/tmp/source-config] [2024-01-19T03:38:50.618Z] 2024-01-19 03:38:50 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T03:38:50.618Z] 2024-01-19 03:38:50 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T03:38:50.619Z] 2024-01-19 03:38:50 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T03:38:50.875Z] 2024-01-19 03:38:50 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T03:38:50.875Z] 2024-01-19 03:38:50 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T03:38:50.875Z] 2024-01-19 03:38:50 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T03:38:51.436Z] 2024-01-19 03:38:51 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T03:38:51.692Z] 2024-01-19 03:38:51 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T03:38:52.255Z] 2024-01-19 03:38:51 +0000:[INFO] [{BackupMultipleNsWithSameLabel}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T03:38:52.255Z] 2024/01/19 03:38:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:39:04.424Z] 2024/01/19 03:39:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:39:12.508Z] 2024/01/19 03:39:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:39:22.451Z] 2024/01/19 03:39:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:39:32.392Z] 2024/01/19 03:39:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:39:42.336Z] 2024/01/19 03:39:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:39:54.566Z] 2024/01/19 03:39:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:40:02.649Z] 2024/01/19 03:40:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:40:12.658Z] 2024/01/19 03:40:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:40:22.611Z] 2024/01/19 03:40:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:40:32.553Z] 2024/01/19 03:40:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:40:44.747Z] 2024/01/19 03:40:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:40:52.835Z] 2024/01/19 03:40:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:41:02.791Z] 2024/01/19 03:41:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:41:12.790Z] 2024/01/19 03:41:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:41:22.746Z] 2024/01/19 03:41:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:41:32.791Z] 2024/01/19 03:41:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:41:42.783Z] 2024/01/19 03:41:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:41:54.953Z] 2024/01/19 03:41:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:42:03.042Z] 2024/01/19 03:42:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:42:12.987Z] 2024/01/19 03:42:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:42:22.934Z] 2024/01/19 03:42:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:42:35.133Z] 2024/01/19 03:42:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:42:43.221Z] 2024/01/19 03:42:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:42:53.269Z] 2024/01/19 03:42:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:43:03.219Z] 2024/01/19 03:43:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:43:13.169Z] 2024/01/19 03:43:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:43:25.350Z] 2024/01/19 03:43:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:43:33.436Z] 2024/01/19 03:43:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:43:43.377Z] 2024/01/19 03:43:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:43:53.328Z] 2024-01-19 03:43:53 +0000:[ERROR] [{BackupMultipleNsWithSameLabel}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.328Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.328Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.328Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.328Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.328Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.328Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.329Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.329Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.329Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.329Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.329Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.329Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.329Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.329Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.329Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.329Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.329Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.329Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.329Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.329Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.329Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.329Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.329Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.329Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.329Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.329Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.330Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.330Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:43:53.330Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T03:43:53.330Z] • Failure [1094.587 seconds] [2024-01-19T03:43:53.330Z] {BackupMultipleNsWithSameLabel} [2024-01-19T03:43:53.330Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:1336 [2024-01-19T03:43:53.330Z] Taking backup and restoring multiple namespace having same labels [It] [2024-01-19T03:43:53.330Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:1373 [2024-01-19T03:43:53.330Z] [2024-01-19T03:43:53.330Z] Unexpected error: [2024-01-19T03:43:53.330Z] <*scheduler.ErrFailedToValidateApp | 0xc001b24768>: { [2024-01-19T03:43:53.330Z] App: { [2024-01-19T03:43:53.330Z] Key: "mysql-ibm", [2024-01-19T03:43:53.330Z] SpecList: [ [2024-01-19T03:43:53.330Z] <*v1.StorageClass | 0xc0021ce000>{ [2024-01-19T03:43:53.330Z] TypeMeta: {Kind: "StorageClass", APIVersion: ""}, [2024-01-19T03:43:53.330Z] ObjectMeta: { [2024-01-19T03:43:53.330Z] Name: "mysql-sc", [2024-01-19T03:43:53.330Z] GenerateName: "", [2024-01-19T03:43:53.330Z] Namespace: "", [2024-01-19T03:43:53.330Z] SelfLink: "", [2024-01-19T03:43:53.330Z] UID: "7b28f014-7fef-4c59-9607-27a8741f51bd", [2024-01-19T03:43:53.330Z] ResourceVersion: "1548474", [2024-01-19T03:43:53.330Z] Generation: 0, [2024-01-19T03:43:53.330Z] CreationTimestamp: { [2024-01-19T03:43:53.330Z] Time: 2024-01-19T02:09:01Z, [2024-01-19T03:43:53.330Z] }, [2024-01-19T03:43:53.330Z] DeletionTimestamp: nil, [2024-01-19T03:43:53.330Z] DeletionGracePeriodSeconds: nil, [2024-01-19T03:43:53.330Z] Labels: nil, [2024-01-19T03:43:53.330Z] Annotations: { [2024-01-19T03:43:53.330Z] "kubectl.kubernetes.io/last-applied-configuration": "{\"allowVolumeExpansion\":true,\"apiVersion\":\"storage.k8s.io/v1\",\"kind\":\"StorageClass\",\"metadata\":{\"annotations\":{},\"name\":\"mysql-sc\"},\"parameters\":{\"billingType\":\"hourly\",\"classVersion\":\"1\",\"csi.storage.k8s.io/fstype\":\"ext4\",\"encrypted\":\"false\",\"encryptionKey\":\"\",\"profile\":\"10iops-tier\",\"region\":\"\",\"resourceGroup\":\"\",\"tags\":\"\",\"zone\":\"\"},\"provisioner\":\"vpc.block.csi.ibm.io\",\"reclaimPolicy\":\"Delete\",\"volumeBindingMode\":\"Immediate\"}\n", [2024-01-19T03:43:53.330Z] }, [2024-01-19T03:43:53.330Z] OwnerReferences: nil, [2024-01-19T03:43:53.330Z] Finalizers: nil, [2024-01-19T03:43:53.330Z] ManagedFields: [ [2024-01-19T03:43:53.330Z] { [2024-01-19T03:43:53.330Z] Manager: "stork", [2024-01-19T03:43:53.330Z] Operation: "Update", [2024-01-19T03:43:53.330Z] APIVersion: "storage.k8s.io/v1", [2024-01-19T03:43:53.330Z] Time: { [2024-01-19T03:43:53.330Z] Time: 2024-01-19T02:09:01Z, [2024-01-19T03:43:53.330Z] }, [2024-01-19T03:43:53.330Z] FieldsType: "FieldsV1", [2024-01-19T03:43:53.330Z] FieldsV1: { [2024-01-19T03:43:53.331Z] Raw: "{\"f:allowVolumeExpansion\":{},\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:kubectl.kubernetes.io/last-applied-configuration\":{}}},\"f:parameters\":{\".\":{},\"f:billingType\":{},\"f:classVersion\":{},\"f:csi.storage.k8s.io/fstype\":{},\"f:encrypted\":{},\"f:encryptionKey\":{},\"f:profile\":{},\"f:region\":{},\"f:resourceGroup\":{},\"f:tags\":{},\"f:zone\":{}},\"f:provisioner\":{},\"f:reclaimPolicy\":{},\"f:volumeBindingMode\":{}}", [2024-01-19T03:43:53.331Z] }, [2024-01-19T03:43:53.331Z] Subresource: "", [2024-01-19T03:43:53.331Z] }, [2024-01-19T03:43:53.331Z] ], [2024-01-19T03:43:53.331Z] }, [2024-01-19T03:43:53.331Z] Provisioner: "vpc.block.csi.ibm.io", [2024-01-19T03:43:53.331Z] Parameters: { [2024-01-19T03:43:53.331Z] "region": "", [2024-01-19T03:43:53.331Z] "resourceGroup": "", [2024-01-19T03:43:53.331Z] "tags": "", [2024-01-19T03:43:53.331Z] "zone": "", [2024-01-19T03:43:53.331Z] "billingType": "hourly", [2024-01-19T03:43:53.331Z] "classVersion": "1", [2024-01-19T03:43:53.331Z] "encryptionKey": "", [2024-01-19T03:43:53.331Z] "csi.storage.k8s.io/fstype": "ext4", [2024-01-19T03:43:53.331Z] "encrypted": "false", [2024-01-19T03:43:53.331Z] "profile": "10iops-tier", [2024-01-19T03:43:53.331Z] }, [2024-01-19T03:43:53.331Z] ReclaimPolicy: "Delete", [2024-01-19T03:43:53.331Z] MountOptions: nil, [2024-01-19T03:43:53.331Z] AllowVolumeExpansion: true, [2024-01-19T03:43:53.331Z] VolumeBindingMode: "Immediate", [2024-01-19T03:43:53.331Z] AllowedTopologies: nil, [2024-01-19T03:43:53.331Z] }, [2024-01-19T03:43:53.331Z] <*v1.PersistentVolumeClaim | 0xc000ee36c0>{ [2024-01-19T03:43:53.331Z] TypeMeta: { [2024-01-19T03:43:53.331Z] Kind: "PersistentVolumeClaim", [2024-01-19T03:43:53.331Z] APIVersion: "", [2024-01-19T03:43:53.331Z] }, [2024-01-19T03:43:53.331Z] ObjectMeta: { [2024-01-19T03:43:53.331Z] Name: "mysql-data", [2024-01-19T03:43:53.331Z] GenerateName: "", [2024-01-19T03:43:53.331Z] Namespace: "mysql-ibm-pxb-5-84851-01-19-03h25m38s", [2024-01-19T03:43:53.331Z] SelfLink: "", [2024-01-19T03:43:53.331Z] UID: "c6180d49-7015-4e39-82c0-13ae... [2024-01-19T03:43:53.331Z] [2024-01-19T03:43:53.331Z] Gomega truncated this representation as it exceeds 'format.MaxLength'. [2024-01-19T03:43:53.331Z] Consider having the object provide a custom 'GomegaStringer' representation [2024-01-19T03:43:53.331Z] or adjust the parameters in Gomega's 'format' package. [2024-01-19T03:43:53.331Z] [2024-01-19T03:43:53.331Z] Learn more here: https://onsi.github.io/gomega/#adjusting-output [2024-01-19T03:43:53.331Z] [2024-01-19T03:43:53.331Z] Failed to validate app: mysql-ibm due to err: Failed to validate Deployment: mysql,Namespace: mysql-ibm-pxb-5-84851-01-19-03h25m38s. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.331Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.331Z] } [2024-01-19T03:43:53.331Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.331Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.331Z] } [2024-01-19T03:43:53.331Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.331Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.331Z] } [2024-01-19T03:43:53.331Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.331Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.331Z] } [2024-01-19T03:43:53.332Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.332Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.332Z] } [2024-01-19T03:43:53.332Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.332Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.332Z] } [2024-01-19T03:43:53.332Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.332Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.332Z] } [2024-01-19T03:43:53.332Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.332Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.332Z] } [2024-01-19T03:43:53.332Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.332Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.332Z] } [2024-01-19T03:43:53.332Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.332Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.332Z] } [2024-01-19T03:43:53.332Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.332Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.332Z] } [2024-01-19T03:43:53.332Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.332Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.332Z] } [2024-01-19T03:43:53.332Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.332Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.332Z] } [2024-01-19T03:43:53.332Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.332Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.332Z] } [2024-01-19T03:43:53.332Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.332Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.332Z] } [2024-01-19T03:43:53.332Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.332Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.332Z] } [2024-01-19T03:43:53.332Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.332Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.332Z] } [2024-01-19T03:43:53.332Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.332Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.332Z] } [2024-01-19T03:43:53.332Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.333Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.333Z] } [2024-01-19T03:43:53.333Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.333Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.333Z] } [2024-01-19T03:43:53.333Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.333Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.333Z] } [2024-01-19T03:43:53.333Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.333Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.333Z] } [2024-01-19T03:43:53.333Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.333Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.333Z] } [2024-01-19T03:43:53.333Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.333Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.333Z] } [2024-01-19T03:43:53.333Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.333Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.333Z] } [2024-01-19T03:43:53.333Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.333Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.333Z] } [2024-01-19T03:43:53.333Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.333Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.333Z] } [2024-01-19T03:43:53.333Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.333Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.333Z] } [2024-01-19T03:43:53.333Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.333Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.333Z] } [2024-01-19T03:43:53.333Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.333Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.333Z] } [2024-01-19T03:43:53.333Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.333Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.333Z] } [2024-01-19T03:43:53.333Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.333Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.333Z] } [2024-01-19T03:43:53.333Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.333Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.333Z] } [2024-01-19T03:43:53.333Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.333Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.334Z] } [2024-01-19T03:43:53.334Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.334Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.334Z] } [2024-01-19T03:43:53.334Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.334Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.334Z] } [2024-01-19T03:43:53.334Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.334Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.334Z] } [2024-01-19T03:43:53.334Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.334Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.334Z] } [2024-01-19T03:43:53.334Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.334Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.334Z] } [2024-01-19T03:43:53.334Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.334Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.334Z] } [2024-01-19T03:43:53.334Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.334Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.334Z] } [2024-01-19T03:43:53.334Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.334Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.334Z] } [2024-01-19T03:43:53.334Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.334Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.334Z] } [2024-01-19T03:43:53.334Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.334Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.334Z] } [2024-01-19T03:43:53.334Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.334Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.334Z] } [2024-01-19T03:43:53.334Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.334Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.334Z] } [2024-01-19T03:43:53.334Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.334Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.334Z] } [2024-01-19T03:43:53.334Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.334Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.334Z] } [2024-01-19T03:43:53.334Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.334Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.334Z] } [2024-01-19T03:43:53.334Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.334Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.335Z] } [2024-01-19T03:43:53.335Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.335Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.335Z] } [2024-01-19T03:43:53.335Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.335Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.335Z] } [2024-01-19T03:43:53.335Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.335Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.335Z] } [2024-01-19T03:43:53.335Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.335Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.335Z] } [2024-01-19T03:43:53.335Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.335Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.335Z] } [2024-01-19T03:43:53.335Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.335Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.335Z] } [2024-01-19T03:43:53.335Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.335Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.335Z] } [2024-01-19T03:43:53.335Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.335Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.335Z] } [2024-01-19T03:43:53.335Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.335Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.335Z] } [2024-01-19T03:43:53.335Z] {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:43:53.335Z] pod name:mysql-7d766cd65c-l7fkm namespace:mysql-ibm-pxb-5-84851-01-19-03h25m38s running:true ready:false node: [2024-01-19T03:43:53.335Z] } [2024-01-19T03:43:53.335Z] occurred [2024-01-19T03:43:53.335Z] [2024-01-19T03:43:53.335Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T03:43:53.335Z] [2024-01-19T03:43:53.335Z] Full Stack Trace [2024-01-19T03:43:53.335Z] github.com/portworx/torpedo/pkg/log.FailOnError({0x88ae500, 0xc001b24768}, {0x7eb1033?, 0xc001a1a000?}, {0x0?, 0x6cc0320?, 0xc000f2a290?}) [2024-01-19T03:43:53.335Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 +0x290 [2024-01-19T03:43:53.335Z] github.com/portworx/torpedo/tests.processError({0x88ae500?, 0xc001b24768?}, {0x0?, 0x0?, 0x0?}) [2024-01-19T03:43:53.335Z] /go/src/github.com/portworx/torpedo/tests/common.go:600 +0x48 [2024-01-19T03:43:53.335Z] github.com/portworx/torpedo/tests.ValidateContext.func2.2() [2024-01-19T03:43:53.335Z] /go/src/github.com/portworx/torpedo/tests/common.go:668 +0xc5 [2024-01-19T03:43:53.335Z] github.com/portworx/torpedo/tests.ValidateContext.func2() [2024-01-19T03:43:53.335Z] /go/src/github.com/portworx/torpedo/tests/common.go:663 +0x37c [2024-01-19T03:43:53.335Z] github.com/portworx/torpedo/tests.ValidateContext(0xc000d8fb00, {0x0, 0x0, 0x0}) [2024-01-19T03:43:53.335Z] /go/src/github.com/portworx/torpedo/tests/common.go:644 +0x169 [2024-01-19T03:43:53.335Z] github.com/portworx/torpedo/tests.ValidateApplications.func1() [2024-01-19T03:43:53.335Z] /go/src/github.com/portworx/torpedo/tests/common.go:1902 +0x5f [2024-01-19T03:43:53.335Z] github.com/portworx/torpedo/tests.ValidateApplications({0xc000fd9580, 0xa, 0x10}) [2024-01-19T03:43:53.335Z] /go/src/github.com/portworx/torpedo/tests/common.go:1899 +0xb9 [2024-01-19T03:43:53.335Z] github.com/portworx/torpedo/tests/backup.glob..func62.2.1() [2024-01-19T03:43:53.335Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:1377 +0x4a [2024-01-19T03:43:53.335Z] github.com/portworx/torpedo/tests/backup.glob..func62.2() [2024-01-19T03:43:53.336Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:1375 +0x197 [2024-01-19T03:43:53.336Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T03:43:53.336Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T03:43:53.336Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T03:43:53.336Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T03:43:53.336Z] created by testing.(*T).Run [2024-01-19T03:43:53.336Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T03:43:53.336Z] ------------------------------ [2024-01-19T03:43:53.336Z] STEP: schedule applications [2024-01-19T03:43:53.336Z] STEP: schedule applications [2024-01-19T03:43:53.336Z] STEP: schedule applications [2024-01-19T03:43:53.336Z] STEP: schedule applications [2024-01-19T03:43:53.336Z] STEP: schedule applications [2024-01-19T03:43:53.336Z] STEP: schedule applications [2024-01-19T03:43:53.336Z] STEP: schedule applications [2024-01-19T03:43:53.336Z] STEP: schedule applications [2024-01-19T03:43:53.336Z] STEP: schedule applications [2024-01-19T03:43:53.336Z] STEP: schedule applications [2024-01-19T03:43:53.336Z] STEP: Validate applications [2024-01-19T03:43:53.336Z] STEP: validate applications [2024-01-19T03:43:53.336Z] STEP: validate mysql-ibm app's volumes [2024-01-19T03:43:53.336Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T03:43:53.336Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T03:43:53.336Z] STEP: get mysql-ibm app's volume: pvc-50476e22-eb11-4390-a4fd-f9d7c465a7a6 inspected by the volume driver [2024-01-19T03:43:53.336Z] STEP: wait for mysql-ibm app to start running [2024-01-19T03:43:53.336Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T03:43:53.336Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T03:43:53.336Z] STEP: Validate Px pod restart count [2024-01-19T03:43:53.336Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T03:43:53.336Z] STEP: validate mysql-ibm app's volumes [2024-01-19T03:43:53.336Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T03:43:53.336Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T03:43:53.336Z] STEP: get mysql-ibm app's volume: pvc-bcfb05d0-6934-4c79-946d-5bdb0bbf09f4 inspected by the volume driver [2024-01-19T03:43:53.336Z] STEP: wait for mysql-ibm app to start running [2024-01-19T03:43:53.336Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T03:43:53.336Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T03:43:53.336Z] STEP: Validate Px pod restart count [2024-01-19T03:43:53.336Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T03:43:53.336Z] STEP: validate mysql-ibm app's volumes [2024-01-19T03:43:53.336Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T03:43:53.336Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T03:43:53.336Z] STEP: get mysql-ibm app's volume: pvc-5ce533bf-5d74-41f0-9fb9-abf6ffbf77e2 inspected by the volume driver [2024-01-19T03:43:53.336Z] STEP: wait for mysql-ibm app to start running [2024-01-19T03:43:53.336Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T03:43:53.336Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T03:43:53.336Z] STEP: Validate Px pod restart count [2024-01-19T03:43:53.336Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T03:43:53.336Z] STEP: validate mysql-ibm app's volumes [2024-01-19T03:43:53.336Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T03:43:53.336Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T03:43:53.336Z] STEP: get mysql-ibm app's volume: pvc-83ed1e7e-b342-44ac-a9c3-4fa9fe290627 inspected by the volume driver [2024-01-19T03:43:53.336Z] STEP: wait for mysql-ibm app to start running [2024-01-19T03:43:53.336Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T03:43:53.336Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T03:43:53.336Z] STEP: Validate Px pod restart count [2024-01-19T03:43:53.336Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T03:43:53.336Z] STEP: validate mysql-ibm app's volumes [2024-01-19T03:43:53.336Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T03:43:53.336Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T03:43:53.336Z] STEP: get mysql-ibm app's volume: pvc-4ae1a253-dd57-4c3c-9555-f2cbd9b4a2ee inspected by the volume driver [2024-01-19T03:43:53.336Z] STEP: wait for mysql-ibm app to start running [2024-01-19T03:43:53.336Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T03:43:53.337Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T03:43:53.337Z] STEP: Validate Px pod restart count [2024-01-19T03:43:53.337Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T03:43:53.337Z] STEP: validate mysql-ibm app's volumes [2024-01-19T03:43:53.337Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T03:43:53.337Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T03:43:53.337Z] STEP: get mysql-ibm app's volume: pvc-c6180d49-7015-4e39-82c0-13ae8a4c5037 inspected by the volume driver [2024-01-19T03:43:53.337Z] STEP: wait for mysql-ibm app to start running [2024-01-19T03:43:53.337Z] STEP: destroy apps [2024-01-19T03:43:53.337Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T03:43:53.337Z] STEP: start destroying mysql-ibm app [2024-01-19T03:43:53.337Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:43:53.337Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T03:43:53.337Z] STEP: start destroying mysql-ibm app [2024-01-19T03:43:53.337Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:43:53.337Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T03:43:53.337Z] STEP: start destroying mysql-ibm app [2024-01-19T03:43:53.337Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:43:53.337Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T03:43:53.337Z] STEP: start destroying mysql-ibm app [2024-01-19T03:43:53.337Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:43:53.337Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T03:43:53.337Z] STEP: start destroying mysql-ibm app [2024-01-19T03:43:53.337Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:43:53.337Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T03:43:53.337Z] STEP: start destroying mysql-ibm app [2024-01-19T03:43:53.337Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:43:53.337Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T03:43:53.337Z] STEP: start destroying mysql-ibm app [2024-01-19T03:43:53.337Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:43:53.337Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T03:43:53.337Z] STEP: start destroying mysql-ibm app [2024-01-19T03:43:53.337Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:43:53.337Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T03:43:53.337Z] STEP: start destroying mysql-ibm app [2024-01-19T03:43:53.337Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:43:53.337Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T03:43:53.337Z] STEP: start destroying mysql-ibm app [2024-01-19T03:43:53.337Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:43:53.337Z] 2024-01-19 03:43:53 +0000:[INFO] [{CancelClusterBackupShare}] Log Dir: /testresults//CancelClusterBackupShare.log [2024-01-19T03:43:53.337Z] INFO[2024-01-19 03:43:53] --------Test Start------ [2024-01-19T03:43:53.337Z] INFO[2024-01-19 03:43:53] #Test: CancelClusterBackupShare [2024-01-19T03:43:53.337Z] INFO[2024-01-19 03:43:53] #Description: Share all backups at cluster level with a user group and revoke it and validate [2024-01-19T03:43:53.337Z] INFO[2024-01-19 03:43:53] ------------------------ [2024-01-19T03:43:53.337Z] 2024-01-19 03:43:53 +0000:[INFO] [{CancelClusterBackupShare}] Deploy applications [2024-01-19T03:43:53.337Z] 2024-01-19 03:43:53 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T03:43:53.337Z] 2024-01-19 03:43:53 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T03:43:53.337Z] 2024-01-19 03:43:53 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T03:43:53.593Z] 2024-01-19 03:43:53 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T03:43:53.594Z] 2024-01-19 03:43:53 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T03:43:53.594Z] 2024-01-19 03:43:53 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T03:43:53.594Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T03:43:53.594Z] 2024-01-19 03:43:53 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T03:43:53.595Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T03:43:53.595Z] 2024-01-19 03:43:53 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T03:43:53.595Z] 2024-01-19 03:43:53 +0000:[INFO] [{CancelClusterBackupShare}] Validate applications [2024-01-19T03:43:53.595Z] 2024-01-19 03:43:53 +0000:[INFO] [{CancelClusterBackupShare}] Validate applications [2024-01-19T03:43:53.595Z] 2024-01-19 03:43:53 +0000:[INFO] [{CancelClusterBackupShare}] Validating mysql-ibm app [2024-01-19T03:43:53.595Z] 2024-01-19 03:43:53 +0000:[INFO] [{CancelClusterBackupShare}] Validating mysql-ibm app's volumes [2024-01-19T03:43:53.595Z] 2024-01-19 03:43:53 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:43:53.851Z] 2024/01/19 03:43:53 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T03:44:03.820Z] 2024/01/19 03:44:03 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T03:44:13.763Z] 2024-01-19 03:44:13 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-82935-01-19-03h43m53s [2024-01-19T03:44:14.019Z] 2024-01-19 03:44:13 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T03:44:14.019Z] 2024-01-19 03:44:13 +0000:[WARNING] [{CancelClusterBackupShare}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T03:44:14.019Z] 2024-01-19 03:44:13 +0000:[INFO] [{CancelClusterBackupShare}] wait for mysql-ibm app to start running [2024-01-19T03:44:14.019Z] 2024-01-19 03:44:13 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T03:44:14.019Z] 2024/01/19 03:44:13 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:44:14.019Z] pod name:mysql-7d766cd65c-jlgfd namespace:mysql-ibm-pxb-0-82935-01-19-03h43m53s running:false ready:false node:10.240.64.8 [2024-01-19T03:44:14.019Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:44:26.211Z] 2024/01/19 03:44:23 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:44:26.211Z] pod name:mysql-7d766cd65c-jlgfd namespace:mysql-ibm-pxb-0-82935-01-19-03h43m53s running:true ready:false node:10.240.64.8 [2024-01-19T03:44:26.211Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:44:34.301Z] 2024/01/19 03:44:34 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:44:34.301Z] pod name:mysql-7d766cd65c-jlgfd namespace:mysql-ibm-pxb-0-82935-01-19-03h43m53s running:true ready:false node:10.240.64.8 [2024-01-19T03:44:34.301Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:44:44.253Z] 2024/01/19 03:44:44 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:44:44.253Z] pod name:mysql-7d766cd65c-jlgfd namespace:mysql-ibm-pxb-0-82935-01-19-03h43m53s running:true ready:false node:10.240.64.8 [2024-01-19T03:44:44.253Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:44:52.335Z] INFO[2024-01-19 03:44:51] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T03:44:52.335Z] INFO[2024-01-19 03:44:51] sched stats: tick interval: 43851 < 1.2s (100%) [2024-01-19T03:44:52.335Z] INFO[2024-01-19 03:44:51] sched stats: tick process durations: 43850 < 500ms (100%) [2024-01-19T03:44:54.855Z] 2024/01/19 03:44:54 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:44:54.855Z] pod name:mysql-7d766cd65c-jlgfd namespace:mysql-ibm-pxb-0-82935-01-19-03h43m53s running:true ready:false node:10.240.64.8 [2024-01-19T03:44:54.855Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:45:04.808Z] 2024/01/19 03:45:04 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:45:04.808Z] pod name:mysql-7d766cd65c-jlgfd namespace:mysql-ibm-pxb-0-82935-01-19-03h43m53s running:true ready:false node:10.240.64.8 [2024-01-19T03:45:04.808Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:45:14.754Z] 2024/01/19 03:45:14 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:45:14.754Z] pod name:mysql-7d766cd65c-jlgfd namespace:mysql-ibm-pxb-0-82935-01-19-03h43m53s running:true ready:false node:10.240.64.8 [2024-01-19T03:45:14.754Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:45:24.698Z] 2024/01/19 03:45:24 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:45:24.698Z] pod name:mysql-7d766cd65c-jlgfd namespace:mysql-ibm-pxb-0-82935-01-19-03h43m53s running:true ready:false node:10.240.64.8 [2024-01-19T03:45:24.698Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:45:36.897Z] 2024-01-19 03:45:34 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T03:45:36.897Z] 2024-01-19 03:45:34 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T03:45:36.897Z] 2024-01-19 03:45:34 +0000:[INFO] [{CancelClusterBackupShare}] validate if mysql-ibm app's volumes are setup [2024-01-19T03:45:36.897Z] 2024-01-19 03:45:34 +0000:[WARNING] [{CancelClusterBackupShare}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:45:36.897Z] 2024-01-19 03:45:34 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:45:36.897Z] 2024-01-19 03:45:34 +0000:[INFO] [{CancelClusterBackupShare}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T03:45:36.897Z] 2024-01-19 03:45:34 +0000:[WARNING] [{CancelClusterBackupShare}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T03:45:36.897Z] 2024-01-19 03:45:35 +0000:[INFO] [{CancelClusterBackupShare}] Creating 10 users to be added to the group [2024-01-19T03:45:37.459Z] 2024-01-19 03:45:37 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddUser:#728] - Response for user [testuser1] creation - [2024-01-19T03:45:37.459Z] 2024-01-19 03:45:37 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddUser:#733] - User [testuser1] created [2024-01-19T03:45:39.351Z] 2024-01-19 03:45:39 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddUser:#728] - Response for user [testuser2] creation - [2024-01-19T03:45:39.351Z] 2024-01-19 03:45:39 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddUser:#733] - User [testuser2] created [2024-01-19T03:45:41.867Z] 2024-01-19 03:45:41 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddUser:#728] - Response for user [testuser3] creation - [2024-01-19T03:45:41.867Z] 2024-01-19 03:45:41 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddUser:#733] - User [testuser3] created [2024-01-19T03:45:43.764Z] 2024-01-19 03:45:43 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddUser:#728] - Response for user [testuser4] creation - [2024-01-19T03:45:43.764Z] 2024-01-19 03:45:43 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddUser:#733] - User [testuser4] created [2024-01-19T03:45:45.655Z] 2024-01-19 03:45:45 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddUser:#728] - Response for user [testuser5] creation - [2024-01-19T03:45:45.655Z] 2024-01-19 03:45:45 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddUser:#733] - User [testuser5] created [2024-01-19T03:45:47.545Z] 2024-01-19 03:45:47 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddUser:#728] - Response for user [testuser6] creation - [2024-01-19T03:45:47.545Z] 2024-01-19 03:45:47 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddUser:#733] - User [testuser6] created [2024-01-19T03:45:49.438Z] 2024-01-19 03:45:49 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddUser:#728] - Response for user [testuser7] creation - [2024-01-19T03:45:49.438Z] 2024-01-19 03:45:49 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddUser:#733] - User [testuser7] created [2024-01-19T03:45:51.330Z] 2024-01-19 03:45:51 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddUser:#728] - Response for user [testuser8] creation - [2024-01-19T03:45:51.330Z] 2024-01-19 03:45:51 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddUser:#733] - User [testuser8] created [2024-01-19T03:45:53.850Z] 2024-01-19 03:45:53 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddUser:#728] - Response for user [testuser9] creation - [2024-01-19T03:45:53.850Z] 2024-01-19 03:45:53 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddUser:#733] - User [testuser9] created [2024-01-19T03:45:55.741Z] 2024-01-19 03:45:55 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddUser:#728] - Response for user [testuser10] creation - [2024-01-19T03:45:55.742Z] 2024-01-19 03:45:55 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddUser:#733] - User [testuser10] created [2024-01-19T03:45:55.742Z] 2024-01-19 03:45:55 +0000:[INFO] [{CancelClusterBackupShare}] Creating a user with username - [autogenerated-user] who is not part of any group [2024-01-19T03:45:55.742Z] 2024-01-19 03:45:55 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddUser:#728] - Response for user [autogenerated-user] creation - [2024-01-19T03:45:55.742Z] 2024-01-19 03:45:55 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddUser:#733] - User [autogenerated-user] created [2024-01-19T03:45:55.742Z] 2024-01-19 03:45:55 +0000:[INFO] [{CancelClusterBackupShare}] Creating 1 groups [2024-01-19T03:45:56.000Z] 2024-01-19 03:45:55 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddGroup:#989] - Creating group response - [2024-01-19T03:45:56.000Z] 2024-01-19 03:45:55 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddGroup:#994] - Group [testGroup1] created [2024-01-19T03:45:56.000Z] 2024-01-19 03:45:55 +0000:[INFO] [{CancelClusterBackupShare}] Adding users to groups [2024-01-19T03:45:58.531Z] 2024-01-19 03:45:58 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 488b3823-aa13-4e9b-bbc1-5c7907b46ab5 [2024-01-19T03:45:58.531Z] 2024-01-19 03:45:58 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1 - 4e9ea7cd-1326-490a-85c7-d9e61856f1e4 [2024-01-19T03:45:58.531Z] 2024-01-19 03:45:58 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-19T03:45:58.531Z] 2024-01-19 03:45:58 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddGroupToUser:#1119] - Added User [testuser1] to Group [testGroup1] [2024-01-19T03:46:00.431Z] 2024-01-19 03:46:00 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 488b3823-aa13-4e9b-bbc1-5c7907b46ab5 [2024-01-19T03:46:00.431Z] 2024-01-19 03:46:00 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2 - 5e6ccc21-5b96-4d45-b1d0-6a6b5bab6483 [2024-01-19T03:46:00.992Z] 2024-01-19 03:46:00 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-19T03:46:00.992Z] 2024-01-19 03:46:00 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddGroupToUser:#1119] - Added User [testuser2] to Group [testGroup1] [2024-01-19T03:46:02.355Z] 2024-01-19 03:46:01 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 488b3823-aa13-4e9b-bbc1-5c7907b46ab5 [2024-01-19T03:46:02.355Z] 2024-01-19 03:46:02 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser3 - eaf0e1a5-f802-468c-8ece-e121482c86f7 [2024-01-19T03:46:03.283Z] 2024-01-19 03:46:03 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-19T03:46:03.284Z] 2024-01-19 03:46:03 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddGroupToUser:#1119] - Added User [testuser3] to Group [testGroup1] [2024-01-19T03:46:04.214Z] 2024-01-19 03:46:03 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 488b3823-aa13-4e9b-bbc1-5c7907b46ab5 [2024-01-19T03:46:04.780Z] 2024-01-19 03:46:04 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser4 - 46d6f682-9c4b-46bc-a6ae-9926aa2cf49f [2024-01-19T03:46:05.710Z] 2024-01-19 03:46:05 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-19T03:46:05.710Z] 2024-01-19 03:46:05 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddGroupToUser:#1119] - Added User [testuser4] to Group [testGroup1] [2024-01-19T03:46:06.285Z] 2024-01-19 03:46:06 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 488b3823-aa13-4e9b-bbc1-5c7907b46ab5 [2024-01-19T03:46:07.212Z] 2024-01-19 03:46:07 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser5 - 484091d9-264e-4711-a439-f2c90e40c5d9 [2024-01-19T03:46:08.140Z] 2024-01-19 03:46:07 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-19T03:46:08.140Z] 2024-01-19 03:46:07 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddGroupToUser:#1119] - Added User [testuser5] to Group [testGroup1] [2024-01-19T03:46:08.700Z] 2024-01-19 03:46:08 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 488b3823-aa13-4e9b-bbc1-5c7907b46ab5 [2024-01-19T03:46:09.628Z] 2024-01-19 03:46:09 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser6 - d7aaae10-a3c8-437d-9ee6-51f853d7ef6d [2024-01-19T03:46:11.020Z] 2024-01-19 03:46:10 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-19T03:46:11.020Z] 2024-01-19 03:46:10 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddGroupToUser:#1119] - Added User [testuser6] to Group [testGroup1] [2024-01-19T03:46:11.276Z] 2024-01-19 03:46:11 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 488b3823-aa13-4e9b-bbc1-5c7907b46ab5 [2024-01-19T03:46:12.204Z] 2024-01-19 03:46:11 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser7 - 7ea98068-ca40-46db-a151-6398e592f58e [2024-01-19T03:46:13.569Z] 2024-01-19 03:46:13 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 488b3823-aa13-4e9b-bbc1-5c7907b46ab5 [2024-01-19T03:46:13.569Z] 2024-01-19 03:46:13 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-19T03:46:13.569Z] 2024-01-19 03:46:13 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddGroupToUser:#1119] - Added User [testuser7] to Group [testGroup1] [2024-01-19T03:46:14.933Z] 2024-01-19 03:46:14 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser8 - 06bb4316-9885-4247-9861-529a3be2112b [2024-01-19T03:46:15.494Z] 2024-01-19 03:46:15 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 488b3823-aa13-4e9b-bbc1-5c7907b46ab5 [2024-01-19T03:46:16.863Z] 2024-01-19 03:46:16 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-19T03:46:16.863Z] 2024-01-19 03:46:16 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddGroupToUser:#1119] - Added User [testuser8] to Group [testGroup1] [2024-01-19T03:46:17.120Z] 2024-01-19 03:46:17 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser9 - 1297b31b-c0e4-473b-b6af-fc1ab802763d [2024-01-19T03:46:17.681Z] 2024-01-19 03:46:17 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 488b3823-aa13-4e9b-bbc1-5c7907b46ab5 [2024-01-19T03:46:19.046Z] 2024-01-19 03:46:18 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-19T03:46:19.046Z] 2024-01-19 03:46:18 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddGroupToUser:#1119] - Added User [testuser9] to Group [testGroup1] [2024-01-19T03:46:19.302Z] 2024-01-19 03:46:19 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser10 - e91065a4-15d9-4fed-8e03-f29e8cb0bb67 [2024-01-19T03:46:20.230Z] 2024-01-19 03:46:19 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddGroupToUser:#1114] - Adding user to group response - [2024-01-19T03:46:20.230Z] 2024-01-19 03:46:19 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AddGroupToUser:#1119] - Added User [testuser10] to Group [testGroup1] [2024-01-19T03:46:21.158Z] 2024-01-19 03:46:21 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 488b3823-aa13-4e9b-bbc1-5c7907b46ab5 [2024-01-19T03:46:21.721Z] 2024-01-19 03:46:21 +0000:[DEBUG] [{CancelClusterBackupShare}] [backup.GetMembersOfGroup:#961] - list of members : [testuser1 testuser10 testuser2 testuser3 testuser4 testuser5 testuser6 testuser7 testuser8 testuser9] [2024-01-19T03:46:21.721Z] 2024-01-19 03:46:21 +0000:[INFO] [{CancelClusterBackupShare}] [backup.glob..func85.2.4:#1356] - Group [testGroup1] contains the following users: [2024-01-19T03:46:21.721Z] [testuser1 testuser10 testuser2 testuser3 testuser4 testuser5 testuser6 testuser7 testuser8 testuser9] [2024-01-19T03:46:21.721Z] 2024-01-19 03:46:21 +0000:[INFO] [{CancelClusterBackupShare}] Creating cloud credentials and backup location [2024-01-19T03:46:23.093Z] 2024-01-19 03:46:22 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-1705635982] for org [default] with [aws] as provider [2024-01-19T03:46:23.093Z] 2024-01-19 03:46:22 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T03:46:23.093Z] 2024-01-19 03:46:22 +0000:[INFO] [{CancelClusterBackupShare}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-1705635982] is present [2024-01-19T03:46:23.093Z] INFO[2024-01-19 03:46:22] Verifying : Description : Verifying creation of cloud credential named [autogenerated-cred-1705635982] for org [default] with [aws] as provider [2024-01-19T03:46:23.093Z] INFO[2024-01-19 03:46:22] Actual:true, Expected: true [2024-01-19T03:46:23.093Z] 2024-01-19 03:46:22 +0000:[INFO] [{CancelClusterBackupShare}] Created Cloud Credentials with name - autogenerated-cred-1705635982 [2024-01-19T03:46:26.364Z] INFO[2024-01-19 03:46:25] Verifying : Description : Creating backup location autogenerated-backup-location-1705635982 [2024-01-19T03:46:26.364Z] INFO[2024-01-19 03:46:25] Actual:true, Expected: true [2024-01-19T03:46:26.364Z] 2024-01-19 03:46:25 +0000:[INFO] [{CancelClusterBackupShare}] Created Backup Location with name - autogenerated-backup-location-1705635982 [2024-01-19T03:46:26.364Z] 2024-01-19 03:46:25 +0000:[INFO] [{CancelClusterBackupShare}] Registering Source and Destination clusters and verifying the status [2024-01-19T03:46:26.364Z] INFO[2024-01-19 03:46:26] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T03:46:26.364Z] INFO[2024-01-19 03:46:26] Actual:true, Expected: true [2024-01-19T03:46:26.364Z] 2024-01-19 03:46:26 +0000:[INFO] [{CancelClusterBackupShare}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T03:46:26.364Z] 2024-01-19 03:46:26 +0000:[INFO] [{CancelClusterBackupShare}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T03:46:26.365Z] 2024-01-19 03:46:26 +0000:[INFO] [{CancelClusterBackupShare}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T03:46:26.365Z] 2024-01-19 03:46:26 +0000:[INFO] [{CancelClusterBackupShare}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T03:46:26.365Z] 2024-01-19 03:46:26 +0000:[INFO] [{CancelClusterBackupShare}] Create cluster [source-cluster] in org [default] [2024-01-19T03:46:26.365Z] 2024-01-19 03:46:26 +0000:[INFO] [{CancelClusterBackupShare}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T03:46:26.365Z] 2024-01-19 03:46:26 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T03:46:26.365Z] 2024-01-19 03:46:26 +0000:[INFO] [{CancelClusterBackupShare}] Create cluster [destination-cluster] in org [default] [2024-01-19T03:46:26.365Z] 2024-01-19 03:46:26 +0000:[INFO] [{CancelClusterBackupShare}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T03:46:26.365Z] 2024-01-19 03:46:26 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T03:46:26.365Z] 2024-01-19 03:46:26 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-cvqse] for IBM [2024-01-19T03:46:26.365Z] 2024-01-19 03:46:26 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-cvqse] for org [default] with [ibm] as provider [2024-01-19T03:46:26.365Z] 2024-01-19 03:46:26 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T03:46:26.365Z] 2024-01-19 03:46:26 +0000:[INFO] [{CancelClusterBackupShare}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T03:46:26.622Z] 2024-01-19 03:46:26 +0000:[INFO] [{CancelClusterBackupShare}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-cvqse] is present [2024-01-19T03:46:26.622Z] 2024-01-19 03:46:26 +0000:[INFO] [{CancelClusterBackupShare}] Create cluster [source-cluster] in org [default] [2024-01-19T03:46:27.189Z] 2024-01-19 03:46:26 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-jzfsm] for IBM [2024-01-19T03:46:27.189Z] 2024-01-19 03:46:26 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-jzfsm] for org [default] with [ibm] as provider [2024-01-19T03:46:27.189Z] 2024-01-19 03:46:26 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T03:46:27.189Z] 2024-01-19 03:46:26 +0000:[INFO] [{CancelClusterBackupShare}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T03:46:27.189Z] 2024-01-19 03:46:27 +0000:[INFO] [{CancelClusterBackupShare}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-jzfsm] is present [2024-01-19T03:46:27.189Z] 2024-01-19 03:46:27 +0000:[INFO] [{CancelClusterBackupShare}] Create cluster [destination-cluster] in org [default] [2024-01-19T03:46:28.115Z] INFO[2024-01-19 03:46:27] Verifying : Description : Creating source and destination cluster [2024-01-19T03:46:28.115Z] INFO[2024-01-19 03:46:27] Actual:true, Expected: true [2024-01-19T03:46:28.115Z] INFO[2024-01-19 03:46:27] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T03:46:28.115Z] INFO[2024-01-19 03:46:27] Actual:Online, Expected: Online [2024-01-19T03:46:28.115Z] INFO[2024-01-19 03:46:27] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T03:46:28.115Z] INFO[2024-01-19 03:46:27] Actual:true, Expected: true [2024-01-19T03:46:28.115Z] 2024-01-19 03:46:27 +0000:[INFO] [{CancelClusterBackupShare}] Taking backup of applications [2024-01-19T03:46:40.292Z] 2024-01-19 03:46:38 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T03:46:40.292Z] 2024-01-19 03:46:38 +0000:[INFO] [{CancelClusterBackupShare}] Volume snapshot classes found in the cluster - [2024-01-19T03:46:40.292Z] 2024-01-19 03:46:38 +0000:[INFO] [{CancelClusterBackupShare}] ibmc-vpcblock-snapshot [2024-01-19T03:46:40.292Z] 2024-01-19 03:46:38 +0000:[INFO] [{CancelClusterBackupShare}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T03:46:40.293Z] 2024/01/19 03:46:38 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705635998] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T03:46:48.381Z] 2024-01-19 03:46:48 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T03:46:48.381Z] 2024-01-19 03:46:48 +0000:[INFO] [{CancelClusterBackupShare}] Volume snapshot classes found in the cluster - [2024-01-19T03:46:48.381Z] 2024-01-19 03:46:48 +0000:[INFO] [{CancelClusterBackupShare}] ibmc-vpcblock-snapshot [2024-01-19T03:46:48.381Z] 2024-01-19 03:46:48 +0000:[INFO] [{CancelClusterBackupShare}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T03:46:48.637Z] 2024/01/19 03:46:48 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636008] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T03:46:58.577Z] 2024-01-19 03:46:58 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T03:46:58.577Z] 2024-01-19 03:46:58 +0000:[INFO] [{CancelClusterBackupShare}] Volume snapshot classes found in the cluster - [2024-01-19T03:46:58.577Z] 2024-01-19 03:46:58 +0000:[INFO] [{CancelClusterBackupShare}] ibmc-vpcblock-snapshot [2024-01-19T03:46:58.577Z] 2024-01-19 03:46:58 +0000:[INFO] [{CancelClusterBackupShare}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T03:46:58.577Z] 2024/01/19 03:46:58 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636018] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T03:47:08.524Z] 2024-01-19 03:47:08 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T03:47:08.524Z] 2024-01-19 03:47:08 +0000:[INFO] [{CancelClusterBackupShare}] Volume snapshot classes found in the cluster - [2024-01-19T03:47:08.524Z] 2024-01-19 03:47:08 +0000:[INFO] [{CancelClusterBackupShare}] ibmc-vpcblock-snapshot [2024-01-19T03:47:08.524Z] 2024-01-19 03:47:08 +0000:[INFO] [{CancelClusterBackupShare}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T03:47:08.798Z] 2024/01/19 03:47:08 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705635998] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:47:08.798Z] 2024/01/19 03:47:08 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636028] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T03:47:18.767Z] 2024-01-19 03:47:18 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T03:47:18.767Z] 2024-01-19 03:47:18 +0000:[INFO] [{CancelClusterBackupShare}] Volume snapshot classes found in the cluster - [2024-01-19T03:47:18.767Z] 2024-01-19 03:47:18 +0000:[INFO] [{CancelClusterBackupShare}] ibmc-vpcblock-snapshot [2024-01-19T03:47:18.767Z] 2024-01-19 03:47:18 +0000:[INFO] [{CancelClusterBackupShare}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T03:47:18.767Z] 2024/01/19 03:47:18 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636038] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T03:47:18.767Z] 2024/01/19 03:47:18 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636008] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:47:28.728Z] 2024-01-19 03:47:28 +0000:[INFO] [{CancelClusterBackupShare}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T03:47:28.728Z] 2024-01-19 03:47:28 +0000:[INFO] [{CancelClusterBackupShare}] Volume snapshot classes found in the cluster - [2024-01-19T03:47:28.728Z] 2024-01-19 03:47:28 +0000:[INFO] [{CancelClusterBackupShare}] ibmc-vpcblock-snapshot [2024-01-19T03:47:28.728Z] 2024-01-19 03:47:28 +0000:[INFO] [{CancelClusterBackupShare}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T03:47:28.728Z] 2024/01/19 03:47:28 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636048] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T03:47:28.728Z] 2024/01/19 03:47:28 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636018] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:47:40.902Z] 2024/01/19 03:47:38 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705635998] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:47:40.903Z] 2024/01/19 03:47:38 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636028] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:47:48.996Z] 2024/01/19 03:47:48 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636038] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:47:48.996Z] 2024/01/19 03:47:48 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636008] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:47:58.939Z] 2024/01/19 03:47:58 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636048] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:47:58.939Z] 2024/01/19 03:47:58 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636018] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:48:08.884Z] 2024/01/19 03:48:08 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705635998] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:48:08.884Z] 2024/01/19 03:48:08 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636028] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:48:18.832Z] 2024/01/19 03:48:18 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636038] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:48:18.832Z] 2024/01/19 03:48:18 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636008] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:48:28.780Z] 2024/01/19 03:48:28 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636048] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:48:28.780Z] 2024/01/19 03:48:28 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636018] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:48:38.725Z] 2024/01/19 03:48:38 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705635998] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:48:38.982Z] 2024/01/19 03:48:38 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636028] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:48:48.947Z] 2024/01/19 03:48:48 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636038] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:48:48.947Z] 2024/01/19 03:48:48 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636008] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:48:58.658Z] 2024/01/19 03:48:58 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636048] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:48:58.658Z] 2024/01/19 03:48:58 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636018] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:49:10.828Z] 2024/01/19 03:49:08 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705635998] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:49:10.828Z] 2024/01/19 03:49:08 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636028] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:49:18.912Z] 2024/01/19 03:49:18 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636038] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:49:18.912Z] 2024/01/19 03:49:18 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636008] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:49:28.898Z] 2024/01/19 03:49:28 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636048] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:49:28.898Z] 2024/01/19 03:49:28 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636018] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:49:38.847Z] 2024/01/19 03:49:38 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705635998] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:49:38.847Z] 2024/01/19 03:49:38 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636028] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:49:48.794Z] 2024/01/19 03:49:48 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636038] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:49:48.794Z] 2024/01/19 03:49:48 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636008] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:49:58.747Z] 2024/01/19 03:49:58 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636048] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:49:58.747Z] 2024/01/19 03:49:58 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636018] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:50:10.915Z] 2024/01/19 03:50:08 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705635998] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:50:10.915Z] 2024/01/19 03:50:08 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636028] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:50:18.993Z] 2024/01/19 03:50:18 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636038] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:50:18.993Z] 2024/01/19 03:50:18 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636008] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:50:28.953Z] 2024/01/19 03:50:28 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636048] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:50:28.953Z] 2024/01/19 03:50:28 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636018] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:50:38.911Z] 2024-01-19 03:50:38 +0000:[INFO] [{CancelClusterBackupShare}] [backup.CreateBackup:#313] - Backup [tp-backup-1705635998] created successfully [2024-01-19T03:50:38.911Z] 2024-01-19 03:50:38 +0000:[INFO] [{CancelClusterBackupShare}] Validating backup [tp-backup-1705635998] in org [default] [2024-01-19T03:50:38.911Z] 2024-01-19 03:50:38 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705635998] [2024-01-19T03:50:38.911Z] 2024/01/19 03:50:38 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636028] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:50:38.911Z] 2024-01-19 03:50:38 +0000:[INFO] [{CancelClusterBackupShare}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82935-01-19-03h43m53s] in backup [tp-backup-1705635998] [2024-01-19T03:50:38.911Z] 2024-01-19 03:50:38 +0000:[INFO] [{CancelClusterBackupShare}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82935-01-19-03h43m53s] in backup [tp-backup-1705635998] [2024-01-19T03:50:38.911Z] 2024-01-19 03:50:38 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82935-01-19-03h43m53s] [2024-01-19T03:50:39.169Z] 2024-01-19 03:50:38 +0000:[WARNING] [{CancelClusterBackupShare}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:50:39.169Z] 2024-01-19 03:50:38 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:50:39.169Z] 2024-01-19 03:50:38 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82935-01-19-03h43m53s] are [[mysql-data]] [2024-01-19T03:50:39.169Z] 2024-01-19 03:50:38 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T03:50:39.169Z] 2024-01-19 03:50:38 +0000:[INFO] [{CancelClusterBackupShare}] Validating CR cleanup [2024-01-19T03:50:39.169Z] 2024-01-19 03:50:39 +0000:[INFO] [{CancelClusterBackupShare}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T03:50:39.169Z] 2024-01-19 03:50:39 +0000:[INFO] [{CancelClusterBackupShare}] All CRs in [mysql-ibm-pxb-0-82935-01-19-03h43m53s] are [[tp-backup-1705636018-59dcc5e tp-backup-1705636028-c2682f0 tp-backup-1705636038-e1406d0 tp-backup-1705636048-1fc8542]] [2024-01-19T03:50:39.169Z] INFO[2024-01-19 03:50:39] Verifying : Description : Creation and Validation of backup [tp-backup-1705635998] [2024-01-19T03:50:39.169Z] INFO[2024-01-19 03:50:39] Actual:true, Expected: true [2024-01-19T03:50:49.156Z] 2024/01/19 03:50:48 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636038] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:50:49.156Z] 2024-01-19 03:50:48 +0000:[INFO] [{CancelClusterBackupShare}] [backup.CreateBackup:#313] - Backup [tp-backup-1705636008] created successfully [2024-01-19T03:50:49.156Z] 2024-01-19 03:50:48 +0000:[INFO] [{CancelClusterBackupShare}] Validating backup [tp-backup-1705636008] in org [default] [2024-01-19T03:50:49.156Z] 2024-01-19 03:50:48 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705636008] [2024-01-19T03:50:49.156Z] 2024-01-19 03:50:48 +0000:[INFO] [{CancelClusterBackupShare}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82935-01-19-03h43m53s] in backup [tp-backup-1705636008] [2024-01-19T03:50:49.157Z] 2024-01-19 03:50:48 +0000:[INFO] [{CancelClusterBackupShare}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82935-01-19-03h43m53s] in backup [tp-backup-1705636008] [2024-01-19T03:50:49.157Z] 2024-01-19 03:50:48 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82935-01-19-03h43m53s] [2024-01-19T03:50:49.157Z] 2024-01-19 03:50:48 +0000:[WARNING] [{CancelClusterBackupShare}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:50:49.157Z] 2024-01-19 03:50:48 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:50:49.157Z] 2024-01-19 03:50:48 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82935-01-19-03h43m53s] are [[mysql-data]] [2024-01-19T03:50:49.157Z] 2024-01-19 03:50:48 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T03:50:49.157Z] 2024-01-19 03:50:48 +0000:[INFO] [{CancelClusterBackupShare}] Validating CR cleanup [2024-01-19T03:50:49.157Z] 2024-01-19 03:50:48 +0000:[INFO] [{CancelClusterBackupShare}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T03:50:49.157Z] 2024-01-19 03:50:48 +0000:[INFO] [{CancelClusterBackupShare}] All CRs in [mysql-ibm-pxb-0-82935-01-19-03h43m53s] are [[tp-backup-1705636018-59dcc5e tp-backup-1705636028-c2682f0 tp-backup-1705636038-e1406d0 tp-backup-1705636048-1fc8542]] [2024-01-19T03:50:49.157Z] INFO[2024-01-19 03:50:48] Verifying : Description : Creation and Validation of backup [tp-backup-1705636008] [2024-01-19T03:50:49.157Z] INFO[2024-01-19 03:50:48] Actual:true, Expected: true [2024-01-19T03:50:59.105Z] 2024/01/19 03:50:58 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705636048] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T03:50:59.105Z] 2024-01-19 03:50:58 +0000:[INFO] [{CancelClusterBackupShare}] [backup.CreateBackup:#313] - Backup [tp-backup-1705636018] created successfully [2024-01-19T03:50:59.105Z] 2024-01-19 03:50:58 +0000:[INFO] [{CancelClusterBackupShare}] Validating backup [tp-backup-1705636018] in org [default] [2024-01-19T03:50:59.105Z] 2024-01-19 03:50:58 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705636018] [2024-01-19T03:50:59.105Z] 2024-01-19 03:50:58 +0000:[INFO] [{CancelClusterBackupShare}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82935-01-19-03h43m53s] in backup [tp-backup-1705636018] [2024-01-19T03:50:59.105Z] 2024-01-19 03:50:58 +0000:[INFO] [{CancelClusterBackupShare}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82935-01-19-03h43m53s] in backup [tp-backup-1705636018] [2024-01-19T03:50:59.105Z] 2024-01-19 03:50:58 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82935-01-19-03h43m53s] [2024-01-19T03:50:59.105Z] 2024-01-19 03:50:58 +0000:[WARNING] [{CancelClusterBackupShare}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:50:59.105Z] 2024-01-19 03:50:58 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:50:59.105Z] 2024-01-19 03:50:58 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82935-01-19-03h43m53s] are [[mysql-data]] [2024-01-19T03:50:59.106Z] 2024-01-19 03:50:58 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T03:50:59.106Z] 2024-01-19 03:50:58 +0000:[INFO] [{CancelClusterBackupShare}] Validating CR cleanup [2024-01-19T03:50:59.106Z] 2024-01-19 03:50:58 +0000:[INFO] [{CancelClusterBackupShare}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T03:50:59.106Z] 2024-01-19 03:50:58 +0000:[INFO] [{CancelClusterBackupShare}] All CRs in [mysql-ibm-pxb-0-82935-01-19-03h43m53s] are [[tp-backup-1705636038-e1406d0 tp-backup-1705636048-1fc8542]] [2024-01-19T03:50:59.106Z] INFO[2024-01-19 03:50:58] Verifying : Description : Creation and Validation of backup [tp-backup-1705636018] [2024-01-19T03:50:59.106Z] INFO[2024-01-19 03:50:58] Actual:true, Expected: true [2024-01-19T03:51:09.057Z] 2024-01-19 03:51:08 +0000:[INFO] [{CancelClusterBackupShare}] [backup.CreateBackup:#313] - Backup [tp-backup-1705636028] created successfully [2024-01-19T03:51:09.057Z] 2024-01-19 03:51:08 +0000:[INFO] [{CancelClusterBackupShare}] Validating backup [tp-backup-1705636028] in org [default] [2024-01-19T03:51:09.057Z] 2024-01-19 03:51:08 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705636028] [2024-01-19T03:51:09.057Z] 2024-01-19 03:51:08 +0000:[INFO] [{CancelClusterBackupShare}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82935-01-19-03h43m53s] in backup [tp-backup-1705636028] [2024-01-19T03:51:09.057Z] 2024-01-19 03:51:08 +0000:[INFO] [{CancelClusterBackupShare}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82935-01-19-03h43m53s] in backup [tp-backup-1705636028] [2024-01-19T03:51:09.057Z] 2024-01-19 03:51:08 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82935-01-19-03h43m53s] [2024-01-19T03:51:09.057Z] 2024-01-19 03:51:08 +0000:[WARNING] [{CancelClusterBackupShare}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:51:09.057Z] 2024-01-19 03:51:08 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:51:09.057Z] 2024-01-19 03:51:08 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82935-01-19-03h43m53s] are [[mysql-data]] [2024-01-19T03:51:09.057Z] 2024-01-19 03:51:08 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T03:51:09.057Z] 2024-01-19 03:51:08 +0000:[INFO] [{CancelClusterBackupShare}] Validating CR cleanup [2024-01-19T03:51:09.057Z] 2024-01-19 03:51:09 +0000:[INFO] [{CancelClusterBackupShare}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T03:51:09.312Z] 2024-01-19 03:51:09 +0000:[INFO] [{CancelClusterBackupShare}] All CRs in [mysql-ibm-pxb-0-82935-01-19-03h43m53s] are [[]] [2024-01-19T03:51:09.313Z] INFO[2024-01-19 03:51:09] Verifying : Description : Creation and Validation of backup [tp-backup-1705636028] [2024-01-19T03:51:09.313Z] INFO[2024-01-19 03:51:09] Actual:true, Expected: true [2024-01-19T03:51:19.256Z] 2024-01-19 03:51:18 +0000:[INFO] [{CancelClusterBackupShare}] [backup.CreateBackup:#313] - Backup [tp-backup-1705636038] created successfully [2024-01-19T03:51:19.256Z] 2024-01-19 03:51:18 +0000:[INFO] [{CancelClusterBackupShare}] Validating backup [tp-backup-1705636038] in org [default] [2024-01-19T03:51:19.256Z] 2024-01-19 03:51:18 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705636038] [2024-01-19T03:51:19.256Z] 2024-01-19 03:51:18 +0000:[INFO] [{CancelClusterBackupShare}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82935-01-19-03h43m53s] in backup [tp-backup-1705636038] [2024-01-19T03:51:19.256Z] 2024-01-19 03:51:18 +0000:[INFO] [{CancelClusterBackupShare}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82935-01-19-03h43m53s] in backup [tp-backup-1705636038] [2024-01-19T03:51:19.256Z] 2024-01-19 03:51:18 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82935-01-19-03h43m53s] [2024-01-19T03:51:19.256Z] 2024-01-19 03:51:18 +0000:[WARNING] [{CancelClusterBackupShare}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:51:19.256Z] 2024-01-19 03:51:18 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:51:19.256Z] 2024-01-19 03:51:18 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82935-01-19-03h43m53s] are [[mysql-data]] [2024-01-19T03:51:19.256Z] 2024-01-19 03:51:18 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T03:51:19.256Z] 2024-01-19 03:51:18 +0000:[INFO] [{CancelClusterBackupShare}] Validating CR cleanup [2024-01-19T03:51:19.257Z] 2024-01-19 03:51:18 +0000:[INFO] [{CancelClusterBackupShare}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T03:51:19.257Z] 2024-01-19 03:51:18 +0000:[INFO] [{CancelClusterBackupShare}] All CRs in [mysql-ibm-pxb-0-82935-01-19-03h43m53s] are [[]] [2024-01-19T03:51:19.257Z] INFO[2024-01-19 03:51:18] Verifying : Description : Creation and Validation of backup [tp-backup-1705636038] [2024-01-19T03:51:19.257Z] INFO[2024-01-19 03:51:18] Actual:true, Expected: true [2024-01-19T03:51:29.197Z] 2024-01-19 03:51:28 +0000:[INFO] [{CancelClusterBackupShare}] [backup.CreateBackup:#313] - Backup [tp-backup-1705636048] created successfully [2024-01-19T03:51:29.197Z] 2024-01-19 03:51:28 +0000:[INFO] [{CancelClusterBackupShare}] Validating backup [tp-backup-1705636048] in org [default] [2024-01-19T03:51:29.197Z] 2024-01-19 03:51:28 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705636048] [2024-01-19T03:51:29.197Z] 2024-01-19 03:51:28 +0000:[INFO] [{CancelClusterBackupShare}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82935-01-19-03h43m53s] in backup [tp-backup-1705636048] [2024-01-19T03:51:29.197Z] 2024-01-19 03:51:28 +0000:[INFO] [{CancelClusterBackupShare}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82935-01-19-03h43m53s] in backup [tp-backup-1705636048] [2024-01-19T03:51:29.197Z] 2024-01-19 03:51:28 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82935-01-19-03h43m53s] [2024-01-19T03:51:29.197Z] 2024-01-19 03:51:28 +0000:[WARNING] [{CancelClusterBackupShare}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:51:29.197Z] 2024-01-19 03:51:28 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:51:29.197Z] 2024-01-19 03:51:28 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82935-01-19-03h43m53s] are [[mysql-data]] [2024-01-19T03:51:29.197Z] 2024-01-19 03:51:28 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T03:51:29.197Z] 2024-01-19 03:51:28 +0000:[INFO] [{CancelClusterBackupShare}] Validating CR cleanup [2024-01-19T03:51:29.197Z] 2024-01-19 03:51:28 +0000:[INFO] [{CancelClusterBackupShare}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T03:51:29.197Z] 2024-01-19 03:51:29 +0000:[INFO] [{CancelClusterBackupShare}] All CRs in [mysql-ibm-pxb-0-82935-01-19-03h43m53s] are [[]] [2024-01-19T03:51:29.197Z] INFO[2024-01-19 03:51:29] Verifying : Description : Creation and Validation of backup [tp-backup-1705636048] [2024-01-19T03:51:29.197Z] INFO[2024-01-19 03:51:29] Actual:true, Expected: true [2024-01-19T03:51:29.197Z] 2024-01-19 03:51:29 +0000:[INFO] [{CancelClusterBackupShare}] [backup.glob..func85.2.7:#1416] - List of backups - [tp-backup-1705635998 tp-backup-1705636008 tp-backup-1705636018 tp-backup-1705636028 tp-backup-1705636038 tp-backup-1705636048] [2024-01-19T03:51:29.197Z] 2024-01-19 03:51:29 +0000:[INFO] [{CancelClusterBackupShare}] Share all backups with Full Access in source cluster with a group and a user who is not part of the group [2024-01-19T03:51:29.759Z] 2024-01-19 03:51:29 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 488b3823-aa13-4e9b-bbc1-5c7907b46ab5 [2024-01-19T03:51:30.688Z] 2024-01-19 03:51:30 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfUser:#496] - Fetching ID of user autogenerated-user - eaa4bb92-2e0c-40e7-9c87-e1414def7714 [2024-01-19T03:51:30.688Z] 2024/01/19 03:51:30 DoRetryWithTimeout - Error: {cluster backup share status for cluster source-cluster is still InProgress}, Next try in [10s], timeout [1m0s] [2024-01-19T03:51:40.672Z] 2024-01-19 03:51:40 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ClusterUpdateBackupShare.func1:#856] - Cluster source-cluster has status - [3] [2024-01-19T03:51:40.672Z] 2024-01-19 03:51:40 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ClusterUpdateBackupShare:#863] - Cluster backup share check complete [2024-01-19T03:51:40.672Z] 2024-01-19 03:51:40 +0000:[INFO] [{CancelClusterBackupShare}] Validate Full Access of backups shared at cluster level for a user of a group [2024-01-19T03:51:40.965Z] 2024-01-19 03:51:40 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 488b3823-aa13-4e9b-bbc1-5c7907b46ab5 [2024-01-19T03:51:40.965Z] 2024-01-19 03:51:40 +0000:[DEBUG] [{CancelClusterBackupShare}] [backup.GetMembersOfGroup:#961] - list of members : [testuser1 testuser10 testuser2 testuser3 testuser4 testuser5 testuser6 testuser7 testuser8 testuser9] [2024-01-19T03:51:40.965Z] 2024-01-19 03:51:40 +0000:[INFO] [{CancelClusterBackupShare}] [backup.glob..func85.2.9:#1433] - User chosen to validate full access - testuser9 [2024-01-19T03:51:42.331Z] 2024-01-19 03:51:41 +0000:[INFO] [{CancelClusterBackupShare}] Registering Source and Destination clusters from user context [2024-01-19T03:51:42.331Z] INFO[2024-01-19 03:51:41] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T03:51:42.331Z] INFO[2024-01-19 03:51:41] Actual:true, Expected: true [2024-01-19T03:51:42.331Z] 2024-01-19 03:51:41 +0000:[INFO] [{CancelClusterBackupShare}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T03:51:42.331Z] 2024-01-19 03:51:42 +0000:[INFO] [{CancelClusterBackupShare}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T03:51:42.331Z] 2024-01-19 03:51:42 +0000:[INFO] [{CancelClusterBackupShare}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T03:51:42.331Z] 2024-01-19 03:51:42 +0000:[INFO] [{CancelClusterBackupShare}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T03:51:42.331Z] 2024-01-19 03:51:42 +0000:[INFO] [{CancelClusterBackupShare}] Create cluster [source-cluster] in org [default] [2024-01-19T03:51:42.332Z] 2024-01-19 03:51:42 +0000:[INFO] [{CancelClusterBackupShare}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T03:51:42.332Z] 2024-01-19 03:51:42 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T03:51:42.332Z] 2024-01-19 03:51:42 +0000:[INFO] [{CancelClusterBackupShare}] Create cluster [destination-cluster] in org [default] [2024-01-19T03:51:42.332Z] 2024-01-19 03:51:42 +0000:[INFO] [{CancelClusterBackupShare}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T03:51:42.332Z] 2024-01-19 03:51:42 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T03:51:42.332Z] 2024-01-19 03:51:42 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-wwrvv] for IBM [2024-01-19T03:51:42.332Z] 2024-01-19 03:51:42 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-wwrvv] for org [default] with [ibm] as provider [2024-01-19T03:51:42.332Z] 2024-01-19 03:51:42 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T03:51:42.332Z] 2024-01-19 03:51:42 +0000:[INFO] [{CancelClusterBackupShare}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T03:51:42.332Z] 2024-01-19 03:51:42 +0000:[WARNING] [{CancelClusterBackupShare}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-wwrvv] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName9 LastName9 with roles: [px-backup-app.user]] [2024-01-19T03:51:42.332Z] 2024-01-19 03:51:42 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName9 LastName9 with roles: [px-backup-app.user] [2024-01-19T03:51:43.700Z] 2024-01-19 03:51:43 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-wwrvv from admin context and sharing with all the users [2024-01-19T03:51:43.700Z] 2024-01-19 03:51:43 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-wwrvv] for org [default] with [ibm] as provider [2024-01-19T03:51:43.700Z] 2024-01-19 03:51:43 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T03:51:43.700Z] 2024-01-19 03:51:43 +0000:[INFO] [{CancelClusterBackupShare}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T03:51:43.700Z] 2024-01-19 03:51:43 +0000:[INFO] [{CancelClusterBackupShare}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-wwrvv] is present [2024-01-19T03:51:43.961Z] 2024-01-19 03:51:43 +0000:[INFO] [{CancelClusterBackupShare}] Create cluster [source-cluster] in org [default] [2024-01-19T03:51:44.521Z] 2024-01-19 03:51:44 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-pnlwn] for IBM [2024-01-19T03:51:44.522Z] 2024-01-19 03:51:44 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-pnlwn] for org [default] with [ibm] as provider [2024-01-19T03:51:44.522Z] 2024-01-19 03:51:44 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T03:51:44.522Z] 2024-01-19 03:51:44 +0000:[INFO] [{CancelClusterBackupShare}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T03:51:44.522Z] 2024-01-19 03:51:44 +0000:[WARNING] [{CancelClusterBackupShare}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-pnlwn] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName9 LastName9 with roles: [px-backup-app.user]] [2024-01-19T03:51:44.522Z] 2024-01-19 03:51:44 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName9 LastName9 with roles: [px-backup-app.user] [2024-01-19T03:51:45.082Z] 2024-01-19 03:51:44 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-pnlwn from admin context and sharing with all the users [2024-01-19T03:51:45.082Z] 2024-01-19 03:51:44 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-pnlwn] for org [default] with [ibm] as provider [2024-01-19T03:51:45.082Z] 2024-01-19 03:51:44 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T03:51:45.083Z] 2024-01-19 03:51:44 +0000:[INFO] [{CancelClusterBackupShare}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T03:51:45.083Z] 2024-01-19 03:51:44 +0000:[INFO] [{CancelClusterBackupShare}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-pnlwn] is present [2024-01-19T03:51:45.339Z] 2024-01-19 03:51:45 +0000:[INFO] [{CancelClusterBackupShare}] Create cluster [destination-cluster] in org [default] [2024-01-19T03:51:45.899Z] INFO[2024-01-19 03:51:45] Verifying : Description : Creating source and destination cluster [2024-01-19T03:51:45.899Z] INFO[2024-01-19 03:51:45] Actual:true, Expected: true [2024-01-19T03:51:46.459Z] 2024/01/19 03:51:46 DoRetryWithTimeout - Error: {restore status for [tp-restore-1705636305] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T03:52:18.480Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] [backup.CreateRestore:#936] - Restore [tp-restore-1705636305] created successfully [2024-01-19T03:52:18.480Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] Switching cluster context to cluster [destination-cluster] [2024-01-19T03:52:18.481Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T03:52:18.481Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-82935-01-19-03h43m53s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-19T03:52:18.481Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-82935-01-19-03h43m53s] [2024-01-19T03:52:18.481Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T03:52:18.481Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T03:52:18.481Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] Validating restore [tp-restore-1705636305] in org [default] [2024-01-19T03:52:18.481Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-restore-1705636305] [2024-01-19T03:52:18.481Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-restore-1705636305] [2024-01-19T03:52:18.481Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-82935-01-19-03h43m53s] in restore [tp-restore-1705636305] [2024-01-19T03:52:18.481Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-82935-01-19-03h43m53s]) was found in restore [tp-restore-1705636305], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82935-01-19-03h43m53s] [2024-01-19T03:52:18.481Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-82935-01-19-03h43m53s]) found in the restore [tp-restore-1705636305] was also present on the cluster/namespace [mysql-ibm-pxb-0-82935-01-19-03h43m53s] [2024-01-19T03:52:18.481Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-82935-01-19-03h43m53s]) was found in restore [tp-restore-1705636305], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82935-01-19-03h43m53s] [2024-01-19T03:52:18.481Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-82935-01-19-03h43m53s]) found in the restore [tp-restore-1705636305] was also present on the cluster/namespace [mysql-ibm-pxb-0-82935-01-19-03h43m53s] [2024-01-19T03:52:18.481Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-82935-01-19-03h43m53s]) was found in restore [tp-restore-1705636305], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82935-01-19-03h43m53s] [2024-01-19T03:52:18.481Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-82935-01-19-03h43m53s]) found in the restore [tp-restore-1705636305] was also present on the cluster/namespace [mysql-ibm-pxb-0-82935-01-19-03h43m53s] [2024-01-19T03:52:18.481Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-82935-01-19-03h43m53s]) was found in restore [tp-restore-1705636305], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-82935-01-19-03h43m53s] [2024-01-19T03:52:18.482Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-82935-01-19-03h43m53s]) found in the restore [tp-restore-1705636305] was also present on the cluster/namespace [mysql-ibm-pxb-0-82935-01-19-03h43m53s] [2024-01-19T03:52:18.482Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-82935-01-19-03h43m53s] in restore [tp-restore-1705636305] [2024-01-19T03:52:18.482Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-82935-01-19-03h43m53s] in restore [tp-restore-1705636305] [2024-01-19T03:52:18.482Z] 2024-01-19 03:52:16 +0000:[WARNING] [{CancelClusterBackupShare}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T03:52:18.482Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:52:18.482Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82935-01-19-03h43m53s] are [[mysql-data]] [2024-01-19T03:52:18.482Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] [backup.ValidateRestore:#2240] - in restore [tp-restore-1705636305], the status of the restored volume [pvc-260099ea-8990-4ee4-a875-09d25701a318] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-19T03:52:18.482Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] Validate applications in restored namespace [mysql-ibm-pxb-0-82935-01-19-03h43m53s] due to restore [tp-restore-1705636305] [2024-01-19T03:52:18.482Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] Validating mysql-ibm app [2024-01-19T03:52:18.482Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] wait for mysql-ibm app to start running [2024-01-19T03:52:18.482Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T03:52:18.482Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T03:52:18.482Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T03:52:18.482Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] Switching cluster context back to cluster path [/tmp/destination-config] [2024-01-19T03:52:18.482Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T03:52:18.482Z] 2024-01-19 03:52:16 +0000:[ERROR] [{CancelClusterBackupShare}] [backup.glob..func85.2.9:#1454] - Restoring of backup [tp-backup-1705636048] has failed with name - [tp-restore-1705636305]. Err: ValidateRestore Errors: {in restore [tp-restore-1705636305], said restored volume [pvc-260099ea-8990-4ee4-a875-09d25701a318] cannot be found in the actual cluster [mysql-ibm-pxb-0-82935-01-19-03h43m53s]} [2024-01-19T03:52:18.482Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] Deleting the deployed apps after the testcase [2024-01-19T03:52:18.482Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] Destroying apps [2024-01-19T03:52:18.482Z] 2024-01-19 03:52:16 +0000:[INFO] [{CancelClusterBackupShare}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T03:52:18.482Z] 2024-01-19 03:52:17 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T03:52:18.482Z] 2024-01-19 03:52:17 +0000:[INFO] [{CancelClusterBackupShare}] start destroying mysql-ibm app [2024-01-19T03:52:18.482Z] 2024-01-19 03:52:17 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T03:52:18.482Z] 2024-01-19 03:52:17 +0000:[INFO] [{CancelClusterBackupShare}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:52:18.482Z] 2024-01-19 03:52:17 +0000:[WARNING] [{CancelClusterBackupShare}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T03:52:18.483Z] INFO[2024-01-19 03:52:17] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T03:52:18.483Z] INFO[2024-01-19 03:52:17] Actual:true, Expected: true [2024-01-19T03:52:18.483Z] 2024-01-19 03:52:17 +0000:[INFO] [{CancelClusterBackupShare}] [backup.glob..func85.3:#1738] - Cleaning up users [2024-01-19T03:52:18.483Z] I0119 03:52:18.315230 14 request.go:682] Waited for 1.07415658s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/namespaces/px-backup/secrets/pxc-backup-secret [2024-01-19T03:52:26.572Z] 2024-01-19 03:52:26 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser7 - 7ea98068-ca40-46db-a151-6398e592f58e [2024-01-19T03:52:26.827Z] 2024-01-19 03:52:26 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1 - 4e9ea7cd-1326-490a-85c7-d9e61856f1e4 [2024-01-19T03:52:26.827Z] 2024-01-19 03:52:26 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser10 - e91065a4-15d9-4fed-8e03-f29e8cb0bb67 [2024-01-19T03:52:27.083Z] 2024-01-19 03:52:26 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2 - 5e6ccc21-5b96-4d45-b1d0-6a6b5bab6483 [2024-01-19T03:52:27.339Z] 2024-01-19 03:52:27 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser3 - eaf0e1a5-f802-468c-8ece-e121482c86f7 [2024-01-19T03:52:27.596Z] 2024-01-19 03:52:27 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser6 - d7aaae10-a3c8-437d-9ee6-51f853d7ef6d [2024-01-19T03:52:27.596Z] 2024-01-19 03:52:27 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser5 - 484091d9-264e-4711-a439-f2c90e40c5d9 [2024-01-19T03:52:27.853Z] 2024-01-19 03:52:27 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser9 - 1297b31b-c0e4-473b-b6af-fc1ab802763d [2024-01-19T03:52:28.109Z] 2024-01-19 03:52:27 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser4 - 46d6f682-9c4b-46bc-a6ae-9926aa2cf49f [2024-01-19T03:52:28.364Z] 2024-01-19 03:52:28 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser8 - 06bb4316-9885-4247-9861-529a3be2112b [2024-01-19T03:52:28.365Z] I0119 03:52:28.315412 14 request.go:682] Waited for 1.962763771s due to client-side throttling, not priority and fairness, request: GET:https://c100-e.containers.test.cloud.ibm.com:30485/api/v1/services [2024-01-19T03:52:30.882Z] 2024-01-19 03:52:30 +0000:[INFO] [{CancelClusterBackupShare}] [backup.DeleteUser:#755] - Response for user [testuser7] deletion - [2024-01-19T03:52:30.882Z] 2024-01-19 03:52:30 +0000:[INFO] [{CancelClusterBackupShare}] [backup.DeleteUser:#759] - Deleted User - testuser7 [2024-01-19T03:52:30.882Z] INFO[2024-01-19 03:52:30] Verifying : Description : Deleting user testuser7 [2024-01-19T03:52:30.882Z] INFO[2024-01-19 03:52:30] Actual:true, Expected: true [2024-01-19T03:52:30.882Z] 2024-01-19 03:52:30 +0000:[INFO] [{CancelClusterBackupShare}] [backup.DeleteUser:#755] - Response for user [testuser1] deletion - [2024-01-19T03:52:30.882Z] 2024-01-19 03:52:30 +0000:[INFO] [{CancelClusterBackupShare}] [backup.DeleteUser:#759] - Deleted User - testuser1 [2024-01-19T03:52:30.882Z] INFO[2024-01-19 03:52:30] Verifying : Description : Deleting user testuser1 [2024-01-19T03:52:30.882Z] INFO[2024-01-19 03:52:30] Actual:true, Expected: true [2024-01-19T03:52:30.882Z] 2024-01-19 03:52:30 +0000:[INFO] [{CancelClusterBackupShare}] [backup.DeleteUser:#755] - Response for user [testuser10] deletion - [2024-01-19T03:52:30.882Z] 2024-01-19 03:52:30 +0000:[INFO] [{CancelClusterBackupShare}] [backup.DeleteUser:#759] - Deleted User - testuser10 [2024-01-19T03:52:30.882Z] INFO[2024-01-19 03:52:30] Verifying : Description : Deleting user testuser10 [2024-01-19T03:52:30.882Z] INFO[2024-01-19 03:52:30] Actual:true, Expected: true [2024-01-19T03:52:31.142Z] 2024-01-19 03:52:31 +0000:[INFO] [{CancelClusterBackupShare}] [backup.DeleteUser:#755] - Response for user [testuser2] deletion - [2024-01-19T03:52:31.142Z] 2024-01-19 03:52:31 +0000:[INFO] [{CancelClusterBackupShare}] [backup.DeleteUser:#759] - Deleted User - testuser2 [2024-01-19T03:52:31.142Z] INFO[2024-01-19 03:52:31] Verifying : Description : Deleting user testuser2 [2024-01-19T03:52:31.142Z] INFO[2024-01-19 03:52:31] Actual:true, Expected: true [2024-01-19T03:52:31.397Z] 2024-01-19 03:52:31 +0000:[INFO] [{CancelClusterBackupShare}] [backup.DeleteUser:#755] - Response for user [testuser3] deletion - [2024-01-19T03:52:31.397Z] 2024-01-19 03:52:31 +0000:[INFO] [{CancelClusterBackupShare}] [backup.DeleteUser:#759] - Deleted User - testuser3 [2024-01-19T03:52:31.397Z] INFO[2024-01-19 03:52:31] Verifying : Description : Deleting user testuser3 [2024-01-19T03:52:31.397Z] INFO[2024-01-19 03:52:31] Actual:true, Expected: true [2024-01-19T03:52:31.653Z] 2024-01-19 03:52:31 +0000:[INFO] [{CancelClusterBackupShare}] [backup.DeleteUser:#755] - Response for user [testuser6] deletion - [2024-01-19T03:52:31.653Z] 2024-01-19 03:52:31 +0000:[INFO] [{CancelClusterBackupShare}] [backup.DeleteUser:#759] - Deleted User - testuser6 [2024-01-19T03:52:31.653Z] INFO[2024-01-19 03:52:31] Verifying : Description : Deleting user testuser6 [2024-01-19T03:52:31.653Z] INFO[2024-01-19 03:52:31] Actual:true, Expected: true [2024-01-19T03:52:31.908Z] 2024-01-19 03:52:31 +0000:[INFO] [{CancelClusterBackupShare}] [backup.DeleteUser:#755] - Response for user [testuser5] deletion - [2024-01-19T03:52:31.908Z] 2024-01-19 03:52:31 +0000:[INFO] [{CancelClusterBackupShare}] [backup.DeleteUser:#759] - Deleted User - testuser5 [2024-01-19T03:52:31.908Z] INFO[2024-01-19 03:52:31] Verifying : Description : Deleting user testuser5 [2024-01-19T03:52:31.908Z] INFO[2024-01-19 03:52:31] Actual:true, Expected: true [2024-01-19T03:52:31.908Z] 2024-01-19 03:52:31 +0000:[INFO] [{CancelClusterBackupShare}] [backup.DeleteUser:#755] - Response for user [testuser9] deletion - [2024-01-19T03:52:31.908Z] 2024-01-19 03:52:31 +0000:[INFO] [{CancelClusterBackupShare}] [backup.DeleteUser:#759] - Deleted User - testuser9 [2024-01-19T03:52:31.908Z] INFO[2024-01-19 03:52:31] Verifying : Description : Deleting user testuser9 [2024-01-19T03:52:31.908Z] INFO[2024-01-19 03:52:31] Actual:true, Expected: true [2024-01-19T03:52:32.165Z] 2024-01-19 03:52:32 +0000:[INFO] [{CancelClusterBackupShare}] [backup.DeleteUser:#755] - Response for user [testuser4] deletion - [2024-01-19T03:52:32.165Z] 2024-01-19 03:52:32 +0000:[INFO] [{CancelClusterBackupShare}] [backup.DeleteUser:#759] - Deleted User - testuser4 [2024-01-19T03:52:32.165Z] INFO[2024-01-19 03:52:32] Verifying : Description : Deleting user testuser4 [2024-01-19T03:52:32.165Z] INFO[2024-01-19 03:52:32] Actual:true, Expected: true [2024-01-19T03:52:32.427Z] 2024-01-19 03:52:32 +0000:[INFO] [{CancelClusterBackupShare}] [backup.DeleteUser:#755] - Response for user [testuser8] deletion - [2024-01-19T03:52:32.427Z] 2024-01-19 03:52:32 +0000:[INFO] [{CancelClusterBackupShare}] [backup.DeleteUser:#759] - Deleted User - testuser8 [2024-01-19T03:52:32.427Z] INFO[2024-01-19 03:52:32] Verifying : Description : Deleting user testuser8 [2024-01-19T03:52:32.427Z] INFO[2024-01-19 03:52:32] Actual:true, Expected: true [2024-01-19T03:52:33.797Z] 2024-01-19 03:52:33 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfUser:#496] - Fetching ID of user autogenerated-user - eaa4bb92-2e0c-40e7-9c87-e1414def7714 [2024-01-19T03:52:34.057Z] 2024-01-19 03:52:33 +0000:[INFO] [{CancelClusterBackupShare}] [backup.DeleteUser:#755] - Response for user [autogenerated-user] deletion - [2024-01-19T03:52:34.057Z] 2024-01-19 03:52:33 +0000:[INFO] [{CancelClusterBackupShare}] [backup.DeleteUser:#759] - Deleted User - autogenerated-user [2024-01-19T03:52:34.057Z] INFO[2024-01-19 03:52:33] Verifying : Description : Deleting user autogenerated-user [2024-01-19T03:52:34.057Z] INFO[2024-01-19 03:52:33] Actual:true, Expected: true [2024-01-19T03:52:34.057Z] 2024-01-19 03:52:33 +0000:[INFO] [{CancelClusterBackupShare}] [backup.glob..func85.3:#1752] - Cleaning up groups [2024-01-19T03:52:34.983Z] 2024-01-19 03:52:34 +0000:[INFO] [{CancelClusterBackupShare}] [backup.FetchIDOfGroup:#1137] - Fetching ID of group testGroup1 - 488b3823-aa13-4e9b-bbc1-5c7907b46ab5 [2024-01-19T03:52:35.544Z] 2024-01-19 03:52:35 +0000:[INFO] [{CancelClusterBackupShare}] [backup.DeleteGroup:#1018] - Deleted Group - testGroup1 [2024-01-19T03:52:35.544Z] INFO[2024-01-19 03:52:35] Verifying : Description : Deleting group testGroup1 [2024-01-19T03:52:35.544Z] INFO[2024-01-19 03:52:35] Actual:true, Expected: true [2024-01-19T03:52:35.544Z] 2024-01-19 03:52:35 +0000:[INFO] [{CancelClusterBackupShare}] [backup.glob..func85.3:#1763] - Deleting registered clusters for non-admin context [2024-01-19T03:52:35.544Z] 2024-01-19 03:52:35 +0000:[INFO] [{CancelClusterBackupShare}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T03:52:35.544Z] INFO[2024-01-19 03:52:35] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T03:52:35.544Z] INFO[2024-01-19 03:52:35] Actual:true, Expected: true [2024-01-19T03:52:35.800Z] INFO[2024-01-19 03:52:35] Verifying : Description : Deleting cluster source-cluster [2024-01-19T03:52:35.800Z] INFO[2024-01-19 03:52:35] Actual:true, Expected: true [2024-01-19T03:52:37.166Z] INFO[2024-01-19 03:52:36] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-wwrvv] [2024-01-19T03:52:37.166Z] INFO[2024-01-19 03:52:36] Actual:true, Expected: true [2024-01-19T03:52:37.166Z] INFO[2024-01-19 03:52:37] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T03:52:37.166Z] INFO[2024-01-19 03:52:37] Actual:true, Expected: true [2024-01-19T03:52:38.564Z] INFO[2024-01-19 03:52:38] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-pnlwn] [2024-01-19T03:52:38.564Z] INFO[2024-01-19 03:52:38] Actual:true, Expected: true [2024-01-19T03:52:39.931Z] 2024-01-19 03:52:39 +0000:[INFO] [{CancelClusterBackupShare}] [backup.glob..func85.3:#1770] - Removing the backups from the backupNames list which have already been deleted as part of FullAccess Validation [2024-01-19T03:52:39.931Z] 2024-01-19 03:52:39 +0000:[INFO] [{CancelClusterBackupShare}] [backup.glob..func85.3:#1772] - Deleting the backups created [2024-01-19T03:52:39.931Z] 2024-01-19 03:52:39 +0000:[INFO] [{CancelClusterBackupShare}] [backup.glob..func85.3:#1777] - About to delete backup - tp-backup-1705635998 [2024-01-19T03:52:39.931Z] INFO[2024-01-19 03:52:39] Verifying : Description : Deleting backup - [tp-backup-1705635998] [2024-01-19T03:52:39.931Z] INFO[2024-01-19 03:52:39] Actual:true, Expected: true [2024-01-19T03:52:39.931Z] 2024-01-19 03:52:39 +0000:[INFO] [{CancelClusterBackupShare}] [backup.glob..func85.3:#1777] - About to delete backup - tp-backup-1705636008 [2024-01-19T03:52:39.931Z] INFO[2024-01-19 03:52:39] Verifying : Description : Deleting backup - [tp-backup-1705636008] [2024-01-19T03:52:39.931Z] INFO[2024-01-19 03:52:39] Actual:true, Expected: true [2024-01-19T03:52:39.931Z] 2024-01-19 03:52:39 +0000:[INFO] [{CancelClusterBackupShare}] [backup.glob..func85.3:#1777] - About to delete backup - tp-backup-1705636018 [2024-01-19T03:52:39.931Z] INFO[2024-01-19 03:52:39] Verifying : Description : Deleting backup - [tp-backup-1705636018] [2024-01-19T03:52:39.931Z] INFO[2024-01-19 03:52:39] Actual:true, Expected: true [2024-01-19T03:52:39.931Z] 2024-01-19 03:52:39 +0000:[INFO] [{CancelClusterBackupShare}] [backup.glob..func85.3:#1777] - About to delete backup - tp-backup-1705636028 [2024-01-19T03:52:40.188Z] INFO[2024-01-19 03:52:39] Verifying : Description : Deleting backup - [tp-backup-1705636028] [2024-01-19T03:52:40.188Z] INFO[2024-01-19 03:52:39] Actual:true, Expected: true [2024-01-19T03:52:40.188Z] 2024-01-19 03:52:39 +0000:[INFO] [{CancelClusterBackupShare}] Cleaning backup locations in map [map[b4816cdb-0621-48dc-a333-0ee37ec2c86c:autogenerated-backup-location-1705635982]], cloud credential [autogenerated-cred-1705635982], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T03:52:40.188Z] INFO[2024-01-19 03:52:40] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-1705635982] [2024-01-19T03:52:40.188Z] ERRO[2024-01-19 03:52:40] Actual:rpc error: code = Unknown desc = restore object [tp-restore-1705636305] has reference to backuplocation [autogenerated-backup-location-1705635982:b4816cdb-0621-48dc-a333-0ee37ec2c86c], Expected:  [2024-01-19T03:52:40.188Z] INFO[2024-01-19 03:52:40] --------Test End------ [2024-01-19T03:52:40.188Z] INFO[2024-01-19 03:52:40] #Test: [2024-01-19T03:52:40.188Z] INFO[2024-01-19 03:52:40] #Description: [2024-01-19T03:52:40.188Z] INFO[2024-01-19 03:52:40] ------------------------ [2024-01-19T03:52:40.188Z] 2024-01-19 03:52:40 +0000:[INFO] [{CancelClusterBackupShare}] [tests.EndPxBackupTorpedoTest:#7261] - >>>> FAILED TEST: {CancelClusterBackupShare} Share all backups at cluster level with a user group and revoke it and validate [2024-01-19T03:52:40.445Z] 2024-01-19 03:52:40 +0000:[INFO] [{CancelClusterBackupShare}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-82935-01-19-03h43m53s] [2024-01-19T03:52:40.445Z] 2024-01-19 03:52:40 +0000:[INFO] [{CancelClusterBackupShare}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T03:52:40.445Z] 2024-01-19 03:52:40 +0000:[INFO] [{CancelClusterBackupShare}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T03:52:40.445Z] 2024-01-19 03:52:40 +0000:[INFO] [{CancelClusterBackupShare}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-82935-01-19-03h43m53s] [2024-01-19T03:52:40.446Z] 2024-01-19 03:52:40 +0000:[INFO] [{CancelClusterBackupShare}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {CancelClusterBackupShare} Share all backups at cluster level with a user group and revoke it and validate [2024-01-19T03:52:40.446Z] 2024-01-19 03:52:40 +0000:[INFO] [{CancelClusterBackupShare}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T03:52:40.446Z] 2024/01/19 03:52:40 DoRetryWithTimeout - Error: {dial tcp 10.240.128.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:52:45.695Z] 2024/01/19 03:52:45 DoRetryWithTimeout - Error: {dial tcp 10.240.128.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:52:50.942Z] 2024/01/19 03:52:50 DoRetryWithTimeout - Error: {dial tcp 10.240.128.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:52:56.189Z] 2024/01/19 03:52:55 DoRetryWithTimeout - Error: {dial tcp 10.240.128.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:53:01.431Z] 2024/01/19 03:53:00 DoRetryWithTimeout - Error: {dial tcp 10.240.128.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:53:05.615Z] 2024/01/19 03:53:05 DoRetryWithTimeout - Error: {dial tcp 10.240.128.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:53:10.874Z] 2024/01/19 03:53:10 DoRetryWithTimeout - Error: {dial tcp 10.240.128.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:53:16.121Z] 2024/01/19 03:53:15 DoRetryWithTimeout - Error: {dial tcp 10.240.128.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T03:53:21.371Z] 2024/01/19 03:53:20 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.128.6. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.6 10.240.128.6]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T03:53:21.371Z] {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T03:53:21.371Z] {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T03:53:21.371Z] {dial tcp 10.240.128.6:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T03:53:25.543Z] 2024-01-19 03:53:25 +0000:[WARNING] [{CancelClusterBackupShare}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.6. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.6 10.240.128.6]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T03:53:25.543Z] {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T03:53:25.543Z] {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T03:53:25.543Z] {dial tcp 10.240.128.6:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T03:53:25.543Z] 2024-01-19 03:53:25 +0000:[ERROR] [{CancelClusterBackupShare}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.6. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.6 10.240.128.6]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T03:53:25.543Z] {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T03:53:25.543Z] {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T03:53:25.543Z] {dial tcp 10.240.128.6:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T03:53:25.543Z] 2024-01-19 03:53:25 +0000:[INFO] [{CancelClusterBackupShare}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T03:53:25.543Z] 2024-01-19 03:53:25 +0000:[INFO] [{CancelClusterBackupShare}] Switching context to [/tmp/source-config] [2024-01-19T03:53:25.543Z] 2024-01-19 03:53:25 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T03:53:25.543Z] 2024-01-19 03:53:25 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T03:53:25.799Z] 2024-01-19 03:53:25 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T03:53:25.799Z] 2024-01-19 03:53:25 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T03:53:25.799Z] 2024-01-19 03:53:25 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T03:53:26.055Z] 2024-01-19 03:53:25 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T03:53:26.311Z] 2024-01-19 03:53:26 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T03:53:26.873Z] 2024-01-19 03:53:26 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T03:53:27.131Z] 2024-01-19 03:53:27 +0000:[INFO] [{CancelClusterBackupShare}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T03:53:27.390Z] 2024/01/19 03:53:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:53:37.339Z] 2024/01/19 03:53:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:53:49.509Z] 2024/01/19 03:53:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:53:57.632Z] 2024/01/19 03:53:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:54:07.592Z] 2024/01/19 03:54:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:54:17.536Z] 2024/01/19 03:54:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:54:27.477Z] 2024/01/19 03:54:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:54:39.648Z] 2024/01/19 03:54:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:54:47.733Z] 2024/01/19 03:54:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:54:52.978Z] INFO[2024-01-19 03:54:52] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T03:54:52.978Z] INFO[2024-01-19 03:54:52] sched stats: tick interval: 44452 < 1.2s (100%) [2024-01-19T03:54:52.978Z] INFO[2024-01-19 03:54:52] sched stats: tick process durations: 44451 < 500ms (100%) [2024-01-19T03:54:58.249Z] 2024/01/19 03:54:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:55:08.188Z] 2024/01/19 03:55:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:55:18.133Z] 2024/01/19 03:55:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:55:28.081Z] 2024/01/19 03:55:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:55:38.053Z] 2024/01/19 03:55:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:55:48.125Z] 2024/01/19 03:55:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:55:58.097Z] 2024/01/19 03:55:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:56:08.065Z] 2024/01/19 03:56:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:56:18.008Z] 2024/01/19 03:56:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:56:27.947Z] 2024/01/19 03:56:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:56:37.888Z] 2024/01/19 03:56:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:56:50.077Z] 2024/01/19 03:56:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:56:58.163Z] 2024/01/19 03:56:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:57:08.109Z] 2024/01/19 03:57:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:57:20.281Z] 2024/01/19 03:57:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:57:28.377Z] 2024/01/19 03:57:28 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:57:38.319Z] 2024/01/19 03:57:38 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:57:48.261Z] 2024/01/19 03:57:48 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:58:00.454Z] 2024/01/19 03:57:58 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:58:08.539Z] 2024/01/19 03:58:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:58:18.488Z] 2024/01/19 03:58:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T03:58:28.431Z] 2024-01-19 03:58:28 +0000:[ERROR] [{CancelClusterBackupShare}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.431Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.431Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.431Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.431Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.431Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.431Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.431Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.431Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.431Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.431Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.431Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.431Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.431Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.431Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.431Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.431Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.431Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.431Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.431Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.431Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.431Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.432Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.432Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.432Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.432Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.432Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.432Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.432Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T03:58:28.432Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T03:58:28.432Z] • Failure [875.148 seconds] [2024-01-19T03:58:28.432Z] {CancelClusterBackupShare} [2024-01-19T03:58:28.432Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:1241 [2024-01-19T03:58:28.432Z] Share all backups at cluster level with a user group and revoke it and validate [It] [2024-01-19T03:58:28.432Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:1284 [2024-01-19T03:58:28.432Z] [2024-01-19T03:58:28.432Z] Unexpected error: [2024-01-19T03:58:28.432Z] <*errors.errorString | 0xc0010fb720>: { [2024-01-19T03:58:28.432Z] s: "ValidateRestore Errors: {in restore [tp-restore-1705636305], said restored volume [pvc-260099ea-8990-4ee4-a875-09d25701a318] cannot be found in the actual cluster [mysql-ibm-pxb-0-82935-01-19-03h43m53s]}", [2024-01-19T03:58:28.432Z] } [2024-01-19T03:58:28.432Z] ValidateRestore Errors: {in restore [tp-restore-1705636305], said restored volume [pvc-260099ea-8990-4ee4-a875-09d25701a318] cannot be found in the actual cluster [mysql-ibm-pxb-0-82935-01-19-03h43m53s]} [2024-01-19T03:58:28.432Z] occurred [2024-01-19T03:58:28.432Z] [2024-01-19T03:58:28.432Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T03:58:28.432Z] [2024-01-19T03:58:28.432Z] Full Stack Trace [2024-01-19T03:58:28.432Z] github.com/portworx/torpedo/pkg/log.FailOnError({0x88a5060, 0xc0010fb720}, {0x7fb35bb?, 0x7?}, {0xc0014f3020?, 0x1?, 0x1?}) [2024-01-19T03:58:28.432Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 +0x290 [2024-01-19T03:58:28.432Z] github.com/portworx/torpedo/tests/backup.glob..func85.2.9() [2024-01-19T03:58:28.432Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:1454 +0x774 [2024-01-19T03:58:28.432Z] github.com/portworx/torpedo/tests/backup.glob..func85.2() [2024-01-19T03:58:28.432Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:1427 +0x9f1 [2024-01-19T03:58:28.432Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T03:58:28.432Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T03:58:28.432Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T03:58:28.432Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T03:58:28.432Z] created by testing.(*T).Run [2024-01-19T03:58:28.432Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T03:58:28.432Z] ------------------------------ [2024-01-19T03:58:28.432Z] STEP: schedule applications [2024-01-19T03:58:28.432Z] STEP: Validate applications and get their labels [2024-01-19T03:58:28.432Z] STEP: validate applications [2024-01-19T03:58:28.432Z] STEP: validate mysql-ibm app's volumes [2024-01-19T03:58:28.432Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T03:58:28.432Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T03:58:28.433Z] STEP: get mysql-ibm app's volume: pvc-90e3a732-2ebc-496e-a729-97557156d692 inspected by the volume driver [2024-01-19T03:58:28.433Z] STEP: wait for mysql-ibm app to start running [2024-01-19T03:58:28.433Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T03:58:28.433Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T03:58:28.433Z] STEP: Validate Px pod restart count [2024-01-19T03:58:28.433Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T03:58:28.433Z] STEP: Create Users [2024-01-19T03:58:28.433Z] STEP: Create Groups [2024-01-19T03:58:28.433Z] STEP: Add users to group [2024-01-19T03:58:28.433Z] STEP: Adding Credentials and Registering Backup Location [2024-01-19T03:58:28.433Z] STEP: Register source and destination cluster for backup [2024-01-19T03:58:28.433Z] STEP: Taking backup of applications [2024-01-19T03:58:28.433Z] STEP: Share all backups with Full Access in source cluster with a group and a user who is not part of the group [2024-01-19T03:58:28.433Z] STEP: Validate Full Access of backups shared at cluster level [2024-01-19T03:58:28.433Z] STEP: validate mysql-ibm app's volumes [2024-01-19T03:58:28.433Z] STEP: wait for mysql-ibm app to start running [2024-01-19T03:58:28.433Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T03:58:28.433Z] STEP: Validate Px pod restart count [2024-01-19T03:58:28.433Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T03:58:28.433Z] STEP: destroy apps [2024-01-19T03:58:28.433Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T03:58:28.433Z] STEP: start destroying mysql-ibm app [2024-01-19T03:58:28.433Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T03:58:28.433Z] 2024-01-19 03:58:28 +0000:[INFO] [{VerifyRBACForPxAdmin}] Log Dir: /testresults//VerifyRBACForPxAdmin.log [2024-01-19T03:58:28.433Z] INFO[2024-01-19 03:58:28] --------Test Start------ [2024-01-19T03:58:28.433Z] INFO[2024-01-19 03:58:28] #Test: VerifyRBACForPxAdmin [2024-01-19T03:58:28.433Z] INFO[2024-01-19 03:58:28] #Description: Validates the RBAC operation for px-admin group user. [2024-01-19T03:58:28.433Z] INFO[2024-01-19 03:58:28] ------------------------ [2024-01-19T03:58:28.433Z] 2024-01-19 03:58:28 +0000:[INFO] [{VerifyRBACForPxAdmin}] scheduling applications [2024-01-19T03:58:28.433Z] 2024-01-19 03:58:28 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T03:58:28.433Z] 2024-01-19 03:58:28 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T03:58:28.689Z] 2024-01-19 03:58:28 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T03:58:28.689Z] 2024-01-19 03:58:28 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T03:58:28.689Z] 2024-01-19 03:58:28 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T03:58:28.690Z] 2024-01-19 03:58:28 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T03:58:28.690Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T03:58:28.690Z] 2024-01-19 03:58:28 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T03:58:28.691Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T03:58:28.947Z] 2024-01-19 03:58:28 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T03:58:28.947Z] 2024-01-19 03:58:28 +0000:[INFO] [{VerifyRBACForPxAdmin}] Validate applications [2024-01-19T03:58:28.947Z] 2024-01-19 03:58:28 +0000:[INFO] [{VerifyRBACForPxAdmin}] Validating mysql-ibm app [2024-01-19T03:58:28.947Z] 2024-01-19 03:58:28 +0000:[INFO] [{VerifyRBACForPxAdmin}] Validating mysql-ibm app's volumes [2024-01-19T03:58:28.947Z] 2024-01-19 03:58:28 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T03:58:28.947Z] 2024/01/19 03:58:28 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T03:58:41.114Z] 2024/01/19 03:58:38 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T03:58:49.200Z] 2024-01-19 03:58:48 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-87887-01-19-03h58m28s [2024-01-19T03:58:49.200Z] 2024-01-19 03:58:48 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T03:58:49.200Z] 2024-01-19 03:58:49 +0000:[WARNING] [{VerifyRBACForPxAdmin}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T03:58:49.200Z] 2024-01-19 03:58:49 +0000:[INFO] [{VerifyRBACForPxAdmin}] wait for mysql-ibm app to start running [2024-01-19T03:58:49.200Z] 2024-01-19 03:58:49 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T03:58:49.457Z] 2024/01/19 03:58:49 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:58:49.457Z] pod name:mysql-7d766cd65c-pbf5j namespace:mysql-ibm-pxb-0-87887-01-19-03h58m28s running:false ready:false node:10.240.64.8 [2024-01-19T03:58:49.457Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:59:01.625Z] 2024/01/19 03:58:59 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:59:01.625Z] pod name:mysql-7d766cd65c-pbf5j namespace:mysql-ibm-pxb-0-87887-01-19-03h58m28s running:true ready:false node:10.240.64.8 [2024-01-19T03:59:01.625Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:59:09.710Z] 2024/01/19 03:59:09 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:59:09.710Z] pod name:mysql-7d766cd65c-pbf5j namespace:mysql-ibm-pxb-0-87887-01-19-03h58m28s running:true ready:false node:10.240.64.8 [2024-01-19T03:59:09.710Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:59:21.931Z] 2024/01/19 03:59:19 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:59:21.931Z] pod name:mysql-7d766cd65c-pbf5j namespace:mysql-ibm-pxb-0-87887-01-19-03h58m28s running:true ready:false node:10.240.64.8 [2024-01-19T03:59:21.931Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:59:30.015Z] 2024/01/19 03:59:29 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:59:30.015Z] pod name:mysql-7d766cd65c-pbf5j namespace:mysql-ibm-pxb-0-87887-01-19-03h58m28s running:true ready:false node:10.240.64.8 [2024-01-19T03:59:30.015Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:59:42.205Z] 2024/01/19 03:59:40 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:59:42.205Z] pod name:mysql-7d766cd65c-pbf5j namespace:mysql-ibm-pxb-0-87887-01-19-03h58m28s running:true ready:false node:10.240.64.8 [2024-01-19T03:59:42.205Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T03:59:50.323Z] 2024/01/19 03:59:50 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T03:59:50.323Z] pod name:mysql-7d766cd65c-pbf5j namespace:mysql-ibm-pxb-0-87887-01-19-03h58m28s running:true ready:false node:10.240.64.8 [2024-01-19T03:59:50.323Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T04:00:02.497Z] 2024/01/19 04:00:00 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T04:00:02.497Z] pod name:mysql-7d766cd65c-pbf5j namespace:mysql-ibm-pxb-0-87887-01-19-03h58m28s running:true ready:false node:10.240.64.8 [2024-01-19T04:00:02.497Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T04:00:10.586Z] 2024-01-19 04:00:10 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T04:00:10.841Z] 2024-01-19 04:00:10 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T04:00:10.841Z] 2024-01-19 04:00:10 +0000:[INFO] [{VerifyRBACForPxAdmin}] validate if mysql-ibm app's volumes are setup [2024-01-19T04:00:10.841Z] 2024-01-19 04:00:10 +0000:[WARNING] [{VerifyRBACForPxAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:00:10.841Z] 2024-01-19 04:00:10 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:00:10.842Z] 2024-01-19 04:00:10 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T04:00:10.842Z] 2024-01-19 04:00:10 +0000:[WARNING] [{VerifyRBACForPxAdmin}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T04:00:11.100Z] 2024-01-19 04:00:10 +0000:[INFO] [{VerifyRBACForPxAdmin}] Verify Px-Admin group user has permission to create cloud credential and backup location [2024-01-19T04:00:11.355Z] 2024-01-19 04:00:11 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-eajcq] for org [default] with [aws] as provider [2024-01-19T04:00:11.355Z] 2024-01-19 04:00:11 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T04:00:11.355Z] 2024-01-19 04:00:11 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-eajcq] is present [2024-01-19T04:00:11.355Z] INFO[2024-01-19 04:00:11] Verifying : Description : Verifying creation of create cloud credential autogenerated-cred-eajcq using provider aws for the px-admin [2024-01-19T04:00:11.355Z] INFO[2024-01-19 04:00:11] Actual:true, Expected: true [2024-01-19T04:00:13.869Z] INFO[2024-01-19 04:00:13] Verifying : Description : Verifying creation of create backup location autogenerated-backup-location-mnpna using provider aws for the px-admin [2024-01-19T04:00:13.869Z] INFO[2024-01-19 04:00:13] Actual:true, Expected: true [2024-01-19T04:00:13.869Z] 2024-01-19 04:00:13 +0000:[INFO] [{VerifyRBACForPxAdmin}] Verify Px-Admin User has permission to create a schedule policy [2024-01-19T04:00:14.430Z] INFO[2024-01-19 04:00:14] Verifying : Description : Verifying creation of periodic schedule policy of interval [15] minutes named [periodic-enoyp] [2024-01-19T04:00:14.430Z] INFO[2024-01-19 04:00:14] Actual:true, Expected: true [2024-01-19T04:00:14.430Z] 2024-01-19 04:00:14 +0000:[INFO] [{VerifyRBACForPxAdmin}] Verify Px-Admin User has permission to create pre and post exec rules for applications [2024-01-19T04:00:14.990Z] 2024-01-19 04:00:14 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.CreateRuleForBackupWithMultipleApplications:#4309] - Validate pre-rules for backup [2024-01-19T04:00:14.991Z] 2024-01-19 04:00:14 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.CreateRuleForBackupWithMultipleApplications:#4322] - Validate post-rules for backup [2024-01-19T04:00:14.991Z] INFO[2024-01-19 04:00:14] Verifying : Description : Verifying creation of pre and post exec rules for applications from px-admin [2024-01-19T04:00:14.991Z] INFO[2024-01-19 04:00:14] Actual:true, Expected: true [2024-01-19T04:00:14.991Z] 2024-01-19 04:00:14 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.glob..func44.2.4:#856] - Pre backup rule [pre-rule-dzrbc] uid: [582eecdf-8aea-4640-b2f2-75bf2c40fd1d] [2024-01-19T04:00:15.246Z] 2024-01-19 04:00:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.glob..func44.2.4:#861] - Post backup rule [post-rule-hakcj] uid: [3680e75e-91ed-41a4-bf39-1d5fa4da4fd8] [2024-01-19T04:00:15.246Z] 2024-01-19 04:00:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] Verify Px-Admin User has permission to create new custom role [2024-01-19T04:00:18.512Z] INFO[2024-01-19 04:00:17] Verifying : Description : Verifying creation of role [custom-px-admin-role-wedd] by the px-admin [2024-01-19T04:00:18.512Z] INFO[2024-01-19 04:00:17] Actual:true, Expected: true [2024-01-19T04:00:18.512Z] 2024-01-19 04:00:17 +0000:[INFO] [{VerifyRBACForPxAdmin}] Create Users with Different types of roles [2024-01-19T04:00:18.512Z] 2024-01-19 04:00:17 +0000:[INFO] [{VerifyRBACForPxAdmin}] Creating 1 users [2024-01-19T04:00:18.767Z] 2024-01-19 04:00:18 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.AddUser:#728] - Response for user [testuser1-1705636817] creation - [2024-01-19T04:00:18.767Z] 2024-01-19 04:00:18 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.AddUser:#733] - User [testuser1-1705636817] created [2024-01-19T04:00:18.767Z] INFO[2024-01-19 04:00:18] Verifying : Description : Creating user - testuser1-1705636817 [2024-01-19T04:00:18.767Z] INFO[2024-01-19 04:00:18] Actual:true, Expected: true [2024-01-19T04:00:19.694Z] 2024-01-19 04:00:19 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705636817 - 3a8eee2e-3ae6-47b2-99f5-ae682fc2b14d [2024-01-19T04:00:21.059Z] 2024-01-19 04:00:20 +0000:[INFO] [{VerifyRBACForPxAdmin}] Creating 1 users [2024-01-19T04:00:21.989Z] 2024-01-19 04:00:21 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.AddUser:#728] - Response for user [testuser1-1705636820] creation - [2024-01-19T04:00:21.989Z] 2024-01-19 04:00:21 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.AddUser:#733] - User [testuser1-1705636820] created [2024-01-19T04:00:21.989Z] INFO[2024-01-19 04:00:21] Verifying : Description : Creating user - testuser1-1705636820 [2024-01-19T04:00:21.989Z] INFO[2024-01-19 04:00:21] Actual:true, Expected: true [2024-01-19T04:00:22.555Z] 2024-01-19 04:00:22 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705636820 - 52e2390b-1c29-44bd-969c-e1fb4ceb50b3 [2024-01-19T04:00:24.481Z] 2024-01-19 04:00:24 +0000:[INFO] [{VerifyRBACForPxAdmin}] Creating 1 users [2024-01-19T04:00:25.046Z] 2024-01-19 04:00:24 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.AddUser:#728] - Response for user [testuser1-1705636824] creation - [2024-01-19T04:00:25.046Z] 2024-01-19 04:00:24 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.AddUser:#733] - User [testuser1-1705636824] created [2024-01-19T04:00:25.047Z] INFO[2024-01-19 04:00:24] Verifying : Description : Creating user - testuser1-1705636824 [2024-01-19T04:00:25.047Z] INFO[2024-01-19 04:00:24] Actual:true, Expected: true [2024-01-19T04:00:25.982Z] 2024-01-19 04:00:25 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705636824 - 744d6f61-7c58-41c2-99b5-b79fe60034fa [2024-01-19T04:00:27.352Z] 2024-01-19 04:00:27 +0000:[INFO] [{VerifyRBACForPxAdmin}] Verify px-admin User has permission to share RBAC resources with non-admin users [2024-01-19T04:00:28.715Z] 2024-01-19 04:00:28 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.glob..func44.2.7:#896] - Update CloudAccount - autogenerated-cred-eajcq ownership for users - [[testuser1-1705636817 testuser1-1705636820 testuser1-1705636824]] [2024-01-19T04:00:29.649Z] 2024-01-19 04:00:29 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705636817 - 3a8eee2e-3ae6-47b2-99f5-ae682fc2b14d [2024-01-19T04:00:30.585Z] 2024-01-19 04:00:30 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705636820 - 52e2390b-1c29-44bd-969c-e1fb4ceb50b3 [2024-01-19T04:00:31.146Z] 2024-01-19 04:00:31 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705636824 - 744d6f61-7c58-41c2-99b5-b79fe60034fa [2024-01-19T04:00:31.403Z] INFO[2024-01-19 04:00:31] Verifying : Description : Verifying updation of owbership for CloudCredential- autogenerated-cred-eajcq [2024-01-19T04:00:31.403Z] INFO[2024-01-19 04:00:31] Actual:true, Expected: true [2024-01-19T04:00:31.403Z] 2024-01-19 04:00:31 +0000:[INFO] [{VerifyRBACForPxAdmin}] Update BackupLocation - autogenerated-backup-location-mnpna ownership for users - [[testuser1-1705636817 testuser1-1705636820 testuser1-1705636824]] [2024-01-19T04:00:31.964Z] 2024-01-19 04:00:31 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705636817 - 3a8eee2e-3ae6-47b2-99f5-ae682fc2b14d [2024-01-19T04:00:32.890Z] 2024-01-19 04:00:32 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705636820 - 52e2390b-1c29-44bd-969c-e1fb4ceb50b3 [2024-01-19T04:00:33.816Z] 2024-01-19 04:00:33 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705636824 - 744d6f61-7c58-41c2-99b5-b79fe60034fa [2024-01-19T04:00:33.816Z] INFO[2024-01-19 04:00:33] Verifying : Description : Verifying updation of ownership for backuplocation - autogenerated-backup-location-mnpna [2024-01-19T04:00:33.816Z] INFO[2024-01-19 04:00:33] Actual:true, Expected: true [2024-01-19T04:00:33.816Z] 2024-01-19 04:00:33 +0000:[INFO] [{VerifyRBACForPxAdmin}] Update SchedulePolicy - periodic-enoyp ownership for users - [[testuser1-1705636817 testuser1-1705636820 testuser1-1705636824]] [2024-01-19T04:00:34.376Z] 2024-01-19 04:00:34 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705636817 - 3a8eee2e-3ae6-47b2-99f5-ae682fc2b14d [2024-01-19T04:00:35.302Z] 2024-01-19 04:00:35 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705636820 - 52e2390b-1c29-44bd-969c-e1fb4ceb50b3 [2024-01-19T04:00:36.230Z] 2024-01-19 04:00:35 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705636824 - 744d6f61-7c58-41c2-99b5-b79fe60034fa [2024-01-19T04:00:36.230Z] INFO[2024-01-19 04:00:35] Verifying : Description : Verifying updation of ownership for schedulepolicy - periodic-enoyp [2024-01-19T04:00:36.230Z] INFO[2024-01-19 04:00:35] Actual:true, Expected: true [2024-01-19T04:00:36.230Z] 2024-01-19 04:00:35 +0000:[INFO] [{VerifyRBACForPxAdmin}] Update Application Rules ownership for users - [[testuser1-1705636817 testuser1-1705636820 testuser1-1705636824]] [2024-01-19T04:00:36.790Z] 2024-01-19 04:00:36 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705636817 - 3a8eee2e-3ae6-47b2-99f5-ae682fc2b14d [2024-01-19T04:00:37.715Z] 2024-01-19 04:00:37 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705636820 - 52e2390b-1c29-44bd-969c-e1fb4ceb50b3 [2024-01-19T04:00:38.277Z] 2024-01-19 04:00:38 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705636824 - 744d6f61-7c58-41c2-99b5-b79fe60034fa [2024-01-19T04:00:38.533Z] INFO[2024-01-19 04:00:38] Verifying : Description : Verifying updation of ownership for pre-rule of application [2024-01-19T04:00:38.533Z] INFO[2024-01-19 04:00:38] Actual:true, Expected: true [2024-01-19T04:00:39.531Z] 2024-01-19 04:00:39 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705636817 - 3a8eee2e-3ae6-47b2-99f5-ae682fc2b14d [2024-01-19T04:00:40.100Z] 2024-01-19 04:00:39 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705636820 - 52e2390b-1c29-44bd-969c-e1fb4ceb50b3 [2024-01-19T04:00:40.660Z] 2024-01-19 04:00:40 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705636824 - 744d6f61-7c58-41c2-99b5-b79fe60034fa [2024-01-19T04:00:40.916Z] INFO[2024-01-19 04:00:40] Verifying : Description : Verifying updation of ownership for post-rule of application [2024-01-19T04:00:40.916Z] INFO[2024-01-19 04:00:40] Actual:true, Expected: true [2024-01-19T04:00:40.916Z] 2024-01-19 04:00:40 +0000:[INFO] [{VerifyRBACForPxAdmin}] Create source and destination cluster from the px-admin [2024-01-19T04:00:42.282Z] INFO[2024-01-19 04:00:42] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T04:00:42.282Z] INFO[2024-01-19 04:00:42] Actual:true, Expected: true [2024-01-19T04:00:42.282Z] 2024-01-19 04:00:42 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T04:00:42.282Z] 2024-01-19 04:00:42 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T04:00:42.282Z] 2024-01-19 04:00:42 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T04:00:42.282Z] 2024-01-19 04:00:42 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T04:00:42.282Z] 2024-01-19 04:00:42 +0000:[INFO] [{VerifyRBACForPxAdmin}] Create cluster [source-cluster] in org [default] [2024-01-19T04:00:42.282Z] 2024-01-19 04:00:42 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T04:00:42.282Z] 2024-01-19 04:00:42 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T04:00:42.282Z] 2024-01-19 04:00:42 +0000:[INFO] [{VerifyRBACForPxAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-19T04:00:42.282Z] 2024-01-19 04:00:42 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T04:00:42.282Z] 2024-01-19 04:00:42 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T04:00:42.282Z] 2024-01-19 04:00:42 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-vmlxp] for IBM [2024-01-19T04:00:42.282Z] 2024-01-19 04:00:42 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-vmlxp] for org [default] with [ibm] as provider [2024-01-19T04:00:42.282Z] 2024-01-19 04:00:42 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T04:00:42.282Z] 2024-01-19 04:00:42 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T04:00:42.538Z] 2024-01-19 04:00:42 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-vmlxp] is present [2024-01-19T04:00:42.538Z] 2024-01-19 04:00:42 +0000:[INFO] [{VerifyRBACForPxAdmin}] Create cluster [source-cluster] in org [default] [2024-01-19T04:00:42.538Z] 2024-01-19 04:00:42 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-sztik] for IBM [2024-01-19T04:00:42.538Z] 2024-01-19 04:00:42 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-sztik] for org [default] with [ibm] as provider [2024-01-19T04:00:42.538Z] 2024-01-19 04:00:42 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T04:00:42.538Z] 2024-01-19 04:00:42 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T04:00:42.538Z] 2024-01-19 04:00:42 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-sztik] is present [2024-01-19T04:00:42.538Z] 2024-01-19 04:00:42 +0000:[INFO] [{VerifyRBACForPxAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-19T04:00:42.796Z] INFO[2024-01-19 04:00:42] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T04:00:42.796Z] INFO[2024-01-19 04:00:42] Actual:Online, Expected: Online [2024-01-19T04:00:42.796Z] INFO[2024-01-19 04:00:42] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T04:00:42.796Z] INFO[2024-01-19 04:00:42] Actual:true, Expected: true [2024-01-19T04:00:42.796Z] 2024-01-19 04:00:42 +0000:[INFO] [{VerifyRBACForPxAdmin}] Taking backup of applications from the px-admin [2024-01-19T04:00:43.733Z] 2024-01-19 04:00:43 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T04:00:43.733Z] 2024-01-19 04:00:43 +0000:[INFO] [{VerifyRBACForPxAdmin}] Volume snapshot classes found in the cluster - [2024-01-19T04:00:43.733Z] 2024-01-19 04:00:43 +0000:[INFO] [{VerifyRBACForPxAdmin}] ibmc-vpcblock-snapshot [2024-01-19T04:00:43.733Z] 2024-01-19 04:00:43 +0000:[INFO] [{VerifyRBACForPxAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T04:00:44.294Z] 2024/01/19 04:00:44 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-with-rules-bfdr] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T04:01:16.323Z] 2024/01/19 04:01:14 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-with-rules-bfdr] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:01:48.336Z] 2024/01/19 04:01:44 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-single-ns-with-rules-bfdr] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:02:14.934Z] 2024-01-19 04:02:14 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.CreateBackup:#313] - Backup [tp-backup-manual-single-ns-with-rules-bfdr] created successfully [2024-01-19T04:02:14.934Z] 2024-01-19 04:02:14 +0000:[INFO] [{VerifyRBACForPxAdmin}] Validating backup [tp-backup-manual-single-ns-with-rules-bfdr] in org [default] [2024-01-19T04:02:14.934Z] 2024-01-19 04:02:14 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-manual-single-ns-with-rules-bfdr] [2024-01-19T04:02:14.934Z] 2024-01-19 04:02:14 +0000:[INFO] [{VerifyRBACForPxAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87887-01-19-03h58m28s] in backup [tp-backup-manual-single-ns-with-rules-bfdr] [2024-01-19T04:02:14.934Z] 2024-01-19 04:02:14 +0000:[INFO] [{VerifyRBACForPxAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87887-01-19-03h58m28s] in backup [tp-backup-manual-single-ns-with-rules-bfdr] [2024-01-19T04:02:14.934Z] 2024-01-19 04:02:14 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87887-01-19-03h58m28s] [2024-01-19T04:02:14.934Z] 2024-01-19 04:02:14 +0000:[WARNING] [{VerifyRBACForPxAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:02:14.934Z] 2024-01-19 04:02:14 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:02:14.934Z] 2024-01-19 04:02:14 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87887-01-19-03h58m28s] are [[mysql-data]] [2024-01-19T04:02:14.934Z] 2024-01-19 04:02:14 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:02:14.934Z] 2024-01-19 04:02:14 +0000:[INFO] [{VerifyRBACForPxAdmin}] Validating CR cleanup [2024-01-19T04:02:14.934Z] 2024-01-19 04:02:14 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T04:02:14.934Z] 2024-01-19 04:02:14 +0000:[INFO] [{VerifyRBACForPxAdmin}] All CRs in [mysql-ibm-pxb-0-87887-01-19-03h58m28s] are [[]] [2024-01-19T04:02:14.935Z] INFO[2024-01-19 04:02:14] Verifying : Description : Verifying creation and validation of backup [tp-backup-manual-single-ns-with-rules-bfdr] of namespace (scheduled Context) [[mysql-ibm-pxb-0-87887-01-19-03h58m28s]] [2024-01-19T04:02:14.935Z] INFO[2024-01-19 04:02:14] Actual:true, Expected: true [2024-01-19T04:02:14.935Z] 2024-01-19 04:02:14 +0000:[INFO] [{VerifyRBACForPxAdmin}] Taking restore of applications from the px-admin [2024-01-19T04:02:15.193Z] 2024/01/19 04:02:15 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-manual-single-ns-with-rules-bfdr] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T04:02:47.225Z] 2024/01/19 04:02:45 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-manual-single-ns-with-rules-bfdr] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:03:19.288Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.CreateRestore:#936] - Restore [tp-restore-tp-backup-manual-single-ns-with-rules-bfdr] created successfully [2024-01-19T04:03:19.288Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] Switching cluster context to cluster [destination-cluster] [2024-01-19T04:03:19.288Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T04:03:19.288Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-87887-01-19-03h58m28s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-19T04:03:19.288Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-87887-01-19-03h58m28s] [2024-01-19T04:03:19.288Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T04:03:19.289Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T04:03:19.289Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] Validating restore [tp-restore-tp-backup-manual-single-ns-with-rules-bfdr] in org [default] [2024-01-19T04:03:19.289Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-restore-tp-backup-manual-single-ns-with-rules-bfdr] [2024-01-19T04:03:19.289Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-restore-tp-backup-manual-single-ns-with-rules-bfdr] [2024-01-19T04:03:19.289Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-87887-01-19-03h58m28s] in restore [tp-restore-tp-backup-manual-single-ns-with-rules-bfdr] [2024-01-19T04:03:19.289Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-87887-01-19-03h58m28s]) was found in restore [tp-restore-tp-backup-manual-single-ns-with-rules-bfdr], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87887-01-19-03h58m28s] [2024-01-19T04:03:19.289Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-87887-01-19-03h58m28s]) found in the restore [tp-restore-tp-backup-manual-single-ns-with-rules-bfdr] was also present on the cluster/namespace [mysql-ibm-pxb-0-87887-01-19-03h58m28s] [2024-01-19T04:03:19.289Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-87887-01-19-03h58m28s]) was found in restore [tp-restore-tp-backup-manual-single-ns-with-rules-bfdr], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87887-01-19-03h58m28s] [2024-01-19T04:03:19.289Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-87887-01-19-03h58m28s]) found in the restore [tp-restore-tp-backup-manual-single-ns-with-rules-bfdr] was also present on the cluster/namespace [mysql-ibm-pxb-0-87887-01-19-03h58m28s] [2024-01-19T04:03:19.289Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-87887-01-19-03h58m28s]) was found in restore [tp-restore-tp-backup-manual-single-ns-with-rules-bfdr], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87887-01-19-03h58m28s] [2024-01-19T04:03:19.289Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-87887-01-19-03h58m28s]) found in the restore [tp-restore-tp-backup-manual-single-ns-with-rules-bfdr] was also present on the cluster/namespace [mysql-ibm-pxb-0-87887-01-19-03h58m28s] [2024-01-19T04:03:19.289Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-87887-01-19-03h58m28s]) was found in restore [tp-restore-tp-backup-manual-single-ns-with-rules-bfdr], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87887-01-19-03h58m28s] [2024-01-19T04:03:19.289Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-87887-01-19-03h58m28s]) found in the restore [tp-restore-tp-backup-manual-single-ns-with-rules-bfdr] was also present on the cluster/namespace [mysql-ibm-pxb-0-87887-01-19-03h58m28s] [2024-01-19T04:03:19.289Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-87887-01-19-03h58m28s] in restore [tp-restore-tp-backup-manual-single-ns-with-rules-bfdr] [2024-01-19T04:03:19.290Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-87887-01-19-03h58m28s] in restore [tp-restore-tp-backup-manual-single-ns-with-rules-bfdr] [2024-01-19T04:03:19.290Z] 2024-01-19 04:03:15 +0000:[WARNING] [{VerifyRBACForPxAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:03:19.290Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:03:19.290Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87887-01-19-03h58m28s] are [[mysql-data]] [2024-01-19T04:03:19.290Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.ValidateRestore:#2240] - in restore [tp-restore-tp-backup-manual-single-ns-with-rules-bfdr], the status of the restored volume [pvc-672c8c45-9895-4cfc-b350-5afaba0ef385] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-19T04:03:19.290Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] Validate applications in restored namespace [mysql-ibm-pxb-0-87887-01-19-03h58m28s] due to restore [tp-restore-tp-backup-manual-single-ns-with-rules-bfdr] [2024-01-19T04:03:19.290Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] Validating mysql-ibm app [2024-01-19T04:03:19.290Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] wait for mysql-ibm app to start running [2024-01-19T04:03:19.290Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T04:03:19.290Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T04:03:19.290Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T04:03:19.290Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] Switching cluster context back to cluster path [/tmp/destination-config] [2024-01-19T04:03:19.290Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T04:03:19.290Z] INFO[2024-01-19 04:03:15] Verifying : Description : Verifying creation of restore tp-restore-tp-backup-manual-single-ns-with-rules-bfdr of backup tp-backup-manual-single-ns-with-rules-bfdr [2024-01-19T04:03:19.290Z] ERRO[2024-01-19 04:03:15] ValidateRestore Errors: {in restore [tp-restore-tp-backup-manual-single-ns-with-rules-bfdr], said restored volume [pvc-672c8c45-9895-4cfc-b350-5afaba0ef385] cannot be found in the actual cluster [mysql-ibm-pxb-0-87887-01-19-03h58m28s]} [2024-01-19T04:03:19.290Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] Deleting the deployed apps after the testcase [2024-01-19T04:03:19.290Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] Destroying apps [2024-01-19T04:03:19.290Z] 2024-01-19 04:03:15 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T04:03:19.290Z] 2024-01-19 04:03:16 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T04:03:19.290Z] 2024-01-19 04:03:16 +0000:[INFO] [{VerifyRBACForPxAdmin}] start destroying mysql-ibm app [2024-01-19T04:03:19.290Z] 2024-01-19 04:03:16 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T04:03:19.290Z] 2024-01-19 04:03:16 +0000:[INFO] [{VerifyRBACForPxAdmin}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T04:03:19.290Z] 2024-01-19 04:03:16 +0000:[WARNING] [{VerifyRBACForPxAdmin}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T04:03:19.290Z] INFO[2024-01-19 04:03:16] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T04:03:19.290Z] INFO[2024-01-19 04:03:16] Actual:true, Expected: true [2024-01-19T04:03:19.290Z] 2024-01-19 04:03:16 +0000:[INFO] [{VerifyRBACForPxAdmin}] Deleting the px-backup objects [2024-01-19T04:03:19.291Z] 2024-01-19 04:03:16 +0000:[INFO] [{VerifyRBACForPxAdmin}] Cleaning backup locations in map [map[899901ac-46db-4a00-93cb-17daff12eadc:autogenerated-backup-location-mnpna]], cloud credential [autogenerated-cred-eajcq], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T04:03:19.291Z] INFO[2024-01-19 04:03:16] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-mnpna] [2024-01-19T04:03:19.291Z] INFO[2024-01-19 04:03:16] Actual:true, Expected: true [2024-01-19T04:03:19.291Z] 2024-01-19 04:03:16 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-mnpna] is present [2024-01-19T04:03:19.291Z] 2024/01/19 04:03:16 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-mnpna is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T04:03:51.378Z] 2024-01-19 04:03:46 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-mnpna] is present [2024-01-19T04:03:51.378Z] 2024/01/19 04:03:46 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-mnpna is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T04:04:17.877Z] 2024-01-19 04:04:16 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-mnpna] is present [2024-01-19T04:04:17.878Z] 2024/01/19 04:04:16 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-mnpna is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T04:04:49.914Z] 2024-01-19 04:04:46 +0000:[INFO] [{VerifyRBACForPxAdmin}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 aws-global-aws-px-backup-system-test-4cac76dc-bl autogenerated-backup-location-1705628002 autogenerated-backup-location-1705634221 autogenerated-backup-location-1705635982] [2024-01-19T04:04:49.914Z] INFO[2024-01-19 04:04:46] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-mnpna [2024-01-19T04:04:49.914Z] INFO[2024-01-19 04:04:46] Actual:true, Expected: true [2024-01-19T04:04:49.914Z] 2024-01-19 04:04:47 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-eajcq] is present [2024-01-19T04:04:49.914Z] INFO[2024-01-19 04:04:47] Verifying : Description : Verifying if cloud cred [autogenerated-cred-eajcq] is present [2024-01-19T04:04:49.914Z] INFO[2024-01-19 04:04:47] Actual:true, Expected: true [2024-01-19T04:04:49.914Z] INFO[2024-01-19 04:04:47] Verifying : Description : Verifying deletion of cloud cred [autogenerated-cred-eajcq] [2024-01-19T04:04:49.914Z] INFO[2024-01-19 04:04:47] Actual:true, Expected: true [2024-01-19T04:04:49.914Z] INFO[2024-01-19 04:04:47] Verifying : Description : Deleting cloud cred autogenerated-cred-eajcq [2024-01-19T04:04:49.914Z] INFO[2024-01-19 04:04:47] Actual:true, Expected: true [2024-01-19T04:04:49.915Z] INFO[2024-01-19 04:04:47] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T04:04:49.915Z] INFO[2024-01-19 04:04:47] Actual:true, Expected: true [2024-01-19T04:04:49.915Z] INFO[2024-01-19 04:04:47] Verifying : Description : Deleting cluster source-cluster [2024-01-19T04:04:49.915Z] INFO[2024-01-19 04:04:47] Actual:true, Expected: true [2024-01-19T04:04:49.915Z] INFO[2024-01-19 04:04:47] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-cvqse] [2024-01-19T04:04:49.915Z] INFO[2024-01-19 04:04:47] Actual:true, Expected: true [2024-01-19T04:04:49.915Z] INFO[2024-01-19 04:04:47] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T04:04:49.915Z] INFO[2024-01-19 04:04:47] Actual:true, Expected: true [2024-01-19T04:04:49.915Z] INFO[2024-01-19 04:04:48] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-jzfsm] [2024-01-19T04:04:49.915Z] INFO[2024-01-19 04:04:48] Actual:true, Expected: true [2024-01-19T04:04:49.915Z] 2024-01-19 04:04:48 +0000:[INFO] [{VerifyRBACForPxAdmin}] Switching context to destination cluster for clean up [2024-01-19T04:04:49.915Z] 2024-01-19 04:04:48 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T04:04:49.915Z] 2024-01-19 04:04:48 +0000:[INFO] [{VerifyRBACForPxAdmin}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T04:04:49.915Z] 2024-01-19 04:04:48 +0000:[INFO] [{VerifyRBACForPxAdmin}] Destroying apps [2024-01-19T04:04:49.915Z] 2024-01-19 04:04:48 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T04:04:49.915Z] 2024-01-19 04:04:48 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).DeleteVolumes:#4024] - [mysql-ibm] PVC is not found: mysql-data, skipping deletion [2024-01-19T04:04:49.915Z] 2024-01-19 04:04:48 +0000:[INFO] [{VerifyRBACForPxAdmin}] start destroying mysql-ibm app [2024-01-19T04:04:49.915Z] 2024-01-19 04:04:48 +0000:[INFO] [{VerifyRBACForPxAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T04:04:49.915Z] 2024-01-19 04:04:48 +0000:[INFO] [{VerifyRBACForPxAdmin}] Switching context to [/tmp/source-config] [2024-01-19T04:04:49.915Z] 2024-01-19 04:04:48 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T04:04:49.915Z] 2024-01-19 04:04:48 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T04:04:49.915Z] 2024-01-19 04:04:48 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T04:04:49.915Z] 2024-01-19 04:04:48 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T04:04:49.915Z] 2024-01-19 04:04:48 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T04:04:49.915Z] 2024-01-19 04:04:49 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T04:04:49.915Z] 2024-01-19 04:04:49 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T04:04:49.915Z] 2024-01-19 04:04:49 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T04:04:50.477Z] 2024-01-19 04:04:50 +0000:[INFO] [{VerifyRBACForPxAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T04:04:50.477Z] 2024/01/19 04:04:50 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:04:53.747Z] INFO[2024-01-19 04:04:53] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T04:04:53.747Z] INFO[2024-01-19 04:04:53] sched stats: tick interval: 45053 < 1.2s (100%) [2024-01-19T04:04:53.747Z] INFO[2024-01-19 04:04:53] sched stats: tick process durations: 45052 < 500ms (100%) [2024-01-19T04:05:01.840Z] 2024/01/19 04:05:00 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:05:11.806Z] 2024/01/19 04:05:10 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:05:21.748Z] 2024/01/19 04:05:20 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:05:31.689Z] 2024/01/19 04:05:30 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:05:41.659Z] 2024/01/19 04:05:40 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:05:51.602Z] 2024/01/19 04:05:50 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:06:01.543Z] 2024/01/19 04:06:00 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:06:11.488Z] 2024/01/19 04:06:10 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:06:21.428Z] 2024/01/19 04:06:20 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:06:31.395Z] 2024/01/19 04:06:30 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:06:41.331Z] 2024/01/19 04:06:40 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:06:51.277Z] 2024/01/19 04:06:50 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:07:01.219Z] 2024/01/19 04:07:00 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:07:11.159Z] 2024/01/19 04:07:10 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:07:21.095Z] 2024/01/19 04:07:20 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:07:33.263Z] 2024/01/19 04:07:31 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:07:41.348Z] 2024/01/19 04:07:41 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:07:51.293Z] 2024/01/19 04:07:51 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:08:01.245Z] 2024/01/19 04:08:01 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:08:11.192Z] 2024/01/19 04:08:11 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:08:23.358Z] 2024/01/19 04:08:21 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:08:31.438Z] 2024/01/19 04:08:31 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:08:41.382Z] 2024/01/19 04:08:41 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:08:51.328Z] 2024/01/19 04:08:51 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:09:03.495Z] 2024/01/19 04:09:01 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:09:11.576Z] 2024/01/19 04:09:11 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:09:21.514Z] 2024/01/19 04:09:21 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:09:31.466Z] 2024/01/19 04:09:31 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:09:43.641Z] 2024/01/19 04:09:41 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:09:51.723Z] 2024-01-19 04:09:51 +0000:[ERROR] [{VerifyRBACForPxAdmin}] [backup.glob..func44.3.1:#1075] - Unable to switch context to source cluster [source-cluster]. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.723Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.723Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.723Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.723Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.723Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.723Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.723Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.723Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.723Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.723Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.723Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.723Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.723Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.723Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.723Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.723Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.723Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.723Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.724Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.724Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.724Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.724Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.724Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.724Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.724Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.724Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.724Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.724Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:09:51.724Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T04:09:51.724Z] • Failure [683.119 seconds] [2024-01-19T04:09:51.724Z] {VerifyRBACForPxAdmin} [2024-01-19T04:09:51.724Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_rbac_test.go:767 [2024-01-19T04:09:51.724Z] Validates the RBAC operation for px-admin group user. [It] [2024-01-19T04:09:51.724Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_rbac_test.go:813 [2024-01-19T04:09:51.724Z] [2024-01-19T04:09:51.724Z] Unexpected error: [2024-01-19T04:09:51.724Z] <*errors.errorString | 0xc00020d630>: { [2024-01-19T04:09:51.724Z] s: "error: ValidateRestore Errors: {in restore [tp-restore-tp-backup-manual-single-ns-with-rules-bfdr], said restored volume [pvc-672c8c45-9895-4cfc-b350-5afaba0ef385] cannot be found in the actual cluster [mysql-ibm-pxb-0-87887-01-19-03h58m28s]}, Description: Verifying creation of restore tp-restore-tp-backup-manual-single-ns-with-rules-bfdr of backup tp-backup-manual-single-ns-with-rules-bfdr", [2024-01-19T04:09:51.724Z] } [2024-01-19T04:09:51.724Z] error: ValidateRestore Errors: {in restore [tp-restore-tp-backup-manual-single-ns-with-rules-bfdr], said restored volume [pvc-672c8c45-9895-4cfc-b350-5afaba0ef385] cannot be found in the actual cluster [mysql-ibm-pxb-0-87887-01-19-03h58m28s]}, Description: Verifying creation of restore tp-restore-tp-backup-manual-single-ns-with-rules-bfdr of backup tp-backup-manual-single-ns-with-rules-bfdr [2024-01-19T04:09:51.724Z] occurred [2024-01-19T04:09:51.724Z] [2024-01-19T04:09:51.724Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-19T04:09:51.724Z] [2024-01-19T04:09:51.724Z] Full Stack Trace [2024-01-19T04:09:51.724Z] github.com/portworx/torpedo/pkg/aetosutil.(*Dashboard).VerifyFatal(0x7f8804b?, {0x702af20?, 0xc00020d1f0}, {0x0?, 0x0}, {0xc001594c60, 0x88}) [2024-01-19T04:09:51.725Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 +0x143 [2024-01-19T04:09:51.725Z] github.com/portworx/torpedo/tests/backup.glob..func44.2.10() [2024-01-19T04:09:51.725Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_rbac_test.go:947 +0x437 [2024-01-19T04:09:51.725Z] github.com/portworx/torpedo/tests/backup.glob..func44.2() [2024-01-19T04:09:51.725Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_rbac_test.go:940 +0xba3 [2024-01-19T04:09:51.725Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T04:09:51.725Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T04:09:51.725Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T04:09:51.725Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T04:09:51.725Z] created by testing.(*T).Run [2024-01-19T04:09:51.725Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T04:09:51.725Z] ------------------------------ [2024-01-19T04:09:51.725Z] STEP: schedule applications [2024-01-19T04:09:51.725Z] STEP: Validate applications [2024-01-19T04:09:51.725Z] STEP: validate applications [2024-01-19T04:09:51.725Z] STEP: validate mysql-ibm app's volumes [2024-01-19T04:09:51.725Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T04:09:51.725Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T04:09:51.725Z] STEP: get mysql-ibm app's volume: pvc-7f2a47e6-0c07-4e6c-a44e-b8523f5cf6cf inspected by the volume driver [2024-01-19T04:09:51.725Z] STEP: wait for mysql-ibm app to start running [2024-01-19T04:09:51.725Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T04:09:51.725Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T04:09:51.725Z] STEP: Validate Px pod restart count [2024-01-19T04:09:51.725Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T04:09:51.725Z] STEP: Verify Px-Admin group user has permission to create cloud credential and backup location [2024-01-19T04:09:51.725Z] STEP: Verify Px-Admin User has permission to create a schedule policy [2024-01-19T04:09:51.725Z] STEP: Verify Px-Admin User has permission to create pre and post exec rules for applications [2024-01-19T04:09:51.725Z] STEP: Verify Px-Admin User has permission to create new custom role [2024-01-19T04:09:51.725Z] STEP: Create Users with Different types of roles [2024-01-19T04:09:51.725Z] STEP: Verify px-admin User has permission to share RBAC resources with non-admin users [2024-01-19T04:09:51.725Z] STEP: Create source and destination cluster from the px-admin [2024-01-19T04:09:51.725Z] STEP: Take backup of applications from the px-admin [2024-01-19T04:09:51.725Z] STEP: Take restore of applications from the px-admin [2024-01-19T04:09:51.725Z] STEP: validate mysql-ibm app's volumes [2024-01-19T04:09:51.725Z] STEP: wait for mysql-ibm app to start running [2024-01-19T04:09:51.725Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T04:09:51.725Z] STEP: Validate Px pod restart count [2024-01-19T04:09:51.725Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T04:09:51.725Z] STEP: destroy apps [2024-01-19T04:09:51.725Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T04:09:51.725Z] STEP: start destroying mysql-ibm app [2024-01-19T04:09:51.725Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T04:09:51.725Z] STEP: destroy apps [2024-01-19T04:09:51.725Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T04:09:51.725Z] STEP: start destroying mysql-ibm app [2024-01-19T04:09:51.725Z] S2024-01-19 04:09:51 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Log Dir: /testresults//DeleteBackupOfUserSharedRBAC.log [2024-01-19T04:09:51.725Z] INFO[2024-01-19 04:09:51] --------Test Start------ [2024-01-19T04:09:51.725Z] INFO[2024-01-19 04:09:51] #Test: DeleteBackupOfUserSharedRBAC [2024-01-19T04:09:51.725Z] INFO[2024-01-19 04:09:51] #Description: Delete backups,restores,schedules,clusters created by non-admin user with shared RBAC resources from px-admin [2024-01-19T04:09:51.725Z] INFO[2024-01-19 04:09:51] ------------------------ [2024-01-19T04:09:51.725Z] 2024-01-19 04:09:51 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Deploy applications [2024-01-19T04:09:51.725Z] 2024-01-19 04:09:51 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T04:09:51.725Z] 2024-01-19 04:09:51 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T04:09:51.725Z] 2024-01-19 04:09:51 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T04:09:51.726Z] 2024-01-19 04:09:51 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T04:09:51.726Z] 2024-01-19 04:09:51 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T04:09:51.726Z] 2024-01-19 04:09:51 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T04:09:51.726Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T04:09:51.983Z] 2024-01-19 04:09:51 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T04:09:51.983Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T04:09:51.983Z] 2024-01-19 04:09:51 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T04:09:51.983Z] 2024-01-19 04:09:51 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T04:09:51.983Z] 2024-01-19 04:09:51 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T04:09:51.983Z] 2024-01-19 04:09:51 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T04:09:51.983Z] 2024-01-19 04:09:51 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T04:09:51.983Z] 2024-01-19 04:09:51 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T04:09:51.983Z] 2024-01-19 04:09:51 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T04:09:51.984Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T04:09:52.243Z] 2024-01-19 04:09:51 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T04:09:52.244Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T04:09:52.244Z] 2024-01-19 04:09:52 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T04:09:52.244Z] 2024-01-19 04:09:52 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Created namespaces [mysql-ibm-pxb-0-87560-01-19-04h09m51s mysql-ibm-pxb-1-87560-01-19-04h09m51s] [2024-01-19T04:09:52.244Z] 2024-01-19 04:09:52 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validate applications [2024-01-19T04:09:52.244Z] 2024-01-19 04:09:52 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validate applications [2024-01-19T04:09:52.244Z] 2024-01-19 04:09:52 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating mysql-ibm app [2024-01-19T04:09:52.244Z] 2024-01-19 04:09:52 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating mysql-ibm app's volumes [2024-01-19T04:09:52.244Z] 2024-01-19 04:09:52 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:09:52.244Z] 2024/01/19 04:09:52 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T04:10:02.189Z] 2024/01/19 04:10:02 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T04:10:14.359Z] 2024-01-19 04:10:12 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-87560-01-19-04h09m51s [2024-01-19T04:10:14.359Z] 2024-01-19 04:10:12 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T04:10:14.359Z] 2024-01-19 04:10:12 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T04:10:14.359Z] 2024-01-19 04:10:12 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] wait for mysql-ibm app to start running [2024-01-19T04:10:14.359Z] 2024-01-19 04:10:12 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T04:10:14.359Z] 2024/01/19 04:10:12 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T04:10:14.359Z] pod name:mysql-7d766cd65c-blxvz namespace:mysql-ibm-pxb-0-87560-01-19-04h09m51s running:false ready:false node:10.240.64.8 [2024-01-19T04:10:14.359Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T04:10:24.304Z] 2024/01/19 04:10:22 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T04:10:24.304Z] pod name:mysql-7d766cd65c-blxvz namespace:mysql-ibm-pxb-0-87560-01-19-04h09m51s running:true ready:false node:10.240.64.8 [2024-01-19T04:10:24.304Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T04:10:34.274Z] 2024/01/19 04:10:32 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T04:10:34.274Z] pod name:mysql-7d766cd65c-blxvz namespace:mysql-ibm-pxb-0-87560-01-19-04h09m51s running:true ready:false node:10.240.64.8 [2024-01-19T04:10:34.274Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T04:10:44.215Z] 2024/01/19 04:10:42 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T04:10:44.215Z] pod name:mysql-7d766cd65c-blxvz namespace:mysql-ibm-pxb-0-87560-01-19-04h09m51s running:true ready:false node:10.240.64.8 [2024-01-19T04:10:44.215Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T04:10:54.169Z] 2024/01/19 04:10:52 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T04:10:54.169Z] pod name:mysql-7d766cd65c-blxvz namespace:mysql-ibm-pxb-0-87560-01-19-04h09m51s running:true ready:false node:10.240.64.8 [2024-01-19T04:10:54.169Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T04:11:04.120Z] 2024/01/19 04:11:02 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T04:11:04.120Z] pod name:mysql-7d766cd65c-blxvz namespace:mysql-ibm-pxb-0-87560-01-19-04h09m51s running:true ready:false node:10.240.64.8 [2024-01-19T04:11:04.120Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T04:11:14.062Z] 2024/01/19 04:11:13 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T04:11:14.062Z] pod name:mysql-7d766cd65c-blxvz namespace:mysql-ibm-pxb-0-87560-01-19-04h09m51s running:true ready:false node:10.240.64.8 [2024-01-19T04:11:14.062Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T04:11:24.010Z] 2024-01-19 04:11:23 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T04:11:24.010Z] 2024-01-19 04:11:23 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T04:11:24.010Z] 2024-01-19 04:11:23 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] validate if mysql-ibm app's volumes are setup [2024-01-19T04:11:24.010Z] 2024-01-19 04:11:23 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:11:24.010Z] 2024-01-19 04:11:23 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:11:24.010Z] 2024-01-19 04:11:23 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T04:11:24.010Z] 2024-01-19 04:11:23 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T04:11:24.010Z] 2024-01-19 04:11:23 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating mysql-ibm app [2024-01-19T04:11:24.010Z] 2024-01-19 04:11:23 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating mysql-ibm app's volumes [2024-01-19T04:11:24.010Z] 2024-01-19 04:11:23 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:11:24.010Z] 2024-01-19 04:11:23 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:11:24.010Z] 2024-01-19 04:11:23 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-1-87560-01-19-04h09m51s [2024-01-19T04:11:24.010Z] 2024-01-19 04:11:23 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T04:11:24.010Z] 2024-01-19 04:11:23 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T04:11:24.010Z] 2024-01-19 04:11:23 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] wait for mysql-ibm app to start running [2024-01-19T04:11:24.010Z] 2024-01-19 04:11:23 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T04:11:24.266Z] 2024/01/19 04:11:23 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T04:11:24.266Z] pod name:mysql-7d766cd65c-rjk25 namespace:mysql-ibm-pxb-1-87560-01-19-04h09m51s running:true ready:false node:10.240.64.8 [2024-01-19T04:11:24.266Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T04:11:34.206Z] 2024-01-19 04:11:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T04:11:34.462Z] 2024-01-19 04:11:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T04:11:34.462Z] 2024-01-19 04:11:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] validate if mysql-ibm app's volumes are setup [2024-01-19T04:11:34.462Z] 2024-01-19 04:11:34 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:11:34.462Z] 2024-01-19 04:11:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:11:34.462Z] 2024-01-19 04:11:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T04:11:34.462Z] 2024-01-19 04:11:34 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T04:11:34.717Z] 2024-01-19 04:11:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Generate and add labels to namespaces [2024-01-19T04:11:34.717Z] INFO[2024-01-19 04:11:34] Verifying : Description : Adding labels [map[0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a:4172c3ae-23ff-46c1-8d53-9fbf67478e7a]] to namespaces [[mysql-ibm-pxb-0-87560-01-19-04h09m51s mysql-ibm-pxb-1-87560-01-19-04h09m51s]] [2024-01-19T04:11:34.717Z] INFO[2024-01-19 04:11:34] Actual:true, Expected: true [2024-01-19T04:11:34.717Z] 2024-01-19 04:11:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Generating namespace label string from label map [2024-01-19T04:11:34.718Z] 2024-01-19 04:11:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.glob..func32.2.3:#2525] - Generated labels [0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a] [2024-01-19T04:11:34.718Z] 2024-01-19 04:11:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Create Users with Different types of roles [2024-01-19T04:11:34.718Z] 2024-01-19 04:11:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating 1 users [2024-01-19T04:11:34.974Z] 2024-01-19 04:11:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AddUser:#728] - Response for user [testuser1-1705637494] creation - [2024-01-19T04:11:34.974Z] 2024-01-19 04:11:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AddUser:#733] - User [testuser1-1705637494] created [2024-01-19T04:11:34.974Z] INFO[2024-01-19 04:11:34] Verifying : Description : Creating user - testuser1-1705637494 [2024-01-19T04:11:34.974Z] INFO[2024-01-19 04:11:34] Actual:true, Expected: true [2024-01-19T04:11:35.536Z] 2024-01-19 04:11:35 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637494 - 7ab1a700-5d59-4cf3-a59b-46256c94f61b [2024-01-19T04:11:38.049Z] 2024-01-19 04:11:37 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637494 - 7ab1a700-5d59-4cf3-a59b-46256c94f61b [2024-01-19T04:11:38.049Z] 2024-01-19 04:11:37 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating 1 users [2024-01-19T04:11:38.977Z] 2024-01-19 04:11:38 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AddUser:#728] - Response for user [testuser1-1705637497] creation - [2024-01-19T04:11:38.977Z] 2024-01-19 04:11:38 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AddUser:#733] - User [testuser1-1705637497] created [2024-01-19T04:11:38.977Z] INFO[2024-01-19 04:11:38] Verifying : Description : Creating user - testuser1-1705637497 [2024-01-19T04:11:38.977Z] INFO[2024-01-19 04:11:38] Actual:true, Expected: true [2024-01-19T04:11:39.538Z] 2024-01-19 04:11:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637497 - c3f9f3d9-1651-4a0d-800d-9afbf122ffa0 [2024-01-19T04:11:42.054Z] 2024-01-19 04:11:41 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637497 - c3f9f3d9-1651-4a0d-800d-9afbf122ffa0 [2024-01-19T04:11:42.054Z] 2024-01-19 04:11:41 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating 1 users [2024-01-19T04:11:42.984Z] 2024-01-19 04:11:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AddUser:#728] - Response for user [testuser1-1705637501] creation - [2024-01-19T04:11:42.984Z] 2024-01-19 04:11:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AddUser:#733] - User [testuser1-1705637501] created [2024-01-19T04:11:42.984Z] INFO[2024-01-19 04:11:42] Verifying : Description : Creating user - testuser1-1705637501 [2024-01-19T04:11:42.984Z] INFO[2024-01-19 04:11:42] Actual:true, Expected: true [2024-01-19T04:11:43.546Z] 2024-01-19 04:11:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637501 - ad3bd280-b2ab-44a8-8e03-cb49edc964e5 [2024-01-19T04:11:46.063Z] 2024-01-19 04:11:45 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637501 - ad3bd280-b2ab-44a8-8e03-cb49edc964e5 [2024-01-19T04:11:46.063Z] 2024-01-19 04:11:45 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating 1 users [2024-01-19T04:11:46.989Z] 2024-01-19 04:11:46 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AddUser:#728] - Response for user [testuser1-1705637505] creation - [2024-01-19T04:11:46.989Z] 2024-01-19 04:11:46 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AddUser:#733] - User [testuser1-1705637505] created [2024-01-19T04:11:46.989Z] INFO[2024-01-19 04:11:46] Verifying : Description : Creating user - testuser1-1705637505 [2024-01-19T04:11:46.989Z] INFO[2024-01-19 04:11:46] Actual:true, Expected: true [2024-01-19T04:11:47.550Z] 2024-01-19 04:11:47 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637505 - b19c7b6e-9dcf-46db-8832-33dfc89f3250 [2024-01-19T04:11:50.064Z] 2024-01-19 04:11:49 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637505 - b19c7b6e-9dcf-46db-8832-33dfc89f3250 [2024-01-19T04:11:50.064Z] 2024-01-19 04:11:49 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating 1 users [2024-01-19T04:11:50.989Z] 2024-01-19 04:11:50 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AddUser:#728] - Response for user [testuser1-1705637509] creation - [2024-01-19T04:11:50.990Z] 2024-01-19 04:11:50 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AddUser:#733] - User [testuser1-1705637509] created [2024-01-19T04:11:50.990Z] INFO[2024-01-19 04:11:50] Verifying : Description : Creating user - testuser1-1705637509 [2024-01-19T04:11:50.990Z] INFO[2024-01-19 04:11:50] Actual:true, Expected: true [2024-01-19T04:11:51.551Z] 2024-01-19 04:11:51 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637509 - 74751a01-c85e-4c1e-9914-936792573b6c [2024-01-19T04:11:54.063Z] 2024-01-19 04:11:53 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637509 - 74751a01-c85e-4c1e-9914-936792573b6c [2024-01-19T04:11:54.063Z] 2024-01-19 04:11:53 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating 1 users [2024-01-19T04:11:54.996Z] 2024-01-19 04:11:54 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AddUser:#728] - Response for user [testuser1-1705637513] creation - [2024-01-19T04:11:54.996Z] 2024-01-19 04:11:54 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AddUser:#733] - User [testuser1-1705637513] created [2024-01-19T04:11:54.996Z] INFO[2024-01-19 04:11:54] Verifying : Description : Creating user - testuser1-1705637513 [2024-01-19T04:11:54.996Z] INFO[2024-01-19 04:11:54] Actual:true, Expected: true [2024-01-19T04:11:55.559Z] 2024-01-19 04:11:55 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637513 - 42526588-0699-4a99-b785-affbe774f988 [2024-01-19T04:11:58.073Z] 2024-01-19 04:11:57 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637513 - 42526588-0699-4a99-b785-affbe774f988 [2024-01-19T04:11:58.073Z] 2024-01-19 04:11:57 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating cloud credentials and backup location from px-admin user [2024-01-19T04:11:59.437Z] 2024-01-19 04:11:59 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-meoyc] for org [default] with [aws] as provider [2024-01-19T04:11:59.437Z] 2024-01-19 04:11:59 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T04:11:59.692Z] 2024-01-19 04:11:59 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-meoyc] is present [2024-01-19T04:12:04.935Z] 2024-01-19 04:12:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating a schedule policy from px-admin [2024-01-19T04:12:04.935Z] INFO[2024-01-19 04:12:04] Verifying : Description : Verifying creation of periodic schedule policy of interval [15] minutes named [periodic-upgbo] [2024-01-19T04:12:04.936Z] INFO[2024-01-19 04:12:04] Actual:true, Expected: true [2024-01-19T04:12:04.936Z] INFO[2024-01-19 04:12:04] Verifying : Description : Fetching uid of periodic schedule policy named [periodic-upgbo] [2024-01-19T04:12:04.936Z] INFO[2024-01-19 04:12:04] Actual:true, Expected: true [2024-01-19T04:12:04.936Z] 2024-01-19 04:12:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Create pre and post exec rules for applications from px-admin [2024-01-19T04:12:05.191Z] 2024-01-19 04:12:05 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.CreateRuleForBackupWithMultipleApplications:#4309] - Validate pre-rules for backup [2024-01-19T04:12:05.191Z] 2024-01-19 04:12:05 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.CreateRuleForBackupWithMultipleApplications:#4322] - Validate post-rules for backup [2024-01-19T04:12:05.447Z] INFO[2024-01-19 04:12:05] Verifying : Description : Verifying creation of pre and post exec rules for applications from px-admin [2024-01-19T04:12:05.447Z] INFO[2024-01-19 04:12:05] Actual:true, Expected: true [2024-01-19T04:12:05.447Z] 2024-01-19 04:12:05 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.glob..func32.2.7:#2580] - Pre backup rule [pre-rule-jfikz] uid: [54ac2577-61cb-40b9-b501-cfd8a9a9ada8] [2024-01-19T04:12:05.447Z] 2024-01-19 04:12:05 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.glob..func32.2.7:#2585] - Post backup rule [post-rule-thgqc] uid: [3586714c-66f2-42ad-b9cc-0321c2caa97a] [2024-01-19T04:12:05.447Z] 2024-01-19 04:12:05 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] update ownership for RBAC resource with non-admin users from px-admin [2024-01-19T04:12:06.373Z] 2024-01-19 04:12:06 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Update BackupLocation - autogenerated-backup-location-wkyfv ownership for users - [[testuser1-1705637494 testuser1-1705637497 testuser1-1705637501 testuser1-1705637505 testuser1-1705637509 testuser1-1705637513]] [2024-01-19T04:12:07.298Z] 2024-01-19 04:12:07 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637494 - 7ab1a700-5d59-4cf3-a59b-46256c94f61b [2024-01-19T04:12:08.225Z] 2024-01-19 04:12:07 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637497 - c3f9f3d9-1651-4a0d-800d-9afbf122ffa0 [2024-01-19T04:12:08.785Z] 2024-01-19 04:12:08 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637501 - ad3bd280-b2ab-44a8-8e03-cb49edc964e5 [2024-01-19T04:12:09.716Z] 2024-01-19 04:12:09 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637505 - b19c7b6e-9dcf-46db-8832-33dfc89f3250 [2024-01-19T04:12:10.647Z] 2024-01-19 04:12:10 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637509 - 74751a01-c85e-4c1e-9914-936792573b6c [2024-01-19T04:12:11.208Z] 2024-01-19 04:12:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637513 - 42526588-0699-4a99-b785-affbe774f988 [2024-01-19T04:12:11.774Z] INFO[2024-01-19 04:12:11] Verifying : Description : Verifying update of owbership for backuplocation - autogenerated-backup-location-wkyfv [2024-01-19T04:12:11.774Z] INFO[2024-01-19 04:12:11] Actual:true, Expected: true [2024-01-19T04:12:11.774Z] 2024-01-19 04:12:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Update SchedulePolicy - periodic-upgbo ownership for users - [[testuser1-1705637494 testuser1-1705637497 testuser1-1705637501 testuser1-1705637505 testuser1-1705637509 testuser1-1705637513]] [2024-01-19T04:12:12.030Z] 2024-01-19 04:12:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637494 - 7ab1a700-5d59-4cf3-a59b-46256c94f61b [2024-01-19T04:12:12.963Z] 2024-01-19 04:12:12 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637497 - c3f9f3d9-1651-4a0d-800d-9afbf122ffa0 [2024-01-19T04:12:13.890Z] 2024-01-19 04:12:13 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637501 - ad3bd280-b2ab-44a8-8e03-cb49edc964e5 [2024-01-19T04:12:14.452Z] 2024-01-19 04:12:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637505 - b19c7b6e-9dcf-46db-8832-33dfc89f3250 [2024-01-19T04:12:15.379Z] 2024-01-19 04:12:15 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637509 - 74751a01-c85e-4c1e-9914-936792573b6c [2024-01-19T04:12:15.940Z] 2024-01-19 04:12:15 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637513 - 42526588-0699-4a99-b785-affbe774f988 [2024-01-19T04:12:16.197Z] INFO[2024-01-19 04:12:16] Verifying : Description : Verifying update of ownership for schedulepolicy - periodic-upgbo [2024-01-19T04:12:16.197Z] INFO[2024-01-19 04:12:16] Actual:true, Expected: true [2024-01-19T04:12:16.197Z] 2024-01-19 04:12:16 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Update Application Rules ownership for users - [[testuser1-1705637494 testuser1-1705637497 testuser1-1705637501 testuser1-1705637505 testuser1-1705637509 testuser1-1705637513]] [2024-01-19T04:12:16.758Z] 2024-01-19 04:12:16 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637494 - 7ab1a700-5d59-4cf3-a59b-46256c94f61b [2024-01-19T04:12:17.684Z] 2024-01-19 04:12:17 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637497 - c3f9f3d9-1651-4a0d-800d-9afbf122ffa0 [2024-01-19T04:12:18.610Z] 2024-01-19 04:12:18 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637501 - ad3bd280-b2ab-44a8-8e03-cb49edc964e5 [2024-01-19T04:12:19.172Z] 2024-01-19 04:12:19 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637505 - b19c7b6e-9dcf-46db-8832-33dfc89f3250 [2024-01-19T04:12:20.099Z] 2024-01-19 04:12:19 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637509 - 74751a01-c85e-4c1e-9914-936792573b6c [2024-01-19T04:12:21.024Z] 2024-01-19 04:12:20 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637513 - 42526588-0699-4a99-b785-affbe774f988 [2024-01-19T04:12:21.024Z] INFO[2024-01-19 04:12:20] Verifying : Description : Verifying update of ownership for pre-rule of application [2024-01-19T04:12:21.024Z] INFO[2024-01-19 04:12:20] Actual:true, Expected: true [2024-01-19T04:12:21.585Z] 2024-01-19 04:12:21 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637494 - 7ab1a700-5d59-4cf3-a59b-46256c94f61b [2024-01-19T04:12:22.511Z] 2024-01-19 04:12:22 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637497 - c3f9f3d9-1651-4a0d-800d-9afbf122ffa0 [2024-01-19T04:12:23.438Z] 2024-01-19 04:12:23 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637501 - ad3bd280-b2ab-44a8-8e03-cb49edc964e5 [2024-01-19T04:12:23.998Z] 2024-01-19 04:12:23 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637505 - b19c7b6e-9dcf-46db-8832-33dfc89f3250 [2024-01-19T04:12:24.925Z] 2024-01-19 04:12:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637509 - 74751a01-c85e-4c1e-9914-936792573b6c [2024-01-19T04:12:25.851Z] 2024-01-19 04:12:25 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637513 - 42526588-0699-4a99-b785-affbe774f988 [2024-01-19T04:12:25.851Z] INFO[2024-01-19 04:12:25] Verifying : Description : Verifying update of ownership for post-rule of application [2024-01-19T04:12:25.851Z] INFO[2024-01-19 04:12:25] Actual:true, Expected: true [2024-01-19T04:12:25.851Z] 2024-01-19 04:12:25 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Register source and destination cluster for backup on non-admin user [2024-01-19T04:12:25.851Z] 2024-01-19 04:12:25 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Registering Source and Destination clusters as user : testuser1-1705637494 and verifying the status [2024-01-19T04:12:26.107Z] INFO[2024-01-19 04:12:25] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T04:12:26.107Z] INFO[2024-01-19 04:12:25] Actual:true, Expected: true [2024-01-19T04:12:26.107Z] 2024-01-19 04:12:25 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T04:12:26.362Z] 2024-01-19 04:12:26 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T04:12:26.362Z] 2024-01-19 04:12:26 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T04:12:26.362Z] 2024-01-19 04:12:26 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T04:12:26.362Z] 2024-01-19 04:12:26 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Create cluster [source-cluster] in org [default] [2024-01-19T04:12:26.362Z] 2024-01-19 04:12:26 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T04:12:26.362Z] 2024-01-19 04:12:26 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T04:12:26.362Z] 2024-01-19 04:12:26 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Create cluster [destination-cluster] in org [default] [2024-01-19T04:12:26.363Z] 2024-01-19 04:12:26 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T04:12:26.363Z] 2024-01-19 04:12:26 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T04:12:26.363Z] 2024-01-19 04:12:26 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-hrfiy] for IBM [2024-01-19T04:12:26.363Z] 2024-01-19 04:12:26 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-hrfiy] for org [default] with [ibm] as provider [2024-01-19T04:12:26.363Z] 2024-01-19 04:12:26 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T04:12:26.363Z] 2024-01-19 04:12:26 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T04:12:26.363Z] 2024-01-19 04:12:26 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-hrfiy] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user px-backup-app.admin]] [2024-01-19T04:12:26.363Z] 2024-01-19 04:12:26 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user px-backup-app.admin] [2024-01-19T04:12:27.728Z] 2024-01-19 04:12:27 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-hrfiy from admin context and sharing with all the users [2024-01-19T04:12:27.728Z] 2024-01-19 04:12:27 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-hrfiy] for org [default] with [ibm] as provider [2024-01-19T04:12:27.728Z] 2024-01-19 04:12:27 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T04:12:27.728Z] 2024-01-19 04:12:27 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T04:12:27.728Z] 2024-01-19 04:12:27 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-hrfiy] is present [2024-01-19T04:12:27.984Z] 2024-01-19 04:12:27 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Create cluster [source-cluster] in org [default] [2024-01-19T04:12:28.545Z] 2024-01-19 04:12:28 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-acnhm] for IBM [2024-01-19T04:12:28.545Z] 2024-01-19 04:12:28 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-acnhm] for org [default] with [ibm] as provider [2024-01-19T04:12:28.545Z] 2024-01-19 04:12:28 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T04:12:28.545Z] 2024-01-19 04:12:28 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T04:12:28.545Z] 2024-01-19 04:12:28 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-acnhm] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user px-backup-app.admin]] [2024-01-19T04:12:28.545Z] 2024-01-19 04:12:28 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user px-backup-app.admin] [2024-01-19T04:12:29.106Z] 2024-01-19 04:12:28 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-acnhm from admin context and sharing with all the users [2024-01-19T04:12:29.106Z] 2024-01-19 04:12:28 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-acnhm] for org [default] with [ibm] as provider [2024-01-19T04:12:29.106Z] 2024-01-19 04:12:28 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T04:12:29.106Z] 2024-01-19 04:12:28 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T04:12:29.106Z] 2024-01-19 04:12:29 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-acnhm] is present [2024-01-19T04:12:29.362Z] 2024-01-19 04:12:29 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Create cluster [destination-cluster] in org [default] [2024-01-19T04:12:29.923Z] INFO[2024-01-19 04:12:29] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T04:12:29.923Z] INFO[2024-01-19 04:12:29] Actual:Online, Expected: Online [2024-01-19T04:12:29.923Z] INFO[2024-01-19 04:12:29] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T04:12:29.923Z] INFO[2024-01-19 04:12:29] Actual:true, Expected: true [2024-01-19T04:12:30.187Z] INFO[2024-01-19 04:12:29] Verifying : Description : Fetching [destination-cluster] cluster uid [2024-01-19T04:12:30.187Z] INFO[2024-01-19 04:12:29] Actual:true, Expected: true [2024-01-19T04:12:30.188Z] 2024-01-19 04:12:29 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Registering Source and Destination clusters as user : testuser1-1705637497 and verifying the status [2024-01-19T04:12:30.188Z] INFO[2024-01-19 04:12:30] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T04:12:30.188Z] INFO[2024-01-19 04:12:30] Actual:true, Expected: true [2024-01-19T04:12:30.188Z] 2024-01-19 04:12:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T04:12:30.188Z] 2024-01-19 04:12:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T04:12:30.188Z] 2024-01-19 04:12:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T04:12:30.188Z] 2024-01-19 04:12:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T04:12:30.188Z] 2024-01-19 04:12:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Create cluster [source-cluster] in org [default] [2024-01-19T04:12:30.188Z] 2024-01-19 04:12:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T04:12:30.188Z] 2024-01-19 04:12:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T04:12:30.188Z] 2024-01-19 04:12:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Create cluster [destination-cluster] in org [default] [2024-01-19T04:12:30.188Z] 2024-01-19 04:12:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T04:12:30.188Z] 2024-01-19 04:12:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T04:12:30.188Z] 2024-01-19 04:12:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-fbvvo] for IBM [2024-01-19T04:12:30.188Z] 2024-01-19 04:12:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-fbvvo] for org [default] with [ibm] as provider [2024-01-19T04:12:30.188Z] 2024-01-19 04:12:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T04:12:30.188Z] 2024-01-19 04:12:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T04:12:30.188Z] 2024-01-19 04:12:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-fbvvo] is present [2024-01-19T04:12:30.188Z] 2024-01-19 04:12:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Create cluster [source-cluster] in org [default] [2024-01-19T04:12:30.750Z] 2024-01-19 04:12:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-soqhu] for IBM [2024-01-19T04:12:30.750Z] 2024-01-19 04:12:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-soqhu] for org [default] with [ibm] as provider [2024-01-19T04:12:30.750Z] 2024-01-19 04:12:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T04:12:30.750Z] 2024-01-19 04:12:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T04:12:31.006Z] 2024-01-19 04:12:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-soqhu] is present [2024-01-19T04:12:31.006Z] 2024-01-19 04:12:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Create cluster [destination-cluster] in org [default] [2024-01-19T04:12:31.569Z] INFO[2024-01-19 04:12:31] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T04:12:31.569Z] INFO[2024-01-19 04:12:31] Actual:Online, Expected: Online [2024-01-19T04:12:31.569Z] INFO[2024-01-19 04:12:31] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T04:12:31.569Z] INFO[2024-01-19 04:12:31] Actual:true, Expected: true [2024-01-19T04:12:31.569Z] INFO[2024-01-19 04:12:31] Verifying : Description : Fetching [destination-cluster] cluster uid [2024-01-19T04:12:31.569Z] INFO[2024-01-19 04:12:31] Actual:true, Expected: true [2024-01-19T04:12:31.570Z] 2024-01-19 04:12:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Registering Source and Destination clusters as user : testuser1-1705637501 and verifying the status [2024-01-19T04:12:31.570Z] INFO[2024-01-19 04:12:31] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T04:12:31.570Z] INFO[2024-01-19 04:12:31] Actual:true, Expected: true [2024-01-19T04:12:31.570Z] 2024-01-19 04:12:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T04:12:31.825Z] 2024-01-19 04:12:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T04:12:31.825Z] 2024-01-19 04:12:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T04:12:31.825Z] 2024-01-19 04:12:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T04:12:31.825Z] 2024-01-19 04:12:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Create cluster [source-cluster] in org [default] [2024-01-19T04:12:31.825Z] 2024-01-19 04:12:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T04:12:31.825Z] 2024-01-19 04:12:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T04:12:31.825Z] 2024-01-19 04:12:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Create cluster [destination-cluster] in org [default] [2024-01-19T04:12:31.825Z] 2024-01-19 04:12:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T04:12:31.825Z] 2024-01-19 04:12:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T04:12:31.826Z] 2024-01-19 04:12:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-enzij] for IBM [2024-01-19T04:12:31.826Z] 2024-01-19 04:12:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-enzij] for org [default] with [ibm] as provider [2024-01-19T04:12:31.826Z] 2024-01-19 04:12:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T04:12:31.826Z] 2024-01-19 04:12:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T04:12:31.826Z] 2024-01-19 04:12:31 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-enzij] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user]] [2024-01-19T04:12:31.826Z] 2024-01-19 04:12:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user] [2024-01-19T04:12:31.826Z] 2024-01-19 04:12:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-enzij from admin context and sharing with all the users [2024-01-19T04:12:31.826Z] 2024-01-19 04:12:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-enzij] for org [default] with [ibm] as provider [2024-01-19T04:12:32.081Z] 2024-01-19 04:12:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T04:12:32.082Z] 2024-01-19 04:12:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T04:12:32.082Z] 2024-01-19 04:12:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-enzij] is present [2024-01-19T04:12:32.337Z] 2024-01-19 04:12:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Create cluster [source-cluster] in org [default] [2024-01-19T04:12:32.898Z] 2024-01-19 04:12:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-fcirm] for IBM [2024-01-19T04:12:32.898Z] 2024-01-19 04:12:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-fcirm] for org [default] with [ibm] as provider [2024-01-19T04:12:32.898Z] 2024-01-19 04:12:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T04:12:32.899Z] 2024-01-19 04:12:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T04:12:32.899Z] 2024-01-19 04:12:32 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-fcirm] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user]] [2024-01-19T04:12:32.899Z] 2024-01-19 04:12:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user] [2024-01-19T04:12:33.156Z] 2024-01-19 04:12:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-fcirm from admin context and sharing with all the users [2024-01-19T04:12:33.156Z] 2024-01-19 04:12:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-fcirm] for org [default] with [ibm] as provider [2024-01-19T04:12:33.156Z] 2024-01-19 04:12:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T04:12:33.156Z] 2024-01-19 04:12:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T04:12:33.412Z] 2024-01-19 04:12:33 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-fcirm] is present [2024-01-19T04:12:33.412Z] 2024-01-19 04:12:33 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Create cluster [destination-cluster] in org [default] [2024-01-19T04:12:33.978Z] INFO[2024-01-19 04:12:33] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T04:12:33.978Z] INFO[2024-01-19 04:12:33] Actual:Online, Expected: Online [2024-01-19T04:12:33.978Z] INFO[2024-01-19 04:12:33] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T04:12:33.978Z] INFO[2024-01-19 04:12:33] Actual:true, Expected: true [2024-01-19T04:12:33.978Z] INFO[2024-01-19 04:12:33] Verifying : Description : Fetching [destination-cluster] cluster uid [2024-01-19T04:12:33.978Z] INFO[2024-01-19 04:12:33] Actual:true, Expected: true [2024-01-19T04:12:33.978Z] 2024-01-19 04:12:33 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Registering Source and Destination clusters as user : testuser1-1705637505 and verifying the status [2024-01-19T04:12:34.233Z] INFO[2024-01-19 04:12:34] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T04:12:34.233Z] INFO[2024-01-19 04:12:34] Actual:true, Expected: true [2024-01-19T04:12:34.233Z] 2024-01-19 04:12:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T04:12:34.233Z] 2024-01-19 04:12:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T04:12:34.233Z] 2024-01-19 04:12:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T04:12:34.233Z] 2024-01-19 04:12:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T04:12:34.233Z] 2024-01-19 04:12:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Create cluster [source-cluster] in org [default] [2024-01-19T04:12:34.233Z] 2024-01-19 04:12:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T04:12:34.233Z] 2024-01-19 04:12:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T04:12:34.233Z] 2024-01-19 04:12:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Create cluster [destination-cluster] in org [default] [2024-01-19T04:12:34.233Z] 2024-01-19 04:12:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T04:12:34.233Z] 2024-01-19 04:12:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T04:12:34.233Z] 2024-01-19 04:12:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-wcpfm] for IBM [2024-01-19T04:12:34.234Z] 2024-01-19 04:12:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-wcpfm] for org [default] with [ibm] as provider [2024-01-19T04:12:34.234Z] 2024-01-19 04:12:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T04:12:34.234Z] 2024-01-19 04:12:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T04:12:34.234Z] 2024-01-19 04:12:34 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-wcpfm] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user px-backup-app.admin]] [2024-01-19T04:12:34.234Z] 2024-01-19 04:12:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user px-backup-app.admin] [2024-01-19T04:12:35.160Z] 2024-01-19 04:12:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-wcpfm from admin context and sharing with all the users [2024-01-19T04:12:35.160Z] 2024-01-19 04:12:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-wcpfm] for org [default] with [ibm] as provider [2024-01-19T04:12:35.160Z] 2024-01-19 04:12:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T04:12:35.160Z] 2024-01-19 04:12:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T04:12:35.160Z] 2024-01-19 04:12:35 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-wcpfm] is present [2024-01-19T04:12:35.415Z] 2024-01-19 04:12:35 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Create cluster [source-cluster] in org [default] [2024-01-19T04:12:35.976Z] 2024-01-19 04:12:35 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-vpaxz] for IBM [2024-01-19T04:12:35.976Z] 2024-01-19 04:12:35 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-vpaxz] for org [default] with [ibm] as provider [2024-01-19T04:12:35.976Z] 2024-01-19 04:12:35 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T04:12:35.976Z] 2024-01-19 04:12:35 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T04:12:35.976Z] 2024-01-19 04:12:35 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-vpaxz] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user px-backup-app.admin]] [2024-01-19T04:12:35.976Z] 2024-01-19 04:12:35 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user px-backup-app.admin] [2024-01-19T04:12:36.538Z] 2024-01-19 04:12:36 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-vpaxz from admin context and sharing with all the users [2024-01-19T04:12:36.538Z] 2024-01-19 04:12:36 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-vpaxz] for org [default] with [ibm] as provider [2024-01-19T04:12:36.538Z] 2024-01-19 04:12:36 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T04:12:36.538Z] 2024-01-19 04:12:36 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T04:12:36.538Z] 2024-01-19 04:12:36 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-vpaxz] is present [2024-01-19T04:12:36.794Z] 2024-01-19 04:12:36 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Create cluster [destination-cluster] in org [default] [2024-01-19T04:12:37.354Z] INFO[2024-01-19 04:12:37] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T04:12:37.354Z] INFO[2024-01-19 04:12:37] Actual:Online, Expected: Online [2024-01-19T04:12:37.354Z] INFO[2024-01-19 04:12:37] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T04:12:37.354Z] INFO[2024-01-19 04:12:37] Actual:true, Expected: true [2024-01-19T04:12:37.354Z] INFO[2024-01-19 04:12:37] Verifying : Description : Fetching [destination-cluster] cluster uid [2024-01-19T04:12:37.354Z] INFO[2024-01-19 04:12:37] Actual:true, Expected: true [2024-01-19T04:12:37.354Z] 2024-01-19 04:12:37 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Registering Source and Destination clusters as user : testuser1-1705637509 and verifying the status [2024-01-19T04:12:37.609Z] INFO[2024-01-19 04:12:37] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T04:12:37.610Z] INFO[2024-01-19 04:12:37] Actual:true, Expected: true [2024-01-19T04:12:37.610Z] 2024-01-19 04:12:37 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T04:12:37.610Z] 2024-01-19 04:12:37 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T04:12:37.610Z] 2024-01-19 04:12:37 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T04:12:37.610Z] 2024-01-19 04:12:37 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T04:12:37.610Z] 2024-01-19 04:12:37 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Create cluster [source-cluster] in org [default] [2024-01-19T04:12:37.610Z] 2024-01-19 04:12:37 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T04:12:37.610Z] 2024-01-19 04:12:37 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T04:12:37.610Z] 2024-01-19 04:12:37 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Create cluster [destination-cluster] in org [default] [2024-01-19T04:12:37.610Z] 2024-01-19 04:12:37 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T04:12:37.610Z] 2024-01-19 04:12:37 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T04:12:37.610Z] 2024-01-19 04:12:37 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-vlbwu] for IBM [2024-01-19T04:12:37.610Z] 2024-01-19 04:12:37 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-vlbwu] for org [default] with [ibm] as provider [2024-01-19T04:12:37.610Z] 2024-01-19 04:12:37 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T04:12:37.610Z] 2024-01-19 04:12:37 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T04:12:37.610Z] 2024-01-19 04:12:37 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-vlbwu] is present [2024-01-19T04:12:37.610Z] 2024-01-19 04:12:37 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Create cluster [source-cluster] in org [default] [2024-01-19T04:12:38.535Z] 2024-01-19 04:12:38 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-tjtlo] for IBM [2024-01-19T04:12:38.535Z] 2024-01-19 04:12:38 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-tjtlo] for org [default] with [ibm] as provider [2024-01-19T04:12:38.535Z] 2024-01-19 04:12:38 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T04:12:38.536Z] 2024-01-19 04:12:38 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T04:12:38.536Z] 2024-01-19 04:12:38 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-tjtlo] is present [2024-01-19T04:12:38.536Z] 2024-01-19 04:12:38 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Create cluster [destination-cluster] in org [default] [2024-01-19T04:12:39.097Z] INFO[2024-01-19 04:12:38] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T04:12:39.097Z] INFO[2024-01-19 04:12:38] Actual:Online, Expected: Online [2024-01-19T04:12:39.097Z] INFO[2024-01-19 04:12:38] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T04:12:39.097Z] INFO[2024-01-19 04:12:38] Actual:true, Expected: true [2024-01-19T04:12:39.097Z] INFO[2024-01-19 04:12:38] Verifying : Description : Fetching [destination-cluster] cluster uid [2024-01-19T04:12:39.097Z] INFO[2024-01-19 04:12:38] Actual:true, Expected: true [2024-01-19T04:12:39.097Z] 2024-01-19 04:12:38 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Registering Source and Destination clusters as user : testuser1-1705637513 and verifying the status [2024-01-19T04:12:39.353Z] INFO[2024-01-19 04:12:39] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T04:12:39.353Z] INFO[2024-01-19 04:12:39] Actual:true, Expected: true [2024-01-19T04:12:39.353Z] 2024-01-19 04:12:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T04:12:39.354Z] 2024-01-19 04:12:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T04:12:39.354Z] 2024-01-19 04:12:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T04:12:39.354Z] 2024-01-19 04:12:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T04:12:39.354Z] 2024-01-19 04:12:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Create cluster [source-cluster] in org [default] [2024-01-19T04:12:39.354Z] 2024-01-19 04:12:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T04:12:39.354Z] 2024-01-19 04:12:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T04:12:39.354Z] 2024-01-19 04:12:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Create cluster [destination-cluster] in org [default] [2024-01-19T04:12:39.354Z] 2024-01-19 04:12:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T04:12:39.354Z] 2024-01-19 04:12:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T04:12:39.354Z] 2024-01-19 04:12:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-mhxzi] for IBM [2024-01-19T04:12:39.354Z] 2024-01-19 04:12:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-mhxzi] for org [default] with [ibm] as provider [2024-01-19T04:12:39.354Z] 2024-01-19 04:12:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T04:12:39.354Z] 2024-01-19 04:12:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T04:12:39.355Z] 2024-01-19 04:12:39 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-mhxzi] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user]] [2024-01-19T04:12:39.355Z] 2024-01-19 04:12:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user] [2024-01-19T04:12:39.610Z] 2024-01-19 04:12:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-mhxzi from admin context and sharing with all the users [2024-01-19T04:12:39.610Z] 2024-01-19 04:12:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-mhxzi] for org [default] with [ibm] as provider [2024-01-19T04:12:39.610Z] 2024-01-19 04:12:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T04:12:39.611Z] 2024-01-19 04:12:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T04:12:39.611Z] 2024-01-19 04:12:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-mhxzi] is present [2024-01-19T04:12:39.866Z] 2024-01-19 04:12:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Create cluster [source-cluster] in org [default] [2024-01-19T04:12:40.430Z] 2024-01-19 04:12:40 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-nnofn] for IBM [2024-01-19T04:12:40.430Z] 2024-01-19 04:12:40 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-nnofn] for org [default] with [ibm] as provider [2024-01-19T04:12:40.430Z] 2024-01-19 04:12:40 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T04:12:40.430Z] 2024-01-19 04:12:40 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T04:12:40.430Z] 2024-01-19 04:12:40 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-nnofn] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user]] [2024-01-19T04:12:40.430Z] 2024-01-19 04:12:40 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user] [2024-01-19T04:12:40.685Z] 2024-01-19 04:12:40 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-nnofn from admin context and sharing with all the users [2024-01-19T04:12:40.685Z] 2024-01-19 04:12:40 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-nnofn] for org [default] with [ibm] as provider [2024-01-19T04:12:40.686Z] 2024-01-19 04:12:40 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T04:12:40.686Z] 2024-01-19 04:12:40 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T04:12:40.941Z] 2024-01-19 04:12:40 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-nnofn] is present [2024-01-19T04:12:40.941Z] 2024-01-19 04:12:40 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Create cluster [destination-cluster] in org [default] [2024-01-19T04:12:41.868Z] INFO[2024-01-19 04:12:41] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T04:12:41.868Z] INFO[2024-01-19 04:12:41] Actual:Online, Expected: Online [2024-01-19T04:12:41.868Z] INFO[2024-01-19 04:12:41] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T04:12:41.868Z] INFO[2024-01-19 04:12:41] Actual:true, Expected: true [2024-01-19T04:12:41.868Z] INFO[2024-01-19 04:12:41] Verifying : Description : Fetching [destination-cluster] cluster uid [2024-01-19T04:12:41.868Z] INFO[2024-01-19 04:12:41] Actual:true, Expected: true [2024-01-19T04:12:41.868Z] 2024-01-19 04:12:41 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Taking schedule backup of applications as non-admin user with and without rules [2024-01-19T04:12:54.028Z] 2024-01-19 04:12:51 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Taking schedule backup of single namespace as user : testuser1-1705637494 without-rules [2024-01-19T04:12:54.028Z] 2024-01-19 04:12:51 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating a schedule backup of namespace [mysql-ibm-pxb-0-87560-01-19-04h09m51s] without pre and post exec rules [2024-01-19T04:12:54.028Z] 2024-01-19 04:12:51 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T04:12:54.028Z] 2024-01-19 04:12:51 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Volume snapshot classes found in the cluster - [2024-01-19T04:12:54.029Z] 2024-01-19 04:12:51 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] ibmc-vpcblock-snapshot [2024-01-19T04:12:54.029Z] 2024-01-19 04:12:51 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T04:13:02.111Z] 2024-01-19 04:13:01 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Taking schedule backup of single namespace as user : testuser1-1705637497 without-rules [2024-01-19T04:13:02.111Z] 2024-01-19 04:13:01 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating a schedule backup of namespace [mysql-ibm-pxb-0-87560-01-19-04h09m51s] without pre and post exec rules [2024-01-19T04:13:02.111Z] 2024-01-19 04:13:01 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T04:13:02.112Z] 2024-01-19 04:13:01 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Volume snapshot classes found in the cluster - [2024-01-19T04:13:02.112Z] 2024-01-19 04:13:01 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] ibmc-vpcblock-snapshot [2024-01-19T04:13:02.112Z] 2024-01-19 04:13:01 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T04:13:12.064Z] 2024-01-19 04:13:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Taking schedule backup of single namespace as user : testuser1-1705637501 without-rules [2024-01-19T04:13:12.064Z] 2024-01-19 04:13:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating a schedule backup of namespace [mysql-ibm-pxb-0-87560-01-19-04h09m51s] without pre and post exec rules [2024-01-19T04:13:12.064Z] 2024-01-19 04:13:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T04:13:12.064Z] 2024-01-19 04:13:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Volume snapshot classes found in the cluster - [2024-01-19T04:13:12.064Z] 2024-01-19 04:13:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] ibmc-vpcblock-snapshot [2024-01-19T04:13:12.064Z] 2024-01-19 04:13:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T04:13:22.000Z] 2024-01-19 04:13:21 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Taking schedule backup of single namespace as user : testuser1-1705637505 without-rules [2024-01-19T04:13:22.000Z] 2024-01-19 04:13:21 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating a schedule backup of namespace [mysql-ibm-pxb-0-87560-01-19-04h09m51s] without pre and post exec rules [2024-01-19T04:13:22.000Z] 2024-01-19 04:13:21 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T04:13:22.000Z] 2024-01-19 04:13:21 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Volume snapshot classes found in the cluster - [2024-01-19T04:13:22.000Z] 2024-01-19 04:13:21 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] ibmc-vpcblock-snapshot [2024-01-19T04:13:22.000Z] 2024-01-19 04:13:21 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T04:13:31.934Z] 2024-01-19 04:13:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Taking schedule backup of single namespace as user : testuser1-1705637509 without-rules [2024-01-19T04:13:31.935Z] 2024-01-19 04:13:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating a schedule backup of namespace [mysql-ibm-pxb-0-87560-01-19-04h09m51s] without pre and post exec rules [2024-01-19T04:13:31.935Z] 2024-01-19 04:13:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T04:13:31.935Z] 2024-01-19 04:13:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Volume snapshot classes found in the cluster - [2024-01-19T04:13:31.935Z] 2024-01-19 04:13:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] ibmc-vpcblock-snapshot [2024-01-19T04:13:31.935Z] 2024-01-19 04:13:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T04:13:41.874Z] 2024-01-19 04:13:41 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Taking schedule backup of single namespace as user : testuser1-1705637513 without-rules [2024-01-19T04:13:41.874Z] 2024-01-19 04:13:41 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating a schedule backup of namespace [mysql-ibm-pxb-0-87560-01-19-04h09m51s] without pre and post exec rules [2024-01-19T04:13:41.874Z] 2024-01-19 04:13:41 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T04:13:42.130Z] 2024-01-19 04:13:41 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Volume snapshot classes found in the cluster - [2024-01-19T04:13:42.130Z] 2024-01-19 04:13:41 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] ibmc-vpcblock-snapshot [2024-01-19T04:13:42.130Z] 2024-01-19 04:13:41 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T04:13:54.296Z] 2024-01-19 04:13:52 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] first schedule backup for schedule name [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl] is [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl-7c3a711-interval-2024-01-19-041252] [2024-01-19T04:13:54.296Z] 2024/01/19 04:13:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl-7c3a711-interval-2024-01-19-041252] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:14:04.234Z] 2024-01-19 04:14:02 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] first schedule backup for schedule name [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh] is [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302] [2024-01-19T04:14:04.234Z] 2024/01/19 04:14:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:14:12.333Z] 2024-01-19 04:14:12 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] first schedule backup for schedule name [tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo] is [tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312] [2024-01-19T04:14:12.589Z] 2024/01/19 04:14:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:14:22.527Z] 2024-01-19 04:14:22 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] first schedule backup for schedule name [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo] is [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322] [2024-01-19T04:14:22.527Z] 2024/01/19 04:14:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:14:22.527Z] 2024/01/19 04:14:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl-7c3a711-interval-2024-01-19-041252] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:14:32.511Z] 2024-01-19 04:14:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] first schedule backup for schedule name [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx] is [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332] [2024-01-19T04:14:32.512Z] 2024/01/19 04:14:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:14:32.767Z] 2024/01/19 04:14:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:14:42.721Z] 2024-01-19 04:14:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] first schedule backup for schedule name [tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa] is [tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342] [2024-01-19T04:14:42.721Z] 2024/01/19 04:14:42 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:14:42.721Z] 2024/01/19 04:14:42 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:14:52.674Z] 2024/01/19 04:14:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:14:52.674Z] 2024/01/19 04:14:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl-7c3a711-interval-2024-01-19-041252] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:14:55.201Z] INFO[2024-01-19 04:14:54] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T04:14:55.201Z] INFO[2024-01-19 04:14:54] sched stats: tick interval: 45654 < 1.2s (100%) [2024-01-19T04:14:55.201Z] INFO[2024-01-19 04:14:54] sched stats: tick process durations: 45653 < 500ms (100%) [2024-01-19T04:15:03.288Z] 2024/01/19 04:15:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:15:03.288Z] 2024/01/19 04:15:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:15:13.237Z] 2024/01/19 04:15:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:15:13.237Z] 2024/01/19 04:15:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:15:23.174Z] 2024/01/19 04:15:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:15:23.174Z] 2024/01/19 04:15:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl-7c3a711-interval-2024-01-19-041252] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:15:33.112Z] 2024/01/19 04:15:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:15:33.112Z] 2024/01/19 04:15:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:15:43.052Z] 2024/01/19 04:15:42 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:15:43.052Z] 2024/01/19 04:15:42 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:15:53.025Z] 2024/01/19 04:15:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:15:53.026Z] 2024/01/19 04:15:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl-7c3a711-interval-2024-01-19-041252] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:16:02.979Z] 2024/01/19 04:16:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:16:02.980Z] 2024/01/19 04:16:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:16:12.925Z] 2024/01/19 04:16:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:16:12.925Z] 2024/01/19 04:16:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:16:22.862Z] 2024/01/19 04:16:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:16:22.863Z] 2024/01/19 04:16:22 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl-7c3a711-interval-2024-01-19-041252] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:16:32.805Z] 2024/01/19 04:16:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:16:32.805Z] 2024/01/19 04:16:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:16:42.744Z] 2024/01/19 04:16:42 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:16:42.744Z] 2024/01/19 04:16:42 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:16:52.682Z] 2024/01/19 04:16:52 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:16:52.682Z] 2024-01-19 04:16:52 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backup [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl-7c3a711-interval-2024-01-19-041252] in org [default] [2024-01-19T04:16:52.682Z] 2024-01-19 04:16:52 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl-7c3a711-interval-2024-01-19-041252] [2024-01-19T04:16:52.683Z] 2024-01-19 04:16:52 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl-7c3a711-interval-2024-01-19-041252] [2024-01-19T04:16:52.683Z] 2024-01-19 04:16:52 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl-7c3a711-interval-2024-01-19-041252] [2024-01-19T04:16:52.683Z] 2024-01-19 04:16:52 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:16:52.683Z] 2024-01-19 04:16:52 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:16:52.683Z] 2024-01-19 04:16:52 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:16:52.683Z] 2024-01-19 04:16:52 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:16:52.683Z] 2024-01-19 04:16:52 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:16:52.683Z] 2024-01-19 04:16:52 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating CR cleanup [2024-01-19T04:16:52.939Z] 2024-01-19 04:16:52 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T04:16:52.939Z] 2024-01-19 04:16:52 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302 tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312 tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332 tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342]] [2024-01-19T04:16:52.939Z] INFO[2024-01-19 04:16:52] Verifying : Description : Verifying creation of backup [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl-7c3a711-interval-2024-01-19-041252] [2024-01-19T04:16:52.939Z] INFO[2024-01-19 04:16:52] Actual:true, Expected: true [2024-01-19T04:16:53.197Z] INFO[2024-01-19 04:16:53] Verifying : Description : Suspending Backup Schedule [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl] for user [testuser1-1705637494] [2024-01-19T04:16:53.197Z] INFO[2024-01-19 04:16:53] Actual:true, Expected: true [2024-01-19T04:17:03.133Z] 2024/01/19 04:17:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:17:03.133Z] 2024-01-19 04:17:02 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backup [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302] in org [default] [2024-01-19T04:17:03.133Z] 2024-01-19 04:17:02 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302] [2024-01-19T04:17:03.133Z] 2024-01-19 04:17:02 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302] [2024-01-19T04:17:03.133Z] 2024-01-19 04:17:02 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302] [2024-01-19T04:17:03.133Z] 2024-01-19 04:17:02 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:17:03.133Z] 2024-01-19 04:17:02 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:17:03.133Z] 2024-01-19 04:17:02 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:17:03.133Z] 2024-01-19 04:17:02 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:17:03.134Z] 2024-01-19 04:17:02 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:17:03.134Z] 2024-01-19 04:17:02 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating CR cleanup [2024-01-19T04:17:03.134Z] 2024-01-19 04:17:02 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T04:17:03.134Z] 2024-01-19 04:17:02 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302 tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312 tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332 tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342]] [2024-01-19T04:17:03.134Z] 2024-01-19 04:17:02 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302 tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312 tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332 tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:17:03.134Z] 2024/01/19 04:17:02 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:17:08.383Z] 2024-01-19 04:17:08 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302 tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312 tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332 tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342]] [2024-01-19T04:17:08.383Z] 2024-01-19 04:17:08 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302 tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312 tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332 tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:17:08.383Z] 2024/01/19 04:17:08 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:17:12.556Z] 2024-01-19 04:17:12 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backup [tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312] in org [default] [2024-01-19T04:17:12.556Z] 2024-01-19 04:17:12 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312] [2024-01-19T04:17:12.556Z] 2024/01/19 04:17:12 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:17:12.556Z] 2024-01-19 04:17:12 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312] [2024-01-19T04:17:12.556Z] 2024-01-19 04:17:12 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312] [2024-01-19T04:17:12.556Z] 2024-01-19 04:17:12 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:17:12.814Z] 2024-01-19 04:17:12 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:17:12.814Z] 2024-01-19 04:17:12 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:17:12.814Z] 2024-01-19 04:17:12 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:17:12.814Z] 2024-01-19 04:17:12 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:17:12.814Z] 2024-01-19 04:17:12 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating CR cleanup [2024-01-19T04:17:12.815Z] 2024-01-19 04:17:12 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T04:17:12.815Z] 2024-01-19 04:17:12 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302 tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332 tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342]] [2024-01-19T04:17:12.815Z] INFO[2024-01-19 04:17:12] Verifying : Description : Verifying creation of backup [tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312] [2024-01-19T04:17:12.815Z] INFO[2024-01-19 04:17:12] Actual:true, Expected: true [2024-01-19T04:17:13.378Z] INFO[2024-01-19 04:17:13] Verifying : Description : Suspending Backup Schedule [tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo] for user [testuser1-1705637501] [2024-01-19T04:17:13.378Z] INFO[2024-01-19 04:17:13] Actual:true, Expected: true [2024-01-19T04:17:13.378Z] 2024-01-19 04:17:13 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302 tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332 tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342]] [2024-01-19T04:17:13.378Z] 2024-01-19 04:17:13 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302 tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332 tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:17:13.378Z] 2024/01/19 04:17:13 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:17:18.623Z] 2024-01-19 04:17:18 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332 tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342]] [2024-01-19T04:17:18.623Z] INFO[2024-01-19 04:17:18] Verifying : Description : Verifying creation of backup [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302] [2024-01-19T04:17:18.623Z] INFO[2024-01-19 04:17:18] Actual:true, Expected: true [2024-01-19T04:17:18.623Z] INFO[2024-01-19 04:17:18] Verifying : Description : Suspending Backup Schedule [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh] for user [testuser1-1705637497] [2024-01-19T04:17:18.623Z] INFO[2024-01-19 04:17:18] Actual:true, Expected: true [2024-01-19T04:17:22.813Z] 2024-01-19 04:17:22 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backup [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322] in org [default] [2024-01-19T04:17:22.813Z] 2024-01-19 04:17:22 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322] [2024-01-19T04:17:22.813Z] 2024-01-19 04:17:22 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322] [2024-01-19T04:17:22.813Z] 2024-01-19 04:17:22 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322] [2024-01-19T04:17:22.813Z] 2024-01-19 04:17:22 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:17:22.813Z] 2024-01-19 04:17:22 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:17:22.814Z] 2024-01-19 04:17:22 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:17:22.814Z] 2024-01-19 04:17:22 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:17:22.814Z] 2024-01-19 04:17:22 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:17:22.814Z] 2024-01-19 04:17:22 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating CR cleanup [2024-01-19T04:17:22.814Z] 2024-01-19 04:17:22 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T04:17:22.814Z] 2024-01-19 04:17:22 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332 tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342]] [2024-01-19T04:17:22.814Z] 2024-01-19 04:17:22 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332 tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:17:22.814Z] 2024/01/19 04:17:22 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:17:28.054Z] 2024-01-19 04:17:27 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332 tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342]] [2024-01-19T04:17:28.054Z] 2024-01-19 04:17:27 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332 tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:17:28.054Z] 2024/01/19 04:17:27 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:17:33.292Z] 2024-01-19 04:17:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backup [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332] in org [default] [2024-01-19T04:17:33.292Z] 2024-01-19 04:17:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332] [2024-01-19T04:17:33.292Z] 2024-01-19 04:17:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332] [2024-01-19T04:17:33.293Z] 2024-01-19 04:17:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332] [2024-01-19T04:17:33.293Z] 2024-01-19 04:17:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:17:33.293Z] 2024-01-19 04:17:32 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:17:33.293Z] 2024-01-19 04:17:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:17:33.293Z] 2024-01-19 04:17:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:17:33.293Z] 2024-01-19 04:17:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:17:33.293Z] 2024-01-19 04:17:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating CR cleanup [2024-01-19T04:17:33.293Z] 2024-01-19 04:17:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T04:17:33.293Z] 2024-01-19 04:17:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] [2024-01-19T04:17:33.293Z] 2024-01-19 04:17:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:17:33.293Z] 2024/01/19 04:17:32 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:17:33.293Z] 2024-01-19 04:17:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] [2024-01-19T04:17:33.293Z] 2024-01-19 04:17:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:17:33.293Z] 2024/01/19 04:17:32 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:17:38.536Z] 2024-01-19 04:17:37 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] [2024-01-19T04:17:38.536Z] 2024-01-19 04:17:37 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:17:38.536Z] 2024/01/19 04:17:37 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:17:38.537Z] 2024-01-19 04:17:37 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] [2024-01-19T04:17:38.537Z] 2024-01-19 04:17:37 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:17:38.537Z] 2024/01/19 04:17:37 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:17:42.727Z] 2024-01-19 04:17:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backup [tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342] in org [default] [2024-01-19T04:17:42.727Z] 2024-01-19 04:17:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342] [2024-01-19T04:17:42.727Z] 2024-01-19 04:17:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342] [2024-01-19T04:17:42.727Z] 2024-01-19 04:17:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342] [2024-01-19T04:17:42.727Z] 2024-01-19 04:17:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:17:42.727Z] 2024-01-19 04:17:42 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:17:42.727Z] 2024-01-19 04:17:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:17:42.727Z] 2024-01-19 04:17:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:17:42.727Z] 2024-01-19 04:17:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:17:42.727Z] 2024-01-19 04:17:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating CR cleanup [2024-01-19T04:17:42.727Z] 2024-01-19 04:17:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T04:17:42.727Z] 2024-01-19 04:17:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] [2024-01-19T04:17:42.727Z] INFO[2024-01-19 04:17:42] Verifying : Description : Verifying creation of backup [tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342] [2024-01-19T04:17:42.727Z] INFO[2024-01-19 04:17:42] Actual:true, Expected: true [2024-01-19T04:17:43.004Z] 2024-01-19 04:17:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] [2024-01-19T04:17:43.004Z] 2024-01-19 04:17:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:17:43.004Z] 2024/01/19 04:17:42 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:17:43.004Z] 2024-01-19 04:17:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] [2024-01-19T04:17:43.004Z] 2024-01-19 04:17:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:17:43.004Z] 2024/01/19 04:17:42 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:17:43.004Z] INFO[2024-01-19 04:17:42] Verifying : Description : Suspending Backup Schedule [tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa] for user [testuser1-1705637513] [2024-01-19T04:17:43.004Z] INFO[2024-01-19 04:17:42] Actual:true, Expected: true [2024-01-19T04:17:48.198Z] 2024-01-19 04:17:47 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] [2024-01-19T04:17:48.198Z] 2024-01-19 04:17:47 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:17:48.198Z] 2024/01/19 04:17:47 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:17:48.198Z] 2024-01-19 04:17:47 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] [2024-01-19T04:17:48.199Z] 2024-01-19 04:17:47 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:17:48.199Z] 2024/01/19 04:17:47 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:17:53.439Z] 2024-01-19 04:17:52 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] [2024-01-19T04:17:53.439Z] 2024-01-19 04:17:52 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:17:53.439Z] 2024/01/19 04:17:52 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:17:53.439Z] 2024-01-19 04:17:52 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] [2024-01-19T04:17:53.439Z] 2024-01-19 04:17:52 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:17:53.439Z] 2024/01/19 04:17:52 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:17:58.678Z] 2024-01-19 04:17:57 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] [2024-01-19T04:17:58.679Z] 2024-01-19 04:17:57 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:17:58.679Z] 2024/01/19 04:17:57 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:17:58.679Z] 2024-01-19 04:17:57 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] [2024-01-19T04:17:58.679Z] 2024-01-19 04:17:57 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:17:58.679Z] 2024/01/19 04:17:57 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:18:03.923Z] 2024-01-19 04:18:02 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] [2024-01-19T04:18:03.923Z] 2024-01-19 04:18:02 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:18:03.923Z] 2024/01/19 04:18:02 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:18:03.923Z] 2024-01-19 04:18:02 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] [2024-01-19T04:18:03.923Z] 2024-01-19 04:18:02 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:18:03.923Z] 2024/01/19 04:18:02 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:18:08.092Z] 2024-01-19 04:18:07 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]] [2024-01-19T04:18:08.092Z] INFO[2024-01-19 04:18:07] Verifying : Description : Verifying creation of backup [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332] [2024-01-19T04:18:08.092Z] INFO[2024-01-19 04:18:07] Actual:true, Expected: true [2024-01-19T04:18:08.093Z] 2024-01-19 04:18:07 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]] [2024-01-19T04:18:08.093Z] 2024-01-19 04:18:07 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:18:08.093Z] 2024/01/19 04:18:07 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:18:08.348Z] INFO[2024-01-19 04:18:08] Verifying : Description : Suspending Backup Schedule [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx] for user [testuser1-1705637509] [2024-01-19T04:18:08.348Z] INFO[2024-01-19 04:18:08] Actual:true, Expected: true [2024-01-19T04:18:13.590Z] 2024-01-19 04:18:13 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]] [2024-01-19T04:18:13.590Z] 2024-01-19 04:18:13 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:18:13.590Z] 2024/01/19 04:18:13 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:18:18.831Z] 2024-01-19 04:18:18 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]] [2024-01-19T04:18:18.831Z] 2024-01-19 04:18:18 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:18:18.831Z] 2024/01/19 04:18:18 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:18:24.071Z] 2024-01-19 04:18:23 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[]] [2024-01-19T04:18:24.071Z] INFO[2024-01-19 04:18:23] Verifying : Description : Verifying creation of backup [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322] [2024-01-19T04:18:24.071Z] INFO[2024-01-19 04:18:23] Actual:true, Expected: true [2024-01-19T04:18:24.071Z] INFO[2024-01-19 04:18:23] Verifying : Description : Suspending Backup Schedule [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo] for user [testuser1-1705637505] [2024-01-19T04:18:24.071Z] INFO[2024-01-19 04:18:23] Actual:true, Expected: true [2024-01-19T04:18:34.010Z] 2024-01-19 04:18:33 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Taking schedule backup of multiple namespace as user : testuser1-1705637494 with-rules [2024-01-19T04:18:34.010Z] 2024-01-19 04:18:33 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating a schedule backup of namespace [[mysql-ibm-pxb-0-87560-01-19-04h09m51s mysql-ibm-pxb-1-87560-01-19-04h09m51s]] with pre and post exec rules [2024-01-19T04:18:34.010Z] 2024-01-19 04:18:33 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T04:18:34.011Z] 2024-01-19 04:18:33 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Volume snapshot classes found in the cluster - [2024-01-19T04:18:34.011Z] 2024-01-19 04:18:33 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] ibmc-vpcblock-snapshot [2024-01-19T04:18:34.011Z] 2024-01-19 04:18:33 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T04:18:43.967Z] 2024-01-19 04:18:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Taking schedule backup of multiple namespace as user : testuser1-1705637497 with-rules [2024-01-19T04:18:43.967Z] 2024-01-19 04:18:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating a schedule backup of namespace [[mysql-ibm-pxb-0-87560-01-19-04h09m51s mysql-ibm-pxb-1-87560-01-19-04h09m51s]] with pre and post exec rules [2024-01-19T04:18:43.967Z] 2024-01-19 04:18:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T04:18:43.967Z] 2024-01-19 04:18:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Volume snapshot classes found in the cluster - [2024-01-19T04:18:43.967Z] 2024-01-19 04:18:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] ibmc-vpcblock-snapshot [2024-01-19T04:18:43.967Z] 2024-01-19 04:18:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T04:18:53.908Z] 2024-01-19 04:18:53 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Taking schedule backup of multiple namespace as user : testuser1-1705637501 with-rules [2024-01-19T04:18:53.908Z] 2024-01-19 04:18:53 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating a schedule backup of namespace [[mysql-ibm-pxb-0-87560-01-19-04h09m51s mysql-ibm-pxb-1-87560-01-19-04h09m51s]] with pre and post exec rules [2024-01-19T04:18:53.909Z] 2024-01-19 04:18:53 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T04:18:53.909Z] 2024-01-19 04:18:53 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Volume snapshot classes found in the cluster - [2024-01-19T04:18:53.909Z] 2024-01-19 04:18:53 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] ibmc-vpcblock-snapshot [2024-01-19T04:18:53.909Z] 2024-01-19 04:18:53 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T04:19:00.445Z] Iteration: 63 [2024-01-19T04:19:02.334Z] Logs from iteration: 63 [2024-01-19T04:19:02.896Z] 2024-01-19 04:18:02 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] [2024-01-19T04:19:02.896Z] 2024-01-19 04:18:02 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:19:02.896Z] 2024/01/19 04:18:02 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:19:02.897Z] 2024-01-19 04:18:02 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] [2024-01-19T04:19:02.897Z] 2024-01-19 04:18:02 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:19:02.897Z] 2024/01/19 04:18:02 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:19:02.897Z] 2024-01-19 04:18:07 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]] [2024-01-19T04:19:02.897Z] INFO[2024-01-19 04:18:07] Verifying : Description : Verifying creation of backup [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332] [2024-01-19T04:19:02.897Z] INFO[2024-01-19 04:18:07] Actual:true, Expected: true [2024-01-19T04:19:02.897Z] 2024-01-19 04:18:07 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]] [2024-01-19T04:19:02.897Z] 2024-01-19 04:18:07 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:19:02.897Z] 2024/01/19 04:18:07 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:19:02.897Z] INFO[2024-01-19 04:18:08] Verifying : Description : Suspending Backup Schedule [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx] for user [testuser1-1705637509] [2024-01-19T04:19:02.897Z] INFO[2024-01-19 04:18:08] Actual:true, Expected: true [2024-01-19T04:19:02.897Z] 2024-01-19 04:18:13 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]] [2024-01-19T04:19:02.897Z] 2024-01-19 04:18:13 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:19:02.897Z] 2024/01/19 04:18:13 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:19:02.897Z] 2024-01-19 04:18:18 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]] [2024-01-19T04:19:02.897Z] 2024-01-19 04:18:18 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]] under [mysql-ibm-pxb-0-87560-01-19-04h09m51s] namespace [2024-01-19T04:19:02.897Z] 2024/01/19 04:18:18 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:19:02.897Z] 2024-01-19 04:18:23 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[]] [2024-01-19T04:19:02.898Z] INFO[2024-01-19 04:18:23] Verifying : Description : Verifying creation of backup [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322] [2024-01-19T04:19:02.898Z] INFO[2024-01-19 04:18:23] Actual:true, Expected: true [2024-01-19T04:19:02.898Z] INFO[2024-01-19 04:18:23] Verifying : Description : Suspending Backup Schedule [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo] for user [testuser1-1705637505] [2024-01-19T04:19:02.898Z] INFO[2024-01-19 04:18:23] Actual:true, Expected: true [2024-01-19T04:19:02.898Z] 2024-01-19 04:18:33 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Taking schedule backup of multiple namespace as user : testuser1-1705637494 with-rules [2024-01-19T04:19:02.898Z] 2024-01-19 04:18:33 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating a schedule backup of namespace [[mysql-ibm-pxb-0-87560-01-19-04h09m51s mysql-ibm-pxb-1-87560-01-19-04h09m51s]] with pre and post exec rules [2024-01-19T04:19:02.898Z] 2024-01-19 04:18:33 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T04:19:02.898Z] 2024-01-19 04:18:33 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Volume snapshot classes found in the cluster - [2024-01-19T04:19:02.898Z] 2024-01-19 04:18:33 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] ibmc-vpcblock-snapshot [2024-01-19T04:19:02.898Z] 2024-01-19 04:18:33 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T04:19:02.898Z] 2024-01-19 04:18:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Taking schedule backup of multiple namespace as user : testuser1-1705637497 with-rules [2024-01-19T04:19:02.898Z] 2024-01-19 04:18:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating a schedule backup of namespace [[mysql-ibm-pxb-0-87560-01-19-04h09m51s mysql-ibm-pxb-1-87560-01-19-04h09m51s]] with pre and post exec rules [2024-01-19T04:19:02.898Z] 2024-01-19 04:18:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T04:19:02.898Z] 2024-01-19 04:18:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Volume snapshot classes found in the cluster - [2024-01-19T04:19:02.898Z] 2024-01-19 04:18:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] ibmc-vpcblock-snapshot [2024-01-19T04:19:02.898Z] 2024-01-19 04:18:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T04:19:02.898Z] 2024-01-19 04:18:53 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Taking schedule backup of multiple namespace as user : testuser1-1705637501 with-rules [2024-01-19T04:19:02.898Z] 2024-01-19 04:18:53 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating a schedule backup of namespace [[mysql-ibm-pxb-0-87560-01-19-04h09m51s mysql-ibm-pxb-1-87560-01-19-04h09m51s]] with pre and post exec rules [2024-01-19T04:19:02.898Z] 2024-01-19 04:18:53 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T04:19:02.898Z] 2024-01-19 04:18:53 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Volume snapshot classes found in the cluster - [2024-01-19T04:19:02.898Z] 2024-01-19 04:18:53 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] ibmc-vpcblock-snapshot [2024-01-19T04:19:02.898Z] 2024-01-19 04:18:53 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T04:19:03.459Z] 2024-01-19 04:19:03 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Taking schedule backup of multiple namespace as user : testuser1-1705637505 with-rules [2024-01-19T04:19:03.459Z] 2024-01-19 04:19:03 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating a schedule backup of namespace [[mysql-ibm-pxb-0-87560-01-19-04h09m51s mysql-ibm-pxb-1-87560-01-19-04h09m51s]] with pre and post exec rules [2024-01-19T04:19:03.460Z] 2024-01-19 04:19:03 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T04:19:03.715Z] 2024-01-19 04:19:03 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Volume snapshot classes found in the cluster - [2024-01-19T04:19:03.715Z] 2024-01-19 04:19:03 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] ibmc-vpcblock-snapshot [2024-01-19T04:19:03.715Z] 2024-01-19 04:19:03 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T04:19:13.656Z] 2024-01-19 04:19:13 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Taking schedule backup of multiple namespace as user : testuser1-1705637509 with-rules [2024-01-19T04:19:13.656Z] 2024-01-19 04:19:13 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating a schedule backup of namespace [[mysql-ibm-pxb-0-87560-01-19-04h09m51s mysql-ibm-pxb-1-87560-01-19-04h09m51s]] with pre and post exec rules [2024-01-19T04:19:13.656Z] 2024-01-19 04:19:13 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T04:19:13.656Z] 2024-01-19 04:19:13 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Volume snapshot classes found in the cluster - [2024-01-19T04:19:13.656Z] 2024-01-19 04:19:13 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] ibmc-vpcblock-snapshot [2024-01-19T04:19:13.656Z] 2024-01-19 04:19:13 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T04:19:23.649Z] 2024-01-19 04:19:23 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Taking schedule backup of multiple namespace as user : testuser1-1705637513 with-rules [2024-01-19T04:19:23.649Z] 2024-01-19 04:19:23 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating a schedule backup of namespace [[mysql-ibm-pxb-0-87560-01-19-04h09m51s mysql-ibm-pxb-1-87560-01-19-04h09m51s]] with pre and post exec rules [2024-01-19T04:19:23.649Z] 2024-01-19 04:19:23 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T04:19:23.649Z] 2024-01-19 04:19:23 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Volume snapshot classes found in the cluster - [2024-01-19T04:19:23.649Z] 2024-01-19 04:19:23 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] ibmc-vpcblock-snapshot [2024-01-19T04:19:23.649Z] 2024-01-19 04:19:23 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T04:19:35.817Z] 2024-01-19 04:19:33 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] first schedule backup for schedule name [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn] is [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833] [2024-01-19T04:19:35.817Z] 2024/01/19 04:19:34 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833] expected was [[Success]] but got [InProgress] because of [Pre-Exec rules are being executed]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:19:45.766Z] 2024-01-19 04:19:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] first schedule backup for schedule name [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg] is [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843] [2024-01-19T04:19:45.766Z] 2024/01/19 04:19:43 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843] expected was [[Success]] but got [InProgress] because of [Pre-Exec rules are being executed]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:19:55.713Z] 2024-01-19 04:19:54 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] first schedule backup for schedule name [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb] is [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853] [2024-01-19T04:19:55.713Z] 2024/01/19 04:19:54 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853] expected was [[Success]] but got [InProgress] because of [Pre-Exec rules are being executed]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:20:05.653Z] 2024-01-19 04:20:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] first schedule backup for schedule name [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn] is [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903] [2024-01-19T04:20:05.653Z] 2024/01/19 04:20:04 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:20:05.653Z] 2024/01/19 04:20:04 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903] expected was [[Success]] but got [InProgress] because of [Pre-Exec rules are being executed]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:20:15.632Z] 2024/01/19 04:20:14 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:20:15.632Z] 2024-01-19 04:20:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] first schedule backup for schedule name [tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo] is [tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913] [2024-01-19T04:20:15.632Z] 2024/01/19 04:20:14 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913] expected was [[Success]] but got [InProgress] because of [Pre-Exec rules are being executed]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:20:25.578Z] 2024-01-19 04:20:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] first schedule backup for schedule name [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf] is [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923] [2024-01-19T04:20:25.578Z] 2024/01/19 04:20:24 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923] expected was [[Success]] but got [InProgress] because of [Pre-Exec rules are being executed]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:20:25.578Z] 2024/01/19 04:20:24 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:20:35.568Z] 2024/01/19 04:20:34 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:20:35.568Z] 2024/01/19 04:20:34 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:20:45.507Z] 2024/01/19 04:20:44 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:20:45.507Z] 2024/01/19 04:20:44 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913] expected was [[Success]] but got [InProgress] because of [Pre-Exec rules are being executed]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:20:55.454Z] 2024/01/19 04:20:54 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923] expected was [[Success]] but got [InProgress] because of [Pre-Exec rules are being executed]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:20:55.454Z] 2024/01/19 04:20:54 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:21:05.440Z] 2024/01/19 04:21:04 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:21:05.440Z] 2024/01/19 04:21:04 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:21:15.394Z] 2024/01/19 04:21:14 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:21:15.394Z] 2024/01/19 04:21:14 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:21:25.342Z] 2024/01/19 04:21:24 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:21:25.342Z] 2024/01/19 04:21:24 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:21:35.285Z] 2024/01/19 04:21:34 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:21:35.285Z] 2024/01/19 04:21:34 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:21:45.228Z] 2024/01/19 04:21:44 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:21:45.228Z] 2024/01/19 04:21:44 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:21:55.181Z] 2024/01/19 04:21:54 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:21:55.181Z] 2024/01/19 04:21:54 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:22:05.122Z] 2024/01/19 04:22:04 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:22:05.122Z] 2024/01/19 04:22:04 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:22:15.061Z] 2024/01/19 04:22:14 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:22:15.061Z] 2024/01/19 04:22:14 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:22:25.003Z] 2024/01/19 04:22:24 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:22:25.004Z] 2024/01/19 04:22:24 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:22:34.947Z] 2024/01/19 04:22:34 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:22:34.947Z] 2024/01/19 04:22:34 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:22:44.892Z] 2024/01/19 04:22:44 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:22:44.892Z] 2024/01/19 04:22:44 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:22:54.838Z] 2024/01/19 04:22:54 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:22:54.838Z] 2024/01/19 04:22:54 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:23:04.804Z] 2024/01/19 04:23:04 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:23:04.804Z] 2024/01/19 04:23:04 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:23:14.754Z] 2024/01/19 04:23:14 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:23:14.755Z] 2024/01/19 04:23:14 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:23:24.697Z] 2024/01/19 04:23:24 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:23:24.697Z] 2024/01/19 04:23:24 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:23:34.659Z] 2024/01/19 04:23:34 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:23:34.659Z] 2024/01/19 04:23:34 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:23:44.601Z] 2024/01/19 04:23:44 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:23:44.601Z] 2024/01/19 04:23:44 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:23:54.566Z] 2024/01/19 04:23:54 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:23:54.566Z] 2024/01/19 04:23:54 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:24:04.516Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backup [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833] in org [default] [2024-01-19T04:24:04.516Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833] [2024-01-19T04:24:04.516Z] 2024/01/19 04:24:04 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:24:04.516Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833] [2024-01-19T04:24:04.516Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833] [2024-01-19T04:24:04.516Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:24:04.516Z] 2024-01-19 04:24:04 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:24:04.516Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:24:04.517Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:24:04.517Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:24:04.517Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833] [2024-01-19T04:24:04.517Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833] [2024-01-19T04:24:04.517Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] [2024-01-19T04:24:04.517Z] 2024-01-19 04:24:04 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:24:04.517Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:24:04.517Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:24:04.517Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:24:04.517Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating CR cleanup [2024-01-19T04:24:04.517Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T04:24:04.774Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:04.774Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:04.774Z] 2024/01/19 04:24:04 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:10.018Z] 2024-01-19 04:24:09 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:10.018Z] 2024-01-19 04:24:09 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:10.018Z] 2024/01/19 04:24:09 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:14.220Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backup [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843] in org [default] [2024-01-19T04:24:14.221Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843] [2024-01-19T04:24:14.476Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843] [2024-01-19T04:24:14.477Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843] [2024-01-19T04:24:14.477Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:24:14.477Z] 2024/01/19 04:24:14 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:24:14.477Z] 2024-01-19 04:24:14 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:24:14.477Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:24:14.477Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:24:14.477Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:24:14.477Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843] [2024-01-19T04:24:14.477Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843] [2024-01-19T04:24:14.477Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] [2024-01-19T04:24:14.477Z] 2024-01-19 04:24:14 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:24:14.477Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:24:14.477Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:24:14.477Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:24:14.477Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating CR cleanup [2024-01-19T04:24:14.477Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T04:24:14.735Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:14.736Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:14.736Z] 2024/01/19 04:24:14 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:14.736Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:14.736Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:14.736Z] 2024/01/19 04:24:14 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:19.989Z] 2024-01-19 04:24:19 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:19.989Z] 2024-01-19 04:24:19 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:19.989Z] 2024/01/19 04:24:19 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:19.989Z] 2024-01-19 04:24:19 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:19.990Z] 2024-01-19 04:24:19 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:19.990Z] 2024/01/19 04:24:19 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:25.236Z] 2024/01/19 04:24:24 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:24:25.236Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backup [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853] in org [default] [2024-01-19T04:24:25.236Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853] [2024-01-19T04:24:25.236Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853] [2024-01-19T04:24:25.236Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853] [2024-01-19T04:24:25.236Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:24:25.236Z] 2024-01-19 04:24:24 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:24:25.237Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:24:25.237Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:24:25.237Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:24:25.237Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853] [2024-01-19T04:24:25.237Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853] [2024-01-19T04:24:25.237Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] [2024-01-19T04:24:25.237Z] 2024-01-19 04:24:24 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:24:25.237Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:24:25.237Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:24:25.237Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:24:25.237Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating CR cleanup [2024-01-19T04:24:25.237Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T04:24:25.237Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:25.238Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:25.238Z] 2024/01/19 04:24:24 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:25.238Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:25.238Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:25.238Z] 2024/01/19 04:24:24 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:25.238Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:25.239Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:25.239Z] 2024/01/19 04:24:24 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:30.496Z] 2024-01-19 04:24:29 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:30.496Z] 2024-01-19 04:24:29 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:30.496Z] 2024/01/19 04:24:29 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:30.497Z] 2024-01-19 04:24:29 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:30.497Z] 2024-01-19 04:24:29 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:30.497Z] 2024/01/19 04:24:29 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:30.497Z] 2024-01-19 04:24:29 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:30.497Z] 2024-01-19 04:24:29 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:30.498Z] 2024/01/19 04:24:29 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:34.710Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backup [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903] in org [default] [2024-01-19T04:24:34.710Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903] [2024-01-19T04:24:34.710Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903] [2024-01-19T04:24:34.710Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903] [2024-01-19T04:24:34.710Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:24:34.710Z] 2024-01-19 04:24:34 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:24:34.710Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:24:34.710Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:24:34.710Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:24:34.710Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903] [2024-01-19T04:24:34.710Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903] [2024-01-19T04:24:34.710Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] [2024-01-19T04:24:34.711Z] 2024-01-19 04:24:34 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:24:34.711Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:24:34.711Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:24:34.711Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:24:34.711Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating CR cleanup [2024-01-19T04:24:34.711Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T04:24:34.711Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:34.711Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:34.711Z] 2024/01/19 04:24:34 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:34.968Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:34.968Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:34.968Z] 2024/01/19 04:24:34 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:34.968Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:34.968Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:34.968Z] 2024/01/19 04:24:34 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:34.969Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:34.969Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:34.969Z] 2024/01/19 04:24:34 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:39.151Z] Iteration: 64 [2024-01-19T04:24:39.713Z] Logs from iteration: 64 [2024-01-19T04:24:40.640Z] 2024/01/19 04:23:44 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:24:40.640Z] 2024/01/19 04:23:44 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:24:40.640Z] 2024/01/19 04:23:54 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:24:40.640Z] 2024/01/19 04:23:54 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:24:40.640Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backup [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833] in org [default] [2024-01-19T04:24:40.640Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833] [2024-01-19T04:24:40.640Z] 2024/01/19 04:24:04 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:24:40.640Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833] [2024-01-19T04:24:40.640Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833] [2024-01-19T04:24:40.640Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:24:40.640Z] 2024-01-19 04:24:04 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:24:40.640Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:24:40.640Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:24:40.640Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:24:40.640Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833] [2024-01-19T04:24:40.640Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833] [2024-01-19T04:24:40.641Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] [2024-01-19T04:24:40.641Z] 2024-01-19 04:24:04 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:24:40.641Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:24:40.641Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:24:40.641Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:24:40.641Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating CR cleanup [2024-01-19T04:24:40.641Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T04:24:40.641Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:40.641Z] 2024-01-19 04:24:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:40.641Z] 2024/01/19 04:24:04 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:40.641Z] 2024-01-19 04:24:09 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:40.642Z] 2024-01-19 04:24:09 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:40.642Z] 2024/01/19 04:24:09 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:40.642Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backup [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843] in org [default] [2024-01-19T04:24:40.642Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843] [2024-01-19T04:24:40.642Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843] [2024-01-19T04:24:40.642Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843] [2024-01-19T04:24:40.642Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:24:40.642Z] 2024/01/19 04:24:14 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:24:40.642Z] 2024-01-19 04:24:14 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:24:40.642Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:24:40.642Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:24:40.642Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:24:40.642Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843] [2024-01-19T04:24:40.642Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843] [2024-01-19T04:24:40.643Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] [2024-01-19T04:24:40.643Z] 2024-01-19 04:24:14 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:24:40.643Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:24:40.643Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:24:40.643Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:24:40.643Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating CR cleanup [2024-01-19T04:24:40.643Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T04:24:40.643Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:40.643Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:40.643Z] 2024/01/19 04:24:14 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:40.643Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:40.644Z] 2024-01-19 04:24:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:40.644Z] 2024/01/19 04:24:14 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:40.644Z] 2024-01-19 04:24:19 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:40.644Z] 2024-01-19 04:24:19 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:40.644Z] 2024/01/19 04:24:19 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:40.644Z] 2024-01-19 04:24:19 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:40.645Z] 2024-01-19 04:24:19 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:40.645Z] 2024/01/19 04:24:19 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:40.645Z] 2024/01/19 04:24:24 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:24:40.645Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backup [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853] in org [default] [2024-01-19T04:24:40.645Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853] [2024-01-19T04:24:40.645Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853] [2024-01-19T04:24:40.645Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853] [2024-01-19T04:24:40.645Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:24:40.645Z] 2024-01-19 04:24:24 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:24:40.645Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:24:40.645Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:24:40.645Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:24:40.645Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853] [2024-01-19T04:24:40.645Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853] [2024-01-19T04:24:40.645Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] [2024-01-19T04:24:40.645Z] 2024-01-19 04:24:24 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:24:40.645Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:24:40.645Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:24:40.646Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:24:40.646Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating CR cleanup [2024-01-19T04:24:40.646Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T04:24:40.646Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:40.646Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:40.646Z] 2024/01/19 04:24:24 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:40.646Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:40.647Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:40.647Z] 2024/01/19 04:24:24 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:40.647Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:40.647Z] 2024-01-19 04:24:24 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:40.647Z] 2024/01/19 04:24:24 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:40.647Z] 2024-01-19 04:24:29 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:40.904Z] 2024-01-19 04:24:29 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:40.904Z] 2024/01/19 04:24:29 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:40.904Z] 2024-01-19 04:24:29 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:40.904Z] 2024-01-19 04:24:29 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:40.904Z] 2024/01/19 04:24:29 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:40.904Z] 2024-01-19 04:24:29 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:40.905Z] 2024-01-19 04:24:29 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:40.905Z] 2024/01/19 04:24:29 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:40.905Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backup [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903] in org [default] [2024-01-19T04:24:40.905Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903] [2024-01-19T04:24:40.905Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903] [2024-01-19T04:24:40.905Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903] [2024-01-19T04:24:40.905Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:24:40.905Z] 2024-01-19 04:24:34 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:24:40.905Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:24:40.905Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:24:40.905Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:24:40.905Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903] [2024-01-19T04:24:40.905Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903] [2024-01-19T04:24:40.905Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] [2024-01-19T04:24:40.905Z] 2024-01-19 04:24:34 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:24:40.906Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:24:40.906Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:24:40.906Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:24:40.906Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating CR cleanup [2024-01-19T04:24:40.906Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T04:24:40.906Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:40.906Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:40.906Z] 2024/01/19 04:24:34 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:40.906Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:40.907Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:40.907Z] 2024/01/19 04:24:34 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:40.907Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:40.907Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:40.907Z] 2024/01/19 04:24:34 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:40.907Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:40.908Z] 2024-01-19 04:24:34 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:40.908Z] 2024/01/19 04:24:34 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:40.908Z] 2024-01-19 04:24:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:40.908Z] 2024-01-19 04:24:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:40.908Z] 2024/01/19 04:24:39 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:40.908Z] 2024-01-19 04:24:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:40.908Z] 2024-01-19 04:24:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:40.909Z] 2024/01/19 04:24:39 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:40.909Z] 2024-01-19 04:24:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:40.909Z] 2024-01-19 04:24:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:40.909Z] 2024/01/19 04:24:39 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:40.909Z] 2024-01-19 04:24:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:40.909Z] 2024-01-19 04:24:39 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:40.909Z] 2024/01/19 04:24:39 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:45.081Z] 2024-01-19 04:24:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backup [tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913] in org [default] [2024-01-19T04:24:45.081Z] 2024-01-19 04:24:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913] [2024-01-19T04:24:45.081Z] 2024-01-19 04:24:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913] [2024-01-19T04:24:45.081Z] 2024-01-19 04:24:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913] [2024-01-19T04:24:45.081Z] 2024-01-19 04:24:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:24:45.081Z] 2024-01-19 04:24:44 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:24:45.081Z] 2024-01-19 04:24:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:24:45.081Z] 2024-01-19 04:24:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:24:45.081Z] 2024-01-19 04:24:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:24:45.081Z] 2024-01-19 04:24:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913] [2024-01-19T04:24:45.081Z] 2024-01-19 04:24:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913] [2024-01-19T04:24:45.081Z] 2024-01-19 04:24:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] [2024-01-19T04:24:45.081Z] 2024-01-19 04:24:44 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:24:45.081Z] 2024-01-19 04:24:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:24:45.081Z] 2024-01-19 04:24:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:24:45.081Z] 2024-01-19 04:24:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:24:45.081Z] 2024-01-19 04:24:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating CR cleanup [2024-01-19T04:24:45.082Z] 2024-01-19 04:24:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T04:24:45.082Z] 2024-01-19 04:24:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:45.082Z] 2024-01-19 04:24:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:45.082Z] 2024/01/19 04:24:44 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:45.082Z] 2024-01-19 04:24:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:45.082Z] 2024-01-19 04:24:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:45.082Z] 2024/01/19 04:24:44 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:45.083Z] 2024-01-19 04:24:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:45.083Z] 2024-01-19 04:24:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:45.083Z] 2024/01/19 04:24:44 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:45.083Z] 2024-01-19 04:24:45 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:45.083Z] 2024-01-19 04:24:45 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:45.083Z] 2024/01/19 04:24:45 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:45.084Z] 2024-01-19 04:24:45 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:45.084Z] 2024-01-19 04:24:45 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:45.084Z] 2024/01/19 04:24:45 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:50.346Z] 2024-01-19 04:24:49 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:50.346Z] 2024-01-19 04:24:49 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:50.346Z] 2024/01/19 04:24:49 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:50.346Z] 2024-01-19 04:24:49 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:50.346Z] 2024-01-19 04:24:49 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:50.346Z] 2024/01/19 04:24:49 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:50.347Z] 2024-01-19 04:24:50 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:50.347Z] 2024-01-19 04:24:50 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:50.347Z] 2024/01/19 04:24:50 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:50.347Z] 2024-01-19 04:24:50 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:50.347Z] INFO[2024-01-19 04:24:50] Verifying : Description : Verifying creation of backup [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853] [2024-01-19T04:24:50.347Z] INFO[2024-01-19 04:24:50] Actual:true, Expected: true [2024-01-19T04:24:50.347Z] 2024-01-19 04:24:50 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:50.347Z] 2024-01-19 04:24:50 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:50.348Z] 2024/01/19 04:24:50 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:50.348Z] INFO[2024-01-19 04:24:50] Verifying : Description : Suspending Backup Schedule [tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb] for user [testuser1-1705637501] [2024-01-19T04:24:50.348Z] INFO[2024-01-19 04:24:50] Actual:true, Expected: true [2024-01-19T04:24:54.533Z] 2024-01-19 04:24:54 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backup [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923] in org [default] [2024-01-19T04:24:54.533Z] 2024-01-19 04:24:54 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923] [2024-01-19T04:24:54.533Z] 2024-01-19 04:24:54 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923] [2024-01-19T04:24:54.533Z] 2024-01-19 04:24:54 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923] [2024-01-19T04:24:54.534Z] 2024-01-19 04:24:54 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:24:54.534Z] 2024-01-19 04:24:54 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:24:54.534Z] 2024-01-19 04:24:54 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:24:54.534Z] 2024-01-19 04:24:54 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:24:54.534Z] 2024-01-19 04:24:54 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:24:54.534Z] 2024-01-19 04:24:54 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923] [2024-01-19T04:24:54.534Z] 2024-01-19 04:24:54 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] in backup [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923] [2024-01-19T04:24:54.534Z] 2024-01-19 04:24:54 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] [2024-01-19T04:24:54.534Z] 2024-01-19 04:24:54 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:24:54.534Z] 2024-01-19 04:24:54 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:24:54.534Z] 2024-01-19 04:24:54 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:24:54.534Z] 2024-01-19 04:24:54 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:24:54.534Z] 2024-01-19 04:24:54 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating CR cleanup [2024-01-19T04:24:54.790Z] 2024-01-19 04:24:54 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T04:24:54.790Z] INFO[2024-01-19 04:24:54] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T04:24:54.790Z] INFO[2024-01-19 04:24:54] sched stats: tick interval: 46254 < 1.2s (100%) [2024-01-19T04:24:54.790Z] INFO[2024-01-19 04:24:54] sched stats: tick process durations: 46253 < 500ms (100%) [2024-01-19T04:24:54.790Z] 2024-01-19 04:24:54 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:54.791Z] 2024-01-19 04:24:54 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:54.791Z] 2024/01/19 04:24:54 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:54.791Z] 2024-01-19 04:24:54 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:54.791Z] 2024-01-19 04:24:54 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:54.791Z] 2024/01/19 04:24:54 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:54.791Z] 2024-01-19 04:24:54 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:54.792Z] 2024-01-19 04:24:54 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:54.792Z] 2024/01/19 04:24:54 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:55.353Z] 2024-01-19 04:24:55 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:55.354Z] 2024-01-19 04:24:55 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:55.354Z] 2024/01/19 04:24:55 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:24:55.354Z] 2024-01-19 04:24:55 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:24:55.354Z] 2024-01-19 04:24:55 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:24:55.354Z] 2024/01/19 04:24:55 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:25:01.020Z] 2024-01-19 04:24:59 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:25:01.020Z] 2024-01-19 04:24:59 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:25:01.020Z] 2024/01/19 04:24:59 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:25:01.020Z] 2024-01-19 04:24:59 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:25:01.021Z] INFO[2024-01-19 04:24:59] Verifying : Description : Verifying creation of backup [tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913] [2024-01-19T04:25:01.021Z] INFO[2024-01-19 04:24:59] Actual:true, Expected: true [2024-01-19T04:25:01.021Z] 2024-01-19 04:24:59 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:25:01.021Z] 2024-01-19 04:24:59 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:25:01.021Z] 2024/01/19 04:24:59 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:25:01.021Z] INFO[2024-01-19 04:24:59] Verifying : Description : Suspending Backup Schedule [tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo] for user [testuser1-1705637509] [2024-01-19T04:25:01.021Z] INFO[2024-01-19 04:24:59] Actual:true, Expected: true [2024-01-19T04:25:01.021Z] 2024-01-19 04:25:00 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:25:01.021Z] 2024-01-19 04:25:00 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:25:01.021Z] 2024/01/19 04:25:00 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:25:01.022Z] 2024-01-19 04:25:00 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:25:01.022Z] 2024-01-19 04:25:00 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:25:01.022Z] 2024/01/19 04:25:00 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:25:05.191Z] 2024-01-19 04:25:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:25:05.191Z] 2024-01-19 04:25:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:25:05.191Z] 2024/01/19 04:25:04 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:25:05.191Z] 2024-01-19 04:25:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:25:05.191Z] 2024-01-19 04:25:04 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:25:05.191Z] 2024/01/19 04:25:04 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:25:05.447Z] 2024-01-19 04:25:05 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:25:05.447Z] 2024-01-19 04:25:05 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:25:05.447Z] 2024/01/19 04:25:05 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:25:05.447Z] 2024-01-19 04:25:05 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:25:05.448Z] INFO[2024-01-19 04:25:05] Verifying : Description : Verifying creation of backup [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833] [2024-01-19T04:25:05.448Z] INFO[2024-01-19 04:25:05] Actual:true, Expected: true [2024-01-19T04:25:05.703Z] INFO[2024-01-19 04:25:05] Verifying : Description : Suspending Backup Schedule [tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn] for user [testuser1-1705637494] [2024-01-19T04:25:05.703Z] INFO[2024-01-19 04:25:05] Actual:true, Expected: true [2024-01-19T04:25:10.949Z] 2024-01-19 04:25:09 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:25:10.949Z] 2024-01-19 04:25:09 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:25:10.949Z] 2024/01/19 04:25:09 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:25:10.949Z] 2024-01-19 04:25:09 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:25:10.949Z] 2024-01-19 04:25:09 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:25:10.950Z] 2024/01/19 04:25:09 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:25:10.950Z] 2024-01-19 04:25:10 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:25:10.950Z] INFO[2024-01-19 04:25:10] Verifying : Description : Verifying creation of backup [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843] [2024-01-19T04:25:10.950Z] INFO[2024-01-19 04:25:10] Actual:true, Expected: true [2024-01-19T04:25:10.950Z] INFO[2024-01-19 04:25:10] Verifying : Description : Suspending Backup Schedule [tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg] for user [testuser1-1705637497] [2024-01-19T04:25:10.950Z] INFO[2024-01-19 04:25:10] Actual:true, Expected: true [2024-01-19T04:25:15.116Z] 2024-01-19 04:25:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:25:15.116Z] 2024-01-19 04:25:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:25:15.116Z] 2024/01/19 04:25:14 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:25:15.116Z] 2024-01-19 04:25:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:25:15.116Z] 2024-01-19 04:25:14 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:25:15.117Z] 2024/01/19 04:25:14 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:25:20.361Z] 2024-01-19 04:25:20 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:25:20.361Z] 2024-01-19 04:25:20 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:25:20.361Z] 2024/01/19 04:25:20 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:25:20.361Z] 2024-01-19 04:25:20 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:25:20.361Z] 2024-01-19 04:25:20 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:25:20.361Z] 2024/01/19 04:25:20 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:25:25.610Z] 2024-01-19 04:25:25 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:25:25.610Z] 2024-01-19 04:25:25 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:25:25.610Z] 2024/01/19 04:25:25 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:25:25.610Z] 2024-01-19 04:25:25 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:25:25.610Z] 2024-01-19 04:25:25 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:25:25.610Z] 2024/01/19 04:25:25 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:25:30.854Z] 2024-01-19 04:25:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:25:30.854Z] 2024-01-19 04:25:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:25:30.854Z] 2024/01/19 04:25:30 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:25:30.855Z] 2024-01-19 04:25:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:25:30.855Z] 2024-01-19 04:25:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:25:30.855Z] 2024/01/19 04:25:30 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:25:36.096Z] 2024-01-19 04:25:35 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:25:36.096Z] 2024-01-19 04:25:35 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] under [kube-system] namespace [2024-01-19T04:25:36.096Z] 2024/01/19 04:25:35 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:25:36.096Z] 2024-01-19 04:25:35 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923]] [2024-01-19T04:25:36.096Z] INFO[2024-01-19 04:25:35] Verifying : Description : Verifying creation of backup [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903] [2024-01-19T04:25:36.096Z] INFO[2024-01-19 04:25:35] Actual:true, Expected: true [2024-01-19T04:25:36.096Z] INFO[2024-01-19 04:25:35] Verifying : Description : Suspending Backup Schedule [tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn] for user [testuser1-1705637505] [2024-01-19T04:25:36.096Z] INFO[2024-01-19 04:25:35] Actual:true, Expected: true [2024-01-19T04:25:41.344Z] 2024-01-19 04:25:40 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133]] [2024-01-19T04:25:41.344Z] INFO[2024-01-19 04:25:40] Verifying : Description : Verifying creation of backup [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923] [2024-01-19T04:25:41.344Z] INFO[2024-01-19 04:25:40] Actual:true, Expected: true [2024-01-19T04:25:41.344Z] INFO[2024-01-19 04:25:40] Verifying : Description : Suspending Backup Schedule [tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf] for user [testuser1-1705637513] [2024-01-19T04:25:41.344Z] INFO[2024-01-19 04:25:40] Actual:true, Expected: true [2024-01-19T04:25:41.344Z] 2024-01-19 04:25:40 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Taking namespace label schedule backup of applications with and without rules from non-admin user [2024-01-19T04:25:51.281Z] 2024-01-19 04:25:50 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Taking namespace label schedule backup of applications of user : testuser1-1705637494 [2024-01-19T04:25:51.281Z] 2024-01-19 04:25:50 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating a backup of namespaces [[mysql-ibm-pxb-0-87560-01-19-04h09m51s mysql-ibm-pxb-1-87560-01-19-04h09m51s]] with pre and post exec rules [2024-01-19T04:25:51.282Z] 2024-01-19 04:25:50 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T04:25:51.282Z] 2024-01-19 04:25:50 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Volume snapshot classes found in the cluster - [2024-01-19T04:25:51.282Z] 2024-01-19 04:25:50 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] ibmc-vpcblock-snapshot [2024-01-19T04:25:51.282Z] 2024-01-19 04:25:50 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T04:26:01.223Z] 2024-01-19 04:26:00 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Taking namespace label schedule backup of applications of user : testuser1-1705637497 [2024-01-19T04:26:01.223Z] 2024-01-19 04:26:00 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating a backup of namespaces [[mysql-ibm-pxb-0-87560-01-19-04h09m51s mysql-ibm-pxb-1-87560-01-19-04h09m51s]] with pre and post exec rules [2024-01-19T04:26:01.223Z] 2024-01-19 04:26:00 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T04:26:01.223Z] 2024-01-19 04:26:00 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Volume snapshot classes found in the cluster - [2024-01-19T04:26:01.223Z] 2024-01-19 04:26:00 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] ibmc-vpcblock-snapshot [2024-01-19T04:26:01.223Z] 2024-01-19 04:26:00 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T04:26:11.211Z] 2024-01-19 04:26:10 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Taking namespace label schedule backup of applications of user : testuser1-1705637501 [2024-01-19T04:26:11.211Z] 2024-01-19 04:26:10 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating a backup of namespaces [[mysql-ibm-pxb-0-87560-01-19-04h09m51s mysql-ibm-pxb-1-87560-01-19-04h09m51s]] with pre and post exec rules [2024-01-19T04:26:11.211Z] 2024-01-19 04:26:10 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T04:26:11.211Z] 2024-01-19 04:26:10 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Volume snapshot classes found in the cluster - [2024-01-19T04:26:11.211Z] 2024-01-19 04:26:10 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] ibmc-vpcblock-snapshot [2024-01-19T04:26:11.211Z] 2024-01-19 04:26:10 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T04:26:21.158Z] 2024-01-19 04:26:20 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Taking namespace label schedule backup of applications of user : testuser1-1705637505 [2024-01-19T04:26:21.158Z] 2024-01-19 04:26:20 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating a backup of namespaces [[mysql-ibm-pxb-0-87560-01-19-04h09m51s mysql-ibm-pxb-1-87560-01-19-04h09m51s]] with pre and post exec rules [2024-01-19T04:26:21.158Z] 2024-01-19 04:26:20 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T04:26:21.158Z] 2024-01-19 04:26:20 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Volume snapshot classes found in the cluster - [2024-01-19T04:26:21.158Z] 2024-01-19 04:26:20 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] ibmc-vpcblock-snapshot [2024-01-19T04:26:21.158Z] 2024-01-19 04:26:20 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T04:26:31.153Z] 2024-01-19 04:26:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Taking namespace label schedule backup of applications of user : testuser1-1705637509 [2024-01-19T04:26:31.153Z] 2024-01-19 04:26:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating a backup of namespaces [[mysql-ibm-pxb-0-87560-01-19-04h09m51s mysql-ibm-pxb-1-87560-01-19-04h09m51s]] with pre and post exec rules [2024-01-19T04:26:31.153Z] 2024-01-19 04:26:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T04:26:31.153Z] 2024-01-19 04:26:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Volume snapshot classes found in the cluster - [2024-01-19T04:26:31.153Z] 2024-01-19 04:26:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] ibmc-vpcblock-snapshot [2024-01-19T04:26:31.153Z] 2024-01-19 04:26:30 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T04:26:41.107Z] 2024-01-19 04:26:40 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Taking namespace label schedule backup of applications of user : testuser1-1705637513 [2024-01-19T04:26:41.107Z] 2024-01-19 04:26:40 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Creating a backup of namespaces [[mysql-ibm-pxb-0-87560-01-19-04h09m51s mysql-ibm-pxb-1-87560-01-19-04h09m51s]] with pre and post exec rules [2024-01-19T04:26:41.107Z] 2024-01-19 04:26:40 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T04:26:41.107Z] 2024-01-19 04:26:40 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Volume snapshot classes found in the cluster - [2024-01-19T04:26:41.107Z] 2024-01-19 04:26:40 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] ibmc-vpcblock-snapshot [2024-01-19T04:26:41.107Z] 2024-01-19 04:26:40 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T04:26:53.320Z] 2024-01-19 04:26:51 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] first schedule backup for schedule name [tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs] is [tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-042551] [2024-01-19T04:26:53.320Z] 2024/01/19 04:26:51 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-042551] expected was [[Success]] but got [InProgress] because of [Pre-Exec rules are being executed]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:27:03.292Z] 2024-01-19 04:27:02 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] first schedule backup for schedule name [tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm] is [tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602] [2024-01-19T04:27:03.292Z] 2024/01/19 04:27:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602] expected was [[Success]] but got [InProgress] because of [Pre-Exec rules are being executed]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:27:13.233Z] 2024-01-19 04:27:12 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] first schedule backup for schedule name [tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv] is [tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-042612] [2024-01-19T04:27:13.233Z] INFO[2024-01-19 04:27:12] Verifying : Description : Verifying creation of backup [tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-042612] [2024-01-19T04:27:13.233Z] ERRO[2024-01-19 04:27:12] backup status for [tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-042612] expected was [[Success]] but got [Failed] because of [Error running PreExecRule: error executing PreExecRule for namespace mysql-ibm-pxb-0-87560-01-19-04h09m51s: Pod: [kube-system] pod-cmd-executor-39b19f71-9237-4393-aa55-8044beff691f failed. cmd executor failed because: failed to run command in pod: [mysql-ibm-pxb-0-87560-01-19-04h09m51s] mysql-7d766cd65c-blxvz due to: failed to create wait script in pod: [mysql-ibm-pxb-0-87560-01-19-04h09m51s] mysql-7d766cd65c-blxvz using command: rm -rf /tmp/stork-cmd-done-39b19f71-9237-4393-aa55-8044beff691f /tmp/killme-39b19f71-9237-4393-aa55-8044beff691f && echo 'touch /tmp/stork-cmd-done-39b19f71-9237-4393-aa55-8044beff691f && while [ ! -f /tmp/killme-39b19f71-9237-4393-aa55-8044beff691f ]; do sleep 2; done' > /tmp/wait-39b19f71-9237-4393-aa55-8044beff691f.sh && chmod +x /tmp/wait-39b19f71-9237-4393-aa55-8044beff691f.sh due to err: could not execute: error dialing backend: EOF: ] [2024-01-19T04:27:23.179Z] INFO[2024-01-19 04:27:21] Verifying : Description : Verifying creation of backup [tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-042551] [2024-01-19T04:27:23.179Z] ERRO[2024-01-19 04:27:21] backup status for [tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-042551] expected was [[Success]] but got [Failed] because of [Error running PreExecRule: error executing PreExecRule for namespace mysql-ibm-pxb-1-87560-01-19-04h09m51s: Pod: [kube-system] pod-cmd-executor-a41ea5f9-7284-472e-86c9-0b19a526e360 failed. cmd executor failed because: failed to run command in pod: [mysql-ibm-pxb-1-87560-01-19-04h09m51s] mysql-7d766cd65c-rjk25 due to: failed to create wait script in pod: [mysql-ibm-pxb-1-87560-01-19-04h09m51s] mysql-7d766cd65c-rjk25 using command: rm -rf /tmp/stork-cmd-done-a41ea5f9-7284-472e-86c9-0b19a526e360 /tmp/killme-a41ea5f9-7284-472e-86c9-0b19a526e360 && echo 'touch /tmp/stork-cmd-done-a41ea5f9-7284-472e-86c9-0b19a526e360 && while [ ! -f /tmp/killme-a41ea5f9-7284-472e-86c9-0b19a526e360 ]; do sleep 2; done' > /tmp/wait-a41ea5f9-7284-472e-86c9-0b19a526e360.sh && chmod +x /tmp/wait-a41ea5f9-7284-472e-86c9-0b19a526e360.sh due to err: could not execute: error dialing backend: EOF: ] [2024-01-19T04:27:23.179Z] 2024-01-19 04:27:21 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] first schedule backup for schedule name [tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc] is [tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-042621] [2024-01-19T04:27:23.179Z] INFO[2024-01-19 04:27:21] Verifying : Description : Verifying creation of backup [tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-042621] [2024-01-19T04:27:23.179Z] ERRO[2024-01-19 04:27:21] backup status for [tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-042621] expected was [[Success]] but got [Failed] because of [Error running PreExecRule: error executing PreExecRule for namespace mysql-ibm-pxb-0-87560-01-19-04h09m51s: Pod: [kube-system] pod-cmd-executor-7c09842f-64de-43e4-a022-025c94ce5754 failed. cmd executor failed because: failed to run command: mysql --user=**** --password=$MYSQL_ROOT_PASSWORD -Bse 'FLUSH TABLES WITH READ LOCK;system /tmp/wait-7c09842f-64de-43e4-a022-025c94ce5754.sh;' in pod: [mysql-ibm-pxb-0-87560-01-19-04h09m51s] mysql-7d766cd65c-blxvz due to err: could not execute: error dialing backend: EOF: ] [2024-01-19T04:27:33.118Z] 2024-01-19 04:27:31 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] first schedule backup for schedule name [tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj] is [tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631] [2024-01-19T04:27:33.118Z] 2024/01/19 04:27:31 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631] expected was [[Success]] but got [InProgress] because of [Pre-Exec rules are being executed]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:27:33.119Z] 2024/01/19 04:27:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:27:43.058Z] 2024-01-19 04:27:41 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] first schedule backup for schedule name [tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck] is [tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641] [2024-01-19T04:27:43.058Z] 2024/01/19 04:27:41 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641] expected was [[Success]] but got [InProgress] because of [Pre-Exec rules are being executed]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:28:04.949Z] 2024/01/19 04:28:01 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:28:04.949Z] 2024/01/19 04:28:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:28:13.037Z] 2024/01/19 04:28:11 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:28:34.924Z] 2024/01/19 04:28:31 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:28:34.924Z] 2024/01/19 04:28:32 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:28:43.008Z] 2024/01/19 04:28:41 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:29:04.894Z] 2024/01/19 04:29:01 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:29:04.894Z] 2024/01/19 04:29:02 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:29:13.044Z] 2024/01/19 04:29:11 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:29:34.936Z] 2024/01/19 04:29:31 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:29:34.936Z] 2024-01-19 04:29:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backup [tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602] in org [default] [2024-01-19T04:29:34.936Z] 2024-01-19 04:29:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602] [2024-01-19T04:29:34.936Z] 2024-01-19 04:29:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602] [2024-01-19T04:29:34.936Z] 2024-01-19 04:29:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602] [2024-01-19T04:29:34.936Z] 2024-01-19 04:29:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:29:34.936Z] 2024-01-19 04:29:32 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:29:34.936Z] 2024-01-19 04:29:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:29:34.936Z] 2024-01-19 04:29:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:29:34.936Z] 2024-01-19 04:29:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:29:34.936Z] 2024-01-19 04:29:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] in backup [tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602] [2024-01-19T04:29:34.936Z] 2024-01-19 04:29:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] in backup [tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602] [2024-01-19T04:29:34.936Z] 2024-01-19 04:29:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] [2024-01-19T04:29:34.936Z] 2024-01-19 04:29:32 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:29:34.936Z] 2024-01-19 04:29:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:29:34.936Z] 2024-01-19 04:29:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:29:34.936Z] 2024-01-19 04:29:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:29:34.936Z] 2024-01-19 04:29:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating CR cleanup [2024-01-19T04:29:34.937Z] 2024-01-19 04:29:33 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T04:29:34.937Z] 2024-01-19 04:29:33 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602 tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] [2024-01-19T04:29:34.937Z] 2024-01-19 04:29:33 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602 tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] under [kube-system] namespace [2024-01-19T04:29:34.937Z] 2024/01/19 04:29:33 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:29:38.205Z] 2024-01-19 04:29:38 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602 tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] [2024-01-19T04:29:38.205Z] 2024-01-19 04:29:38 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602 tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] under [kube-system] namespace [2024-01-19T04:29:38.205Z] 2024/01/19 04:29:38 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:29:42.370Z] 2024/01/19 04:29:41 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:29:43.298Z] 2024-01-19 04:29:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602 tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] [2024-01-19T04:29:43.298Z] 2024-01-19 04:29:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602 tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] under [kube-system] namespace [2024-01-19T04:29:43.298Z] 2024/01/19 04:29:43 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:29:48.544Z] 2024-01-19 04:29:48 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602 tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] [2024-01-19T04:29:48.545Z] 2024-01-19 04:29:48 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602 tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] under [kube-system] namespace [2024-01-19T04:29:48.545Z] 2024/01/19 04:29:48 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:29:53.799Z] 2024-01-19 04:29:53 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] [2024-01-19T04:29:53.799Z] INFO[2024-01-19 04:29:53] Verifying : Description : Verifying creation of backup [tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602] [2024-01-19T04:29:53.799Z] INFO[2024-01-19 04:29:53] Actual:true, Expected: true [2024-01-19T04:29:53.800Z] INFO[2024-01-19 04:29:53] Verifying : Description : Suspending Backup Schedule [tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm] for user [testuser1-1705637497] [2024-01-19T04:29:53.800Z] INFO[2024-01-19 04:29:53] Actual:true, Expected: true [2024-01-19T04:30:01.885Z] 2024-01-19 04:30:01 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backup [tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631] in org [default] [2024-01-19T04:30:01.885Z] 2024-01-19 04:30:01 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631] [2024-01-19T04:30:01.885Z] 2024-01-19 04:30:01 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631] [2024-01-19T04:30:01.885Z] 2024-01-19 04:30:01 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631] [2024-01-19T04:30:01.885Z] 2024-01-19 04:30:01 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:30:01.885Z] 2024-01-19 04:30:01 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:30:01.885Z] 2024-01-19 04:30:01 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:30:01.885Z] 2024-01-19 04:30:01 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:30:01.885Z] 2024-01-19 04:30:01 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:30:01.885Z] 2024-01-19 04:30:01 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] in backup [tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631] [2024-01-19T04:30:01.885Z] 2024-01-19 04:30:01 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] in backup [tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631] [2024-01-19T04:30:01.885Z] 2024-01-19 04:30:01 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] [2024-01-19T04:30:01.885Z] 2024-01-19 04:30:01 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:30:01.885Z] 2024-01-19 04:30:01 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:30:01.885Z] 2024-01-19 04:30:01 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:30:01.886Z] 2024-01-19 04:30:01 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:30:01.886Z] 2024-01-19 04:30:01 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating CR cleanup [2024-01-19T04:30:01.886Z] 2024-01-19 04:30:01 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T04:30:02.142Z] 2024-01-19 04:30:01 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] [2024-01-19T04:30:02.142Z] 2024-01-19 04:30:01 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] under [kube-system] namespace [2024-01-19T04:30:02.142Z] 2024/01/19 04:30:01 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:30:07.390Z] 2024-01-19 04:30:06 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] [2024-01-19T04:30:07.390Z] 2024-01-19 04:30:06 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] under [kube-system] namespace [2024-01-19T04:30:07.390Z] 2024/01/19 04:30:06 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:30:12.634Z] 2024-01-19 04:30:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backup [tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641] in org [default] [2024-01-19T04:30:12.634Z] 2024-01-19 04:30:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641] [2024-01-19T04:30:12.634Z] 2024-01-19 04:30:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641] [2024-01-19T04:30:12.634Z] 2024-01-19 04:30:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in backup [tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641] [2024-01-19T04:30:12.634Z] 2024-01-19 04:30:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:30:12.634Z] 2024-01-19 04:30:11 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:30:12.634Z] 2024-01-19 04:30:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:30:12.634Z] 2024-01-19 04:30:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:30:12.634Z] 2024-01-19 04:30:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:30:12.634Z] 2024-01-19 04:30:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] in backup [tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641] [2024-01-19T04:30:12.634Z] 2024-01-19 04:30:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] in backup [tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641] [2024-01-19T04:30:12.634Z] 2024-01-19 04:30:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-87560-01-19-04h09m51s] [2024-01-19T04:30:12.634Z] 2024-01-19 04:30:11 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:30:12.634Z] 2024-01-19 04:30:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:30:12.635Z] 2024-01-19 04:30:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:30:12.635Z] 2024-01-19 04:30:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T04:30:12.635Z] 2024-01-19 04:30:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating CR cleanup [2024-01-19T04:30:12.635Z] 2024-01-19 04:30:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T04:30:12.635Z] 2024-01-19 04:30:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] [2024-01-19T04:30:12.635Z] 2024-01-19 04:30:11 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] under [kube-system] namespace [2024-01-19T04:30:12.635Z] 2024/01/19 04:30:11 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:30:12.635Z] 2024-01-19 04:30:12 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] [2024-01-19T04:30:12.636Z] 2024-01-19 04:30:12 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] under [kube-system] namespace [2024-01-19T04:30:12.636Z] 2024/01/19 04:30:12 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:30:17.880Z] 2024-01-19 04:30:16 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] [2024-01-19T04:30:17.880Z] 2024-01-19 04:30:16 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] under [kube-system] namespace [2024-01-19T04:30:17.881Z] 2024/01/19 04:30:16 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:30:17.881Z] 2024-01-19 04:30:17 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] [2024-01-19T04:30:17.881Z] 2024-01-19 04:30:17 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] under [kube-system] namespace [2024-01-19T04:30:17.881Z] 2024/01/19 04:30:17 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:30:22.048Z] 2024-01-19 04:30:22 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] [2024-01-19T04:30:22.049Z] 2024-01-19 04:30:22 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] under [kube-system] namespace [2024-01-19T04:30:22.049Z] 2024/01/19 04:30:22 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:30:22.305Z] 2024-01-19 04:30:22 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] [2024-01-19T04:30:22.305Z] 2024-01-19 04:30:22 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] under [kube-system] namespace [2024-01-19T04:30:22.305Z] 2024/01/19 04:30:22 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:30:27.553Z] 2024-01-19 04:30:27 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] [2024-01-19T04:30:27.554Z] 2024-01-19 04:30:27 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] under [kube-system] namespace [2024-01-19T04:30:27.554Z] 2024/01/19 04:30:27 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]}, Next try in [5s], timeout [5m0s] [2024-01-19T04:30:27.554Z] 2024-01-19 04:30:27 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641]] [2024-01-19T04:30:27.554Z] INFO[2024-01-19 04:30:27] Verifying : Description : Verifying creation of backup [tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631] [2024-01-19T04:30:27.554Z] INFO[2024-01-19 04:30:27] Actual:true, Expected: true [2024-01-19T04:30:27.554Z] INFO[2024-01-19 04:30:27] Verifying : Description : Suspending Backup Schedule [tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj] for user [testuser1-1705637509] [2024-01-19T04:30:27.554Z] INFO[2024-01-19 04:30:27] Actual:true, Expected: true [2024-01-19T04:30:32.808Z] 2024-01-19 04:30:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133]] [2024-01-19T04:30:32.808Z] INFO[2024-01-19 04:30:32] Verifying : Description : Verifying creation of backup [tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641] [2024-01-19T04:30:32.808Z] INFO[2024-01-19 04:30:32] Actual:true, Expected: true [2024-01-19T04:30:32.808Z] INFO[2024-01-19 04:30:32] Verifying : Description : Suspending Backup Schedule [tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck] for user [testuser1-1705637513] [2024-01-19T04:30:32.808Z] INFO[2024-01-19 04:30:32] Actual:true, Expected: true [2024-01-19T04:30:32.808Z] 2024-01-19 04:30:32 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Restore single namespace backups with different configs [2024-01-19T04:30:42.751Z] INFO[2024-01-19 04:30:42] Verifying : Description : Fetching px-central-admin ctx [2024-01-19T04:30:42.751Z] INFO[2024-01-19 04:30:42] Actual:true, Expected: true [2024-01-19T04:30:42.752Z] 2024-01-19 04:30:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Restoring single namespace backup [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl-7c3a711-interval-2024-01-19-041252] in cluster [destination-cluster] with restore [testuser1-1705637494-test-restore-single-ns-pbdt] and namespace mapping map[] for user [testuser1-1705637494] [2024-01-19T04:30:43.009Z] 2024/01/19 04:30:42 DoRetryWithTimeout - Error: {restore status for [testuser1-1705637494-test-restore-single-ns-pbdt] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T04:31:15.031Z] 2024/01/19 04:31:12 DoRetryWithTimeout - Error: {restore status for [testuser1-1705637494-test-restore-single-ns-pbdt] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T04:31:47.056Z] 2024-01-19 04:31:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.CreateRestore:#936] - Restore [testuser1-1705637494-test-restore-single-ns-pbdt] created successfully [2024-01-19T04:31:47.056Z] 2024-01-19 04:31:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Switching cluster context to cluster [destination-cluster] [2024-01-19T04:31:47.056Z] 2024-01-19 04:31:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T04:31:47.056Z] 2024-01-19 04:31:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-87560-01-19-04h09m51s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-19T04:31:47.056Z] 2024-01-19 04:31:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:31:47.056Z] 2024-01-19 04:31:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T04:31:47.056Z] 2024-01-19 04:31:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T04:31:47.056Z] 2024-01-19 04:31:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating restore [testuser1-1705637494-test-restore-single-ns-pbdt] in org [default] [2024-01-19T04:31:47.056Z] 2024-01-19 04:31:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [testuser1-1705637494-test-restore-single-ns-pbdt] [2024-01-19T04:31:47.056Z] 2024-01-19 04:31:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateRestore:#2110] - Checking status of restore [testuser1-1705637494-test-restore-single-ns-pbdt] [2024-01-19T04:31:47.056Z] 2024-01-19 04:31:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in restore [testuser1-1705637494-test-restore-single-ns-pbdt] [2024-01-19T04:31:47.056Z] 2024-01-19 04:31:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-87560-01-19-04h09m51s]) was found in restore [testuser1-1705637494-test-restore-single-ns-pbdt], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:31:47.057Z] 2024-01-19 04:31:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-87560-01-19-04h09m51s]) found in the restore [testuser1-1705637494-test-restore-single-ns-pbdt] was also present on the cluster/namespace [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:31:47.057Z] 2024-01-19 04:31:42 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-87560-01-19-04h09m51s]) was found in restore [testuser1-1705637494-test-restore-single-ns-pbdt], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:31:47.057Z] 2024-01-19 04:31:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-87560-01-19-04h09m51s]) found in the restore [testuser1-1705637494-test-restore-single-ns-pbdt] was also present on the cluster/namespace [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:31:47.057Z] 2024-01-19 04:31:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-87560-01-19-04h09m51s]) was found in restore [testuser1-1705637494-test-restore-single-ns-pbdt], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:31:47.057Z] 2024-01-19 04:31:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-87560-01-19-04h09m51s]) found in the restore [testuser1-1705637494-test-restore-single-ns-pbdt] was also present on the cluster/namespace [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:31:47.057Z] 2024-01-19 04:31:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-87560-01-19-04h09m51s]) was found in restore [testuser1-1705637494-test-restore-single-ns-pbdt], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:31:47.057Z] 2024-01-19 04:31:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-87560-01-19-04h09m51s]) found in the restore [testuser1-1705637494-test-restore-single-ns-pbdt] was also present on the cluster/namespace [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:31:47.057Z] 2024-01-19 04:31:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in restore [testuser1-1705637494-test-restore-single-ns-pbdt] [2024-01-19T04:31:47.057Z] 2024-01-19 04:31:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-87560-01-19-04h09m51s] in restore [testuser1-1705637494-test-restore-single-ns-pbdt] [2024-01-19T04:31:47.057Z] 2024-01-19 04:31:43 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:31:47.057Z] 2024-01-19 04:31:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:31:47.057Z] 2024-01-19 04:31:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87560-01-19-04h09m51s] are [[mysql-data]] [2024-01-19T04:31:47.058Z] 2024-01-19 04:31:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.ValidateRestore:#2240] - in restore [testuser1-1705637494-test-restore-single-ns-pbdt], the status of the restored volume [pvc-34adb639-4415-4c14-b4a5-c764edb229e5] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-19T04:31:47.058Z] 2024-01-19 04:31:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validate applications in restored namespace [mysql-ibm-pxb-0-87560-01-19-04h09m51s] due to restore [testuser1-1705637494-test-restore-single-ns-pbdt] [2024-01-19T04:31:47.058Z] 2024-01-19 04:31:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Validating mysql-ibm app [2024-01-19T04:31:47.058Z] 2024-01-19 04:31:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] wait for mysql-ibm app to start running [2024-01-19T04:31:47.058Z] 2024-01-19 04:31:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T04:31:47.058Z] 2024-01-19 04:31:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T04:31:47.058Z] 2024-01-19 04:31:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T04:31:47.058Z] 2024-01-19 04:31:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Switching cluster context back to cluster path [/tmp/destination-config] [2024-01-19T04:31:47.058Z] 2024-01-19 04:31:43 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T04:31:47.058Z] INFO[2024-01-19 04:31:43] Verifying : Description : Verifying restoration [testuser1-1705637494-test-restore-single-ns-pbdt] of single namespace backup [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl-7c3a711-interval-2024-01-19-041252] in cluster [destination-cluster] by user [testuser1-1705637494] [2024-01-19T04:31:47.058Z] ERRO[2024-01-19 04:31:43] ValidateRestore Errors: {in restore [testuser1-1705637494-test-restore-single-ns-pbdt], said restored volume [pvc-34adb639-4415-4c14-b4a5-c764edb229e5] cannot be found in the actual cluster [mysql-ibm-pxb-0-87560-01-19-04h09m51s]} [2024-01-19T04:31:47.058Z] 2024-01-19 04:31:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Deleting the deployed apps after the testcase [2024-01-19T04:31:47.058Z] 2024-01-19 04:31:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Destroying apps [2024-01-19T04:31:47.058Z] 2024-01-19 04:31:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T04:31:47.058Z] 2024-01-19 04:31:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T04:31:47.058Z] 2024-01-19 04:31:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] start destroying mysql-ibm app [2024-01-19T04:31:47.058Z] 2024-01-19 04:31:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T04:31:47.058Z] 2024-01-19 04:31:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T04:31:47.058Z] 2024-01-19 04:31:44 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T04:31:47.058Z] INFO[2024-01-19 04:31:44] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T04:31:47.058Z] INFO[2024-01-19 04:31:44] Actual:true, Expected: true [2024-01-19T04:31:47.058Z] 2024-01-19 04:31:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T04:31:47.058Z] 2024-01-19 04:31:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T04:31:47.058Z] 2024-01-19 04:31:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] start destroying mysql-ibm app [2024-01-19T04:31:47.058Z] 2024-01-19 04:31:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T04:31:47.058Z] 2024-01-19 04:31:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T04:31:47.059Z] 2024-01-19 04:31:44 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T04:31:47.059Z] INFO[2024-01-19 04:31:44] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T04:31:47.059Z] INFO[2024-01-19 04:31:44] Actual:true, Expected: true [2024-01-19T04:31:47.059Z] 2024-01-19 04:31:44 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.glob..func32.3:#3011] - Deleting backup schedule policy [2024-01-19T04:31:47.059Z] INFO[2024-01-19 04:31:46] Verifying : Description : Deleting backup schedule policy [periodic-upgbo] [2024-01-19T04:31:47.059Z] ERRO[2024-01-19 04:31:46] Failed to delete schedule policy periodic-upgbo with error [rpc error: code = Unknown desc = backup schedule [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl] has reference to schedule policy/UID [periodic-upgbo/fe8c5da4-117f-48db-85c1-3fdc72c2f155]] [2024-01-19T04:31:47.059Z] 2024-01-19 04:31:46 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [backup.glob..func32.3:#3014] - Deleting pre and post exec rules [2024-01-19T04:31:47.985Z] INFO[2024-01-19 04:31:47] Verifying : Description : Deleting pre exec rule pre-rule-jfikz [2024-01-19T04:31:47.985Z] ERRO[2024-01-19 04:31:47] Failed to delete rule: [rpc error: code = Unknown desc = backup schedule object [tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs] has reference to rule object [pre-rule-jfikz/54ac2577-61cb-40b9-b501-cfd8a9a9ada8]] [2024-01-19T04:31:49.351Z] INFO[2024-01-19 04:31:49] Verifying : Description : Deleting post exec rule post-rule-thgqc [2024-01-19T04:31:49.351Z] ERRO[2024-01-19 04:31:49] Failed to delete rule: [rpc error: code = Unknown desc = backup schedule object [tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs] has reference to rule object [post-rule-thgqc/3586714c-66f2-42ad-b9cc-0321c2caa97a]] [2024-01-19T04:31:49.351Z] 2024-01-19 04:31:49 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Cleaning backup locations in map [map[fb5a14e2-2342-4596-b12d-cfadc2b7ea70:autogenerated-backup-location-wkyfv]], cloud credential [autogenerated-cred-meoyc], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T04:31:49.351Z] INFO[2024-01-19 04:31:49] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-wkyfv] [2024-01-19T04:31:49.351Z] ERRO[2024-01-19 04:31:49] Actual:rpc error: code = Unknown desc = backup schedule object [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl] has reference to backuplocation [autogenerated-backup-location-wkyfv:fb5a14e2-2342-4596-b12d-cfadc2b7ea70], Expected:  [2024-01-19T04:31:49.351Z] INFO[2024-01-19 04:31:49] --------Test End------ [2024-01-19T04:31:49.351Z] INFO[2024-01-19 04:31:49] #Test: [2024-01-19T04:31:49.351Z] INFO[2024-01-19 04:31:49] #Description: [2024-01-19T04:31:49.351Z] INFO[2024-01-19 04:31:49] ------------------------ [2024-01-19T04:31:49.351Z] 2024-01-19 04:31:49 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.EndPxBackupTorpedoTest:#7261] - >>>> FAILED TEST: {DeleteBackupOfUserSharedRBAC} Delete backups by user from admin with shared RBAC objects [2024-01-19T04:31:49.608Z] 2024-01-19 04:31:49 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:31:49.608Z] 2024-01-19 04:31:49 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-1-87560-01-19-04h09m51s] [2024-01-19T04:31:49.864Z] 2024-01-19 04:31:49 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T04:31:49.864Z] 2024-01-19 04:31:49 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T04:31:50.120Z] 2024-01-19 04:31:49 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-87560-01-19-04h09m51s] [2024-01-19T04:31:50.121Z] 2024-01-19 04:31:49 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-1-87560-01-19-04h09m51s] [2024-01-19T04:31:50.380Z] 2024-01-19 04:31:50 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {DeleteBackupOfUserSharedRBAC} Delete backups by user from admin with shared RBAC objects [2024-01-19T04:31:50.380Z] 2024-01-19 04:31:50 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T04:31:50.381Z] 2024/01/19 04:31:50 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T04:31:55.640Z] 2024/01/19 04:31:55 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T04:32:00.883Z] 2024/01/19 04:32:00 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T04:32:06.128Z] 2024/01/19 04:32:05 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T04:32:10.298Z] 2024/01/19 04:32:10 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T04:32:15.543Z] 2024/01/19 04:32:15 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T04:32:20.791Z] 2024/01/19 04:32:20 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T04:32:26.037Z] 2024/01/19 04:32:25 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T04:32:31.278Z] 2024/01/19 04:32:30 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.64.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.8 10.240.64.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T04:32:31.278Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T04:32:31.278Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T04:32:31.278Z] {dial tcp 10.240.64.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T04:32:35.444Z] 2024-01-19 04:32:35 +0000:[WARNING] [{DeleteBackupOfUserSharedRBAC}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.64.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.8 10.240.64.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T04:32:35.444Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T04:32:35.444Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T04:32:35.444Z] {dial tcp 10.240.64.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T04:32:35.444Z] 2024-01-19 04:32:35 +0000:[ERROR] [{DeleteBackupOfUserSharedRBAC}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.64.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.8 10.240.64.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T04:32:35.444Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T04:32:35.444Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T04:32:35.444Z] {dial tcp 10.240.64.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T04:32:35.444Z] 2024-01-19 04:32:35 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T04:32:35.444Z] 2024-01-19 04:32:35 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] Switching context to [/tmp/source-config] [2024-01-19T04:32:35.444Z] 2024-01-19 04:32:35 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T04:32:35.707Z] 2024-01-19 04:32:35 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T04:32:35.707Z] 2024-01-19 04:32:35 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T04:32:35.707Z] 2024-01-19 04:32:35 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T04:32:35.707Z] 2024-01-19 04:32:35 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T04:32:35.707Z] 2024-01-19 04:32:35 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T04:32:36.268Z] 2024-01-19 04:32:36 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T04:32:36.526Z] 2024-01-19 04:32:36 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T04:32:37.089Z] 2024-01-19 04:32:36 +0000:[INFO] [{DeleteBackupOfUserSharedRBAC}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T04:32:37.089Z] 2024/01/19 04:32:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:32:49.255Z] 2024/01/19 04:32:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:32:57.355Z] 2024/01/19 04:32:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:33:07.300Z] 2024/01/19 04:33:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:33:17.238Z] 2024/01/19 04:33:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:33:29.413Z] 2024/01/19 04:33:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:33:37.500Z] 2024/01/19 04:33:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:33:47.436Z] 2024/01/19 04:33:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:33:57.374Z] 2024/01/19 04:33:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:34:09.541Z] 2024/01/19 04:34:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:34:17.630Z] 2024/01/19 04:34:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:34:27.627Z] 2024/01/19 04:34:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:34:37.564Z] 2024/01/19 04:34:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:34:49.729Z] 2024/01/19 04:34:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:34:56.266Z] INFO[2024-01-19 04:34:55] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T04:34:56.266Z] INFO[2024-01-19 04:34:55] sched stats: tick interval: 46855 < 1.2s (100%) [2024-01-19T04:34:56.266Z] INFO[2024-01-19 04:34:55] sched stats: tick process durations: 46854 < 500ms (100%) [2024-01-19T04:34:57.630Z] 2024/01/19 04:34:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:35:07.605Z] 2024/01/19 04:35:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:35:19.774Z] 2024/01/19 04:35:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:35:27.859Z] 2024/01/19 04:35:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:35:37.799Z] 2024/01/19 04:35:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:35:47.741Z] 2024/01/19 04:35:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:35:59.313Z] 2024/01/19 04:35:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:36:09.252Z] 2024/01/19 04:36:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:36:19.191Z] 2024/01/19 04:36:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:36:29.141Z] 2024/01/19 04:36:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:36:39.121Z] 2024/01/19 04:36:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:36:49.055Z] 2024/01/19 04:36:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:36:59.001Z] 2024/01/19 04:36:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:37:08.944Z] 2024/01/19 04:37:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:37:18.894Z] 2024/01/19 04:37:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:37:28.836Z] 2024/01/19 04:37:28 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:37:38.778Z] 2024-01-19 04:37:38 +0000:[ERROR] [{DeleteBackupOfUserSharedRBAC}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.778Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.778Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.778Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.778Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.778Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.778Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.778Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.778Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.778Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.779Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.779Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.779Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.779Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.779Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.779Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.779Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.779Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.779Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.779Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.779Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.779Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.779Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.779Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.779Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.779Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.779Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.779Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.779Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:37:38.779Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T04:37:38.779Z] [2024-01-19T04:37:38.779Z] ------------------------------ [2024-01-19T04:37:38.779Z] • Failure [1666.630 seconds] [2024-01-19T04:37:38.779Z] {DeleteBackupOfUserSharedRBAC} [2024-01-19T04:37:38.779Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_orphaned_test.go:2451 [2024-01-19T04:37:38.779Z] Delete backups by user from admin with shared RBAC objects [It] [2024-01-19T04:37:38.779Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_orphaned_test.go:2510 [2024-01-19T04:37:38.779Z] [2024-01-19T04:37:38.779Z] Unexpected error: [2024-01-19T04:37:38.779Z] <*errors.errorString | 0xc000fb90b0>: { [2024-01-19T04:37:38.780Z] s: "error: backup status for [tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-042612] expected was [[Success]] but got [Failed] because of [Error running PreExecRule: error executing PreExecRule for namespace mysql-ibm-pxb-0-87560-01-19-04h09m51s: Pod: [kube-system] pod-cmd-executor-39b19f71-9237-4393-aa55-8044beff691f failed. cmd executor failed because: failed to run command in pod: [mysql-ibm-pxb-0-87560-01-19-04h09m51s] mysql-7d766cd65c-blxvz due to: failed to create wait script in pod: [mysql-ibm-pxb-0-87560-01-19-04h09m51s] mysql-7d766cd65c-blxvz using command: rm -rf /tmp/stork-cmd-done-39b19f71-9237-4393-aa55-8044beff691f /tmp/killme-39b19f71-9237-4393-aa55-8044beff691f && echo 'touch /tmp/stork-cmd-done-39b19f71-9237-4393-aa55-8044beff691f && while [ ! -f /tmp/killme-39b19f71-9237-4393-aa55-8044beff691f ]; do sleep 2; done' > /tmp/wait-39b19f71-9237-4393-aa55-8044beff691f.sh && chmod +x /tmp/wait-39b19f71-9237-4393-aa55-8044beff691f.sh due to err: could not execute: error dialing backend: EOF: ], Description: Verifying creation of backup [tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-042612]", [2024-01-19T04:37:38.780Z] } [2024-01-19T04:37:38.780Z] error: backup status for [tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-042612] expected was [[Success]] but got [Failed] because of [Error running PreExecRule: error executing PreExecRule for namespace mysql-ibm-pxb-0-87560-01-19-04h09m51s: Pod: [kube-system] pod-cmd-executor-39b19f71-9237-4393-aa55-8044beff691f failed. cmd executor failed because: failed to run command in pod: [mysql-ibm-pxb-0-87560-01-19-04h09m51s] mysql-7d766cd65c-blxvz due to: failed to create wait script in pod: [mysql-ibm-pxb-0-87560-01-19-04h09m51s] mysql-7d766cd65c-blxvz using command: rm -rf /tmp/stork-cmd-done-39b19f71-9237-4393-aa55-8044beff691f /tmp/killme-39b19f71-9237-4393-aa55-8044beff691f && echo 'touch /tmp/stork-cmd-done-39b19f71-9237-4393-aa55-8044beff691f && while [ ! -f /tmp/killme-39b19f71-9237-4393-aa55-8044beff691f ]; do sleep 2; done' > /tmp/wait-39b19f71-9237-4393-aa55-8044beff691f.sh && chmod +x /tmp/wait-39b19f71-9237-4393-aa55-8044beff691f.sh due to err: could not execute: error dialing backend: EOF: ], Description: Verifying creation of backup [tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-042612] [2024-01-19T04:37:38.780Z] occurred [2024-01-19T04:37:38.780Z] [2024-01-19T04:37:38.780Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-19T04:37:38.780Z] [2024-01-19T04:37:38.780Z] Full Stack Trace [2024-01-19T04:37:38.780Z] github.com/portworx/torpedo/pkg/aetosutil.(*Dashboard).VerifyFatal(0x7f34c78?, {0x702af20?, 0xc000fb8fb0}, {0x0?, 0x0}, {0xc001e4c000, 0x81}) [2024-01-19T04:37:38.780Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 +0x143 [2024-01-19T04:37:38.780Z] github.com/portworx/torpedo/tests/backup.glob..func32.2.11.1({0xc002484648, 0x14}) [2024-01-19T04:37:38.780Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_orphaned_test.go:2705 +0x7ab [2024-01-19T04:37:38.780Z] created by github.com/portworx/torpedo/tests/backup.glob..func32.2.11 [2024-01-19T04:37:38.780Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_orphaned_test.go:2692 +0x3e5 [2024-01-19T04:37:38.780Z] ------------------------------ [2024-01-19T04:37:38.780Z] STEP: schedule applications [2024-01-19T04:37:38.780Z] STEP: schedule applications [2024-01-19T04:37:38.781Z] STEP: Validate applications [2024-01-19T04:37:38.781Z] STEP: validate applications [2024-01-19T04:37:38.781Z] STEP: validate mysql-ibm app's volumes [2024-01-19T04:37:38.781Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T04:37:38.781Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T04:37:38.781Z] STEP: get mysql-ibm app's volume: pvc-03378f3a-b457-4bd9-a6df-8a06e43b2a80 inspected by the volume driver [2024-01-19T04:37:38.781Z] STEP: wait for mysql-ibm app to start running [2024-01-19T04:37:38.781Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T04:37:38.781Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T04:37:38.781Z] STEP: Validate Px pod restart count [2024-01-19T04:37:38.781Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T04:37:38.781Z] STEP: validate mysql-ibm app's volumes [2024-01-19T04:37:38.781Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T04:37:38.781Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T04:37:38.781Z] STEP: get mysql-ibm app's volume: pvc-92a4a8b6-9bb3-4d44-8dab-f56b6602acd7 inspected by the volume driver [2024-01-19T04:37:38.781Z] STEP: wait for mysql-ibm app to start running [2024-01-19T04:37:38.781Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T04:37:38.781Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T04:37:38.782Z] STEP: Validate Px pod restart count [2024-01-19T04:37:38.782Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T04:37:38.782Z] STEP: Generate and add labels to namespaces [2024-01-19T04:37:38.782Z] STEP: Generating namespace label string from label map [2024-01-19T04:37:38.782Z] STEP: Create Users with Different types of roles [2024-01-19T04:37:38.782Z] STEP: Adding Credentials and Backup Location from px-admin user [2024-01-19T04:37:38.782Z] STEP: Create schedule policy from px-admin [2024-01-19T04:37:38.782Z] STEP: Create pre and post exec rules for applications from px-admin [2024-01-19T04:37:38.782Z] STEP: Update ownership for RBAC resource with non-admin users from px-admin [2024-01-19T04:37:38.782Z] STEP: Register source and destination cluster for backup on non-admin user [2024-01-19T04:37:38.782Z] STEP: Taking schedule backup of applications as non-admin user with and without rules [2024-01-19T04:37:38.782Z] STEP: Taking namespace label schedule backup of applications with and without rules from non-admin user [2024-01-19T04:37:38.782Z] STEP: Restore single namespace backups with different configs [2024-01-19T04:37:38.782Z] STEP: validate mysql-ibm app's volumes [2024-01-19T04:37:38.782Z] STEP: wait for mysql-ibm app to start running [2024-01-19T04:37:38.782Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T04:37:38.782Z] STEP: Validate Px pod restart count [2024-01-19T04:37:38.782Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T04:37:38.782Z] STEP: destroy apps [2024-01-19T04:37:38.782Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T04:37:38.782Z] STEP: start destroying mysql-ibm app [2024-01-19T04:37:38.782Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T04:37:38.782Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T04:37:38.782Z] STEP: start destroying mysql-ibm app [2024-01-19T04:37:38.782Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T04:37:38.782Z] 2024-01-19 04:37:38 +0000:[INFO] [{ScheduleBackupDeleteAndRecreateNS}] Log Dir: /testresults//ScheduleBackupDeleteAndRecreateNS.log [2024-01-19T04:37:38.782Z] INFO[2024-01-19 04:37:38] --------Test Start------ [2024-01-19T04:37:38.782Z] INFO[2024-01-19 04:37:38] #Test: ScheduleBackupDeleteAndRecreateNS [2024-01-19T04:37:38.782Z] INFO[2024-01-19 04:37:38] #Description: Verification of schedule backups when namespaces are deleted and recreated [2024-01-19T04:37:38.782Z] INFO[2024-01-19 04:37:38] ------------------------ [2024-01-19T04:37:38.782Z] 2024-01-19 04:37:38 +0000:[INFO] [{ScheduleBackupDeleteAndRecreateNS}] Scheduling applications [2024-01-19T04:37:38.782Z] 2024-01-19 04:37:38 +0000:[INFO] [{ScheduleBackupDeleteAndRecreateNS}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T04:37:38.782Z] 2024-01-19 04:37:38 +0000:[INFO] [{ScheduleBackupDeleteAndRecreateNS}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T04:37:38.782Z] 2024-01-19 04:37:38 +0000:[INFO] [{ScheduleBackupDeleteAndRecreateNS}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T04:37:38.782Z] 2024-01-19 04:37:38 +0000:[INFO] [{ScheduleBackupDeleteAndRecreateNS}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T04:37:38.782Z] 2024/01/19 04:37:38 DoRetryWithTimeout - Error: {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated}, Next try in [10s], timeout [2m0s] [2024-01-19T04:37:48.716Z] 2024/01/19 04:37:48 DoRetryWithTimeout - Error: {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated}, Next try in [10s], timeout [2m0s] [2024-01-19T04:37:58.653Z] 2024/01/19 04:37:58 DoRetryWithTimeout - Error: {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated}, Next try in [10s], timeout [2m0s] [2024-01-19T04:38:08.591Z] 2024/01/19 04:38:08 DoRetryWithTimeout - Error: {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated}, Next try in [10s], timeout [2m0s] [2024-01-19T04:38:18.548Z] 2024/01/19 04:38:18 DoRetryWithTimeout - Error: {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated}, Next try in [10s], timeout [2m0s] [2024-01-19T04:38:28.493Z] 2024/01/19 04:38:28 DoRetryWithTimeout - Error: {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated}, Next try in [10s], timeout [2m0s] [2024-01-19T04:38:40.672Z] 2024/01/19 04:38:38 DoRetryWithTimeout - Error: {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated}, Next try in [10s], timeout [2m0s] [2024-01-19T04:38:48.756Z] 2024/01/19 04:38:48 DoRetryWithTimeout - Error: {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated}, Next try in [10s], timeout [2m0s] [2024-01-19T04:38:58.697Z] 2024/01/19 04:38:58 DoRetryWithTimeout - Error: {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated}, Next try in [10s], timeout [2m0s] [2024-01-19T04:39:08.639Z] 2024/01/19 04:39:08 DoRetryWithTimeout - Error: {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated}, Next try in [10s], timeout [2m0s] [2024-01-19T04:39:18.576Z] 2024/01/19 04:39:18 DoRetryWithTimeout - Error: {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated}, Next try in [10s], timeout [2m0s] [2024-01-19T04:39:30.744Z] 2024/01/19 04:39:28 DoRetryWithTimeout - Error: {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated}, Next try in [10s], timeout [2m0s] [2024-01-19T04:39:38.828Z] 2024-01-19 04:39:38 +0000:[ERROR] [{ScheduleBackupDeleteAndRecreateNS}] [tests.processError:#600] - processError. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated} [2024-01-19T04:39:38.828Z] {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated} [2024-01-19T04:39:38.828Z] {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated} [2024-01-19T04:39:38.828Z] {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated} [2024-01-19T04:39:38.829Z] {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated} [2024-01-19T04:39:38.829Z] {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated} [2024-01-19T04:39:38.829Z] {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated} [2024-01-19T04:39:38.829Z] {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated} [2024-01-19T04:39:38.829Z] {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated} [2024-01-19T04:39:38.829Z] {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated} [2024-01-19T04:39:38.829Z] {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated} [2024-01-19T04:39:38.829Z] {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated} [2024-01-19T04:39:39.084Z] INFO[2024-01-19 04:39:38] Verifying : Description : Verification of deleting backup schedule - [2024-01-19T04:39:39.084Z] ERRO[2024-01-19 04:39:38] backup schedule with name '' not found for org 'default' [2024-01-19T04:39:39.084Z] 2024-01-19 04:39:38 +0000:[INFO] [{ScheduleBackupDeleteAndRecreateNS}] [backup.glob..func70.3:#3161] - Deleting backup schedule policy [2024-01-19T04:39:39.646Z] INFO[2024-01-19 04:39:39] Verifying : Description : Deleting backup schedule policies [] [2024-01-19T04:39:39.646Z] ERRO[2024-01-19 04:39:39] Failed to delete schedule policy with error [Unable to find schedule policy Uid] [2024-01-19T04:39:39.646Z] INFO[2024-01-19 04:39:39] Verifying : Description : Getting the backup driver [2024-01-19T04:39:39.646Z] INFO[2024-01-19 04:39:39] Actual:true, Expected: true [2024-01-19T04:39:39.646Z] INFO[2024-01-19 04:39:39] Verifying : Description : Deleting restore [] [2024-01-19T04:39:39.646Z] ERRO[2024-01-19 04:39:39] restore with name '' not found for org 'default' [2024-01-19T04:39:39.646Z] 2024-01-19 04:39:39 +0000:[INFO] [{ScheduleBackupDeleteAndRecreateNS}] Destroying apps [2024-01-19T04:39:39.646Z] 2024-01-19 04:39:39 +0000:[INFO] [{ScheduleBackupDeleteAndRecreateNS}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T04:39:39.646Z] INFO[2024-01-19 04:39:39] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T04:39:39.646Z] INFO[2024-01-19 04:39:39] Actual:true, Expected: true [2024-01-19T04:39:39.646Z] INFO[2024-01-19 04:39:39] Verifying : Description : Deleting cluster source-cluster [2024-01-19T04:39:39.646Z] ERRO[2024-01-19 04:39:39] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl], cannot be deleted, Expected:  [2024-01-19T04:39:41.011Z] INFO[2024-01-19 04:39:40] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-hrfiy] [2024-01-19T04:39:41.011Z] ERRO[2024-01-19 04:39:40] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-hrfiy:72b1b409-7bb1-4f22-92df-0ba5e77c11d3], Expected:  [2024-01-19T04:39:41.267Z] INFO[2024-01-19 04:39:41] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T04:39:41.267Z] INFO[2024-01-19 04:39:41] Actual:true, Expected: true [2024-01-19T04:39:42.631Z] INFO[2024-01-19 04:39:42] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-acnhm] [2024-01-19T04:39:42.631Z] INFO[2024-01-19 04:39:42] Actual:true, Expected: true [2024-01-19T04:39:42.631Z] INFO[2024-01-19 04:39:42] Verifying : Description : Deleting cluster source-cluster [2024-01-19T04:39:42.631Z] ERRO[2024-01-19 04:39:42] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh], cannot be deleted, Expected:  [2024-01-19T04:39:43.996Z] INFO[2024-01-19 04:39:43] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-fbvvo] [2024-01-19T04:39:43.996Z] ERRO[2024-01-19 04:39:43] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-fbvvo:7cddb5aa-c7ff-4044-9d27-458be2bd5cde], Expected:  [2024-01-19T04:39:43.996Z] INFO[2024-01-19 04:39:43] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T04:39:43.996Z] INFO[2024-01-19 04:39:43] Actual:true, Expected: true [2024-01-19T04:39:45.363Z] INFO[2024-01-19 04:39:45] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-soqhu] [2024-01-19T04:39:45.364Z] INFO[2024-01-19 04:39:45] Actual:true, Expected: true [2024-01-19T04:39:45.364Z] INFO[2024-01-19 04:39:45] Verifying : Description : Deleting cluster source-cluster [2024-01-19T04:39:45.364Z] ERRO[2024-01-19 04:39:45] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo], cannot be deleted, Expected:  [2024-01-19T04:39:46.731Z] INFO[2024-01-19 04:39:46] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-enzij] [2024-01-19T04:39:46.731Z] ERRO[2024-01-19 04:39:46] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-enzij:4e0ddfc7-8811-4ed5-8e87-ed0841582111], Expected:  [2024-01-19T04:39:46.986Z] INFO[2024-01-19 04:39:46] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T04:39:46.986Z] INFO[2024-01-19 04:39:46] Actual:true, Expected: true [2024-01-19T04:39:47.912Z] INFO[2024-01-19 04:39:47] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-fcirm] [2024-01-19T04:39:47.913Z] INFO[2024-01-19 04:39:47] Actual:true, Expected: true [2024-01-19T04:39:48.168Z] INFO[2024-01-19 04:39:47] Verifying : Description : Deleting cluster source-cluster [2024-01-19T04:39:48.168Z] ERRO[2024-01-19 04:39:47] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo], cannot be deleted, Expected:  [2024-01-19T04:39:49.531Z] INFO[2024-01-19 04:39:49] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-wcpfm] [2024-01-19T04:39:49.531Z] ERRO[2024-01-19 04:39:49] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-wcpfm:c5ff084a-a913-4412-99fb-e2b0a003a014], Expected:  [2024-01-19T04:39:49.531Z] INFO[2024-01-19 04:39:49] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T04:39:49.531Z] INFO[2024-01-19 04:39:49] Actual:true, Expected: true [2024-01-19T04:39:50.898Z] INFO[2024-01-19 04:39:50] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-vpaxz] [2024-01-19T04:39:50.898Z] INFO[2024-01-19 04:39:50] Actual:true, Expected: true [2024-01-19T04:39:50.898Z] INFO[2024-01-19 04:39:50] Verifying : Description : Deleting cluster source-cluster [2024-01-19T04:39:50.898Z] ERRO[2024-01-19 04:39:50] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx], cannot be deleted, Expected:  [2024-01-19T04:39:52.261Z] INFO[2024-01-19 04:39:52] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-vlbwu] [2024-01-19T04:39:52.262Z] ERRO[2024-01-19 04:39:52] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-vlbwu:f0465bb5-2565-4948-bb27-7e45a4b883ff], Expected:  [2024-01-19T04:39:52.517Z] INFO[2024-01-19 04:39:52] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T04:39:52.517Z] INFO[2024-01-19 04:39:52] Actual:true, Expected: true [2024-01-19T04:39:53.897Z] INFO[2024-01-19 04:39:53] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-tjtlo] [2024-01-19T04:39:53.897Z] INFO[2024-01-19 04:39:53] Actual:true, Expected: true [2024-01-19T04:39:53.897Z] INFO[2024-01-19 04:39:53] Verifying : Description : Deleting cluster source-cluster [2024-01-19T04:39:53.897Z] ERRO[2024-01-19 04:39:53] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa], cannot be deleted, Expected:  [2024-01-19T04:39:55.263Z] INFO[2024-01-19 04:39:54] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-mhxzi] [2024-01-19T04:39:55.263Z] ERRO[2024-01-19 04:39:54] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-mhxzi:f551fb81-1a78-4d7f-b804-759c6e6b562e], Expected:  [2024-01-19T04:39:55.263Z] INFO[2024-01-19 04:39:55] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T04:39:55.263Z] INFO[2024-01-19 04:39:55] Actual:true, Expected: true [2024-01-19T04:39:56.626Z] INFO[2024-01-19 04:39:56] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-nnofn] [2024-01-19T04:39:56.626Z] INFO[2024-01-19 04:39:56] Actual:true, Expected: true [2024-01-19T04:39:56.626Z] INFO[2024-01-19 04:39:56] --------Test End------ [2024-01-19T04:39:56.626Z] INFO[2024-01-19 04:39:56] #Test: [2024-01-19T04:39:56.626Z] INFO[2024-01-19 04:39:56] #Description: [2024-01-19T04:39:56.626Z] INFO[2024-01-19 04:39:56] ------------------------ [2024-01-19T04:39:56.626Z] 2024-01-19 04:39:56 +0000:[INFO] [{ScheduleBackupDeleteAndRecreateNS}] [tests.EndPxBackupTorpedoTest:#7261] - >>>> FAILED TEST: {ScheduleBackupDeleteAndRecreateNS} Validates schedule backups when namespaces are deleted and recreated [2024-01-19T04:39:56.626Z] 2024-01-19 04:39:56 +0000:[INFO] [{ScheduleBackupDeleteAndRecreateNS}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T04:39:56.626Z] 2024-01-19 04:39:56 +0000:[INFO] [{ScheduleBackupDeleteAndRecreateNS}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T04:39:56.882Z] 2024-01-19 04:39:56 +0000:[INFO] [{ScheduleBackupDeleteAndRecreateNS}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {ScheduleBackupDeleteAndRecreateNS} Validates schedule backups when namespaces are deleted and recreated [2024-01-19T04:39:56.882Z] 2024-01-19 04:39:56 +0000:[INFO] [{ScheduleBackupDeleteAndRecreateNS}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T04:39:56.882Z] 2024/01/19 04:39:56 DoRetryWithTimeout - Error: {dial tcp 10.240.0.41:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T04:40:02.155Z] 2024/01/19 04:40:01 DoRetryWithTimeout - Error: {dial tcp 10.240.0.41:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T04:40:07.404Z] 2024/01/19 04:40:06 DoRetryWithTimeout - Error: {dial tcp 10.240.0.41:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T04:40:12.645Z] 2024/01/19 04:40:11 DoRetryWithTimeout - Error: {dial tcp 10.240.0.41:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T04:40:16.811Z] 2024/01/19 04:40:16 DoRetryWithTimeout - Error: {dial tcp 10.240.0.41:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T04:40:22.050Z] 2024/01/19 04:40:21 DoRetryWithTimeout - Error: {dial tcp 10.240.0.41:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T04:40:27.290Z] 2024/01/19 04:40:26 DoRetryWithTimeout - Error: {dial tcp 10.240.0.41:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T04:40:32.541Z] 2024/01/19 04:40:31 DoRetryWithTimeout - Error: {dial tcp 10.240.0.41:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T04:40:36.706Z] 2024/01/19 04:40:36 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.0.41. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.41 10.240.0.41]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.41:22: connect: connection refused} [2024-01-19T04:40:36.706Z] {dial tcp 10.240.0.41:22: connect: connection refused} [2024-01-19T04:40:36.706Z] {dial tcp 10.240.0.41:22: connect: connection refused} [2024-01-19T04:40:36.706Z] {dial tcp 10.240.0.41:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T04:40:41.949Z] 2024-01-19 04:40:41 +0000:[WARNING] [{ScheduleBackupDeleteAndRecreateNS}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.41. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.41 10.240.0.41]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.41:22: connect: connection refused} [2024-01-19T04:40:41.949Z] {dial tcp 10.240.0.41:22: connect: connection refused} [2024-01-19T04:40:41.949Z] {dial tcp 10.240.0.41:22: connect: connection refused} [2024-01-19T04:40:41.949Z] {dial tcp 10.240.0.41:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T04:40:41.949Z] 2024-01-19 04:40:41 +0000:[ERROR] [{ScheduleBackupDeleteAndRecreateNS}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.41. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.41 10.240.0.41]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.41:22: connect: connection refused} [2024-01-19T04:40:41.949Z] {dial tcp 10.240.0.41:22: connect: connection refused} [2024-01-19T04:40:41.949Z] {dial tcp 10.240.0.41:22: connect: connection refused} [2024-01-19T04:40:41.949Z] {dial tcp 10.240.0.41:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T04:40:41.949Z] 2024-01-19 04:40:41 +0000:[INFO] [{ScheduleBackupDeleteAndRecreateNS}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T04:40:41.949Z] 2024-01-19 04:40:41 +0000:[INFO] [{ScheduleBackupDeleteAndRecreateNS}] Switching context to [/tmp/source-config] [2024-01-19T04:40:41.949Z] 2024-01-19 04:40:41 +0000:[INFO] [{ScheduleBackupDeleteAndRecreateNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T04:40:41.950Z] 2024-01-19 04:40:41 +0000:[INFO] [{ScheduleBackupDeleteAndRecreateNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T04:40:42.205Z] 2024-01-19 04:40:41 +0000:[INFO] [{ScheduleBackupDeleteAndRecreateNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T04:40:42.206Z] 2024-01-19 04:40:41 +0000:[INFO] [{ScheduleBackupDeleteAndRecreateNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T04:40:42.206Z] 2024-01-19 04:40:42 +0000:[INFO] [{ScheduleBackupDeleteAndRecreateNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T04:40:42.206Z] 2024-01-19 04:40:42 +0000:[INFO] [{ScheduleBackupDeleteAndRecreateNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T04:40:42.767Z] 2024-01-19 04:40:42 +0000:[INFO] [{ScheduleBackupDeleteAndRecreateNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T04:40:43.023Z] 2024-01-19 04:40:42 +0000:[INFO] [{ScheduleBackupDeleteAndRecreateNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T04:40:43.585Z] 2024-01-19 04:40:43 +0000:[INFO] [{ScheduleBackupDeleteAndRecreateNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T04:40:43.585Z] 2024/01/19 04:40:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:40:55.752Z] 2024/01/19 04:40:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:41:03.835Z] 2024/01/19 04:41:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:41:13.803Z] 2024/01/19 04:41:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:41:23.745Z] 2024/01/19 04:41:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:41:33.741Z] 2024/01/19 04:41:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:41:45.949Z] 2024/01/19 04:41:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:41:54.041Z] 2024/01/19 04:41:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:42:03.982Z] 2024/01/19 04:42:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:42:13.922Z] 2024/01/19 04:42:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:42:26.089Z] 2024/01/19 04:42:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:42:34.193Z] 2024/01/19 04:42:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:42:44.137Z] 2024/01/19 04:42:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:42:56.305Z] 2024/01/19 04:42:54 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:43:04.390Z] 2024/01/19 04:43:04 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:43:14.331Z] 2024/01/19 04:43:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:43:24.267Z] 2024/01/19 04:43:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:43:36.455Z] 2024/01/19 04:43:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:43:44.559Z] 2024/01/19 04:43:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:43:54.499Z] 2024/01/19 04:43:54 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:44:04.440Z] 2024/01/19 04:44:04 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:44:14.428Z] 2024/01/19 04:44:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:44:26.594Z] 2024/01/19 04:44:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:44:34.678Z] 2024/01/19 04:44:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:44:44.618Z] 2024/01/19 04:44:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:44:54.568Z] 2024/01/19 04:44:54 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:44:55.933Z] INFO[2024-01-19 04:44:55] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T04:44:55.933Z] INFO[2024-01-19 04:44:55] sched stats: tick interval: 47455 < 1.2s (100%) [2024-01-19T04:44:55.933Z] INFO[2024-01-19 04:44:55] sched stats: tick process durations: 47454 < 500ms (100%) [2024-01-19T04:45:05.869Z] 2024/01/19 04:45:04 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:45:15.810Z] 2024/01/19 04:45:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:45:25.763Z] 2024/01/19 04:45:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:45:35.705Z] 2024/01/19 04:45:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:45:45.691Z] 2024-01-19 04:45:44 +0000:[ERROR] [{ScheduleBackupDeleteAndRecreateNS}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.691Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.691Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.691Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.691Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.691Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.691Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.691Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.691Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.691Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.691Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.691Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.691Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.691Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.691Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.691Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.691Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.691Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.692Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.692Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.692Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.692Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.692Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.692Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.692Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.692Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.692Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.692Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.692Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:45:45.692Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T04:45:45.692Z] • Failure in Spec Setup (JustBeforeEach) [486.511 seconds] [2024-01-19T04:45:45.692Z] {ScheduleBackupDeleteAndRecreateNS} [2024-01-19T04:45:45.692Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:3000 [2024-01-19T04:45:45.692Z] Validates schedule backups when namespaces are deleted and recreated [JustBeforeEach] [2024-01-19T04:45:45.692Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:3040 [2024-01-19T04:45:45.692Z] [2024-01-19T04:45:45.692Z] Unexpected error: [2024-01-19T04:45:45.692Z] <*task.ErrTimedOut | 0xc000573640>: { [2024-01-19T04:45:45.693Z] Reason: "DoRetryWithTimeout timed out. Errors generated in retries: {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims \"mysql-data\" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated}\n{Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims \"mysql-data\" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated}\n{Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims \"mysql-data\" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated}\n{Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims \"mysql-data\" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated}\n{Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims \"mysql-data\" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated}\n{Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims \"mysql-data\" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated}\n{Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims \"mysql-data\" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated}\n{Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims \"mysql-data\" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated}\n{Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims \"mysql-data\" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated}\n{Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims \"mysql-data\" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated}\n{Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims \"mysql-data\" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated}\n{Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims \"mysql-data\" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated}", [2024-01-19T04:45:45.693Z] } [2024-01-19T04:45:45.693Z] timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated} [2024-01-19T04:45:45.693Z] {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated} [2024-01-19T04:45:45.693Z] {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated} [2024-01-19T04:45:45.693Z] {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated} [2024-01-19T04:45:45.693Z] {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated} [2024-01-19T04:45:45.693Z] {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated} [2024-01-19T04:45:45.693Z] {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated} [2024-01-19T04:45:45.694Z] {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated} [2024-01-19T04:45:45.694Z] {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated} [2024-01-19T04:45:45.694Z] {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated} [2024-01-19T04:45:45.694Z] {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated} [2024-01-19T04:45:45.694Z] {Failed to schedule app: mysql-ibm due to err: Failed to create PVC: mysql-data. Err: persistentvolumeclaims "mysql-data" is forbidden: unable to create new content in namespace test-namespace-src-pxb-0 because it is being terminated} [2024-01-19T04:45:45.694Z] occurred [2024-01-19T04:45:45.694Z] [2024-01-19T04:45:45.694Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T04:45:45.694Z] [2024-01-19T04:45:45.694Z] Full Stack Trace [2024-01-19T04:45:45.694Z] github.com/portworx/torpedo/pkg/log.FailOnError({0x88adf80, 0xc000573640}, {0x7eb1033?, 0x18?}, {0x0?, 0x0?, 0x0?}) [2024-01-19T04:45:45.694Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 +0x290 [2024-01-19T04:45:45.694Z] github.com/portworx/torpedo/tests.processError({0x88adf80?, 0xc000573640?}, {0x0?, 0xb924cc0?, 0x0?}) [2024-01-19T04:45:45.694Z] /go/src/github.com/portworx/torpedo/tests/common.go:600 +0x48 [2024-01-19T04:45:45.694Z] github.com/portworx/torpedo/tests.ScheduleApplicationsOnNamespace.func2() [2024-01-19T04:45:45.694Z] /go/src/github.com/portworx/torpedo/tests/common.go:1843 +0x216 [2024-01-19T04:45:45.694Z] github.com/portworx/torpedo/tests.ScheduleApplicationsOnNamespace({0xc00158e768, 0x18}, {0xc0012e4850, 0x9}, {0x0, 0x0, 0x0}) [2024-01-19T04:45:45.694Z] /go/src/github.com/portworx/torpedo/tests/common.go:1837 +0x223 [2024-01-19T04:45:45.694Z] github.com/portworx/torpedo/tests/backup.glob..func70.1() [2024-01-19T04:45:45.694Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:3029 +0x1e5 [2024-01-19T04:45:45.694Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T04:45:45.694Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T04:45:45.694Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T04:45:45.694Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T04:45:45.694Z] created by testing.(*T).Run [2024-01-19T04:45:45.694Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T04:45:45.694Z] ------------------------------ [2024-01-19T04:45:45.694Z] STEP: schedule applications [2024-01-19T04:45:45.694Z] STEP: destroy apps [2024-01-19T04:45:45.694Z] SS2024-01-19 04:45:44 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Log Dir: /testresults//KubeAndPxNamespacesSkipOnAllNSBackup.log [2024-01-19T04:45:45.694Z] INFO[2024-01-19 04:45:44] --------Test Start------ [2024-01-19T04:45:45.694Z] INFO[2024-01-19 04:45:44] #Test: KubeAndPxNamespacesSkipOnAllNSBackup [2024-01-19T04:45:45.694Z] INFO[2024-01-19 04:45:44] #Description: Verify if kube-system, kube-node-lease, kube-public and Px Namespace is skipped on all namespace backup [2024-01-19T04:45:45.694Z] INFO[2024-01-19 04:45:44] ------------------------ [2024-01-19T04:45:45.694Z] 2024-01-19 04:45:44 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Scheduling applications in destination cluster [2024-01-19T04:45:45.694Z] 2024-01-19 04:45:44 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T04:45:45.694Z] 2024-01-19 04:45:44 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T04:45:45.694Z] 2024-01-19 04:45:44 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T04:45:45.695Z] 2024-01-19 04:45:44 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T04:45:45.695Z] 2024-01-19 04:45:45 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T04:45:45.695Z] 2024-01-19 04:45:45 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T04:45:45.695Z] 2024-01-19 04:45:45 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T04:45:45.695Z] 2024-01-19 04:45:45 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T04:45:45.696Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T04:45:45.696Z] 2024-01-19 04:45:45 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T04:45:45.696Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T04:45:45.696Z] 2024-01-19 04:45:45 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T04:45:45.696Z] 2024-01-19 04:45:45 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Scheduled application [mysql-ibm] in destination cluster in namespace [mysql-ibm-dst-pxb-0-92858-01-19-04h45m44s] [2024-01-19T04:45:45.696Z] 2024-01-19 04:45:45 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T04:45:45.696Z] 2024-01-19 04:45:45 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T04:45:45.696Z] 2024-01-19 04:45:45 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T04:45:45.953Z] 2024-01-19 04:45:45 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T04:45:45.953Z] 2024-01-19 04:45:45 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T04:45:45.953Z] 2024-01-19 04:45:45 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T04:45:45.954Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T04:45:45.954Z] 2024-01-19 04:45:45 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T04:45:45.954Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T04:45:45.954Z] 2024-01-19 04:45:45 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T04:45:45.954Z] 2024-01-19 04:45:45 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Scheduled application [mysql-ibm] in destination cluster in namespace [mysql-ibm-dst-pxb-1-92858-01-19-04h45m44s] [2024-01-19T04:45:45.954Z] 2024-01-19 04:45:45 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T04:45:45.954Z] 2024-01-19 04:45:45 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T04:45:45.954Z] 2024-01-19 04:45:45 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T04:45:46.213Z] 2024-01-19 04:45:45 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T04:45:46.213Z] 2024-01-19 04:45:46 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T04:45:46.213Z] 2024-01-19 04:45:46 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T04:45:46.214Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T04:45:46.214Z] 2024-01-19 04:45:46 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T04:45:46.214Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T04:45:46.214Z] 2024-01-19 04:45:46 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T04:45:46.214Z] 2024-01-19 04:45:46 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Scheduled application [mysql-ibm] in destination cluster in namespace [mysql-ibm-dst-pxb-2-92858-01-19-04h45m44s] [2024-01-19T04:45:46.214Z] 2024-01-19 04:45:46 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Validate applications [2024-01-19T04:45:46.214Z] 2024-01-19 04:45:46 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Validating mysql-ibm app [2024-01-19T04:45:46.214Z] 2024-01-19 04:45:46 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Validating mysql-ibm app's volumes [2024-01-19T04:45:46.477Z] 2024-01-19 04:45:46 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:45:46.478Z] 2024/01/19 04:45:46 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T04:45:56.417Z] 2024/01/19 04:45:56 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T04:46:06.354Z] 2024-01-19 04:46:06 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-dst-pxb-0-92858-01-19-04h45m44s [2024-01-19T04:46:06.613Z] 2024-01-19 04:46:06 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T04:46:06.613Z] 2024-01-19 04:46:06 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T04:46:06.613Z] 2024-01-19 04:46:06 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] wait for mysql-ibm app to start running [2024-01-19T04:46:06.613Z] 2024-01-19 04:46:06 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T04:46:06.871Z] 2024/01/19 04:46:06 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T04:46:06.871Z] pod name:mysql-7d766cd65c-7s56n namespace:mysql-ibm-dst-pxb-0-92858-01-19-04h45m44s running:false ready:false node:10.240.64.27 [2024-01-19T04:46:06.871Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T04:46:16.811Z] 2024/01/19 04:46:16 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T04:46:16.811Z] pod name:mysql-7d766cd65c-7s56n namespace:mysql-ibm-dst-pxb-0-92858-01-19-04h45m44s running:true ready:false node:10.240.64.27 [2024-01-19T04:46:16.811Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T04:46:27.945Z] 2024/01/19 04:46:26 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T04:46:27.945Z] pod name:mysql-7d766cd65c-7s56n namespace:mysql-ibm-dst-pxb-0-92858-01-19-04h45m44s running:true ready:false node:10.240.64.27 [2024-01-19T04:46:27.945Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T04:46:37.881Z] 2024/01/19 04:46:36 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T04:46:37.881Z] pod name:mysql-7d766cd65c-7s56n namespace:mysql-ibm-dst-pxb-0-92858-01-19-04h45m44s running:true ready:false node:10.240.64.27 [2024-01-19T04:46:37.881Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T04:46:47.815Z] 2024/01/19 04:46:47 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T04:46:47.816Z] pod name:mysql-7d766cd65c-7s56n namespace:mysql-ibm-dst-pxb-0-92858-01-19-04h45m44s running:true ready:false node:10.240.64.27 [2024-01-19T04:46:47.816Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T04:46:57.756Z] 2024/01/19 04:46:57 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T04:46:57.756Z] pod name:mysql-7d766cd65c-7s56n namespace:mysql-ibm-dst-pxb-0-92858-01-19-04h45m44s running:true ready:false node:10.240.64.27 [2024-01-19T04:46:57.756Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T04:47:07.698Z] 2024/01/19 04:47:07 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T04:47:07.698Z] pod name:mysql-7d766cd65c-7s56n namespace:mysql-ibm-dst-pxb-0-92858-01-19-04h45m44s running:true ready:false node:10.240.64.27 [2024-01-19T04:47:07.698Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T04:47:17.651Z] 2024/01/19 04:47:17 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T04:47:17.651Z] pod name:mysql-7d766cd65c-7s56n namespace:mysql-ibm-dst-pxb-0-92858-01-19-04h45m44s running:true ready:false node:10.240.64.27 [2024-01-19T04:47:17.651Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T04:47:27.620Z] 2024-01-19 04:47:27 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T04:47:27.875Z] 2024-01-19 04:47:27 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T04:47:28.131Z] 2024-01-19 04:47:27 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] validate if mysql-ibm app's volumes are setup [2024-01-19T04:47:28.131Z] 2024-01-19 04:47:27 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:47:28.131Z] 2024-01-19 04:47:27 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:47:28.131Z] 2024-01-19 04:47:27 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T04:47:28.131Z] 2024-01-19 04:47:27 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T04:47:28.131Z] 2024-01-19 04:47:27 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Validating mysql-ibm app [2024-01-19T04:47:28.131Z] 2024-01-19 04:47:27 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Validating mysql-ibm app's volumes [2024-01-19T04:47:28.131Z] 2024-01-19 04:47:28 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:47:28.131Z] 2024-01-19 04:47:28 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:47:28.132Z] 2024-01-19 04:47:28 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-dst-pxb-1-92858-01-19-04h45m44s [2024-01-19T04:47:28.388Z] 2024-01-19 04:47:28 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T04:47:28.388Z] 2024-01-19 04:47:28 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T04:47:28.388Z] 2024-01-19 04:47:28 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] wait for mysql-ibm app to start running [2024-01-19T04:47:28.388Z] 2024-01-19 04:47:28 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T04:47:28.644Z] 2024-01-19 04:47:28 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T04:47:28.899Z] 2024-01-19 04:47:28 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T04:47:29.826Z] 2024-01-19 04:47:29 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] validate if mysql-ibm app's volumes are setup [2024-01-19T04:47:29.826Z] 2024-01-19 04:47:29 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:47:29.827Z] 2024-01-19 04:47:29 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:47:29.827Z] 2024-01-19 04:47:29 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T04:47:29.827Z] 2024-01-19 04:47:29 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T04:47:29.827Z] 2024-01-19 04:47:29 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Validating mysql-ibm app [2024-01-19T04:47:29.827Z] 2024-01-19 04:47:29 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Validating mysql-ibm app's volumes [2024-01-19T04:47:29.827Z] 2024-01-19 04:47:29 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:47:29.827Z] 2024-01-19 04:47:29 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:47:29.827Z] 2024-01-19 04:47:29 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-dst-pxb-2-92858-01-19-04h45m44s [2024-01-19T04:47:30.083Z] 2024-01-19 04:47:29 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T04:47:30.083Z] 2024-01-19 04:47:29 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T04:47:30.083Z] 2024-01-19 04:47:29 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] wait for mysql-ibm app to start running [2024-01-19T04:47:30.083Z] 2024-01-19 04:47:30 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T04:47:30.339Z] 2024-01-19 04:47:30 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T04:47:30.901Z] 2024-01-19 04:47:30 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T04:47:31.827Z] 2024-01-19 04:47:31 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] validate if mysql-ibm app's volumes are setup [2024-01-19T04:47:31.827Z] 2024-01-19 04:47:31 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:47:31.827Z] 2024-01-19 04:47:31 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:47:31.827Z] 2024-01-19 04:47:31 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T04:47:31.827Z] 2024-01-19 04:47:31 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T04:47:31.827Z] 2024-01-19 04:47:31 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T04:47:31.827Z] 2024-01-19 04:47:31 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Switching context to [/tmp/source-config] [2024-01-19T04:47:31.827Z] 2024-01-19 04:47:31 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T04:47:31.827Z] 2024-01-19 04:47:31 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T04:47:31.827Z] 2024-01-19 04:47:31 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T04:47:32.083Z] 2024-01-19 04:47:31 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T04:47:32.083Z] 2024-01-19 04:47:31 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T04:47:32.083Z] 2024-01-19 04:47:32 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T04:47:32.647Z] 2024-01-19 04:47:32 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T04:47:33.208Z] 2024-01-19 04:47:32 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T04:47:33.465Z] 2024-01-19 04:47:33 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T04:47:33.465Z] 2024/01/19 04:47:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:47:45.708Z] 2024/01/19 04:47:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:47:53.802Z] 2024/01/19 04:47:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:48:03.742Z] 2024/01/19 04:48:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:48:13.716Z] 2024/01/19 04:48:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:48:23.655Z] 2024/01/19 04:48:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:48:35.824Z] 2024/01/19 04:48:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:48:43.908Z] 2024/01/19 04:48:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:48:53.882Z] 2024/01/19 04:48:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:49:03.836Z] 2024/01/19 04:49:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:49:16.010Z] 2024/01/19 04:49:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:49:24.097Z] 2024/01/19 04:49:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:49:34.042Z] 2024/01/19 04:49:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:49:44.031Z] 2024/01/19 04:49:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:49:53.977Z] 2024/01/19 04:49:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:50:03.956Z] 2024/01/19 04:50:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:50:16.156Z] 2024/01/19 04:50:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:50:24.261Z] 2024/01/19 04:50:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:50:34.200Z] 2024/01/19 04:50:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:50:44.222Z] 2024/01/19 04:50:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:50:54.186Z] 2024/01/19 04:50:54 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:51:04.171Z] 2024/01/19 04:51:04 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:51:16.361Z] 2024/01/19 04:51:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:51:24.454Z] 2024/01/19 04:51:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:51:34.408Z] 2024/01/19 04:51:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:51:44.353Z] 2024/01/19 04:51:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:51:54.344Z] 2024/01/19 04:51:54 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:52:04.335Z] 2024/01/19 04:52:04 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:52:16.501Z] 2024/01/19 04:52:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:52:24.589Z] 2024/01/19 04:52:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:52:34.523Z] 2024-01-19 04:52:34 +0000:[ERROR] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [backup.glob..func75.2.2:#3896] - Switching context to source cluster failed. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.523Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.523Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.523Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.523Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.523Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.523Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.523Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.523Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.523Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.523Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.523Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.523Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.523Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.523Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.523Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.524Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.524Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.524Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.524Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.524Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.524Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.524Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.524Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.524Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.524Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.524Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.524Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.524Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:52:34.524Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T04:52:34.524Z] 2024-01-19 04:52:34 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T04:52:34.524Z] 2024-01-19 04:52:34 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T04:52:34.524Z] 2024-01-19 04:52:34 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Deleting deployed namespaces - [mysql-ibm-dst-pxb-0-92858-01-19-04h45m44s mysql-ibm-dst-pxb-1-92858-01-19-04h45m44s mysql-ibm-dst-pxb-2-92858-01-19-04h45m44s] [2024-01-19T04:52:34.524Z] 2024-01-19 04:52:34 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Validating apps [2024-01-19T04:52:34.524Z] 2024-01-19 04:52:34 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Validating mysql-ibm app [2024-01-19T04:52:34.524Z] 2024-01-19 04:52:34 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Validating mysql-ibm app's volumes [2024-01-19T04:52:34.524Z] 2024-01-19 04:52:34 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:52:34.524Z] 2024-01-19 04:52:34 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:52:34.524Z] 2024-01-19 04:52:34 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-dst-pxb-0-92858-01-19-04h45m44s [2024-01-19T04:52:34.780Z] 2024-01-19 04:52:34 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T04:52:34.780Z] 2024-01-19 04:52:34 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T04:52:34.780Z] 2024-01-19 04:52:34 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] wait for mysql-ibm app to start running [2024-01-19T04:52:34.780Z] 2024-01-19 04:52:34 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T04:52:35.039Z] 2024-01-19 04:52:34 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T04:52:35.039Z] 2024-01-19 04:52:34 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T04:52:35.295Z] 2024-01-19 04:52:35 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] validate if mysql-ibm app's volumes are setup [2024-01-19T04:52:35.295Z] 2024-01-19 04:52:35 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:52:35.295Z] 2024-01-19 04:52:35 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:52:35.295Z] 2024-01-19 04:52:35 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T04:52:35.295Z] 2024-01-19 04:52:35 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T04:52:35.295Z] 2024-01-19 04:52:35 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Validating mysql-ibm app [2024-01-19T04:52:35.295Z] 2024-01-19 04:52:35 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Validating mysql-ibm app's volumes [2024-01-19T04:52:35.295Z] 2024-01-19 04:52:35 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:52:35.295Z] 2024-01-19 04:52:35 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:52:35.295Z] 2024-01-19 04:52:35 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-dst-pxb-1-92858-01-19-04h45m44s [2024-01-19T04:52:35.551Z] 2024-01-19 04:52:35 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T04:52:35.807Z] 2024-01-19 04:52:35 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T04:52:35.807Z] 2024-01-19 04:52:35 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] wait for mysql-ibm app to start running [2024-01-19T04:52:36.064Z] 2024-01-19 04:52:35 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T04:52:36.064Z] 2024-01-19 04:52:35 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T04:52:36.064Z] 2024-01-19 04:52:36 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T04:52:36.710Z] 2024-01-19 04:52:36 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] validate if mysql-ibm app's volumes are setup [2024-01-19T04:52:36.966Z] 2024-01-19 04:52:36 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:52:36.966Z] 2024-01-19 04:52:36 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:52:36.966Z] 2024-01-19 04:52:36 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T04:52:36.966Z] 2024-01-19 04:52:36 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T04:52:36.966Z] 2024-01-19 04:52:36 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Validating mysql-ibm app [2024-01-19T04:52:36.967Z] 2024-01-19 04:52:36 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Validating mysql-ibm app's volumes [2024-01-19T04:52:36.967Z] 2024-01-19 04:52:36 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:52:36.967Z] 2024-01-19 04:52:36 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:52:36.967Z] 2024-01-19 04:52:36 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-dst-pxb-2-92858-01-19-04h45m44s [2024-01-19T04:52:37.527Z] 2024-01-19 04:52:37 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T04:52:37.782Z] 2024-01-19 04:52:37 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T04:52:37.782Z] 2024-01-19 04:52:37 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] wait for mysql-ibm app to start running [2024-01-19T04:52:38.043Z] 2024-01-19 04:52:37 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T04:52:38.043Z] 2024-01-19 04:52:37 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T04:52:38.043Z] 2024-01-19 04:52:38 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T04:52:38.969Z] 2024-01-19 04:52:38 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] validate if mysql-ibm app's volumes are setup [2024-01-19T04:52:38.969Z] 2024-01-19 04:52:38 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T04:52:38.969Z] 2024-01-19 04:52:38 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T04:52:38.969Z] 2024-01-19 04:52:38 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T04:52:38.969Z] 2024-01-19 04:52:38 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T04:52:38.969Z] 2024-01-19 04:52:38 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Destroying apps [2024-01-19T04:52:38.969Z] 2024-01-19 04:52:38 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T04:52:38.969Z] 2024-01-19 04:52:38 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T04:52:38.969Z] 2024-01-19 04:52:38 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] start destroying mysql-ibm app [2024-01-19T04:52:39.225Z] 2024-01-19 04:52:39 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T04:52:39.225Z] 2024-01-19 04:52:39 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T04:52:39.225Z] 2024-01-19 04:52:39 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T04:52:39.226Z] INFO[2024-01-19 04:52:39] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T04:52:39.226Z] INFO[2024-01-19 04:52:39] Actual:true, Expected: true [2024-01-19T04:52:39.226Z] 2024-01-19 04:52:39 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T04:52:39.484Z] 2024-01-19 04:52:39 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T04:52:39.484Z] 2024-01-19 04:52:39 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] start destroying mysql-ibm app [2024-01-19T04:52:39.739Z] 2024-01-19 04:52:39 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T04:52:39.739Z] 2024-01-19 04:52:39 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T04:52:39.739Z] 2024-01-19 04:52:39 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T04:52:39.739Z] INFO[2024-01-19 04:52:39] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T04:52:39.739Z] INFO[2024-01-19 04:52:39] Actual:true, Expected: true [2024-01-19T04:52:39.739Z] 2024-01-19 04:52:39 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T04:52:40.301Z] 2024-01-19 04:52:40 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T04:52:40.301Z] 2024-01-19 04:52:40 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] start destroying mysql-ibm app [2024-01-19T04:52:40.301Z] 2024-01-19 04:52:40 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T04:52:40.557Z] 2024-01-19 04:52:40 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T04:52:40.557Z] 2024-01-19 04:52:40 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T04:52:40.557Z] INFO[2024-01-19 04:52:40] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T04:52:40.557Z] INFO[2024-01-19 04:52:40] Actual:true, Expected: true [2024-01-19T04:52:40.557Z] 2024-01-19 04:52:40 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T04:52:40.557Z] 2024-01-19 04:52:40 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Switching context to [/tmp/source-config] [2024-01-19T04:52:40.557Z] 2024-01-19 04:52:40 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T04:52:40.557Z] 2024-01-19 04:52:40 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T04:52:40.557Z] 2024-01-19 04:52:40 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T04:52:40.813Z] 2024-01-19 04:52:40 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T04:52:40.813Z] 2024-01-19 04:52:40 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T04:52:40.813Z] 2024-01-19 04:52:40 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T04:52:41.374Z] 2024-01-19 04:52:41 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T04:52:41.630Z] 2024-01-19 04:52:41 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T04:52:42.190Z] 2024-01-19 04:52:41 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T04:52:42.190Z] 2024/01/19 04:52:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:52:54.363Z] 2024/01/19 04:52:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:53:02.447Z] 2024/01/19 04:53:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:53:12.389Z] 2024/01/19 04:53:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:53:22.335Z] 2024/01/19 04:53:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:53:34.497Z] 2024/01/19 04:53:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:53:42.577Z] 2024/01/19 04:53:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:53:52.514Z] 2024/01/19 04:53:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:54:02.456Z] 2024/01/19 04:54:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:54:14.620Z] 2024/01/19 04:54:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:54:22.708Z] 2024/01/19 04:54:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:54:32.649Z] 2024/01/19 04:54:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:54:42.591Z] 2024/01/19 04:54:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:54:54.760Z] 2024/01/19 04:54:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:54:57.269Z] INFO[2024-01-19 04:54:56] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T04:54:57.269Z] INFO[2024-01-19 04:54:56] sched stats: tick interval: 48056 < 1.2s (100%) [2024-01-19T04:54:57.269Z] INFO[2024-01-19 04:54:56] sched stats: tick process durations: 48055 < 500ms (100%) [2024-01-19T04:55:03.799Z] 2024/01/19 04:55:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:55:13.732Z] 2024/01/19 04:55:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:55:23.671Z] 2024/01/19 04:55:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:55:33.617Z] 2024/01/19 04:55:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:55:43.562Z] 2024/01/19 04:55:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:55:53.519Z] 2024/01/19 04:55:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:56:03.461Z] 2024/01/19 04:56:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:56:13.439Z] 2024/01/19 04:56:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:56:23.404Z] 2024/01/19 04:56:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:56:33.343Z] 2024/01/19 04:56:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:56:43.279Z] 2024/01/19 04:56:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:56:53.224Z] 2024/01/19 04:56:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:57:03.160Z] 2024/01/19 04:57:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:57:13.111Z] 2024/01/19 04:57:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:57:25.282Z] 2024/01/19 04:57:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:57:33.379Z] 2024/01/19 04:57:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:57:43.323Z] 2024-01-19 04:57:43 +0000:[ERROR] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [backup.glob..func75.3:#4091] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.323Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.323Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.323Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.323Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.324Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.325Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.326Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.326Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.326Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.326Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.326Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.327Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.327Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.327Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.327Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.328Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.328Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.328Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.328Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.328Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.328Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.328Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.328Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.328Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.328Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.330Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.330Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.330Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T04:57:43.331Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T04:57:43.331Z] 2024-01-19 04:57:43 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T04:57:43.331Z] 2024-01-19 04:57:43 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Switching context to [/tmp/source-config] [2024-01-19T04:57:43.331Z] 2024-01-19 04:57:43 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T04:57:43.332Z] 2024-01-19 04:57:43 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T04:57:43.332Z] 2024-01-19 04:57:43 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T04:57:43.597Z] 2024-01-19 04:57:43 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T04:57:43.597Z] 2024-01-19 04:57:43 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T04:57:43.597Z] 2024-01-19 04:57:43 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T04:57:44.170Z] 2024-01-19 04:57:43 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T04:57:44.431Z] 2024-01-19 04:57:44 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T04:57:44.993Z] 2024-01-19 04:57:44 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T04:57:44.993Z] 2024/01/19 04:57:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:57:56.532Z] 2024/01/19 04:57:54 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:58:06.495Z] 2024/01/19 04:58:05 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:58:16.430Z] 2024/01/19 04:58:15 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:58:26.367Z] 2024/01/19 04:58:25 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:58:36.308Z] 2024/01/19 04:58:35 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:58:46.256Z] 2024/01/19 04:58:45 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:58:56.193Z] 2024/01/19 04:58:55 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:59:06.131Z] 2024/01/19 04:59:05 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:59:16.073Z] 2024/01/19 04:59:15 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:59:26.013Z] 2024/01/19 04:59:25 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:59:35.952Z] 2024/01/19 04:59:35 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:59:45.897Z] 2024/01/19 04:59:45 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T04:59:55.994Z] 2024/01/19 04:59:55 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:00:05.937Z] 2024/01/19 05:00:05 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:00:15.903Z] 2024/01/19 05:00:15 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:00:25.838Z] 2024/01/19 05:00:25 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:00:35.788Z] 2024/01/19 05:00:35 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:00:45.725Z] 2024/01/19 05:00:45 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:00:55.666Z] 2024/01/19 05:00:55 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:01:07.837Z] 2024/01/19 05:01:05 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:01:15.965Z] 2024/01/19 05:01:15 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:01:25.901Z] 2024/01/19 05:01:25 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:01:35.843Z] 2024/01/19 05:01:35 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:01:45.788Z] 2024/01/19 05:01:45 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:01:57.957Z] 2024/01/19 05:01:55 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:02:06.040Z] 2024/01/19 05:02:05 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:02:16.013Z] 2024/01/19 05:02:15 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:02:25.956Z] 2024/01/19 05:02:25 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:02:38.154Z] 2024/01/19 05:02:35 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:02:46.236Z] 2024-01-19 05:02:45 +0000:[ERROR] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [backup.glob..func75.3.1:#4080] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.236Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.236Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.236Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.236Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.236Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.236Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.236Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.236Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.237Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.237Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.237Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.237Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.237Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.237Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.237Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.237Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.237Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.237Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.237Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.237Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.237Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.237Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.237Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.237Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.237Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.237Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.237Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.237Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:02:46.237Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T05:02:46.237Z] INFO[2024-01-19 05:02:45] --------Test End------ [2024-01-19T05:02:46.237Z] INFO[2024-01-19 05:02:45] #Test: [2024-01-19T05:02:46.238Z] INFO[2024-01-19 05:02:45] #Description: [2024-01-19T05:02:46.238Z] INFO[2024-01-19 05:02:45] ------------------------ [2024-01-19T05:02:46.238Z] 2024-01-19 05:02:45 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.EndPxBackupTorpedoTest:#7261] - >>>> FAILED TEST: {KubeAndPxNamespacesSkipOnAllNSBackup} Verify if kube-system, kube-node-lease, kube-public and Px Namespace is skipped on all namespace backup [2024-01-19T05:02:46.238Z] 2024-01-19 05:02:46 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-dst-pxb-0-92858-01-19-04h45m44s] [2024-01-19T05:02:46.238Z] 2024-01-19 05:02:46 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-dst-pxb-1-92858-01-19-04h45m44s] [2024-01-19T05:02:46.238Z] 2024-01-19 05:02:46 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-dst-pxb-2-92858-01-19-04h45m44s] [2024-01-19T05:02:46.238Z] 2024-01-19 05:02:46 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T05:02:46.238Z] 2024-01-19 05:02:46 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T05:02:46.493Z] 2024-01-19 05:02:46 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-dst-pxb-0-92858-01-19-04h45m44s] [2024-01-19T05:02:46.493Z] 2024-01-19 05:02:46 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-dst-pxb-1-92858-01-19-04h45m44s] [2024-01-19T05:02:46.494Z] 2024-01-19 05:02:46 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-dst-pxb-2-92858-01-19-04h45m44s] [2024-01-19T05:02:46.494Z] 2024-01-19 05:02:46 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {KubeAndPxNamespacesSkipOnAllNSBackup} Verify if kube-system, kube-node-lease, kube-public and Px Namespace is skipped on all namespace backup [2024-01-19T05:02:46.494Z] 2024-01-19 05:02:46 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T05:02:46.494Z] 2024/01/19 05:02:46 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:02:51.740Z] 2024/01/19 05:02:51 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:02:56.983Z] 2024/01/19 05:02:56 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:03:02.228Z] 2024/01/19 05:03:01 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:03:07.468Z] 2024/01/19 05:03:06 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:03:11.631Z] 2024/01/19 05:03:11 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:03:16.904Z] 2024/01/19 05:03:16 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:03:22.155Z] 2024/01/19 05:03:21 DoRetryWithTimeout - Error: {dial tcp 10.240.64.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:03:27.394Z] 2024/01/19 05:03:26 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.64.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.8 10.240.64.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T05:03:27.394Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T05:03:27.394Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T05:03:27.394Z] {dial tcp 10.240.64.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T05:03:31.558Z] 2024-01-19 05:03:31 +0000:[WARNING] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.64.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.8 10.240.64.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T05:03:31.558Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T05:03:31.558Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T05:03:31.558Z] {dial tcp 10.240.64.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T05:03:31.558Z] 2024-01-19 05:03:31 +0000:[ERROR] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.64.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.8 10.240.64.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T05:03:31.558Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T05:03:31.558Z] {dial tcp 10.240.64.8:22: connect: connection refused} [2024-01-19T05:03:31.558Z] {dial tcp 10.240.64.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T05:03:31.558Z] 2024-01-19 05:03:31 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T05:03:31.558Z] 2024-01-19 05:03:31 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] Switching context to [/tmp/source-config] [2024-01-19T05:03:31.558Z] 2024-01-19 05:03:31 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T05:03:31.558Z] 2024-01-19 05:03:31 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T05:03:31.814Z] 2024-01-19 05:03:31 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T05:03:31.814Z] 2024-01-19 05:03:31 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T05:03:31.814Z] 2024-01-19 05:03:31 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T05:03:32.069Z] 2024-01-19 05:03:31 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T05:03:32.324Z] 2024-01-19 05:03:32 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T05:03:32.894Z] 2024-01-19 05:03:32 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T05:03:33.150Z] 2024-01-19 05:03:33 +0000:[INFO] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T05:03:33.407Z] 2024/01/19 05:03:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:03:43.350Z] 2024/01/19 05:03:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:03:55.523Z] 2024/01/19 05:03:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:04:03.620Z] 2024/01/19 05:04:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:04:13.570Z] 2024/01/19 05:04:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:04:23.513Z] 2024/01/19 05:04:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:04:35.679Z] 2024/01/19 05:04:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:04:43.798Z] 2024/01/19 05:04:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:04:53.743Z] 2024/01/19 05:04:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:04:57.943Z] INFO[2024-01-19 05:04:57] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T05:04:57.943Z] INFO[2024-01-19 05:04:57] sched stats: tick interval: 48657 < 1.2s (100%) [2024-01-19T05:04:57.943Z] INFO[2024-01-19 05:04:57] sched stats: tick process durations: 48656 < 500ms (100%) [2024-01-19T05:05:04.480Z] 2024/01/19 05:05:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:05:14.420Z] 2024/01/19 05:05:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:05:24.363Z] 2024/01/19 05:05:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:05:34.297Z] 2024/01/19 05:05:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:05:44.235Z] 2024/01/19 05:05:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:05:54.173Z] 2024/01/19 05:05:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:06:04.112Z] 2024/01/19 05:06:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:06:14.068Z] 2024/01/19 05:06:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:06:24.010Z] 2024/01/19 05:06:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:06:33.952Z] 2024/01/19 05:06:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:06:46.121Z] 2024/01/19 05:06:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:06:54.208Z] 2024/01/19 05:06:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:07:04.156Z] 2024/01/19 05:07:04 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:07:14.099Z] 2024/01/19 05:07:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:07:26.274Z] 2024/01/19 05:07:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:07:34.365Z] 2024/01/19 05:07:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:07:44.310Z] 2024/01/19 05:07:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:07:54.257Z] 2024/01/19 05:07:54 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:08:06.422Z] 2024/01/19 05:08:04 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:08:14.511Z] 2024/01/19 05:08:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:08:24.446Z] 2024/01/19 05:08:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:08:34.386Z] 2024-01-19 05:08:34 +0000:[ERROR] [{KubeAndPxNamespacesSkipOnAllNSBackup}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.386Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.386Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.386Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.388Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.388Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T05:08:34.388Z] [2024-01-19T05:08:34.388Z] ------------------------------ [2024-01-19T05:08:34.388Z] • Failure [1369.684 seconds] [2024-01-19T05:08:34.388Z] {KubeAndPxNamespacesSkipOnAllNSBackup} [2024-01-19T05:08:34.388Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:3840 [2024-01-19T05:08:34.388Z] Verify if kube-system, kube-node-lease, kube-public and Px Namespace is skipped on all namespace backup [It] [2024-01-19T05:08:34.388Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:3873 [2024-01-19T05:08:34.388Z] [2024-01-19T05:08:34.388Z] Unexpected error: [2024-01-19T05:08:34.388Z] <*errors.errorString | 0xc000342c50>: { [2024-01-19T05:08:34.389Z] s: "failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{No... [2024-01-19T05:08:34.389Z] [2024-01-19T05:08:34.389Z] Gomega truncated this representation as it exceeds 'format.MaxLength'. [2024-01-19T05:08:34.389Z] Consider having the object provide a custom 'GomegaStringer' representation [2024-01-19T05:08:34.389Z] or adjust the parameters in Gomega's 'format' package. [2024-01-19T05:08:34.389Z] [2024-01-19T05:08:34.389Z] Learn more here: https://onsi.github.io/gomega/#adjusting-output [2024-01-19T05:08:34.389Z] [2024-01-19T05:08:34.389Z] failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.389Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.389Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.389Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.389Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.389Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.389Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.389Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.389Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.389Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.389Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.389Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.389Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.389Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.389Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.390Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.390Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.390Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.390Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.390Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.390Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.390Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.390Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.390Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.390Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.390Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.390Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.390Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.390Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:08:34.390Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T05:08:34.390Z] occurred [2024-01-19T05:08:34.390Z] [2024-01-19T05:08:34.390Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T05:08:34.390Z] [2024-01-19T05:08:34.390Z] Full Stack Trace [2024-01-19T05:08:34.390Z] github.com/portworx/torpedo/pkg/log.FailOnError({0x88a5060, 0xc000342c50}, {0x7f72c29?, 0x3d?}, {0x0?, 0x0?, 0x0?}) [2024-01-19T05:08:34.390Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 +0x290 [2024-01-19T05:08:34.390Z] github.com/portworx/torpedo/tests/backup.glob..func75.2.2() [2024-01-19T05:08:34.390Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:3896 +0x46 [2024-01-19T05:08:34.390Z] github.com/portworx/torpedo/tests/backup.glob..func75.2() [2024-01-19T05:08:34.390Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:3893 +0x233 [2024-01-19T05:08:34.390Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T05:08:34.390Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T05:08:34.390Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T05:08:34.390Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T05:08:34.390Z] created by testing.(*T).Run [2024-01-19T05:08:34.390Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T05:08:34.390Z] ------------------------------ [2024-01-19T05:08:34.390Z] STEP: Schedule applications in destination cluster [2024-01-19T05:08:34.390Z] STEP: schedule applications [2024-01-19T05:08:34.390Z] STEP: schedule applications [2024-01-19T05:08:34.390Z] STEP: schedule applications [2024-01-19T05:08:34.390Z] STEP: Validate app namespaces in destination cluster [2024-01-19T05:08:34.390Z] STEP: validate applications [2024-01-19T05:08:34.390Z] STEP: validate mysql-ibm app's volumes [2024-01-19T05:08:34.390Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T05:08:34.390Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T05:08:34.391Z] STEP: get mysql-ibm app's volume: pvc-217246da-a6df-4d8f-bbc6-c07db6640768 inspected by the volume driver [2024-01-19T05:08:34.391Z] STEP: wait for mysql-ibm app to start running [2024-01-19T05:08:34.391Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T05:08:34.391Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T05:08:34.391Z] STEP: Validate Px pod restart count [2024-01-19T05:08:34.391Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T05:08:34.391Z] STEP: validate mysql-ibm app's volumes [2024-01-19T05:08:34.391Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T05:08:34.391Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T05:08:34.391Z] STEP: get mysql-ibm app's volume: pvc-9497e52f-e933-4868-8a4e-4dcf7b0107ce inspected by the volume driver [2024-01-19T05:08:34.391Z] STEP: wait for mysql-ibm app to start running [2024-01-19T05:08:34.391Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T05:08:34.391Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T05:08:34.391Z] STEP: Validate Px pod restart count [2024-01-19T05:08:34.391Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T05:08:34.391Z] STEP: validate mysql-ibm app's volumes [2024-01-19T05:08:34.391Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T05:08:34.391Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T05:08:34.391Z] STEP: get mysql-ibm app's volume: pvc-b939ab0f-b0f1-4e58-8e42-4157bc45344b inspected by the volume driver [2024-01-19T05:08:34.391Z] STEP: wait for mysql-ibm app to start running [2024-01-19T05:08:34.391Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T05:08:34.391Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T05:08:34.391Z] STEP: Validate Px pod restart count [2024-01-19T05:08:34.391Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T05:08:34.391Z] STEP: validate apps [2024-01-19T05:08:34.391Z] STEP: validate mysql-ibm app's volumes [2024-01-19T05:08:34.391Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T05:08:34.391Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T05:08:34.391Z] STEP: get mysql-ibm app's volume: pvc-217246da-a6df-4d8f-bbc6-c07db6640768 inspected by the volume driver [2024-01-19T05:08:34.391Z] STEP: wait for mysql-ibm app to start running [2024-01-19T05:08:34.391Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T05:08:34.391Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T05:08:34.391Z] STEP: Validate Px pod restart count [2024-01-19T05:08:34.391Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T05:08:34.391Z] STEP: validate mysql-ibm app's volumes [2024-01-19T05:08:34.391Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T05:08:34.391Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T05:08:34.391Z] STEP: get mysql-ibm app's volume: pvc-9497e52f-e933-4868-8a4e-4dcf7b0107ce inspected by the volume driver [2024-01-19T05:08:34.391Z] STEP: wait for mysql-ibm app to start running [2024-01-19T05:08:34.391Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T05:08:34.391Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T05:08:34.391Z] STEP: Validate Px pod restart count [2024-01-19T05:08:34.391Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T05:08:34.391Z] STEP: validate mysql-ibm app's volumes [2024-01-19T05:08:34.391Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T05:08:34.391Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T05:08:34.391Z] STEP: get mysql-ibm app's volume: pvc-b939ab0f-b0f1-4e58-8e42-4157bc45344b inspected by the volume driver [2024-01-19T05:08:34.392Z] STEP: wait for mysql-ibm app to start running [2024-01-19T05:08:34.392Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T05:08:34.392Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T05:08:34.392Z] STEP: Validate Px pod restart count [2024-01-19T05:08:34.392Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T05:08:34.392Z] STEP: destroy apps [2024-01-19T05:08:34.392Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T05:08:34.392Z] STEP: start destroying mysql-ibm app [2024-01-19T05:08:34.392Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T05:08:34.392Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T05:08:34.392Z] STEP: start destroying mysql-ibm app [2024-01-19T05:08:34.392Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T05:08:34.392Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T05:08:34.392Z] STEP: start destroying mysql-ibm app [2024-01-19T05:08:34.392Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T05:08:34.392Z] S2024-01-19 05:08:34 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Log Dir: /testresults//DeleteBackupSharedByMultipleUsersFromAdmin.log [2024-01-19T05:08:34.392Z] INFO[2024-01-19 05:08:34] --------Test Start------ [2024-01-19T05:08:34.392Z] INFO[2024-01-19 05:08:34] #Test: DeleteBackupSharedByMultipleUsersFromAdmin [2024-01-19T05:08:34.392Z] INFO[2024-01-19 05:08:34] #Description: Delete backups of non admin user from px-admin when backup is shared by multiple users [2024-01-19T05:08:34.392Z] INFO[2024-01-19 05:08:34] ------------------------ [2024-01-19T05:08:34.392Z] 2024-01-19 05:08:34 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Deploy applications [2024-01-19T05:08:34.392Z] 2024-01-19 05:08:34 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T05:08:34.392Z] 2024-01-19 05:08:34 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T05:08:34.392Z] 2024-01-19 05:08:34 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T05:08:34.650Z] 2024-01-19 05:08:34 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T05:08:34.650Z] 2024-01-19 05:08:34 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T05:08:34.650Z] 2024-01-19 05:08:34 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T05:08:34.651Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T05:08:34.651Z] 2024-01-19 05:08:34 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T05:08:34.651Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T05:08:34.907Z] 2024-01-19 05:08:34 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T05:08:34.908Z] 2024-01-19 05:08:34 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Created namespaces [mysql-ibm-pxb-0-87565-01-19-05h08m34s] [2024-01-19T05:08:34.908Z] 2024-01-19 05:08:34 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Validate applications [2024-01-19T05:08:34.908Z] 2024-01-19 05:08:34 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Validate applications [2024-01-19T05:08:34.908Z] 2024-01-19 05:08:34 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Validating mysql-ibm app [2024-01-19T05:08:34.908Z] 2024-01-19 05:08:34 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Validating mysql-ibm app's volumes [2024-01-19T05:08:34.908Z] 2024-01-19 05:08:34 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T05:08:34.908Z] 2024/01/19 05:08:34 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T05:08:47.080Z] 2024/01/19 05:08:44 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T05:08:55.175Z] 2024-01-19 05:08:54 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-87565-01-19-05h08m34s [2024-01-19T05:08:55.175Z] 2024-01-19 05:08:54 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T05:08:55.175Z] 2024-01-19 05:08:55 +0000:[WARNING] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T05:08:55.175Z] 2024-01-19 05:08:55 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] wait for mysql-ibm app to start running [2024-01-19T05:08:55.175Z] 2024-01-19 05:08:55 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T05:08:55.430Z] 2024/01/19 05:08:55 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:08:55.430Z] pod name:mysql-7d766cd65c-594fz namespace:mysql-ibm-pxb-0-87565-01-19-05h08m34s running:false ready:false node:10.240.64.8 [2024-01-19T05:08:55.430Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:09:05.370Z] 2024/01/19 05:09:05 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:09:05.370Z] pod name:mysql-7d766cd65c-594fz namespace:mysql-ibm-pxb-0-87565-01-19-05h08m34s running:true ready:false node:10.240.64.8 [2024-01-19T05:09:05.370Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:09:17.535Z] 2024/01/19 05:09:15 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:09:17.535Z] pod name:mysql-7d766cd65c-594fz namespace:mysql-ibm-pxb-0-87565-01-19-05h08m34s running:true ready:false node:10.240.64.8 [2024-01-19T05:09:17.535Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:09:25.621Z] 2024/01/19 05:09:25 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:09:25.621Z] pod name:mysql-7d766cd65c-594fz namespace:mysql-ibm-pxb-0-87565-01-19-05h08m34s running:true ready:false node:10.240.64.8 [2024-01-19T05:09:25.621Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:09:37.790Z] 2024/01/19 05:09:35 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:09:37.790Z] pod name:mysql-7d766cd65c-594fz namespace:mysql-ibm-pxb-0-87565-01-19-05h08m34s running:true ready:false node:10.240.64.8 [2024-01-19T05:09:37.790Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:09:45.898Z] 2024/01/19 05:09:45 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:09:45.898Z] pod name:mysql-7d766cd65c-594fz namespace:mysql-ibm-pxb-0-87565-01-19-05h08m34s running:true ready:false node:10.240.64.8 [2024-01-19T05:09:45.898Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:09:58.098Z] 2024/01/19 05:09:55 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:09:58.098Z] pod name:mysql-7d766cd65c-594fz namespace:mysql-ibm-pxb-0-87565-01-19-05h08m34s running:true ready:false node:10.240.64.8 [2024-01-19T05:09:58.098Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:10:06.184Z] 2024-01-19 05:10:06 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T05:10:06.184Z] 2024-01-19 05:10:06 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T05:10:06.443Z] 2024-01-19 05:10:06 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] validate if mysql-ibm app's volumes are setup [2024-01-19T05:10:06.443Z] 2024-01-19 05:10:06 +0000:[WARNING] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T05:10:06.443Z] 2024-01-19 05:10:06 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T05:10:06.443Z] 2024-01-19 05:10:06 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T05:10:06.443Z] 2024-01-19 05:10:06 +0000:[WARNING] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T05:10:06.443Z] 2024-01-19 05:10:06 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Create a non-admin users to create the backups and restore [2024-01-19T05:10:06.443Z] 2024-01-19 05:10:06 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Creating 1 users [2024-01-19T05:10:06.699Z] 2024-01-19 05:10:06 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.AddUser:#728] - Response for user [testuser1-1705641006] creation - [2024-01-19T05:10:06.699Z] 2024-01-19 05:10:06 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.AddUser:#733] - User [testuser1-1705641006] created [2024-01-19T05:10:06.699Z] INFO[2024-01-19 05:10:06] Verifying : Description : Creating user - testuser1-1705641006 [2024-01-19T05:10:06.699Z] INFO[2024-01-19 05:10:06] Actual:true, Expected: true [2024-01-19T05:10:06.954Z] 2024-01-19 05:10:06 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705641006 - 8b3d5ca3-d403-499b-9712-afd37bc55594 [2024-01-19T05:10:09.471Z] 2024-01-19 05:10:08 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705641006 - 8b3d5ca3-d403-499b-9712-afd37bc55594 [2024-01-19T05:10:09.471Z] 2024-01-19 05:10:08 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Creating 1 users [2024-01-19T05:10:10.033Z] 2024-01-19 05:10:09 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.AddUser:#728] - Response for user [testuser1-1705641008] creation - [2024-01-19T05:10:10.033Z] 2024-01-19 05:10:09 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.AddUser:#733] - User [testuser1-1705641008] created [2024-01-19T05:10:10.033Z] INFO[2024-01-19 05:10:09] Verifying : Description : Creating user - testuser1-1705641008 [2024-01-19T05:10:10.033Z] INFO[2024-01-19 05:10:09] Actual:true, Expected: true [2024-01-19T05:10:10.592Z] 2024-01-19 05:10:10 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705641008 - 7ead8c09-d3d3-4fa5-bf3e-e6576a1154e5 [2024-01-19T05:10:13.108Z] 2024-01-19 05:10:12 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705641008 - 7ead8c09-d3d3-4fa5-bf3e-e6576a1154e5 [2024-01-19T05:10:13.108Z] 2024-01-19 05:10:12 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Creating 1 users [2024-01-19T05:10:14.043Z] 2024-01-19 05:10:13 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.AddUser:#728] - Response for user [testuser1-1705641012] creation - [2024-01-19T05:10:14.043Z] 2024-01-19 05:10:13 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.AddUser:#733] - User [testuser1-1705641012] created [2024-01-19T05:10:14.043Z] INFO[2024-01-19 05:10:13] Verifying : Description : Creating user - testuser1-1705641012 [2024-01-19T05:10:14.043Z] INFO[2024-01-19 05:10:13] Actual:true, Expected: true [2024-01-19T05:10:14.605Z] 2024-01-19 05:10:14 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705641012 - dd66b4af-edba-49c4-b40f-f2328ceeb36d [2024-01-19T05:10:17.123Z] 2024-01-19 05:10:16 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705641012 - dd66b4af-edba-49c4-b40f-f2328ceeb36d [2024-01-19T05:10:17.123Z] 2024-01-19 05:10:16 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Adding Credentials from px-admin user and sharing with required users [2024-01-19T05:10:18.488Z] 2024-01-19 05:10:18 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-tbjjo] for org [default] with [aws] as provider [2024-01-19T05:10:18.488Z] 2024-01-19 05:10:18 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T05:10:18.488Z] 2024-01-19 05:10:18 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-tbjjo] is present [2024-01-19T05:10:18.488Z] 2024-01-19 05:10:18 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.glob..func34.2.3:#3346] - Updating cloud credential ownership for non-admin users [2024-01-19T05:10:19.416Z] 2024-01-19 05:10:19 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705641006 - 8b3d5ca3-d403-499b-9712-afd37bc55594 [2024-01-19T05:10:19.416Z] INFO[2024-01-19 05:10:19] Verifying : Description : Verifying ownership update for cloud credential autogenerated-cred-tbjjo to user [testuser1-1705641006] [2024-01-19T05:10:19.416Z] INFO[2024-01-19 05:10:19] Actual:true, Expected: true [2024-01-19T05:10:19.417Z] 2024-01-19 05:10:19 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Adding backup location from px-admin user and sharing with app users [2024-01-19T05:10:23.591Z] 2024-01-19 05:10:22 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.glob..func34.2.4:#3363] - Updating backup location ownership for non-admin users [2024-01-19T05:10:23.591Z] 2024-01-19 05:10:23 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705641012 - dd66b4af-edba-49c4-b40f-f2328ceeb36d [2024-01-19T05:10:23.591Z] INFO[2024-01-19 05:10:23] Verifying : Description : Verifying ownership update for backup location autogenerated-backup-location-mcqju to user [testuser1-1705641012] [2024-01-19T05:10:23.591Z] INFO[2024-01-19 05:10:23] Actual:true, Expected: true [2024-01-19T05:10:23.591Z] 2024-01-19 05:10:23 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Adding backup location for app-admin user and shared cloud cred from px-admin [2024-01-19T05:10:26.866Z] 2024-01-19 05:10:26 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Adding cloud credentials and backup location for infra-admin users [2024-01-19T05:10:27.123Z] 2024-01-19 05:10:26 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-testuser1-1705641008-jsxlu] for org [default] with [aws] as provider [2024-01-19T05:10:27.123Z] 2024-01-19 05:10:26 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T05:10:27.123Z] 2024-01-19 05:10:27 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-testuser1-1705641008-jsxlu] is present [2024-01-19T05:10:30.395Z] 2024-01-19 05:10:29 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Creating a schedule policy from px-admin user [2024-01-19T05:10:30.395Z] INFO[2024-01-19 05:10:30] Verifying : Description : Verifying creation of periodic schedule policy of interval [15] minutes named [periodic-iyybk] for px-admin [2024-01-19T05:10:30.395Z] INFO[2024-01-19 05:10:30] Actual:true, Expected: true [2024-01-19T05:10:30.651Z] 2024-01-19 05:10:30 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705641006 - 8b3d5ca3-d403-499b-9712-afd37bc55594 [2024-01-19T05:10:31.211Z] 2024-01-19 05:10:30 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705641008 - 7ead8c09-d3d3-4fa5-bf3e-e6576a1154e5 [2024-01-19T05:10:31.772Z] 2024-01-19 05:10:31 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705641012 - dd66b4af-edba-49c4-b40f-f2328ceeb36d [2024-01-19T05:10:32.028Z] INFO[2024-01-19 05:10:31] Verifying : Description : Verifying update of owbership for SchedulePolicy- periodic-iyybk [2024-01-19T05:10:32.028Z] INFO[2024-01-19 05:10:31] Actual:true, Expected: true [2024-01-19T05:10:32.028Z] 2024-01-19 05:10:31 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Register source and destination cluster for backup on non-admin user [2024-01-19T05:10:32.028Z] 2024-01-19 05:10:31 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Registering Source and Destination clusters as user : testuser1-1705641006 and verifying the status [2024-01-19T05:10:32.284Z] INFO[2024-01-19 05:10:32] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T05:10:32.284Z] INFO[2024-01-19 05:10:32] Actual:true, Expected: true [2024-01-19T05:10:32.284Z] 2024-01-19 05:10:32 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T05:10:32.284Z] 2024-01-19 05:10:32 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T05:10:32.284Z] 2024-01-19 05:10:32 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T05:10:32.540Z] 2024-01-19 05:10:32 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T05:10:32.540Z] 2024-01-19 05:10:32 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Create cluster [source-cluster] in org [default] [2024-01-19T05:10:32.540Z] 2024-01-19 05:10:32 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T05:10:32.540Z] 2024-01-19 05:10:32 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T05:10:32.540Z] 2024-01-19 05:10:32 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-19T05:10:32.540Z] 2024-01-19 05:10:32 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T05:10:32.540Z] 2024-01-19 05:10:32 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T05:10:32.540Z] 2024-01-19 05:10:32 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-smmgg] for IBM [2024-01-19T05:10:32.540Z] 2024-01-19 05:10:32 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-smmgg] for org [default] with [ibm] as provider [2024-01-19T05:10:32.540Z] 2024-01-19 05:10:32 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T05:10:32.540Z] 2024-01-19 05:10:32 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T05:10:32.540Z] 2024-01-19 05:10:32 +0000:[WARNING] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-smmgg] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user px-backup-app.admin]] [2024-01-19T05:10:32.541Z] 2024-01-19 05:10:32 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user px-backup-app.admin] [2024-01-19T05:10:33.909Z] 2024-01-19 05:10:33 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-smmgg from admin context and sharing with all the users [2024-01-19T05:10:33.909Z] 2024-01-19 05:10:33 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-smmgg] for org [default] with [ibm] as provider [2024-01-19T05:10:33.909Z] 2024-01-19 05:10:33 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T05:10:33.909Z] 2024-01-19 05:10:33 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T05:10:33.909Z] 2024-01-19 05:10:33 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-smmgg] is present [2024-01-19T05:10:34.168Z] 2024-01-19 05:10:33 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Create cluster [source-cluster] in org [default] [2024-01-19T05:10:38.355Z] 2024-01-19 05:10:37 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-ymjvy] for IBM [2024-01-19T05:10:38.355Z] 2024-01-19 05:10:37 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-ymjvy] for org [default] with [ibm] as provider [2024-01-19T05:10:38.355Z] 2024-01-19 05:10:37 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T05:10:38.355Z] 2024-01-19 05:10:37 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T05:10:38.355Z] 2024-01-19 05:10:37 +0000:[WARNING] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-ymjvy] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user px-backup-app.admin]] [2024-01-19T05:10:38.355Z] 2024-01-19 05:10:37 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user px-backup-app.admin] [2024-01-19T05:10:38.356Z] 2024-01-19 05:10:37 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-ymjvy from admin context and sharing with all the users [2024-01-19T05:10:38.356Z] 2024-01-19 05:10:37 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-ymjvy] for org [default] with [ibm] as provider [2024-01-19T05:10:38.356Z] 2024-01-19 05:10:37 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T05:10:38.356Z] 2024-01-19 05:10:37 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T05:10:38.356Z] 2024-01-19 05:10:37 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-ymjvy] is present [2024-01-19T05:10:38.356Z] 2024-01-19 05:10:38 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-19T05:10:39.282Z] INFO[2024-01-19 05:10:39] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T05:10:39.282Z] INFO[2024-01-19 05:10:39] Actual:Online, Expected: Online [2024-01-19T05:10:39.282Z] INFO[2024-01-19 05:10:39] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T05:10:39.282Z] INFO[2024-01-19 05:10:39] Actual:true, Expected: true [2024-01-19T05:10:39.282Z] INFO[2024-01-19 05:10:39] Verifying : Description : Fetching [destination-cluster] cluster uid [2024-01-19T05:10:39.282Z] INFO[2024-01-19 05:10:39] Actual:true, Expected: true [2024-01-19T05:10:39.282Z] 2024-01-19 05:10:39 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Registering Source and Destination clusters as user : testuser1-1705641008 and verifying the status [2024-01-19T05:10:39.282Z] INFO[2024-01-19 05:10:39] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T05:10:39.282Z] INFO[2024-01-19 05:10:39] Actual:true, Expected: true [2024-01-19T05:10:39.282Z] 2024-01-19 05:10:39 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T05:10:39.282Z] 2024-01-19 05:10:39 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T05:10:39.282Z] 2024-01-19 05:10:39 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T05:10:39.282Z] 2024-01-19 05:10:39 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T05:10:39.282Z] 2024-01-19 05:10:39 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Create cluster [source-cluster] in org [default] [2024-01-19T05:10:39.282Z] 2024-01-19 05:10:39 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T05:10:39.282Z] 2024-01-19 05:10:39 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T05:10:39.282Z] 2024-01-19 05:10:39 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-19T05:10:39.282Z] 2024-01-19 05:10:39 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T05:10:39.282Z] 2024-01-19 05:10:39 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T05:10:39.282Z] 2024-01-19 05:10:39 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-nimwm] for IBM [2024-01-19T05:10:39.282Z] 2024-01-19 05:10:39 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-nimwm] for org [default] with [ibm] as provider [2024-01-19T05:10:39.282Z] 2024-01-19 05:10:39 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T05:10:39.283Z] 2024-01-19 05:10:39 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T05:10:39.539Z] 2024-01-19 05:10:39 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-nimwm] is present [2024-01-19T05:10:39.539Z] 2024-01-19 05:10:39 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Create cluster [source-cluster] in org [default] [2024-01-19T05:10:43.704Z] 2024-01-19 05:10:43 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-nmuvh] for IBM [2024-01-19T05:10:43.705Z] 2024-01-19 05:10:43 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-nmuvh] for org [default] with [ibm] as provider [2024-01-19T05:10:43.705Z] 2024-01-19 05:10:43 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T05:10:43.705Z] 2024-01-19 05:10:43 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T05:10:43.705Z] 2024-01-19 05:10:43 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-nmuvh] is present [2024-01-19T05:10:43.705Z] 2024-01-19 05:10:43 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-19T05:10:43.962Z] INFO[2024-01-19 05:10:43] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T05:10:43.962Z] INFO[2024-01-19 05:10:43] Actual:Online, Expected: Online [2024-01-19T05:10:43.962Z] INFO[2024-01-19 05:10:43] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T05:10:43.962Z] INFO[2024-01-19 05:10:43] Actual:true, Expected: true [2024-01-19T05:10:43.962Z] INFO[2024-01-19 05:10:43] Verifying : Description : Fetching [destination-cluster] cluster uid [2024-01-19T05:10:43.962Z] INFO[2024-01-19 05:10:43] Actual:true, Expected: true [2024-01-19T05:10:43.963Z] 2024-01-19 05:10:43 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Registering Source and Destination clusters as user : testuser1-1705641012 and verifying the status [2024-01-19T05:10:44.219Z] INFO[2024-01-19 05:10:44] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T05:10:44.219Z] INFO[2024-01-19 05:10:44] Actual:true, Expected: true [2024-01-19T05:10:44.219Z] 2024-01-19 05:10:44 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T05:10:44.219Z] 2024-01-19 05:10:44 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T05:10:44.219Z] 2024-01-19 05:10:44 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T05:10:44.219Z] 2024-01-19 05:10:44 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T05:10:44.219Z] 2024-01-19 05:10:44 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Create cluster [source-cluster] in org [default] [2024-01-19T05:10:44.220Z] 2024-01-19 05:10:44 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T05:10:44.220Z] 2024-01-19 05:10:44 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T05:10:44.220Z] 2024-01-19 05:10:44 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-19T05:10:44.220Z] 2024-01-19 05:10:44 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T05:10:44.220Z] 2024-01-19 05:10:44 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T05:10:44.220Z] 2024-01-19 05:10:44 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-yeitr] for IBM [2024-01-19T05:10:44.220Z] 2024-01-19 05:10:44 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-yeitr] for org [default] with [ibm] as provider [2024-01-19T05:10:44.220Z] 2024-01-19 05:10:44 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T05:10:44.220Z] 2024-01-19 05:10:44 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T05:10:44.220Z] 2024-01-19 05:10:44 +0000:[WARNING] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-yeitr] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user]] [2024-01-19T05:10:44.220Z] 2024-01-19 05:10:44 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user] [2024-01-19T05:10:44.478Z] 2024-01-19 05:10:44 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-yeitr from admin context and sharing with all the users [2024-01-19T05:10:44.478Z] 2024-01-19 05:10:44 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-yeitr] for org [default] with [ibm] as provider [2024-01-19T05:10:44.478Z] 2024-01-19 05:10:44 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T05:10:44.478Z] 2024-01-19 05:10:44 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T05:10:44.734Z] 2024-01-19 05:10:44 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-yeitr] is present [2024-01-19T05:10:44.734Z] 2024-01-19 05:10:44 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Create cluster [source-cluster] in org [default] [2024-01-19T05:10:48.906Z] 2024-01-19 05:10:48 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-ndrog] for IBM [2024-01-19T05:10:48.906Z] 2024-01-19 05:10:48 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-ndrog] for org [default] with [ibm] as provider [2024-01-19T05:10:48.906Z] 2024-01-19 05:10:48 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T05:10:48.906Z] 2024-01-19 05:10:48 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T05:10:48.906Z] 2024-01-19 05:10:48 +0000:[WARNING] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-ndrog] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user]] [2024-01-19T05:10:48.906Z] 2024-01-19 05:10:48 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user] [2024-01-19T05:10:48.906Z] 2024-01-19 05:10:48 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-ndrog from admin context and sharing with all the users [2024-01-19T05:10:48.906Z] 2024-01-19 05:10:48 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-ndrog] for org [default] with [ibm] as provider [2024-01-19T05:10:48.906Z] 2024-01-19 05:10:48 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T05:10:48.906Z] 2024-01-19 05:10:48 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T05:10:48.906Z] 2024-01-19 05:10:48 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-ndrog] is present [2024-01-19T05:10:48.906Z] 2024-01-19 05:10:48 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-19T05:10:49.833Z] INFO[2024-01-19 05:10:49] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T05:10:49.833Z] INFO[2024-01-19 05:10:49] Actual:Online, Expected: Online [2024-01-19T05:10:49.833Z] INFO[2024-01-19 05:10:49] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T05:10:49.833Z] INFO[2024-01-19 05:10:49] Actual:true, Expected: true [2024-01-19T05:10:49.833Z] INFO[2024-01-19 05:10:49] Verifying : Description : Fetching [destination-cluster] cluster uid [2024-01-19T05:10:49.833Z] INFO[2024-01-19 05:10:49] Actual:true, Expected: true [2024-01-19T05:10:49.833Z] 2024-01-19 05:10:49 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Taking manual backup of applications as non-admin user [2024-01-19T05:11:02.007Z] 2024-01-19 05:10:59 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T05:11:02.007Z] 2024-01-19 05:10:59 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Volume snapshot classes found in the cluster - [2024-01-19T05:11:02.007Z] 2024-01-19 05:10:59 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] ibmc-vpcblock-snapshot [2024-01-19T05:11:02.007Z] 2024-01-19 05:10:59 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T05:11:02.007Z] 2024/01/19 05:11:00 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705641006-lfdm] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T05:11:10.092Z] 2024-01-19 05:11:09 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T05:11:10.092Z] 2024-01-19 05:11:09 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Volume snapshot classes found in the cluster - [2024-01-19T05:11:10.092Z] 2024-01-19 05:11:09 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] ibmc-vpcblock-snapshot [2024-01-19T05:11:10.092Z] 2024-01-19 05:11:09 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T05:11:10.348Z] 2024/01/19 05:11:10 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705641008-ugow] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T05:11:20.289Z] 2024-01-19 05:11:19 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T05:11:20.289Z] 2024-01-19 05:11:19 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Volume snapshot classes found in the cluster - [2024-01-19T05:11:20.289Z] 2024-01-19 05:11:19 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] ibmc-vpcblock-snapshot [2024-01-19T05:11:20.289Z] 2024-01-19 05:11:19 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T05:11:20.289Z] 2024/01/19 05:11:20 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705641012-lcvr] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T05:11:32.457Z] 2024/01/19 05:11:30 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705641006-lfdm] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:11:40.536Z] 2024/01/19 05:11:40 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705641008-ugow] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:11:50.474Z] 2024/01/19 05:11:50 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705641012-lcvr] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:12:02.643Z] 2024/01/19 05:12:00 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705641006-lfdm] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:12:10.728Z] 2024/01/19 05:12:10 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705641008-ugow] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:12:20.732Z] 2024/01/19 05:12:20 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705641012-lcvr] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:12:30.693Z] 2024/01/19 05:12:30 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705641006-lfdm] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:12:40.687Z] 2024/01/19 05:12:40 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705641008-ugow] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:12:50.627Z] 2024/01/19 05:12:50 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705641012-lcvr] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:13:00.572Z] 2024/01/19 05:13:00 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705641006-lfdm] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:13:10.522Z] 2024/01/19 05:13:10 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705641008-ugow] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:13:20.458Z] 2024/01/19 05:13:20 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705641012-lcvr] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:13:32.621Z] 2024/01/19 05:13:30 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705641006-lfdm] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:13:40.705Z] 2024/01/19 05:13:40 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705641008-ugow] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:13:50.654Z] 2024/01/19 05:13:50 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705641012-lcvr] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:14:00.634Z] 2024/01/19 05:14:00 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705641006-lfdm] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:14:10.580Z] 2024/01/19 05:14:10 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705641008-ugow] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:14:20.523Z] 2024/01/19 05:14:20 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705641012-lcvr] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:14:31.844Z] 2024/01/19 05:14:30 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705641006-lfdm] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:14:41.791Z] 2024/01/19 05:14:40 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705641008-ugow] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:14:51.726Z] 2024/01/19 05:14:50 DoRetryWithTimeout - Error: {backup status for [tp-backup-manual-testuser1-1705641012-lcvr] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:14:58.265Z] INFO[2024-01-19 05:14:57] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T05:14:58.265Z] INFO[2024-01-19 05:14:57] sched stats: tick interval: 49257 < 1.2s (100%) [2024-01-19T05:14:58.265Z] INFO[2024-01-19 05:14:57] sched stats: tick process durations: 49256 < 500ms (100%) [2024-01-19T05:15:00.781Z] 2024-01-19 05:15:00 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.CreateBackup:#313] - Backup [tp-backup-manual-testuser1-1705641006-lfdm] created successfully [2024-01-19T05:15:00.781Z] 2024-01-19 05:15:00 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Validating backup [tp-backup-manual-testuser1-1705641006-lfdm] in org [default] [2024-01-19T05:15:00.781Z] 2024-01-19 05:15:00 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-manual-testuser1-1705641006-lfdm] [2024-01-19T05:15:00.781Z] 2024-01-19 05:15:00 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87565-01-19-05h08m34s] in backup [tp-backup-manual-testuser1-1705641006-lfdm] [2024-01-19T05:15:00.781Z] 2024-01-19 05:15:00 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87565-01-19-05h08m34s] in backup [tp-backup-manual-testuser1-1705641006-lfdm] [2024-01-19T05:15:00.781Z] 2024-01-19 05:15:00 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87565-01-19-05h08m34s] [2024-01-19T05:15:00.781Z] 2024-01-19 05:15:00 +0000:[WARNING] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T05:15:00.781Z] 2024-01-19 05:15:00 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T05:15:00.781Z] 2024-01-19 05:15:00 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[mysql-data]] [2024-01-19T05:15:01.038Z] 2024-01-19 05:15:00 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T05:15:01.038Z] 2024-01-19 05:15:00 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Validating CR cleanup [2024-01-19T05:15:01.038Z] 2024-01-19 05:15:00 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T05:15:01.038Z] 2024-01-19 05:15:00 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[]] [2024-01-19T05:15:01.038Z] INFO[2024-01-19 05:15:00] Verifying : Description : Verifying creation of backup [tp-backup-manual-testuser1-1705641006-lfdm] [2024-01-19T05:15:01.038Z] INFO[2024-01-19 05:15:00] Actual:true, Expected: true [2024-01-19T05:15:10.978Z] 2024-01-19 05:15:10 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.CreateBackup:#313] - Backup [tp-backup-manual-testuser1-1705641008-ugow] created successfully [2024-01-19T05:15:10.978Z] 2024-01-19 05:15:10 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Validating backup [tp-backup-manual-testuser1-1705641008-ugow] in org [default] [2024-01-19T05:15:10.978Z] 2024-01-19 05:15:10 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-manual-testuser1-1705641008-ugow] [2024-01-19T05:15:10.978Z] 2024-01-19 05:15:10 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87565-01-19-05h08m34s] in backup [tp-backup-manual-testuser1-1705641008-ugow] [2024-01-19T05:15:10.978Z] 2024-01-19 05:15:10 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87565-01-19-05h08m34s] in backup [tp-backup-manual-testuser1-1705641008-ugow] [2024-01-19T05:15:10.978Z] 2024-01-19 05:15:10 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87565-01-19-05h08m34s] [2024-01-19T05:15:10.978Z] 2024-01-19 05:15:10 +0000:[WARNING] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T05:15:10.978Z] 2024-01-19 05:15:10 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T05:15:10.978Z] 2024-01-19 05:15:10 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[mysql-data]] [2024-01-19T05:15:10.978Z] 2024-01-19 05:15:10 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T05:15:10.978Z] 2024-01-19 05:15:10 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Validating CR cleanup [2024-01-19T05:15:10.978Z] 2024-01-19 05:15:10 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T05:15:10.978Z] 2024-01-19 05:15:10 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[]] [2024-01-19T05:15:10.978Z] INFO[2024-01-19 05:15:10] Verifying : Description : Verifying creation of backup [tp-backup-manual-testuser1-1705641008-ugow] [2024-01-19T05:15:10.978Z] INFO[2024-01-19 05:15:10] Actual:true, Expected: true [2024-01-19T05:15:20.919Z] 2024-01-19 05:15:20 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.CreateBackup:#313] - Backup [tp-backup-manual-testuser1-1705641012-lcvr] created successfully [2024-01-19T05:15:20.919Z] 2024-01-19 05:15:20 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Validating backup [tp-backup-manual-testuser1-1705641012-lcvr] in org [default] [2024-01-19T05:15:20.919Z] 2024-01-19 05:15:20 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-manual-testuser1-1705641012-lcvr] [2024-01-19T05:15:20.919Z] 2024-01-19 05:15:20 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87565-01-19-05h08m34s] in backup [tp-backup-manual-testuser1-1705641012-lcvr] [2024-01-19T05:15:20.920Z] 2024-01-19 05:15:20 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87565-01-19-05h08m34s] in backup [tp-backup-manual-testuser1-1705641012-lcvr] [2024-01-19T05:15:20.920Z] 2024-01-19 05:15:20 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87565-01-19-05h08m34s] [2024-01-19T05:15:20.920Z] 2024-01-19 05:15:20 +0000:[WARNING] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T05:15:20.920Z] 2024-01-19 05:15:20 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T05:15:20.920Z] 2024-01-19 05:15:20 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[mysql-data]] [2024-01-19T05:15:20.920Z] 2024-01-19 05:15:20 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T05:15:20.920Z] 2024-01-19 05:15:20 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Validating CR cleanup [2024-01-19T05:15:20.920Z] 2024-01-19 05:15:20 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T05:15:20.920Z] 2024-01-19 05:15:20 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[]] [2024-01-19T05:15:20.920Z] INFO[2024-01-19 05:15:20] Verifying : Description : Verifying creation of backup [tp-backup-manual-testuser1-1705641012-lcvr] [2024-01-19T05:15:20.920Z] INFO[2024-01-19 05:15:20] Actual:true, Expected: true [2024-01-19T05:15:20.920Z] 2024-01-19 05:15:20 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Taking schedule backup of applications as non-admin user [2024-01-19T05:15:30.872Z] 2024-01-19 05:15:30 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Taking schedule backup of single namespace as user : testuser1-1705641006 without-rules [2024-01-19T05:15:30.872Z] 2024-01-19 05:15:30 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Creating a schedule backup of namespace [mysql-ibm-pxb-0-87565-01-19-05h08m34s] without pre and post exec rules [2024-01-19T05:15:30.872Z] 2024-01-19 05:15:30 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T05:15:30.872Z] 2024-01-19 05:15:30 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Volume snapshot classes found in the cluster - [2024-01-19T05:15:30.872Z] 2024-01-19 05:15:30 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] ibmc-vpcblock-snapshot [2024-01-19T05:15:30.872Z] 2024-01-19 05:15:30 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T05:15:40.819Z] 2024-01-19 05:15:40 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Taking schedule backup of single namespace as user : testuser1-1705641008 without-rules [2024-01-19T05:15:40.819Z] 2024-01-19 05:15:40 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Creating a schedule backup of namespace [mysql-ibm-pxb-0-87565-01-19-05h08m34s] without pre and post exec rules [2024-01-19T05:15:40.819Z] 2024-01-19 05:15:40 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T05:15:40.819Z] 2024-01-19 05:15:40 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Volume snapshot classes found in the cluster - [2024-01-19T05:15:40.819Z] 2024-01-19 05:15:40 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] ibmc-vpcblock-snapshot [2024-01-19T05:15:40.819Z] 2024-01-19 05:15:40 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T05:15:50.770Z] 2024-01-19 05:15:50 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Taking schedule backup of single namespace as user : testuser1-1705641012 without-rules [2024-01-19T05:15:50.771Z] 2024-01-19 05:15:50 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Creating a schedule backup of namespace [mysql-ibm-pxb-0-87565-01-19-05h08m34s] without pre and post exec rules [2024-01-19T05:15:50.771Z] 2024-01-19 05:15:50 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T05:15:50.771Z] 2024-01-19 05:15:50 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Volume snapshot classes found in the cluster - [2024-01-19T05:15:50.771Z] 2024-01-19 05:15:50 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] ibmc-vpcblock-snapshot [2024-01-19T05:15:50.771Z] 2024-01-19 05:15:50 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T05:16:37.391Z] 2024-01-19 05:16:31 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] first schedule backup for schedule name [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm] is [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531] [2024-01-19T05:16:37.391Z] 2024/01/19 05:16:31 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:16:41.557Z] 2024-01-19 05:16:41 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] first schedule backup for schedule name [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr] is [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541] [2024-01-19T05:16:41.557Z] 2024/01/19 05:16:41 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:16:51.502Z] 2024-01-19 05:16:51 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] first schedule backup for schedule name [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco] is [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551] [2024-01-19T05:16:51.502Z] 2024/01/19 05:16:51 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:17:03.671Z] 2024/01/19 05:17:01 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:17:11.774Z] 2024/01/19 05:17:11 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:17:21.718Z] 2024/01/19 05:17:21 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:17:31.675Z] 2024/01/19 05:17:31 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:17:41.648Z] 2024/01/19 05:17:41 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:17:51.586Z] 2024/01/19 05:17:51 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:18:01.535Z] 2024-01-19 05:18:01 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Validating backup [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531] in org [default] [2024-01-19T05:18:01.535Z] 2024-01-19 05:18:01 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531] [2024-01-19T05:18:01.790Z] 2024-01-19 05:18:01 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87565-01-19-05h08m34s] in backup [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531] [2024-01-19T05:18:01.790Z] 2024-01-19 05:18:01 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87565-01-19-05h08m34s] in backup [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531] [2024-01-19T05:18:01.790Z] 2024-01-19 05:18:01 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87565-01-19-05h08m34s] [2024-01-19T05:18:01.790Z] 2024-01-19 05:18:01 +0000:[WARNING] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T05:18:01.790Z] 2024-01-19 05:18:01 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T05:18:01.790Z] 2024-01-19 05:18:01 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[mysql-data]] [2024-01-19T05:18:01.790Z] 2024-01-19 05:18:01 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T05:18:01.790Z] 2024-01-19 05:18:01 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Validating CR cleanup [2024-01-19T05:18:02.047Z] 2024-01-19 05:18:01 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T05:18:02.047Z] 2024-01-19 05:18:01 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] [2024-01-19T05:18:02.047Z] 2024-01-19 05:18:01 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] under [mysql-ibm-pxb-0-87565-01-19-05h08m34s] namespace [2024-01-19T05:18:02.047Z] 2024/01/19 05:18:01 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531]}, Next try in [5s], timeout [5m0s] [2024-01-19T05:18:07.323Z] 2024-01-19 05:18:07 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] [2024-01-19T05:18:07.323Z] 2024-01-19 05:18:07 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] under [mysql-ibm-pxb-0-87565-01-19-05h08m34s] namespace [2024-01-19T05:18:07.323Z] 2024/01/19 05:18:07 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531]}, Next try in [5s], timeout [5m0s] [2024-01-19T05:18:11.493Z] 2024/01/19 05:18:11 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:18:12.422Z] 2024-01-19 05:18:12 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] [2024-01-19T05:18:12.422Z] 2024-01-19 05:18:12 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] under [mysql-ibm-pxb-0-87565-01-19-05h08m34s] namespace [2024-01-19T05:18:12.422Z] 2024/01/19 05:18:12 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531]}, Next try in [5s], timeout [5m0s] [2024-01-19T05:18:17.666Z] 2024-01-19 05:18:17 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] [2024-01-19T05:18:17.666Z] 2024-01-19 05:18:17 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] under [mysql-ibm-pxb-0-87565-01-19-05h08m34s] namespace [2024-01-19T05:18:17.666Z] 2024/01/19 05:18:17 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531]}, Next try in [5s], timeout [5m0s] [2024-01-19T05:18:21.834Z] 2024-01-19 05:18:21 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Validating backup [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551] in org [default] [2024-01-19T05:18:21.834Z] 2024-01-19 05:18:21 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551] [2024-01-19T05:18:21.834Z] 2024-01-19 05:18:21 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87565-01-19-05h08m34s] in backup [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551] [2024-01-19T05:18:21.834Z] 2024-01-19 05:18:21 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87565-01-19-05h08m34s] in backup [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551] [2024-01-19T05:18:21.834Z] 2024-01-19 05:18:21 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87565-01-19-05h08m34s] [2024-01-19T05:18:21.834Z] 2024-01-19 05:18:21 +0000:[WARNING] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T05:18:21.834Z] 2024-01-19 05:18:21 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T05:18:21.834Z] 2024-01-19 05:18:21 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[mysql-data]] [2024-01-19T05:18:21.834Z] 2024-01-19 05:18:21 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T05:18:21.834Z] 2024-01-19 05:18:21 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Validating CR cleanup [2024-01-19T05:18:21.834Z] 2024-01-19 05:18:21 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T05:18:21.834Z] 2024-01-19 05:18:21 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] [2024-01-19T05:18:21.834Z] 2024-01-19 05:18:21 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] under [mysql-ibm-pxb-0-87565-01-19-05h08m34s] namespace [2024-01-19T05:18:21.834Z] 2024/01/19 05:18:21 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]}, Next try in [5s], timeout [5m0s] [2024-01-19T05:18:22.399Z] 2024-01-19 05:18:22 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] [2024-01-19T05:18:22.399Z] 2024-01-19 05:18:22 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] under [mysql-ibm-pxb-0-87565-01-19-05h08m34s] namespace [2024-01-19T05:18:22.399Z] 2024/01/19 05:18:22 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531]}, Next try in [5s], timeout [5m0s] [2024-01-19T05:18:27.656Z] 2024-01-19 05:18:26 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] [2024-01-19T05:18:27.656Z] 2024-01-19 05:18:26 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] under [mysql-ibm-pxb-0-87565-01-19-05h08m34s] namespace [2024-01-19T05:18:27.656Z] 2024/01/19 05:18:26 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]}, Next try in [5s], timeout [5m0s] [2024-01-19T05:18:27.656Z] 2024-01-19 05:18:27 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] [2024-01-19T05:18:27.657Z] 2024-01-19 05:18:27 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] under [mysql-ibm-pxb-0-87565-01-19-05h08m34s] namespace [2024-01-19T05:18:27.657Z] 2024/01/19 05:18:27 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531]}, Next try in [5s], timeout [5m0s] [2024-01-19T05:18:31.825Z] 2024-01-19 05:18:31 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] [2024-01-19T05:18:31.825Z] 2024-01-19 05:18:31 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] under [mysql-ibm-pxb-0-87565-01-19-05h08m34s] namespace [2024-01-19T05:18:31.825Z] 2024/01/19 05:18:31 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]}, Next try in [5s], timeout [5m0s] [2024-01-19T05:18:32.386Z] 2024-01-19 05:18:32 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] [2024-01-19T05:18:32.386Z] 2024-01-19 05:18:32 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] under [mysql-ibm-pxb-0-87565-01-19-05h08m34s] namespace [2024-01-19T05:18:32.386Z] 2024/01/19 05:18:32 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531]}, Next try in [5s], timeout [5m0s] [2024-01-19T05:18:37.626Z] 2024-01-19 05:18:36 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] [2024-01-19T05:18:37.626Z] 2024-01-19 05:18:36 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] under [mysql-ibm-pxb-0-87565-01-19-05h08m34s] namespace [2024-01-19T05:18:37.626Z] 2024/01/19 05:18:36 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]}, Next try in [5s], timeout [5m0s] [2024-01-19T05:18:37.627Z] 2024-01-19 05:18:37 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] [2024-01-19T05:18:37.627Z] 2024-01-19 05:18:37 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] under [mysql-ibm-pxb-0-87565-01-19-05h08m34s] namespace [2024-01-19T05:18:37.627Z] 2024/01/19 05:18:37 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531]}, Next try in [5s], timeout [5m0s] [2024-01-19T05:18:41.796Z] 2024/01/19 05:18:41 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:18:41.796Z] 2024-01-19 05:18:41 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] [2024-01-19T05:18:41.796Z] 2024-01-19 05:18:41 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] under [mysql-ibm-pxb-0-87565-01-19-05h08m34s] namespace [2024-01-19T05:18:41.796Z] 2024/01/19 05:18:41 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]}, Next try in [5s], timeout [5m0s] [2024-01-19T05:18:42.726Z] 2024-01-19 05:18:42 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] [2024-01-19T05:18:42.726Z] 2024-01-19 05:18:42 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] under [mysql-ibm-pxb-0-87565-01-19-05h08m34s] namespace [2024-01-19T05:18:42.726Z] 2024/01/19 05:18:42 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531]}, Next try in [5s], timeout [5m0s] [2024-01-19T05:18:46.935Z] 2024-01-19 05:18:46 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] [2024-01-19T05:18:46.935Z] 2024-01-19 05:18:46 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] under [mysql-ibm-pxb-0-87565-01-19-05h08m34s] namespace [2024-01-19T05:18:46.935Z] 2024/01/19 05:18:46 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]}, Next try in [5s], timeout [5m0s] [2024-01-19T05:18:47.526Z] 2024-01-19 05:18:47 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] [2024-01-19T05:18:47.527Z] INFO[2024-01-19 05:18:47] Verifying : Description : Verifying creation of backup [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531] [2024-01-19T05:18:47.527Z] INFO[2024-01-19 05:18:47] Actual:true, Expected: true [2024-01-19T05:18:47.783Z] INFO[2024-01-19 05:18:47] Verifying : Description : Suspending Backup Schedule [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm] for user [testuser1-1705641006] [2024-01-19T05:18:47.783Z] INFO[2024-01-19 05:18:47] Actual:true, Expected: true [2024-01-19T05:18:51.952Z] 2024-01-19 05:18:51 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] [2024-01-19T05:18:51.952Z] 2024-01-19 05:18:51 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] under [mysql-ibm-pxb-0-87565-01-19-05h08m34s] namespace [2024-01-19T05:18:51.952Z] 2024/01/19 05:18:51 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]}, Next try in [5s], timeout [5m0s] [2024-01-19T05:18:57.206Z] 2024-01-19 05:18:56 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] [2024-01-19T05:18:57.206Z] 2024-01-19 05:18:56 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] under [mysql-ibm-pxb-0-87565-01-19-05h08m34s] namespace [2024-01-19T05:18:57.206Z] 2024/01/19 05:18:56 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]}, Next try in [5s], timeout [5m0s] [2024-01-19T05:19:02.454Z] 2024-01-19 05:19:01 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] [2024-01-19T05:19:02.454Z] 2024-01-19 05:19:01 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] under [mysql-ibm-pxb-0-87565-01-19-05h08m34s] namespace [2024-01-19T05:19:02.454Z] 2024/01/19 05:19:01 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]}, Next try in [5s], timeout [5m0s] [2024-01-19T05:19:07.699Z] 2024-01-19 05:19:07 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] [2024-01-19T05:19:07.699Z] 2024-01-19 05:19:07 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] under [mysql-ibm-pxb-0-87565-01-19-05h08m34s] namespace [2024-01-19T05:19:07.699Z] 2024/01/19 05:19:07 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]}, Next try in [5s], timeout [5m0s] [2024-01-19T05:19:11.863Z] 2024/01/19 05:19:11 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:19:12.119Z] 2024-01-19 05:19:12 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] [2024-01-19T05:19:12.119Z] 2024-01-19 05:19:12 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] under [mysql-ibm-pxb-0-87565-01-19-05h08m34s] namespace [2024-01-19T05:19:12.119Z] 2024/01/19 05:19:12 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]}, Next try in [5s], timeout [5m0s] [2024-01-19T05:19:17.376Z] 2024-01-19 05:19:17 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] [2024-01-19T05:19:17.376Z] 2024-01-19 05:19:17 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] under [mysql-ibm-pxb-0-87565-01-19-05h08m34s] namespace [2024-01-19T05:19:17.376Z] 2024/01/19 05:19:17 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]}, Next try in [5s], timeout [5m0s] [2024-01-19T05:19:22.632Z] 2024-01-19 05:19:22 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] [2024-01-19T05:19:22.632Z] 2024-01-19 05:19:22 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] CR found for [[tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] under [mysql-ibm-pxb-0-87565-01-19-05h08m34s] namespace [2024-01-19T05:19:22.632Z] 2024/01/19 05:19:22 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]}, Next try in [5s], timeout [5m0s] [2024-01-19T05:19:27.878Z] 2024-01-19 05:19:27 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] All CRs in [mysql-ibm-pxb-0-87565-01-19-05h08m34s] are [[tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541]] [2024-01-19T05:19:27.878Z] INFO[2024-01-19 05:19:27] Verifying : Description : Verifying creation of backup [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551] [2024-01-19T05:19:27.878Z] INFO[2024-01-19 05:19:27] Actual:true, Expected: true [2024-01-19T05:19:27.878Z] INFO[2024-01-19 05:19:27] Verifying : Description : Suspending Backup Schedule [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco] for user [testuser1-1705641012] [2024-01-19T05:19:27.878Z] INFO[2024-01-19 05:19:27] Actual:true, Expected: true [2024-01-19T05:19:42.713Z] 2024/01/19 05:19:41 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:20:14.759Z] 2024/01/19 05:20:11 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:20:46.808Z] 2024/01/19 05:20:41 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:21:13.314Z] INFO[2024-01-19 05:21:11] Verifying : Description : Verifying creation of backup [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541] [2024-01-19T05:21:13.314Z] ERRO[2024-01-19 05:21:11] backup status for [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541] expected was [[Success]] but got [Failed] because of [snapshot timeout out after 5m0s] [2024-01-19T05:21:13.314Z] 2024-01-19 05:21:11 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Sharing of backup of app-user user with app-admin and infra-admin user [2024-01-19T05:21:13.314Z] 2024-01-19 05:21:11 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.glob..func34.2.11:#3495] - Sharing app-user [testuser1-1705641012] backups [[tp-backup-manual-testuser1-1705641012-lcvr tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551]] with app-admin and infra-admin [2024-01-19T05:21:13.314Z] 2024-01-19 05:21:11 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.ShareBackup:#715] - Backup UID for tp-backup-manual-testuser1-1705641012-lcvr - 3e017c88-af7f-4535-b65f-b48394075def [2024-01-19T05:21:13.314Z] 2024-01-19 05:21:11 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705641006 - 8b3d5ca3-d403-499b-9712-afd37bc55594 [2024-01-19T05:21:13.314Z] 2024-01-19 05:21:11 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705641008 - 7ead8c09-d3d3-4fa5-bf3e-e6576a1154e5 [2024-01-19T05:21:13.314Z] INFO[2024-01-19 05:21:12] Verifying : Description : Verification of sharing backup [tp-backup-manual-testuser1-1705641012-lcvr] of user [testuser1-1705641012] [2024-01-19T05:21:13.315Z] INFO[2024-01-19 05:21:12] Actual:true, Expected: true [2024-01-19T05:21:13.315Z] 2024-01-19 05:21:12 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.ShareBackup:#715] - Backup UID for tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551 - c9ca779a-6c6f-49d2-9366-74b2e17d06de [2024-01-19T05:21:13.315Z] 2024-01-19 05:21:12 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705641006 - 8b3d5ca3-d403-499b-9712-afd37bc55594 [2024-01-19T05:21:13.315Z] 2024-01-19 05:21:13 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705641008 - 7ead8c09-d3d3-4fa5-bf3e-e6576a1154e5 [2024-01-19T05:21:13.315Z] INFO[2024-01-19 05:21:13] Verifying : Description : Verification of sharing backup [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551] of user [testuser1-1705641012] [2024-01-19T05:21:13.315Z] INFO[2024-01-19 05:21:13] Actual:true, Expected: true [2024-01-19T05:21:13.315Z] 2024-01-19 05:21:13 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Sharing of backup of app-admin user with app-user and infra-admin user [2024-01-19T05:21:13.573Z] 2024-01-19 05:21:13 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.glob..func34.2.12:#3506] - Sharing app-admin user [testuser1-1705641006] backups [[tp-backup-manual-testuser1-1705641006-lfdm tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531]] with app-user and infra-admin [2024-01-19T05:21:13.573Z] 2024-01-19 05:21:13 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.ShareBackup:#715] - Backup UID for tp-backup-manual-testuser1-1705641006-lfdm - f7d9c74b-9c09-4301-b337-d4d66d2f0784 [2024-01-19T05:21:14.508Z] 2024-01-19 05:21:14 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705641012 - dd66b4af-edba-49c4-b40f-f2328ceeb36d [2024-01-19T05:21:15.073Z] 2024-01-19 05:21:15 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705641008 - 7ead8c09-d3d3-4fa5-bf3e-e6576a1154e5 [2024-01-19T05:21:15.329Z] INFO[2024-01-19 05:21:15] Verifying : Description : Verification of sharing backup [tp-backup-manual-testuser1-1705641006-lfdm] of user [testuser1-1705641006] [2024-01-19T05:21:15.329Z] INFO[2024-01-19 05:21:15] Actual:true, Expected: true [2024-01-19T05:21:15.329Z] 2024-01-19 05:21:15 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.ShareBackup:#715] - Backup UID for tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 - e444f85d-24c6-43cc-b590-ce969988cda1 [2024-01-19T05:21:15.889Z] 2024-01-19 05:21:15 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705641012 - dd66b4af-edba-49c4-b40f-f2328ceeb36d [2024-01-19T05:21:16.817Z] 2024-01-19 05:21:16 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705641008 - 7ead8c09-d3d3-4fa5-bf3e-e6576a1154e5 [2024-01-19T05:21:16.817Z] INFO[2024-01-19 05:21:16] Verifying : Description : Verification of sharing backup [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531] of user [testuser1-1705641006] [2024-01-19T05:21:16.817Z] INFO[2024-01-19 05:21:16] Actual:true, Expected: true [2024-01-19T05:21:16.817Z] 2024-01-19 05:21:16 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Sharing of backup of infra-admin user with app-user and app-admin user [2024-01-19T05:21:17.379Z] 2024-01-19 05:21:17 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.glob..func34.2.13:#3517] - Sharing infra-admin user [testuser1-1705641008] backups [[tp-backup-manual-testuser1-1705641008-ugow]] with app-admin and app-user [2024-01-19T05:21:17.379Z] 2024-01-19 05:21:17 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.ShareBackup:#715] - Backup UID for tp-backup-manual-testuser1-1705641008-ugow - ad1ac77e-d277-4924-b732-26c37d1c5c7d [2024-01-19T05:21:17.939Z] 2024-01-19 05:21:17 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705641006 - 8b3d5ca3-d403-499b-9712-afd37bc55594 [2024-01-19T05:21:18.864Z] 2024-01-19 05:21:18 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705641012 - dd66b4af-edba-49c4-b40f-f2328ceeb36d [2024-01-19T05:21:18.864Z] INFO[2024-01-19 05:21:18] Verifying : Description : Verification of sharing backup [tp-backup-manual-testuser1-1705641008-ugow] of user [testuser1-1705641008] [2024-01-19T05:21:18.864Z] INFO[2024-01-19 05:21:18] Actual:true, Expected: true [2024-01-19T05:21:18.864Z] 2024-01-19 05:21:18 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Listing and Deletion of shared backup of non-admin users from px-admin user [2024-01-19T05:21:20.227Z] 2024-01-19 05:21:19 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Verifying listing backups of non-admin user [testuser1-1705641006] from px-admin user [2024-01-19T05:21:20.228Z] INFO[2024-01-19 05:21:20] Verifying : Description : Verification of fetching backups [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-manual-testuser1-1705641006-lfdm]] of user [testuser1-1705641006] from px-admin user [2024-01-19T05:21:20.228Z] INFO[2024-01-19 05:21:20] Actual:true, Expected: true [2024-01-19T05:21:20.228Z] 2024-01-19 05:21:20 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.glob..func34.2.14:#3533] - The list of user [testuser1-1705641006] backups from px-admin [[tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531 tp-backup-manual-testuser1-1705641006-lfdm]] [2024-01-19T05:21:20.228Z] 2024-01-19 05:21:20 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Verifying listing backups of non-admin user [testuser1-1705641008] from px-admin user [2024-01-19T05:21:20.228Z] INFO[2024-01-19 05:21:20] Verifying : Description : Verification of fetching backups [[tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-manual-testuser1-1705641008-ugow]] of user [testuser1-1705641008] from px-admin user [2024-01-19T05:21:20.228Z] INFO[2024-01-19 05:21:20] Actual:true, Expected: true [2024-01-19T05:21:20.228Z] 2024-01-19 05:21:20 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.glob..func34.2.14:#3533] - The list of user [testuser1-1705641008] backups from px-admin [[tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541 tp-backup-manual-testuser1-1705641008-ugow]] [2024-01-19T05:21:20.228Z] 2024-01-19 05:21:20 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Verifying listing backups of non-admin user [testuser1-1705641012] from px-admin user [2024-01-19T05:21:20.228Z] INFO[2024-01-19 05:21:20] Verifying : Description : Verification of fetching backups [[tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551 tp-backup-manual-testuser1-1705641012-lcvr]] of user [testuser1-1705641012] from px-admin user [2024-01-19T05:21:20.228Z] INFO[2024-01-19 05:21:20] Actual:true, Expected: true [2024-01-19T05:21:20.228Z] 2024-01-19 05:21:20 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.glob..func34.2.14:#3533] - The list of user [testuser1-1705641012] backups from px-admin [[tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551 tp-backup-manual-testuser1-1705641012-lcvr]] [2024-01-19T05:21:20.228Z] 2024-01-19 05:21:20 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Verifying deletion backup [tp-backup-manual-testuser1-1705641012-lcvr] of non-admin user [testuser1-1705641012] from px-admin user [2024-01-19T05:21:20.228Z] 2024-01-19 05:21:20 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Verifying deletion backup [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531] of non-admin user [testuser1-1705641006] from px-admin user [2024-01-19T05:21:20.228Z] 2024-01-19 05:21:20 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Verifying deletion backup [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551] of non-admin user [testuser1-1705641012] from px-admin user [2024-01-19T05:21:20.228Z] 2024-01-19 05:21:20 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Verifying deletion backup [tp-backup-manual-testuser1-1705641008-ugow] of non-admin user [testuser1-1705641008] from px-admin user [2024-01-19T05:21:20.228Z] 2024-01-19 05:21:20 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Verifying deletion backup [tp-backup-manual-testuser1-1705641006-lfdm] of non-admin user [testuser1-1705641006] from px-admin user [2024-01-19T05:21:20.229Z] 2024-01-19 05:21:20 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Verifying deletion backup [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541] of non-admin user [testuser1-1705641008] from px-admin user [2024-01-19T05:21:20.484Z] 2024/01/19 05:21:20 DoRetryWithTimeout - Error: {backupObject [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:21:20.485Z] 2024/01/19 05:21:20 DoRetryWithTimeout - Error: {backupObject [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:21:20.485Z] 2024/01/19 05:21:20 DoRetryWithTimeout - Error: {backupObject [tp-backup-manual-testuser1-1705641008-ugow] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:21:20.485Z] 2024/01/19 05:21:20 DoRetryWithTimeout - Error: {backupObject [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:21:20.485Z] 2024/01/19 05:21:20 DoRetryWithTimeout - Error: {backupObject [tp-backup-manual-testuser1-1705641012-lcvr] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:21:20.485Z] 2024/01/19 05:21:20 DoRetryWithTimeout - Error: {backupObject [tp-backup-manual-testuser1-1705641006-lfdm] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:21:52.506Z] 2024/01/19 05:21:50 DoRetryWithTimeout - Error: {backupObject [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:21:52.506Z] 2024/01/19 05:21:50 DoRetryWithTimeout - Error: {backupObject [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:21:52.506Z] 2024/01/19 05:21:50 DoRetryWithTimeout - Error: {backupObject [tp-backup-manual-testuser1-1705641008-ugow] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:21:52.507Z] 2024/01/19 05:21:50 DoRetryWithTimeout - Error: {backupObject [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:21:52.507Z] 2024/01/19 05:21:50 DoRetryWithTimeout - Error: {backupObject [tp-backup-manual-testuser1-1705641012-lcvr] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:21:52.507Z] 2024/01/19 05:21:50 DoRetryWithTimeout - Error: {backupObject [tp-backup-manual-testuser1-1705641006-lfdm] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:22:24.534Z] 2024/01/19 05:22:20 DoRetryWithTimeout - Error: {backupObject [tp-backup-manual-testuser1-1705641008-ugow] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:22:24.534Z] 2024/01/19 05:22:20 DoRetryWithTimeout - Error: {backupObject [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm-b710593-interval-2024-01-19-051531] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:22:24.534Z] 2024/01/19 05:22:20 DoRetryWithTimeout - Error: {backupObject [tp-backup-manual-testuser1-1705641012-lcvr] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:22:24.534Z] 2024/01/19 05:22:20 DoRetryWithTimeout - Error: {backupObject [tp-backup-manual-testuser1-1705641006-lfdm] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:22:24.534Z] 2024/01/19 05:22:20 DoRetryWithTimeout - Error: {backupObject [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco-37d2c01-interval-2024-01-19-051551] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:22:24.534Z] 2024/01/19 05:22:20 DoRetryWithTimeout - Error: {backupObject [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:22:51.036Z] 2024-01-19 05:22:50 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Deleting the deployed apps after the testcase [2024-01-19T05:22:51.036Z] 2024-01-19 05:22:50 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Destroying apps [2024-01-19T05:22:51.036Z] 2024-01-19 05:22:50 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T05:22:51.036Z] 2024-01-19 05:22:50 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T05:22:51.036Z] 2024-01-19 05:22:50 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] start destroying mysql-ibm app [2024-01-19T05:22:51.036Z] 2024-01-19 05:22:50 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T05:22:51.036Z] 2024-01-19 05:22:50 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T05:22:51.036Z] 2024-01-19 05:22:50 +0000:[WARNING] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T05:22:51.036Z] INFO[2024-01-19 05:22:50] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T05:22:51.036Z] INFO[2024-01-19 05:22:50] Actual:true, Expected: true [2024-01-19T05:22:51.036Z] INFO[2024-01-19 05:22:50] Verifying : Description : Verification of fetching backup schedules of user [2024-01-19T05:22:51.036Z] INFO[2024-01-19 05:22:50] Actual:true, Expected: true [2024-01-19T05:22:51.292Z] INFO[2024-01-19 05:22:51] Verifying : Description : Verification of fetching backup schedules of user [2024-01-19T05:22:51.292Z] INFO[2024-01-19 05:22:51] Actual:true, Expected: true [2024-01-19T05:22:51.292Z] INFO[2024-01-19 05:22:51] Verifying : Description : Verification of fetching backup schedules of user [2024-01-19T05:22:51.292Z] INFO[2024-01-19 05:22:51] Actual:true, Expected: true [2024-01-19T05:22:51.548Z] 2024-01-19 05:22:51 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Verifying deletion of backup schedule [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm] of non-admin user [testuser1-1705641006] [2024-01-19T05:22:51.548Z] 2024/01/19 05:22:51 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T05:22:51.548Z] 2024-01-19 05:22:51 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Verifying deletion of backup schedule [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr] of non-admin user [testuser1-1705641008] [2024-01-19T05:22:51.806Z] 2024/01/19 05:22:51 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T05:22:51.806Z] 2024-01-19 05:22:51 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Verifying deletion of backup schedule [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco] of non-admin user [testuser1-1705641012] [2024-01-19T05:22:51.807Z] 2024/01/19 05:22:51 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T05:23:01.751Z] 2024/01/19 05:23:01 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T05:23:01.751Z] 2024/01/19 05:23:01 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T05:23:01.751Z] 2024/01/19 05:23:01 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T05:23:11.690Z] 2024/01/19 05:23:11 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T05:23:11.690Z] 2024/01/19 05:23:11 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T05:23:11.946Z] 2024/01/19 05:23:11 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T05:23:21.886Z] 2024/01/19 05:23:21 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T05:23:21.886Z] 2024/01/19 05:23:21 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T05:23:21.886Z] 2024/01/19 05:23:21 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T05:23:31.830Z] INFO[2024-01-19 05:23:31] Verifying : Description : Verification of deleting backup scheudle [tp-backup-schedule-single-ns-testuser1-1705641006-without-rules-bdwm] of user [testuser1-1705641006] from px-admin user [2024-01-19T05:23:31.830Z] INFO[2024-01-19 05:23:31] Actual:true, Expected: true [2024-01-19T05:23:31.830Z] INFO[2024-01-19 05:23:31] Verifying : Description : Verification of deleting backup scheudle [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr] of user [testuser1-1705641008] from px-admin user [2024-01-19T05:23:31.830Z] INFO[2024-01-19 05:23:31] Actual:true, Expected: true [2024-01-19T05:23:31.830Z] INFO[2024-01-19 05:23:31] Verifying : Description : Verification of deleting backup scheudle [tp-backup-schedule-single-ns-testuser1-1705641012-without-rules-frco] of user [testuser1-1705641012] from px-admin user [2024-01-19T05:23:31.830Z] INFO[2024-01-19 05:23:31] Actual:true, Expected: true [2024-01-19T05:23:31.830Z] 2024-01-19 05:23:31 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.glob..func34.3:#3593] - Deleting backup schedule policy [2024-01-19T05:23:32.390Z] INFO[2024-01-19 05:23:32] Verifying : Description : Deleting backup schedule policies [periodic-iyybk] [2024-01-19T05:23:32.390Z] INFO[2024-01-19 05:23:32] Actual:true, Expected: true [2024-01-19T05:23:32.956Z] 2024-01-19 05:23:32 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Cleaning backup locations in map [map[805a6978-2fdc-4fd0-9ad3-4f4760503663:autogenerated-backup-location-mcqju b76bb80a-1bb9-4d55-8b49-ffed029844bd:autogenerated-backup-location-testuser1-1705641008-oeclt f90a71f2-dcf1-44d3-8feb-970d9e8c244e:autogenerated-backup-location-testuser1-1705641006-jlnkb]], cloud credential [autogenerated-cred-tbjjo], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T05:23:32.956Z] INFO[2024-01-19 05:23:32] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-mcqju] [2024-01-19T05:23:32.956Z] INFO[2024-01-19 05:23:32] Actual:true, Expected: true [2024-01-19T05:23:32.956Z] 2024-01-19 05:23:32 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-mcqju] is present [2024-01-19T05:23:33.213Z] 2024/01/19 05:23:32 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-mcqju is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:24:05.234Z] 2024-01-19 05:24:03 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-mcqju] is present [2024-01-19T05:24:05.234Z] 2024/01/19 05:24:03 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-mcqju is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:24:37.263Z] 2024-01-19 05:24:33 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 aws-global-aws-px-backup-system-test-4cac76dc-bl autogenerated-backup-location-1705628002 autogenerated-backup-location-1705634221 autogenerated-backup-location-1705635982 autogenerated-backup-location-wkyfv autogenerated-backup-location-testuser1-1705641006-jlnkb autogenerated-backup-location-testuser1-1705641008-oeclt] [2024-01-19T05:24:37.263Z] INFO[2024-01-19 05:24:33] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-mcqju [2024-01-19T05:24:37.263Z] INFO[2024-01-19 05:24:33] Actual:true, Expected: true [2024-01-19T05:24:37.263Z] INFO[2024-01-19 05:24:33] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-testuser1-1705641006-jlnkb] [2024-01-19T05:24:37.263Z] INFO[2024-01-19 05:24:33] Actual:true, Expected: true [2024-01-19T05:24:37.263Z] 2024-01-19 05:24:33 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-testuser1-1705641006-jlnkb] is present [2024-01-19T05:24:37.263Z] 2024/01/19 05:24:33 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-testuser1-1705641006-jlnkb is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:24:59.143Z] INFO[2024-01-19 05:24:58] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T05:24:59.143Z] INFO[2024-01-19 05:24:58] sched stats: tick interval: 49858 < 1.2s (100%) [2024-01-19T05:24:59.143Z] INFO[2024-01-19 05:24:58] sched stats: tick process durations: 49857 < 500ms (100%) [2024-01-19T05:25:04.388Z] 2024-01-19 05:25:03 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-testuser1-1705641006-jlnkb] is present [2024-01-19T05:25:04.388Z] 2024/01/19 05:25:03 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-testuser1-1705641006-jlnkb is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:25:36.417Z] 2024-01-19 05:25:33 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 aws-global-aws-px-backup-system-test-4cac76dc-bl autogenerated-backup-location-1705628002 autogenerated-backup-location-1705634221 autogenerated-backup-location-1705635982 autogenerated-backup-location-wkyfv autogenerated-backup-location-testuser1-1705641008-oeclt] [2024-01-19T05:25:36.417Z] INFO[2024-01-19 05:25:33] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-testuser1-1705641006-jlnkb [2024-01-19T05:25:36.417Z] INFO[2024-01-19 05:25:33] Actual:true, Expected: true [2024-01-19T05:25:36.417Z] INFO[2024-01-19 05:25:33] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-testuser1-1705641008-oeclt] [2024-01-19T05:25:36.417Z] INFO[2024-01-19 05:25:33] Actual:true, Expected: true [2024-01-19T05:25:36.418Z] 2024-01-19 05:25:33 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-testuser1-1705641008-oeclt] is present [2024-01-19T05:25:36.418Z] 2024/01/19 05:25:33 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-testuser1-1705641008-oeclt is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:26:08.456Z] 2024-01-19 05:26:03 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-testuser1-1705641008-oeclt] is present [2024-01-19T05:26:08.456Z] 2024/01/19 05:26:03 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-testuser1-1705641008-oeclt is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:26:34.987Z] 2024-01-19 05:26:33 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 aws-global-aws-px-backup-system-test-4cac76dc-bl autogenerated-backup-location-1705628002 autogenerated-backup-location-1705634221 autogenerated-backup-location-1705635982 autogenerated-backup-location-wkyfv] [2024-01-19T05:26:34.987Z] INFO[2024-01-19 05:26:33] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-testuser1-1705641008-oeclt [2024-01-19T05:26:34.987Z] INFO[2024-01-19 05:26:33] Actual:true, Expected: true [2024-01-19T05:26:34.987Z] 2024-01-19 05:26:34 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-tbjjo] is present [2024-01-19T05:26:34.987Z] INFO[2024-01-19 05:26:34] Verifying : Description : Verifying if cloud cred [autogenerated-cred-tbjjo] is present [2024-01-19T05:26:34.987Z] INFO[2024-01-19 05:26:34] Actual:true, Expected: true [2024-01-19T05:26:34.987Z] INFO[2024-01-19 05:26:34] Verifying : Description : Verifying deletion of cloud cred [autogenerated-cred-tbjjo] [2024-01-19T05:26:34.987Z] INFO[2024-01-19 05:26:34] Actual:true, Expected: true [2024-01-19T05:26:34.987Z] INFO[2024-01-19 05:26:34] Verifying : Description : Deleting cloud cred autogenerated-cred-tbjjo [2024-01-19T05:26:34.987Z] INFO[2024-01-19 05:26:34] Actual:true, Expected: true [2024-01-19T05:26:34.987Z] INFO[2024-01-19 05:26:34] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T05:26:34.987Z] INFO[2024-01-19 05:26:34] Actual:true, Expected: true [2024-01-19T05:26:34.987Z] INFO[2024-01-19 05:26:34] Verifying : Description : Deleting cluster source-cluster [2024-01-19T05:26:34.987Z] ERRO[2024-01-19 05:26:34] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backup [tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-052605], cannot be deleted, Expected:  [2024-01-19T05:26:34.987Z] INFO[2024-01-19 05:26:34] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-hrfiy] [2024-01-19T05:26:34.987Z] ERRO[2024-01-19 05:26:34] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-hrfiy:72b1b409-7bb1-4f22-92df-0ba5e77c11d3], Expected:  [2024-01-19T05:26:34.987Z] INFO[2024-01-19 05:26:34] Verifying : Description : Deleting cluster source-cluster [2024-01-19T05:26:34.987Z] ERRO[2024-01-19 05:26:34] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh], cannot be deleted, Expected:  [2024-01-19T05:26:35.245Z] INFO[2024-01-19 05:26:35] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-fbvvo] [2024-01-19T05:26:35.245Z] ERRO[2024-01-19 05:26:35] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-fbvvo:7cddb5aa-c7ff-4044-9d27-458be2bd5cde], Expected:  [2024-01-19T05:26:35.502Z] INFO[2024-01-19 05:26:35] Verifying : Description : Deleting cluster source-cluster [2024-01-19T05:26:35.502Z] ERRO[2024-01-19 05:26:35] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backup [tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-052626], cannot be deleted, Expected:  [2024-01-19T05:26:36.877Z] INFO[2024-01-19 05:26:36] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-enzij] [2024-01-19T05:26:36.877Z] ERRO[2024-01-19 05:26:36] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-enzij:4e0ddfc7-8811-4ed5-8e87-ed0841582111], Expected:  [2024-01-19T05:26:36.877Z] INFO[2024-01-19 05:26:36] Verifying : Description : Deleting cluster source-cluster [2024-01-19T05:26:36.877Z] ERRO[2024-01-19 05:26:36] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backup [tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-052636], cannot be deleted, Expected:  [2024-01-19T05:26:38.245Z] INFO[2024-01-19 05:26:37] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-wcpfm] [2024-01-19T05:26:38.245Z] ERRO[2024-01-19 05:26:37] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-wcpfm:c5ff084a-a913-4412-99fb-e2b0a003a014], Expected:  [2024-01-19T05:26:38.245Z] INFO[2024-01-19 05:26:38] Verifying : Description : Deleting cluster source-cluster [2024-01-19T05:26:38.245Z] ERRO[2024-01-19 05:26:38] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx], cannot be deleted, Expected:  [2024-01-19T05:26:39.614Z] INFO[2024-01-19 05:26:39] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-vlbwu] [2024-01-19T05:26:39.614Z] ERRO[2024-01-19 05:26:39] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-vlbwu:f0465bb5-2565-4948-bb27-7e45a4b883ff], Expected:  [2024-01-19T05:26:39.614Z] INFO[2024-01-19 05:26:39] Verifying : Description : Deleting cluster source-cluster [2024-01-19T05:26:39.614Z] ERRO[2024-01-19 05:26:39] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa], cannot be deleted, Expected:  [2024-01-19T05:26:40.980Z] INFO[2024-01-19 05:26:40] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-mhxzi] [2024-01-19T05:26:40.980Z] ERRO[2024-01-19 05:26:40] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-mhxzi:f551fb81-1a78-4d7f-b804-759c6e6b562e], Expected:  [2024-01-19T05:26:41.235Z] INFO[2024-01-19 05:26:41] Verifying : Description : Deleting cluster source-cluster [2024-01-19T05:26:41.235Z] INFO[2024-01-19 05:26:41] Actual:true, Expected: true [2024-01-19T05:26:42.597Z] INFO[2024-01-19 05:26:42] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-smmgg] [2024-01-19T05:26:42.597Z] INFO[2024-01-19 05:26:42] Actual:true, Expected: true [2024-01-19T05:26:42.597Z] INFO[2024-01-19 05:26:42] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T05:26:42.598Z] INFO[2024-01-19 05:26:42] Actual:true, Expected: true [2024-01-19T05:26:43.959Z] INFO[2024-01-19 05:26:43] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-ymjvy] [2024-01-19T05:26:43.959Z] INFO[2024-01-19 05:26:43] Actual:true, Expected: true [2024-01-19T05:26:43.959Z] INFO[2024-01-19 05:26:43] Verifying : Description : Deleting cluster source-cluster [2024-01-19T05:26:43.959Z] INFO[2024-01-19 05:26:43] Actual:true, Expected: true [2024-01-19T05:26:45.334Z] INFO[2024-01-19 05:26:45] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-nimwm] [2024-01-19T05:26:45.334Z] INFO[2024-01-19 05:26:45] Actual:true, Expected: true [2024-01-19T05:26:45.590Z] INFO[2024-01-19 05:26:45] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T05:26:45.590Z] INFO[2024-01-19 05:26:45] Actual:true, Expected: true [2024-01-19T05:26:46.514Z] INFO[2024-01-19 05:26:46] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-nmuvh] [2024-01-19T05:26:46.514Z] INFO[2024-01-19 05:26:46] Actual:true, Expected: true [2024-01-19T05:26:46.769Z] INFO[2024-01-19 05:26:46] Verifying : Description : Deleting cluster source-cluster [2024-01-19T05:26:46.770Z] INFO[2024-01-19 05:26:46] Actual:true, Expected: true [2024-01-19T05:26:48.132Z] INFO[2024-01-19 05:26:47] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-yeitr] [2024-01-19T05:26:48.132Z] INFO[2024-01-19 05:26:47] Actual:true, Expected: true [2024-01-19T05:26:48.132Z] INFO[2024-01-19 05:26:48] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T05:26:48.132Z] INFO[2024-01-19 05:26:48] Actual:true, Expected: true [2024-01-19T05:26:49.498Z] INFO[2024-01-19 05:26:49] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-ndrog] [2024-01-19T05:26:49.498Z] INFO[2024-01-19 05:26:49] Actual:true, Expected: true [2024-01-19T05:26:49.498Z] INFO[2024-01-19 05:26:49] --------Test End------ [2024-01-19T05:26:49.498Z] INFO[2024-01-19 05:26:49] #Test: [2024-01-19T05:26:49.498Z] INFO[2024-01-19 05:26:49] #Description: [2024-01-19T05:26:49.498Z] INFO[2024-01-19 05:26:49] ------------------------ [2024-01-19T05:26:49.498Z] 2024-01-19 05:26:49 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.EndPxBackupTorpedoTest:#7261] - >>>> FAILED TEST: {DeleteBackupSharedByMultipleUsersFromAdmin} Delete backups of non admin user from px-admin [2024-01-19T05:26:49.498Z] 2024-01-19 05:26:49 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-87565-01-19-05h08m34s] [2024-01-19T05:26:49.755Z] 2024-01-19 05:26:49 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T05:26:49.755Z] 2024-01-19 05:26:49 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T05:26:50.012Z] 2024-01-19 05:26:49 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-87565-01-19-05h08m34s] [2024-01-19T05:26:50.012Z] 2024-01-19 05:26:49 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {DeleteBackupSharedByMultipleUsersFromAdmin} Delete backups of non admin user from px-admin [2024-01-19T05:26:50.012Z] 2024-01-19 05:26:49 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T05:26:50.012Z] 2024/01/19 05:26:49 DoRetryWithTimeout - Error: {dial tcp 10.240.64.27:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:26:55.289Z] 2024/01/19 05:26:54 DoRetryWithTimeout - Error: {dial tcp 10.240.64.27:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:27:00.538Z] 2024/01/19 05:26:59 DoRetryWithTimeout - Error: {dial tcp 10.240.64.27:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:27:05.791Z] 2024/01/19 05:27:04 DoRetryWithTimeout - Error: {dial tcp 10.240.64.27:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:27:09.963Z] 2024/01/19 05:27:09 DoRetryWithTimeout - Error: {dial tcp 10.240.64.27:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:27:15.226Z] 2024/01/19 05:27:14 DoRetryWithTimeout - Error: {dial tcp 10.240.64.27:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:27:20.474Z] 2024/01/19 05:27:19 DoRetryWithTimeout - Error: {dial tcp 10.240.64.27:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:27:25.732Z] 2024/01/19 05:27:24 DoRetryWithTimeout - Error: {dial tcp 10.240.64.27:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:27:30.984Z] 2024/01/19 05:27:29 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.64.27. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.27 10.240.64.27]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:27:30.984Z] {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:27:30.984Z] {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:27:30.984Z] {dial tcp 10.240.64.27:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T05:27:35.153Z] 2024-01-19 05:27:34 +0000:[WARNING] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.64.27. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.27 10.240.64.27]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:27:35.153Z] {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:27:35.153Z] {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:27:35.153Z] {dial tcp 10.240.64.27:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T05:27:35.153Z] 2024-01-19 05:27:34 +0000:[ERROR] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.64.27. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.27 10.240.64.27]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:27:35.153Z] {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:27:35.153Z] {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:27:35.153Z] {dial tcp 10.240.64.27:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T05:27:35.153Z] 2024-01-19 05:27:34 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T05:27:35.153Z] 2024-01-19 05:27:34 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] Switching context to [/tmp/source-config] [2024-01-19T05:27:35.153Z] 2024-01-19 05:27:35 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T05:27:35.154Z] 2024-01-19 05:27:35 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T05:27:35.154Z] 2024-01-19 05:27:35 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T05:27:35.409Z] 2024-01-19 05:27:35 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T05:27:35.409Z] 2024-01-19 05:27:35 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T05:27:35.409Z] 2024-01-19 05:27:35 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T05:27:35.970Z] 2024-01-19 05:27:35 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T05:27:36.226Z] 2024-01-19 05:27:36 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T05:27:36.792Z] 2024-01-19 05:27:36 +0000:[INFO] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T05:27:36.792Z] 2024/01/19 05:27:36 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:27:48.959Z] 2024/01/19 05:27:46 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:27:57.056Z] 2024/01/19 05:27:56 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:28:07.005Z] 2024/01/19 05:28:06 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:28:16.947Z] 2024/01/19 05:28:16 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:28:29.117Z] 2024/01/19 05:28:26 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:28:37.227Z] 2024/01/19 05:28:36 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:28:47.203Z] 2024/01/19 05:28:46 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:28:57.164Z] 2024/01/19 05:28:56 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:29:07.116Z] 2024/01/19 05:29:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:29:19.282Z] 2024/01/19 05:29:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:29:28.574Z] 2024/01/19 05:29:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:29:38.512Z] 2024/01/19 05:29:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:29:48.449Z] 2024/01/19 05:29:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:29:58.483Z] 2024/01/19 05:29:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:30:08.421Z] 2024/01/19 05:30:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:30:18.364Z] 2024/01/19 05:30:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:30:28.310Z] 2024/01/19 05:30:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:30:38.253Z] 2024/01/19 05:30:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:30:48.194Z] 2024/01/19 05:30:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:30:58.136Z] 2024/01/19 05:30:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:31:08.082Z] 2024/01/19 05:31:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:31:18.020Z] 2024/01/19 05:31:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:31:28.161Z] 2024/01/19 05:31:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:31:38.105Z] 2024/01/19 05:31:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:31:48.046Z] 2024/01/19 05:31:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:31:57.985Z] 2024/01/19 05:31:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:32:07.927Z] 2024/01/19 05:32:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:32:17.873Z] 2024/01/19 05:32:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:32:27.833Z] 2024/01/19 05:32:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:32:37.774Z] 2024-01-19 05:32:37 +0000:[ERROR] [{DeleteBackupSharedByMultipleUsersFromAdmin}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.774Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.774Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.774Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.774Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.774Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.774Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.775Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.775Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.775Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.775Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.775Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.775Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.775Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.775Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.775Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.775Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.775Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.775Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.775Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.775Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.775Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.775Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.775Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.775Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.775Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.775Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.775Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.775Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:32:37.775Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T05:32:37.775Z] [2024-01-19T05:32:37.776Z] ------------------------------ [2024-01-19T05:32:37.776Z] • Failure [1443.269 seconds] [2024-01-19T05:32:37.776Z] {DeleteBackupSharedByMultipleUsersFromAdmin} [2024-01-19T05:32:37.776Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_orphaned_test.go:3257 [2024-01-19T05:32:37.776Z] Delete backups of non admin user from px-admin [It] [2024-01-19T05:32:37.776Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_orphaned_test.go:3310 [2024-01-19T05:32:37.776Z] [2024-01-19T05:32:37.776Z] Unexpected error: [2024-01-19T05:32:37.776Z] <*errors.errorString | 0xc0013da210>: { [2024-01-19T05:32:37.776Z] s: "error: backup status for [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541] expected was [[Success]] but got [Failed] because of [snapshot timeout out after 5m0s], Description: Verifying creation of backup [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541]", [2024-01-19T05:32:37.776Z] } [2024-01-19T05:32:37.776Z] error: backup status for [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541] expected was [[Success]] but got [Failed] because of [snapshot timeout out after 5m0s], Description: Verifying creation of backup [tp-backup-schedule-single-ns-testuser1-1705641008-without-rules-telr-e4327ae-interval-2024-01-19-051541] [2024-01-19T05:32:37.776Z] occurred [2024-01-19T05:32:37.776Z] [2024-01-19T05:32:37.776Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-19T05:32:37.776Z] [2024-01-19T05:32:37.776Z] Full Stack Trace [2024-01-19T05:32:37.776Z] github.com/portworx/torpedo/pkg/aetosutil.(*Dashboard).VerifyFatal(0x7f34c78?, {0x702af20?, 0xc0013da0f0}, {0x0?, 0x0}, {0xc001216000, 0x86}) [2024-01-19T05:32:37.776Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 +0x143 [2024-01-19T05:32:37.776Z] github.com/portworx/torpedo/tests/backup.glob..func34.2.10.1({0xc00158e540, 0x14}) [2024-01-19T05:32:37.776Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_orphaned_test.go:3482 +0x758 [2024-01-19T05:32:37.776Z] created by github.com/portworx/torpedo/tests/backup.glob..func34.2.10 [2024-01-19T05:32:37.776Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_orphaned_test.go:3469 +0x2ee [2024-01-19T05:32:37.776Z] ------------------------------ [2024-01-19T05:32:37.776Z] STEP: schedule applications [2024-01-19T05:32:37.776Z] STEP: Validate applications [2024-01-19T05:32:37.776Z] STEP: validate applications [2024-01-19T05:32:37.776Z] STEP: validate mysql-ibm app's volumes [2024-01-19T05:32:37.776Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T05:32:37.776Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T05:32:37.776Z] STEP: get mysql-ibm app's volume: pvc-ea1eb4e5-e4c3-4e11-92b5-b76730c8c471 inspected by the volume driver [2024-01-19T05:32:37.776Z] STEP: wait for mysql-ibm app to start running [2024-01-19T05:32:37.776Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T05:32:37.776Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T05:32:37.776Z] STEP: Validate Px pod restart count [2024-01-19T05:32:37.776Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T05:32:37.776Z] STEP: Create a non-admin users to create the backups and restore [2024-01-19T05:32:37.776Z] STEP: Adding Credentials from px-admin user and sharing with required users [2024-01-19T05:32:37.776Z] STEP: Adding backup location from px-admin user and sharing with app users [2024-01-19T05:32:37.776Z] STEP: Adding backup location for app-admin user and shared cloud cred from px-admin [2024-01-19T05:32:37.776Z] STEP: Adding cloud credentials and backup location for infra-admin users [2024-01-19T05:32:37.776Z] STEP: Create schedule policy from px-admin user [2024-01-19T05:32:37.776Z] STEP: Register source and destination cluster for backup on non-admin user [2024-01-19T05:32:37.776Z] STEP: Taking manual backup of applications as non-admin user [2024-01-19T05:32:37.776Z] STEP: Taking schedule backup of applications as non-admin user [2024-01-19T05:32:37.776Z] STEP: Sharing of backup of app-user user with app-admin and infra-admin user [2024-01-19T05:32:37.776Z] STEP: Sharing of backup of app-admin user with app-user and infra-admin user [2024-01-19T05:32:37.777Z] STEP: Sharing of backup of infra-admin user with app-user and app-admin user [2024-01-19T05:32:37.777Z] STEP: Listing and Deletion of shared backup of non-admin users from px-admin user [2024-01-19T05:32:37.777Z] STEP: destroy apps [2024-01-19T05:32:37.777Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T05:32:37.777Z] STEP: start destroying mysql-ibm app [2024-01-19T05:32:37.777Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T05:32:37.777Z] S2024-01-19 05:32:37 +0000:[INFO] [{DuplicateSharedBackup}] Log Dir: /testresults//DuplicateSharedBackup.log [2024-01-19T05:32:37.777Z] INFO[2024-01-19 05:32:37] --------Test Start------ [2024-01-19T05:32:37.777Z] INFO[2024-01-19 05:32:37] #Test: DuplicateSharedBackup [2024-01-19T05:32:37.777Z] INFO[2024-01-19 05:32:37] #Description: Share backup with user and duplicate it [2024-01-19T05:32:37.777Z] INFO[2024-01-19 05:32:37] ------------------------ [2024-01-19T05:32:37.777Z] 2024-01-19 05:32:37 +0000:[INFO] [{DuplicateSharedBackup}] Deploy applications [2024-01-19T05:32:37.777Z] 2024-01-19 05:32:37 +0000:[INFO] [{DuplicateSharedBackup}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T05:32:37.777Z] 2024-01-19 05:32:37 +0000:[INFO] [{DuplicateSharedBackup}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T05:32:37.777Z] 2024-01-19 05:32:37 +0000:[INFO] [{DuplicateSharedBackup}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T05:32:37.777Z] 2024-01-19 05:32:37 +0000:[INFO] [{DuplicateSharedBackup}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T05:32:37.777Z] 2024-01-19 05:32:37 +0000:[INFO] [{DuplicateSharedBackup}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T05:32:38.035Z] 2024-01-19 05:32:37 +0000:[INFO] [{DuplicateSharedBackup}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T05:32:38.036Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T05:32:38.036Z] 2024-01-19 05:32:37 +0000:[INFO] [{DuplicateSharedBackup}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T05:32:38.037Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T05:32:38.037Z] 2024-01-19 05:32:37 +0000:[INFO] [{DuplicateSharedBackup}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T05:32:38.292Z] 2024-01-19 05:32:38 +0000:[INFO] [{DuplicateSharedBackup}] Validate applications [2024-01-19T05:32:38.292Z] 2024-01-19 05:32:38 +0000:[INFO] [{DuplicateSharedBackup}] Validate applications [2024-01-19T05:32:38.292Z] 2024-01-19 05:32:38 +0000:[INFO] [{DuplicateSharedBackup}] Validating mysql-ibm app [2024-01-19T05:32:38.292Z] 2024-01-19 05:32:38 +0000:[INFO] [{DuplicateSharedBackup}] Validating mysql-ibm app's volumes [2024-01-19T05:32:38.551Z] 2024-01-19 05:32:38 +0000:[INFO] [{DuplicateSharedBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T05:32:38.551Z] 2024/01/19 05:32:38 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T05:32:48.489Z] 2024/01/19 05:32:48 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T05:32:58.430Z] 2024-01-19 05:32:58 +0000:[INFO] [{DuplicateSharedBackup}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-82942-01-19-05h32m37s [2024-01-19T05:32:58.687Z] 2024-01-19 05:32:58 +0000:[INFO] [{DuplicateSharedBackup}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T05:32:58.687Z] 2024-01-19 05:32:58 +0000:[WARNING] [{DuplicateSharedBackup}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T05:32:58.687Z] 2024-01-19 05:32:58 +0000:[INFO] [{DuplicateSharedBackup}] wait for mysql-ibm app to start running [2024-01-19T05:32:58.687Z] 2024-01-19 05:32:58 +0000:[INFO] [{DuplicateSharedBackup}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T05:32:58.687Z] 2024/01/19 05:32:58 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:32:58.687Z] pod name:mysql-7d766cd65c-q89z2 namespace:mysql-ibm-pxb-0-82942-01-19-05h32m37s running:false ready:false node:10.240.64.8 [2024-01-19T05:32:58.687Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:33:10.853Z] 2024/01/19 05:33:08 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:33:10.853Z] pod name:mysql-7d766cd65c-q89z2 namespace:mysql-ibm-pxb-0-82942-01-19-05h32m37s running:true ready:false node:10.240.64.8 [2024-01-19T05:33:10.853Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:33:18.939Z] 2024/01/19 05:33:18 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:33:18.939Z] pod name:mysql-7d766cd65c-q89z2 namespace:mysql-ibm-pxb-0-82942-01-19-05h32m37s running:true ready:false node:10.240.64.8 [2024-01-19T05:33:18.939Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:33:31.132Z] 2024/01/19 05:33:28 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:33:31.132Z] pod name:mysql-7d766cd65c-q89z2 namespace:mysql-ibm-pxb-0-82942-01-19-05h32m37s running:true ready:false node:10.240.64.8 [2024-01-19T05:33:31.132Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:33:39.223Z] 2024/01/19 05:33:39 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:33:39.223Z] pod name:mysql-7d766cd65c-q89z2 namespace:mysql-ibm-pxb-0-82942-01-19-05h32m37s running:true ready:false node:10.240.64.8 [2024-01-19T05:33:39.223Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:33:51.387Z] 2024/01/19 05:33:49 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:33:51.387Z] pod name:mysql-7d766cd65c-q89z2 namespace:mysql-ibm-pxb-0-82942-01-19-05h32m37s running:true ready:false node:10.240.64.8 [2024-01-19T05:33:51.387Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:33:59.474Z] 2024/01/19 05:33:59 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:33:59.474Z] pod name:mysql-7d766cd65c-q89z2 namespace:mysql-ibm-pxb-0-82942-01-19-05h32m37s running:true ready:false node:10.240.64.8 [2024-01-19T05:33:59.474Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:34:11.644Z] 2024/01/19 05:34:09 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:34:11.644Z] pod name:mysql-7d766cd65c-q89z2 namespace:mysql-ibm-pxb-0-82942-01-19-05h32m37s running:true ready:false node:10.240.64.8 [2024-01-19T05:34:11.644Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:34:19.730Z] 2024-01-19 05:34:19 +0000:[INFO] [{DuplicateSharedBackup}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T05:34:19.985Z] 2024-01-19 05:34:19 +0000:[INFO] [{DuplicateSharedBackup}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T05:34:19.985Z] 2024-01-19 05:34:19 +0000:[INFO] [{DuplicateSharedBackup}] validate if mysql-ibm app's volumes are setup [2024-01-19T05:34:20.245Z] 2024-01-19 05:34:19 +0000:[WARNING] [{DuplicateSharedBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T05:34:20.245Z] 2024-01-19 05:34:19 +0000:[INFO] [{DuplicateSharedBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T05:34:20.245Z] 2024-01-19 05:34:19 +0000:[INFO] [{DuplicateSharedBackup}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T05:34:20.245Z] 2024-01-19 05:34:19 +0000:[WARNING] [{DuplicateSharedBackup}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T05:34:20.500Z] 2024-01-19 05:34:20 +0000:[INFO] [{DuplicateSharedBackup}] [backup.AddUser:#728] - Response for user [testuser1] creation - [2024-01-19T05:34:20.501Z] 2024-01-19 05:34:20 +0000:[INFO] [{DuplicateSharedBackup}] [backup.AddUser:#733] - User [testuser1] created [2024-01-19T05:34:20.501Z] 2024-01-19 05:34:20 +0000:[INFO] [{DuplicateSharedBackup}] Creating cloud credentials and backup location [2024-01-19T05:34:20.501Z] 2024-01-19 05:34:20 +0000:[INFO] [{DuplicateSharedBackup}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-1705642460] for org [default] with [aws] as provider [2024-01-19T05:34:20.501Z] 2024-01-19 05:34:20 +0000:[INFO] [{DuplicateSharedBackup}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T05:34:20.758Z] 2024-01-19 05:34:20 +0000:[INFO] [{DuplicateSharedBackup}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-1705642460] is present [2024-01-19T05:34:20.758Z] INFO[2024-01-19 05:34:20] Verifying : Description : Verifying creation of cloud credential named [autogenerated-cred-1705642460] for org [default] with [aws] as provider [2024-01-19T05:34:20.758Z] INFO[2024-01-19 05:34:20] Actual:true, Expected: true [2024-01-19T05:34:20.758Z] 2024-01-19 05:34:20 +0000:[INFO] [{DuplicateSharedBackup}] Created Cloud Credentials with name - autogenerated-cred-1705642460 [2024-01-19T05:34:24.022Z] INFO[2024-01-19 05:34:23] Verifying : Description : Creating backup location autogenerated-backup-location-1705642460 [2024-01-19T05:34:24.022Z] INFO[2024-01-19 05:34:23] Actual:true, Expected: true [2024-01-19T05:34:24.022Z] 2024-01-19 05:34:23 +0000:[INFO] [{DuplicateSharedBackup}] Created Backup Location with name - autogenerated-backup-location-1705642460 [2024-01-19T05:34:24.022Z] 2024-01-19 05:34:23 +0000:[INFO] [{DuplicateSharedBackup}] Registering Source and Destination clusters and verifying the status [2024-01-19T05:34:24.022Z] INFO[2024-01-19 05:34:23] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T05:34:24.022Z] INFO[2024-01-19 05:34:23] Actual:true, Expected: true [2024-01-19T05:34:24.022Z] 2024-01-19 05:34:23 +0000:[INFO] [{DuplicateSharedBackup}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T05:34:24.022Z] 2024-01-19 05:34:23 +0000:[INFO] [{DuplicateSharedBackup}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T05:34:24.022Z] 2024-01-19 05:34:23 +0000:[INFO] [{DuplicateSharedBackup}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T05:34:24.022Z] 2024-01-19 05:34:23 +0000:[INFO] [{DuplicateSharedBackup}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T05:34:24.022Z] 2024-01-19 05:34:23 +0000:[INFO] [{DuplicateSharedBackup}] Create cluster [source-cluster] in org [default] [2024-01-19T05:34:24.022Z] 2024-01-19 05:34:23 +0000:[INFO] [{DuplicateSharedBackup}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T05:34:24.023Z] 2024-01-19 05:34:23 +0000:[INFO] [{DuplicateSharedBackup}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T05:34:24.023Z] 2024-01-19 05:34:23 +0000:[INFO] [{DuplicateSharedBackup}] Create cluster [destination-cluster] in org [default] [2024-01-19T05:34:24.023Z] 2024-01-19 05:34:23 +0000:[INFO] [{DuplicateSharedBackup}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T05:34:24.023Z] 2024-01-19 05:34:23 +0000:[INFO] [{DuplicateSharedBackup}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T05:34:24.023Z] 2024-01-19 05:34:23 +0000:[INFO] [{DuplicateSharedBackup}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-waqxf] for IBM [2024-01-19T05:34:24.023Z] 2024-01-19 05:34:23 +0000:[INFO] [{DuplicateSharedBackup}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-waqxf] for org [default] with [ibm] as provider [2024-01-19T05:34:24.023Z] 2024-01-19 05:34:23 +0000:[INFO] [{DuplicateSharedBackup}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T05:34:24.023Z] 2024-01-19 05:34:23 +0000:[INFO] [{DuplicateSharedBackup}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T05:34:24.023Z] 2024-01-19 05:34:23 +0000:[INFO] [{DuplicateSharedBackup}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-waqxf] is present [2024-01-19T05:34:24.023Z] 2024-01-19 05:34:23 +0000:[INFO] [{DuplicateSharedBackup}] Create cluster [source-cluster] in org [default] [2024-01-19T05:34:28.201Z] 2024-01-19 05:34:27 +0000:[INFO] [{DuplicateSharedBackup}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-mpqdp] for IBM [2024-01-19T05:34:28.201Z] 2024-01-19 05:34:27 +0000:[INFO] [{DuplicateSharedBackup}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-mpqdp] for org [default] with [ibm] as provider [2024-01-19T05:34:28.201Z] 2024-01-19 05:34:27 +0000:[INFO] [{DuplicateSharedBackup}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T05:34:28.201Z] 2024-01-19 05:34:27 +0000:[INFO] [{DuplicateSharedBackup}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T05:34:28.201Z] 2024-01-19 05:34:27 +0000:[INFO] [{DuplicateSharedBackup}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-mpqdp] is present [2024-01-19T05:34:28.201Z] 2024-01-19 05:34:27 +0000:[INFO] [{DuplicateSharedBackup}] Create cluster [destination-cluster] in org [default] [2024-01-19T05:34:28.201Z] INFO[2024-01-19 05:34:27] Verifying : Description : Creating source and destination cluster [2024-01-19T05:34:28.201Z] INFO[2024-01-19 05:34:27] Actual:true, Expected: true [2024-01-19T05:34:28.201Z] INFO[2024-01-19 05:34:28] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T05:34:28.201Z] INFO[2024-01-19 05:34:28] Actual:Online, Expected: Online [2024-01-19T05:34:28.201Z] INFO[2024-01-19 05:34:28] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T05:34:28.201Z] INFO[2024-01-19 05:34:28] Actual:true, Expected: true [2024-01-19T05:34:28.201Z] 2024-01-19 05:34:28 +0000:[INFO] [{DuplicateSharedBackup}] Taking Backup of application [2024-01-19T05:34:28.201Z] 2024-01-19 05:34:28 +0000:[INFO] [{DuplicateSharedBackup}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T05:34:28.201Z] 2024-01-19 05:34:28 +0000:[INFO] [{DuplicateSharedBackup}] Volume snapshot classes found in the cluster - [2024-01-19T05:34:28.201Z] 2024-01-19 05:34:28 +0000:[INFO] [{DuplicateSharedBackup}] ibmc-vpcblock-snapshot [2024-01-19T05:34:28.201Z] 2024-01-19 05:34:28 +0000:[INFO] [{DuplicateSharedBackup}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T05:34:28.762Z] 2024/01/19 05:34:28 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705642358] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T05:35:00.784Z] 2024/01/19 05:34:58 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705642358] expected was [[Success]] but got [InProgress] because of [Volume backups are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:35:00.784Z] INFO[2024-01-19 05:34:58] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T05:35:00.784Z] INFO[2024-01-19 05:34:58] sched stats: tick interval: 50458 < 1.2s (100%) [2024-01-19T05:35:00.784Z] INFO[2024-01-19 05:34:58] sched stats: tick process durations: 50457 < 500ms (100%) [2024-01-19T05:35:32.827Z] 2024/01/19 05:35:28 DoRetryWithTimeout - Error: {backup status for [tp-backup-1705642358] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:35:59.397Z] 2024-01-19 05:35:58 +0000:[INFO] [{DuplicateSharedBackup}] [backup.CreateBackup:#313] - Backup [tp-backup-1705642358] created successfully [2024-01-19T05:35:59.397Z] 2024-01-19 05:35:58 +0000:[INFO] [{DuplicateSharedBackup}] Validating backup [tp-backup-1705642358] in org [default] [2024-01-19T05:35:59.397Z] 2024-01-19 05:35:58 +0000:[INFO] [{DuplicateSharedBackup}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-1705642358] [2024-01-19T05:35:59.397Z] 2024-01-19 05:35:58 +0000:[INFO] [{DuplicateSharedBackup}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82942-01-19-05h32m37s] in backup [tp-backup-1705642358] [2024-01-19T05:35:59.397Z] 2024-01-19 05:35:58 +0000:[INFO] [{DuplicateSharedBackup}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82942-01-19-05h32m37s] in backup [tp-backup-1705642358] [2024-01-19T05:35:59.398Z] 2024-01-19 05:35:58 +0000:[INFO] [{DuplicateSharedBackup}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82942-01-19-05h32m37s] [2024-01-19T05:35:59.398Z] 2024-01-19 05:35:58 +0000:[WARNING] [{DuplicateSharedBackup}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T05:35:59.398Z] 2024-01-19 05:35:58 +0000:[INFO] [{DuplicateSharedBackup}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T05:35:59.398Z] 2024-01-19 05:35:58 +0000:[INFO] [{DuplicateSharedBackup}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82942-01-19-05h32m37s] are [[mysql-data]] [2024-01-19T05:35:59.398Z] 2024-01-19 05:35:58 +0000:[INFO] [{DuplicateSharedBackup}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T05:35:59.398Z] 2024-01-19 05:35:58 +0000:[INFO] [{DuplicateSharedBackup}] Validating CR cleanup [2024-01-19T05:35:59.398Z] 2024-01-19 05:35:58 +0000:[INFO] [{DuplicateSharedBackup}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T05:35:59.398Z] 2024-01-19 05:35:58 +0000:[INFO] [{DuplicateSharedBackup}] All CRs in [mysql-ibm-pxb-0-82942-01-19-05h32m37s] are [[]] [2024-01-19T05:35:59.398Z] INFO[2024-01-19 05:35:58] Verifying : Description : Creation and Validation of backup [tp-backup-1705642358] [2024-01-19T05:35:59.398Z] INFO[2024-01-19 05:35:58] Actual:true, Expected: true [2024-01-19T05:35:59.398Z] 2024-01-19 05:35:58 +0000:[INFO] [{DuplicateSharedBackup}] Share backup with user having full access [2024-01-19T05:35:59.398Z] 2024-01-19 05:35:59 +0000:[INFO] [{DuplicateSharedBackup}] [backup.ShareBackup:#715] - Backup UID for tp-backup-1705642358 - 065b51fa-68df-41f8-94df-84fd5c8cb0f5 [2024-01-19T05:35:59.398Z] 2024-01-19 05:35:59 +0000:[INFO] [{DuplicateSharedBackup}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1 - 712cd31b-5794-4f53-b894-922ecc09a610 [2024-01-19T05:35:59.398Z] 2024-01-19 05:35:59 +0000:[INFO] [{DuplicateSharedBackup}] Validating to duplicate share backup without adding cluster [2024-01-19T05:35:59.656Z] 2024-01-19 05:35:59 +0000:[INFO] [{DuplicateSharedBackup}] [backup.glob..func81.2.7:#242] - Validating that backups are shared with testuser1 user [2024-01-19T05:35:59.656Z] INFO[2024-01-19 05:35:59] Verifying : Description : Validating that user [testuser1] has access to all shared backups [[tp-backup-1705642358]] [2024-01-19T05:35:59.656Z] INFO[2024-01-19 05:35:59] Actual:1, Expected: 1 [2024-01-19T05:35:59.656Z] 2024-01-19 05:35:59 +0000:[INFO] [{DuplicateSharedBackup}] [backup.glob..func81.2.7:#248] - Duplicate shared backup [2024-01-19T05:35:59.656Z] 2024-01-19 05:35:59 +0000:[INFO] [{DuplicateSharedBackup}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T05:35:59.656Z] 2024-01-19 05:35:59 +0000:[INFO] [{DuplicateSharedBackup}] Volume snapshot classes found in the cluster - [2024-01-19T05:35:59.656Z] 2024-01-19 05:35:59 +0000:[INFO] [{DuplicateSharedBackup}] ibmc-vpcblock-snapshot [2024-01-19T05:35:59.656Z] 2024-01-19 05:35:59 +0000:[INFO] [{DuplicateSharedBackup}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T05:35:59.912Z] 2024-01-19 05:35:59 +0000:[INFO] [{DuplicateSharedBackup}] [backup.glob..func81.2.7:#251] - user not able to duplicate shared backup without adding cluster with err - rpc error: code = NotFound desc = failed to retrieve cluster [source-cluster]: object not found [2024-01-19T05:35:59.912Z] INFO[2024-01-19 05:35:59] Verifying : Description : Verifying that shared backup can't be duplicated without adding cluster [2024-01-19T05:35:59.912Z] INFO[2024-01-19 05:35:59] Actual:true, Expected: true [2024-01-19T05:35:59.912Z] 2024-01-19 05:35:59 +0000:[INFO] [{DuplicateSharedBackup}] Deleting the deployed apps after the testcase [2024-01-19T05:35:59.912Z] 2024-01-19 05:35:59 +0000:[INFO] [{DuplicateSharedBackup}] Destroying apps [2024-01-19T05:35:59.912Z] 2024-01-19 05:35:59 +0000:[INFO] [{DuplicateSharedBackup}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T05:35:59.912Z] 2024-01-19 05:35:59 +0000:[INFO] [{DuplicateSharedBackup}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T05:35:59.912Z] 2024-01-19 05:35:59 +0000:[INFO] [{DuplicateSharedBackup}] start destroying mysql-ibm app [2024-01-19T05:35:59.912Z] 2024-01-19 05:35:59 +0000:[INFO] [{DuplicateSharedBackup}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T05:36:00.168Z] 2024-01-19 05:35:59 +0000:[INFO] [{DuplicateSharedBackup}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T05:36:00.168Z] 2024-01-19 05:35:59 +0000:[WARNING] [{DuplicateSharedBackup}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T05:36:00.168Z] INFO[2024-01-19 05:35:59] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T05:36:00.168Z] INFO[2024-01-19 05:35:59] Actual:true, Expected: true [2024-01-19T05:36:01.094Z] 2024-01-19 05:36:00 +0000:[INFO] [{DuplicateSharedBackup}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1 - 712cd31b-5794-4f53-b894-922ecc09a610 [2024-01-19T05:36:01.350Z] 2024-01-19 05:36:01 +0000:[INFO] [{DuplicateSharedBackup}] [backup.DeleteUser:#755] - Response for user [testuser1] deletion - [2024-01-19T05:36:01.350Z] 2024-01-19 05:36:01 +0000:[INFO] [{DuplicateSharedBackup}] [backup.DeleteUser:#759] - Deleted User - testuser1 [2024-01-19T05:36:02.716Z] 2024-01-19 05:36:02 +0000:[INFO] [{DuplicateSharedBackup}] Cleaning backup locations in map [map[6419e637-c220-4ab8-ae98-e55887bd3c9f:autogenerated-backup-location-1705642460]], cloud credential [autogenerated-cred-1705642460], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T05:36:02.972Z] INFO[2024-01-19 05:36:02] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-1705642460] [2024-01-19T05:36:02.972Z] INFO[2024-01-19 05:36:02] Actual:true, Expected: true [2024-01-19T05:36:02.972Z] 2024-01-19 05:36:02 +0000:[INFO] [{DuplicateSharedBackup}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705642460] is present [2024-01-19T05:36:02.972Z] 2024/01/19 05:36:02 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705642460 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:36:34.998Z] 2024-01-19 05:36:32 +0000:[INFO] [{DuplicateSharedBackup}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 aws-global-aws-px-backup-system-test-4cac76dc-bl autogenerated-backup-location-1705628002 autogenerated-backup-location-1705634221 autogenerated-backup-location-1705635982 autogenerated-backup-location-wkyfv] [2024-01-19T05:36:34.998Z] INFO[2024-01-19 05:36:32] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-1705642460 [2024-01-19T05:36:34.998Z] INFO[2024-01-19 05:36:32] Actual:true, Expected: true [2024-01-19T05:36:34.998Z] 2024-01-19 05:36:33 +0000:[INFO] [{DuplicateSharedBackup}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-1705642460] is present [2024-01-19T05:36:34.998Z] INFO[2024-01-19 05:36:33] Verifying : Description : Verifying if cloud cred [autogenerated-cred-1705642460] is present [2024-01-19T05:36:34.998Z] INFO[2024-01-19 05:36:33] Actual:true, Expected: true [2024-01-19T05:36:34.998Z] INFO[2024-01-19 05:36:33] Verifying : Description : Verifying deletion of cloud cred [autogenerated-cred-1705642460] [2024-01-19T05:36:34.998Z] INFO[2024-01-19 05:36:33] Actual:true, Expected: true [2024-01-19T05:36:34.998Z] INFO[2024-01-19 05:36:33] Verifying : Description : Deleting cloud cred autogenerated-cred-1705642460 [2024-01-19T05:36:34.998Z] INFO[2024-01-19 05:36:33] Actual:true, Expected: true [2024-01-19T05:36:34.998Z] INFO[2024-01-19 05:36:33] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T05:36:34.998Z] INFO[2024-01-19 05:36:33] Actual:true, Expected: true [2024-01-19T05:36:34.998Z] INFO[2024-01-19 05:36:33] Verifying : Description : Deleting cluster source-cluster [2024-01-19T05:36:34.998Z] ERRO[2024-01-19 05:36:33] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl], cannot be deleted, Expected:  [2024-01-19T05:36:34.998Z] INFO[2024-01-19 05:36:33] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-hrfiy] [2024-01-19T05:36:34.998Z] ERRO[2024-01-19 05:36:33] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-hrfiy:72b1b409-7bb1-4f22-92df-0ba5e77c11d3], Expected:  [2024-01-19T05:36:34.998Z] INFO[2024-01-19 05:36:33] Verifying : Description : Deleting cluster source-cluster [2024-01-19T05:36:34.999Z] ERRO[2024-01-19 05:36:33] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh], cannot be deleted, Expected:  [2024-01-19T05:36:34.999Z] INFO[2024-01-19 05:36:34] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-fbvvo] [2024-01-19T05:36:34.999Z] ERRO[2024-01-19 05:36:34] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-fbvvo:7cddb5aa-c7ff-4044-9d27-458be2bd5cde], Expected:  [2024-01-19T05:36:34.999Z] INFO[2024-01-19 05:36:34] Verifying : Description : Deleting cluster source-cluster [2024-01-19T05:36:34.999Z] ERRO[2024-01-19 05:36:34] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo], cannot be deleted, Expected:  [2024-01-19T05:36:35.926Z] INFO[2024-01-19 05:36:35] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-enzij] [2024-01-19T05:36:35.926Z] ERRO[2024-01-19 05:36:35] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-enzij:4e0ddfc7-8811-4ed5-8e87-ed0841582111], Expected:  [2024-01-19T05:36:35.926Z] INFO[2024-01-19 05:36:35] Verifying : Description : Deleting cluster source-cluster [2024-01-19T05:36:35.926Z] ERRO[2024-01-19 05:36:35] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo], cannot be deleted, Expected:  [2024-01-19T05:36:37.290Z] INFO[2024-01-19 05:36:37] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-wcpfm] [2024-01-19T05:36:37.290Z] ERRO[2024-01-19 05:36:37] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-wcpfm:c5ff084a-a913-4412-99fb-e2b0a003a014], Expected:  [2024-01-19T05:36:37.290Z] INFO[2024-01-19 05:36:37] Verifying : Description : Deleting cluster source-cluster [2024-01-19T05:36:37.290Z] ERRO[2024-01-19 05:36:37] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx], cannot be deleted, Expected:  [2024-01-19T05:36:38.658Z] INFO[2024-01-19 05:36:38] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-vlbwu] [2024-01-19T05:36:38.658Z] ERRO[2024-01-19 05:36:38] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-vlbwu:f0465bb5-2565-4948-bb27-7e45a4b883ff], Expected:  [2024-01-19T05:36:38.658Z] INFO[2024-01-19 05:36:38] Verifying : Description : Deleting cluster source-cluster [2024-01-19T05:36:38.658Z] ERRO[2024-01-19 05:36:38] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa], cannot be deleted, Expected:  [2024-01-19T05:36:40.021Z] INFO[2024-01-19 05:36:39] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-mhxzi] [2024-01-19T05:36:40.021Z] ERRO[2024-01-19 05:36:39] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-mhxzi:f551fb81-1a78-4d7f-b804-759c6e6b562e], Expected:  [2024-01-19T05:36:40.277Z] INFO[2024-01-19 05:36:40] Verifying : Description : Deleting cluster source-cluster [2024-01-19T05:36:40.277Z] INFO[2024-01-19 05:36:40] Actual:true, Expected: true [2024-01-19T05:36:41.640Z] INFO[2024-01-19 05:36:41] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-waqxf] [2024-01-19T05:36:41.640Z] INFO[2024-01-19 05:36:41] Actual:true, Expected: true [2024-01-19T05:36:41.640Z] INFO[2024-01-19 05:36:41] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T05:36:41.640Z] INFO[2024-01-19 05:36:41] Actual:true, Expected: true [2024-01-19T05:36:43.014Z] INFO[2024-01-19 05:36:42] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-mpqdp] [2024-01-19T05:36:43.014Z] INFO[2024-01-19 05:36:42] Actual:true, Expected: true [2024-01-19T05:36:43.014Z] INFO[2024-01-19 05:36:42] --------Test End------ [2024-01-19T05:36:43.014Z] INFO[2024-01-19 05:36:42] #Test: [2024-01-19T05:36:43.014Z] INFO[2024-01-19 05:36:42] #Description: [2024-01-19T05:36:43.014Z] INFO[2024-01-19 05:36:42] ------------------------ [2024-01-19T05:36:43.014Z] 2024-01-19 05:36:42 +0000:[INFO] [{DuplicateSharedBackup}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-82942-01-19-05h32m37s] [2024-01-19T05:36:43.014Z] 2024-01-19 05:36:42 +0000:[INFO] [{DuplicateSharedBackup}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T05:36:43.014Z] 2024-01-19 05:36:42 +0000:[INFO] [{DuplicateSharedBackup}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T05:36:43.270Z] 2024-01-19 05:36:43 +0000:[INFO] [{DuplicateSharedBackup}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-82942-01-19-05h32m37s] [2024-01-19T05:36:43.525Z] 2024-01-19 05:36:43 +0000:[INFO] [{DuplicateSharedBackup}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {DuplicateSharedBackup} Validate shared backup is not duplicated without cluster [2024-01-19T05:36:43.525Z] 2024-01-19 05:36:43 +0000:[INFO] [{DuplicateSharedBackup}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T05:36:43.526Z] 2024/01/19 05:36:43 DoRetryWithTimeout - Error: {dial tcp 10.240.64.27:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:36:48.765Z] 2024/01/19 05:36:48 DoRetryWithTimeout - Error: {dial tcp 10.240.64.27:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:36:54.017Z] 2024/01/19 05:36:53 DoRetryWithTimeout - Error: {dial tcp 10.240.64.27:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:36:59.263Z] 2024/01/19 05:36:58 DoRetryWithTimeout - Error: {dial tcp 10.240.64.27:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:37:03.458Z] 2024/01/19 05:37:03 DoRetryWithTimeout - Error: {dial tcp 10.240.64.27:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:37:08.698Z] 2024/01/19 05:37:08 DoRetryWithTimeout - Error: {dial tcp 10.240.64.27:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:37:13.939Z] 2024/01/19 05:37:13 DoRetryWithTimeout - Error: {dial tcp 10.240.64.27:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:37:19.181Z] 2024/01/19 05:37:18 DoRetryWithTimeout - Error: {dial tcp 10.240.64.27:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:37:24.426Z] 2024/01/19 05:37:23 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.64.27. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.27 10.240.64.27]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:37:24.426Z] {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:37:24.426Z] {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:37:24.427Z] {dial tcp 10.240.64.27:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T05:37:28.589Z] 2024-01-19 05:37:28 +0000:[WARNING] [{DuplicateSharedBackup}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.64.27. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.27 10.240.64.27]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:37:28.589Z] {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:37:28.589Z] {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:37:28.589Z] {dial tcp 10.240.64.27:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T05:37:28.589Z] 2024-01-19 05:37:28 +0000:[ERROR] [{DuplicateSharedBackup}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.64.27. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.27 10.240.64.27]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:37:28.589Z] {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:37:28.589Z] {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:37:28.589Z] {dial tcp 10.240.64.27:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T05:37:28.589Z] 2024-01-19 05:37:28 +0000:[INFO] [{DuplicateSharedBackup}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T05:37:28.589Z] 2024-01-19 05:37:28 +0000:[INFO] [{DuplicateSharedBackup}] Switching context to [/tmp/source-config] [2024-01-19T05:37:28.589Z] 2024-01-19 05:37:28 +0000:[INFO] [{DuplicateSharedBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T05:37:28.589Z] 2024-01-19 05:37:28 +0000:[INFO] [{DuplicateSharedBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T05:37:28.589Z] 2024-01-19 05:37:28 +0000:[INFO] [{DuplicateSharedBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T05:37:28.845Z] 2024-01-19 05:37:28 +0000:[INFO] [{DuplicateSharedBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T05:37:28.845Z] 2024-01-19 05:37:28 +0000:[INFO] [{DuplicateSharedBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T05:37:28.845Z] 2024-01-19 05:37:28 +0000:[INFO] [{DuplicateSharedBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T05:37:29.409Z] 2024-01-19 05:37:29 +0000:[INFO] [{DuplicateSharedBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T05:37:29.666Z] 2024-01-19 05:37:29 +0000:[INFO] [{DuplicateSharedBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T05:37:30.230Z] 2024-01-19 05:37:30 +0000:[INFO] [{DuplicateSharedBackup}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T05:37:30.230Z] 2024/01/19 05:37:30 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:37:42.397Z] 2024/01/19 05:37:40 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:37:50.501Z] 2024/01/19 05:37:50 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:38:00.447Z] 2024/01/19 05:38:00 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:38:10.395Z] 2024/01/19 05:38:10 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:38:22.561Z] 2024/01/19 05:38:20 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:38:30.649Z] 2024/01/19 05:38:30 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:38:40.587Z] 2024/01/19 05:38:40 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:38:50.531Z] 2024/01/19 05:38:50 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:39:02.704Z] 2024/01/19 05:39:00 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:39:10.800Z] 2024/01/19 05:39:10 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:39:20.750Z] 2024/01/19 05:39:20 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:39:30.706Z] 2024/01/19 05:39:30 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:39:40.651Z] 2024/01/19 05:39:40 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:39:52.824Z] 2024/01/19 05:39:50 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:40:00.910Z] 2024/01/19 05:40:00 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:40:12.307Z] 2024/01/19 05:40:10 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:40:22.246Z] 2024/01/19 05:40:20 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:40:32.198Z] 2024/01/19 05:40:30 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:40:42.139Z] 2024/01/19 05:40:40 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:40:52.080Z] 2024/01/19 05:40:50 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:41:02.024Z] 2024/01/19 05:41:00 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:41:11.967Z] 2024/01/19 05:41:10 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:41:21.920Z] 2024/01/19 05:41:20 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:41:31.941Z] 2024/01/19 05:41:30 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:41:41.881Z] 2024/01/19 05:41:40 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:41:51.825Z] 2024/01/19 05:41:51 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:42:01.767Z] 2024/01/19 05:42:01 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:42:11.714Z] 2024/01/19 05:42:11 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:42:21.666Z] 2024/01/19 05:42:21 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:42:31.610Z] 2024-01-19 05:42:31 +0000:[ERROR] [{DuplicateSharedBackup}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.610Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.610Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.611Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.611Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.611Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.611Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.611Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.611Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.611Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.611Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.611Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.611Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.611Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.611Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.611Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.611Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.611Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.611Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.611Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.611Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.611Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.611Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.611Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.611Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.611Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.611Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.612Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.612Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.612Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T05:42:31.612Z] [2024-01-19T05:42:31.612Z] ------------------------------ [2024-01-19T05:42:31.612Z] • Failure [593.520 seconds] [2024-01-19T05:42:31.612Z] {DuplicateSharedBackup} [2024-01-19T05:42:31.612Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:142 [2024-01-19T05:42:31.612Z] Validate shared backup is not duplicated without cluster [] [2024-01-19T05:42:31.612Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:179 [2024-01-19T05:42:31.612Z] [2024-01-19T05:42:31.612Z] Unexpected error: [2024-01-19T05:42:31.612Z] <*errors.errorString | 0xc00181a7b0>: { [2024-01-19T05:42:31.613Z] s: "failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{No... [2024-01-19T05:42:31.613Z] [2024-01-19T05:42:31.613Z] Gomega truncated this representation as it exceeds 'format.MaxLength'. [2024-01-19T05:42:31.613Z] Consider having the object provide a custom 'GomegaStringer' representation [2024-01-19T05:42:31.613Z] or adjust the parameters in Gomega's 'format' package. [2024-01-19T05:42:31.613Z] [2024-01-19T05:42:31.613Z] Learn more here: https://onsi.github.io/gomega/#adjusting-output [2024-01-19T05:42:31.613Z] [2024-01-19T05:42:31.613Z] failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.613Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.613Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.613Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.613Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.613Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.613Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.613Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.613Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.613Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.613Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.613Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.613Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.614Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.614Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.614Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.614Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.614Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.614Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.614Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.614Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.614Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.614Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.614Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.614Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.614Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.614Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.614Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.614Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:42:31.614Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T05:42:31.614Z] occurred [2024-01-19T05:42:31.614Z] [2024-01-19T05:42:31.614Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T05:42:31.614Z] [2024-01-19T05:42:31.614Z] Full Stack Trace [2024-01-19T05:42:31.614Z] github.com/portworx/torpedo/pkg/log.FailOnError({0x88a5060, 0xc00181a7b0}, {0x7f74bfd?, 0x6cc0320?}, {0x0?, 0x6cc0320?, 0xc000581d90?}) [2024-01-19T05:42:31.614Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 +0x290 [2024-01-19T05:42:31.614Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest.func1() [2024-01-19T05:42:31.614Z] /go/src/github.com/portworx/torpedo/tests/common.go:7283 +0x32 [2024-01-19T05:42:31.614Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest({0xc000015298, 0x1, 0x1}) [2024-01-19T05:42:31.614Z] /go/src/github.com/portworx/torpedo/tests/common.go:7299 +0x778 [2024-01-19T05:42:31.614Z] github.com/portworx/torpedo/tests/backup.glob..func81.3() [2024-01-19T05:42:31.614Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:277 +0x358 [2024-01-19T05:42:31.614Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T05:42:31.614Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T05:42:31.614Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T05:42:31.615Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T05:42:31.615Z] created by testing.(*T).Run [2024-01-19T05:42:31.615Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T05:42:31.615Z] ------------------------------ [2024-01-19T05:42:31.615Z] STEP: schedule applications [2024-01-19T05:42:31.615Z] STEP: Validate applications [2024-01-19T05:42:31.615Z] STEP: validate applications [2024-01-19T05:42:31.615Z] STEP: validate mysql-ibm app's volumes [2024-01-19T05:42:31.615Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T05:42:31.615Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T05:42:31.615Z] STEP: get mysql-ibm app's volume: pvc-4a225d63-40e5-4a65-9117-8c1ff8a807a8 inspected by the volume driver [2024-01-19T05:42:31.615Z] STEP: wait for mysql-ibm app to start running [2024-01-19T05:42:31.615Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T05:42:31.615Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T05:42:31.615Z] STEP: Validate Px pod restart count [2024-01-19T05:42:31.615Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T05:42:31.615Z] STEP: Create User [2024-01-19T05:42:31.615Z] STEP: Adding Credentials and Registering Backup Location [2024-01-19T05:42:31.615Z] STEP: Register source and destination cluster for backup [2024-01-19T05:42:31.615Z] STEP: Taking backup of applications [2024-01-19T05:42:31.615Z] STEP: Share backup with user [2024-01-19T05:42:31.615Z] STEP: Duplicate shared backup [2024-01-19T05:42:31.615Z] STEP: destroy apps [2024-01-19T05:42:31.615Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T05:42:31.615Z] STEP: start destroying mysql-ibm app [2024-01-19T05:42:31.615Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T05:42:31.615Z] 2024-01-19 05:42:31 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Log Dir: /testresults//DeleteSharedBackupOfUserFromAdmin.log [2024-01-19T05:42:31.615Z] INFO[2024-01-19 05:42:31] --------Test Start------ [2024-01-19T05:42:31.615Z] INFO[2024-01-19 05:42:31] #Test: DeleteSharedBackupOfUserFromAdmin [2024-01-19T05:42:31.615Z] INFO[2024-01-19 05:42:31] #Description: Delete backups shared by the user from the admin [2024-01-19T05:42:31.615Z] INFO[2024-01-19 05:42:31] ------------------------ [2024-01-19T05:42:31.615Z] 2024-01-19 05:42:31 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T05:42:31.615Z] 2024-01-19 05:42:31 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T05:42:31.615Z] 2024-01-19 05:42:31 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T05:42:31.615Z] 2024-01-19 05:42:31 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T05:42:31.615Z] 2024-01-19 05:42:31 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T05:42:31.615Z] 2024-01-19 05:42:31 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T05:42:31.616Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T05:42:31.616Z] 2024-01-19 05:42:31 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T05:42:31.616Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T05:42:31.616Z] 2024-01-19 05:42:31 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T05:42:31.616Z] 2024-01-19 05:42:31 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Scheduled application namespaces: [mysql-ibm-pxb-0-87562-01-19-05h42m31s] [2024-01-19T05:42:31.616Z] 2024-01-19 05:42:31 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Validating applications [2024-01-19T05:42:31.616Z] 2024-01-19 05:42:31 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Validate applications [2024-01-19T05:42:31.616Z] 2024-01-19 05:42:31 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Validating mysql-ibm app [2024-01-19T05:42:31.617Z] 2024-01-19 05:42:31 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Validating mysql-ibm app's volumes [2024-01-19T05:42:31.617Z] 2024-01-19 05:42:31 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T05:42:31.873Z] 2024/01/19 05:42:31 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T05:42:41.812Z] 2024/01/19 05:42:41 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T05:42:51.747Z] 2024-01-19 05:42:51 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-87562-01-19-05h42m31s [2024-01-19T05:42:51.747Z] 2024-01-19 05:42:51 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T05:42:52.003Z] 2024-01-19 05:42:51 +0000:[WARNING] [{DeleteSharedBackupOfUserFromAdmin}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T05:42:52.003Z] 2024-01-19 05:42:51 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] wait for mysql-ibm app to start running [2024-01-19T05:42:52.003Z] 2024-01-19 05:42:51 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T05:42:52.003Z] 2024/01/19 05:42:51 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:42:52.003Z] pod name:mysql-7d766cd65c-wpxmf namespace:mysql-ibm-pxb-0-87562-01-19-05h42m31s running:false ready:false node:10.240.64.8 [2024-01-19T05:42:52.003Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:43:04.173Z] 2024/01/19 05:43:02 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:43:04.173Z] pod name:mysql-7d766cd65c-wpxmf namespace:mysql-ibm-pxb-0-87562-01-19-05h42m31s running:true ready:false node:10.240.64.8 [2024-01-19T05:43:04.173Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:43:12.256Z] 2024/01/19 05:43:12 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:43:12.256Z] pod name:mysql-7d766cd65c-wpxmf namespace:mysql-ibm-pxb-0-87562-01-19-05h42m31s running:true ready:false node:10.240.64.8 [2024-01-19T05:43:12.256Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:43:24.423Z] 2024/01/19 05:43:22 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:43:24.423Z] pod name:mysql-7d766cd65c-wpxmf namespace:mysql-ibm-pxb-0-87562-01-19-05h42m31s running:true ready:false node:10.240.64.8 [2024-01-19T05:43:24.423Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:43:32.542Z] 2024/01/19 05:43:32 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:43:32.542Z] pod name:mysql-7d766cd65c-wpxmf namespace:mysql-ibm-pxb-0-87562-01-19-05h42m31s running:true ready:false node:10.240.64.8 [2024-01-19T05:43:32.542Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:43:44.778Z] 2024/01/19 05:43:42 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:43:44.778Z] pod name:mysql-7d766cd65c-wpxmf namespace:mysql-ibm-pxb-0-87562-01-19-05h42m31s running:true ready:false node:10.240.64.8 [2024-01-19T05:43:44.778Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:43:52.861Z] 2024/01/19 05:43:52 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:43:52.861Z] pod name:mysql-7d766cd65c-wpxmf namespace:mysql-ibm-pxb-0-87562-01-19-05h42m31s running:true ready:false node:10.240.64.8 [2024-01-19T05:43:52.861Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:44:05.056Z] 2024-01-19 05:44:02 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T05:44:05.056Z] 2024-01-19 05:44:02 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T05:44:05.056Z] 2024-01-19 05:44:03 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] validate if mysql-ibm app's volumes are setup [2024-01-19T05:44:05.056Z] 2024-01-19 05:44:03 +0000:[WARNING] [{DeleteSharedBackupOfUserFromAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T05:44:05.056Z] 2024-01-19 05:44:03 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T05:44:05.056Z] 2024-01-19 05:44:03 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T05:44:05.056Z] 2024-01-19 05:44:03 +0000:[WARNING] [{DeleteSharedBackupOfUserFromAdmin}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T05:44:05.056Z] 2024-01-19 05:44:03 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Creating 3 users with px-backup-infra.admin role [2024-01-19T05:44:05.056Z] 2024-01-19 05:44:03 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Creating 1 users [2024-01-19T05:44:05.056Z] 2024-01-19 05:44:03 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.AddUser:#728] - Response for user [testuser1-1705643043] creation - [2024-01-19T05:44:05.056Z] 2024-01-19 05:44:03 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.AddUser:#733] - User [testuser1-1705643043] created [2024-01-19T05:44:05.056Z] INFO[2024-01-19 05:44:03] Verifying : Description : Creating user - testuser1-1705643043 [2024-01-19T05:44:05.057Z] INFO[2024-01-19 05:44:03] Actual:true, Expected: true [2024-01-19T05:44:05.057Z] 2024-01-19 05:44:03 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705643043 - d5e6c621-e3e7-4f40-a7e9-38162428157a [2024-01-19T05:44:05.313Z] 2024-01-19 05:44:05 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Creating 1 users [2024-01-19T05:44:06.243Z] 2024-01-19 05:44:05 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.AddUser:#728] - Response for user [testuser1-1705643045] creation - [2024-01-19T05:44:06.244Z] 2024-01-19 05:44:05 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.AddUser:#733] - User [testuser1-1705643045] created [2024-01-19T05:44:06.244Z] INFO[2024-01-19 05:44:05] Verifying : Description : Creating user - testuser1-1705643045 [2024-01-19T05:44:06.244Z] INFO[2024-01-19 05:44:05] Actual:true, Expected: true [2024-01-19T05:44:06.805Z] 2024-01-19 05:44:06 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705643043 - d5e6c621-e3e7-4f40-a7e9-38162428157a [2024-01-19T05:44:08.695Z] 2024-01-19 05:44:08 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Creating 1 users [2024-01-19T05:44:09.260Z] 2024-01-19 05:44:09 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.AddUser:#728] - Response for user [testuser1-1705643048] creation - [2024-01-19T05:44:09.260Z] 2024-01-19 05:44:09 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.AddUser:#733] - User [testuser1-1705643048] created [2024-01-19T05:44:09.260Z] INFO[2024-01-19 05:44:09] Verifying : Description : Creating user - testuser1-1705643048 [2024-01-19T05:44:09.261Z] INFO[2024-01-19 05:44:09] Actual:true, Expected: true [2024-01-19T05:44:10.189Z] 2024-01-19 05:44:09 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705643043 - d5e6c621-e3e7-4f40-a7e9-38162428157a [2024-01-19T05:44:11.560Z] 2024-01-19 05:44:11 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Creating cloud credential and backup location from the user testuser1-1705643043 [2024-01-19T05:44:12.121Z] 2024-01-19 05:44:11 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-testuser1-1705643043-dvtop] for org [default] with [aws] as provider [2024-01-19T05:44:12.121Z] 2024-01-19 05:44:11 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T05:44:12.121Z] 2024-01-19 05:44:11 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-testuser1-1705643043-dvtop] is present [2024-01-19T05:44:14.643Z] 2024-01-19 05:44:14 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Creating source and destination cluster from the user testuser1-1705643043 [2024-01-19T05:44:14.643Z] INFO[2024-01-19 05:44:14] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T05:44:14.644Z] INFO[2024-01-19 05:44:14] Actual:true, Expected: true [2024-01-19T05:44:14.644Z] 2024-01-19 05:44:14 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T05:44:14.644Z] 2024-01-19 05:44:14 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T05:44:14.644Z] 2024-01-19 05:44:14 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T05:44:14.644Z] 2024-01-19 05:44:14 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T05:44:14.644Z] 2024-01-19 05:44:14 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Create cluster [source-cluster] in org [default] [2024-01-19T05:44:14.644Z] 2024-01-19 05:44:14 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T05:44:14.644Z] 2024-01-19 05:44:14 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T05:44:14.644Z] 2024-01-19 05:44:14 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-19T05:44:14.644Z] 2024-01-19 05:44:14 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T05:44:14.644Z] 2024-01-19 05:44:14 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T05:44:14.644Z] 2024-01-19 05:44:14 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-nahtg] for IBM [2024-01-19T05:44:14.644Z] 2024-01-19 05:44:14 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-nahtg] for org [default] with [ibm] as provider [2024-01-19T05:44:14.644Z] 2024-01-19 05:44:14 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T05:44:14.644Z] 2024-01-19 05:44:14 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T05:44:14.644Z] 2024-01-19 05:44:14 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-nahtg] is present [2024-01-19T05:44:14.644Z] 2024-01-19 05:44:14 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Create cluster [source-cluster] in org [default] [2024-01-19T05:44:18.810Z] 2024-01-19 05:44:18 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-uuakz] for IBM [2024-01-19T05:44:18.810Z] 2024-01-19 05:44:18 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-uuakz] for org [default] with [ibm] as provider [2024-01-19T05:44:18.810Z] 2024-01-19 05:44:18 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T05:44:18.810Z] 2024-01-19 05:44:18 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T05:44:18.810Z] 2024-01-19 05:44:18 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-uuakz] is present [2024-01-19T05:44:18.810Z] 2024-01-19 05:44:18 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-19T05:44:19.066Z] INFO[2024-01-19 05:44:18] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T05:44:19.066Z] INFO[2024-01-19 05:44:18] Actual:Online, Expected: Online [2024-01-19T05:44:19.066Z] INFO[2024-01-19 05:44:18] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T05:44:19.066Z] INFO[2024-01-19 05:44:18] Actual:true, Expected: true [2024-01-19T05:44:19.066Z] INFO[2024-01-19 05:44:18] Verifying : Description : Fetching [destination-cluster] cluster uid [2024-01-19T05:44:19.066Z] INFO[2024-01-19 05:44:18] Actual:true, Expected: true [2024-01-19T05:44:19.066Z] 2024-01-19 05:44:18 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Taking backup of applications from the user testuser1-1705643043 [2024-01-19T05:44:19.322Z] 2024-01-19 05:44:19 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T05:44:19.322Z] 2024-01-19 05:44:19 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Volume snapshot classes found in the cluster - [2024-01-19T05:44:19.322Z] 2024-01-19 05:44:19 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] ibmc-vpcblock-snapshot [2024-01-19T05:44:19.322Z] 2024-01-19 05:44:19 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T05:44:19.579Z] 2024/01/19 05:44:19 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87562-01-19-05h42m31s-0-testuser1-1705643043-tjafu] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T05:44:51.614Z] 2024/01/19 05:44:49 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87562-01-19-05h42m31s-0-testuser1-1705643043-tjafu] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:44:59.710Z] INFO[2024-01-19 05:44:58] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T05:44:59.710Z] INFO[2024-01-19 05:44:58] sched stats: tick interval: 51058 < 1.2s (100%) [2024-01-19T05:44:59.710Z] INFO[2024-01-19 05:44:58] sched stats: tick process durations: 51057 < 500ms (100%) [2024-01-19T05:45:21.649Z] 2024/01/19 05:45:19 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87562-01-19-05h42m31s-0-testuser1-1705643043-tjafu] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T05:45:53.683Z] 2024-01-19 05:45:49 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.CreateBackup:#313] - Backup [tp-backup-mysql-ibm-pxb-0-87562-01-19-05h42m31s-0-testuser1-1705643043-tjafu] created successfully [2024-01-19T05:45:53.683Z] 2024-01-19 05:45:49 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Validating backup [tp-backup-mysql-ibm-pxb-0-87562-01-19-05h42m31s-0-testuser1-1705643043-tjafu] in org [default] [2024-01-19T05:45:53.683Z] 2024-01-19 05:45:49 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-mysql-ibm-pxb-0-87562-01-19-05h42m31s-0-testuser1-1705643043-tjafu] [2024-01-19T05:45:53.683Z] 2024-01-19 05:45:49 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87562-01-19-05h42m31s] in backup [tp-backup-mysql-ibm-pxb-0-87562-01-19-05h42m31s-0-testuser1-1705643043-tjafu] [2024-01-19T05:45:53.683Z] 2024-01-19 05:45:49 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87562-01-19-05h42m31s] in backup [tp-backup-mysql-ibm-pxb-0-87562-01-19-05h42m31s-0-testuser1-1705643043-tjafu] [2024-01-19T05:45:53.684Z] 2024-01-19 05:45:49 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87562-01-19-05h42m31s] [2024-01-19T05:45:53.684Z] 2024-01-19 05:45:49 +0000:[WARNING] [{DeleteSharedBackupOfUserFromAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T05:45:53.684Z] 2024-01-19 05:45:49 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T05:45:53.684Z] 2024-01-19 05:45:49 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87562-01-19-05h42m31s] are [[mysql-data]] [2024-01-19T05:45:53.684Z] 2024-01-19 05:45:49 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T05:45:53.684Z] 2024-01-19 05:45:49 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Validating CR cleanup [2024-01-19T05:45:53.684Z] 2024-01-19 05:45:49 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T05:45:53.684Z] 2024-01-19 05:45:49 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] All CRs in [mysql-ibm-pxb-0-87562-01-19-05h42m31s] are [[]] [2024-01-19T05:45:53.684Z] INFO[2024-01-19 05:45:49] Verifying : Description : Verifying creation and validation of backup [tp-backup-mysql-ibm-pxb-0-87562-01-19-05h42m31s-0-testuser1-1705643043-tjafu] of namespace (scheduled Context) [mysql-ibm-pxb-0-87562-01-19-05h42m31s] [2024-01-19T05:45:53.684Z] INFO[2024-01-19 05:45:49] Actual:true, Expected: true [2024-01-19T05:45:53.684Z] 2024-01-19 05:45:49 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.glob..func30.2.5:#1672] - The list of user backups taken are: map[testuser1-1705643043:map[tp-backup-mysql-ibm-pxb-0-87562-01-19-05h42m31s-0-testuser1-1705643043-tjafu:mysql-ibm-pxb-0-87562-01-19-05h42m31s]] [2024-01-19T05:45:53.684Z] 2024-01-19 05:45:49 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Sharing user testuser1-1705643043 backups [backup-share] with user testuser1-1705643045 with ViewOnlyAccess [2024-01-19T05:45:53.684Z] 2024-01-19 05:45:50 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.ShareBackup:#715] - Backup UID for tp-backup-mysql-ibm-pxb-0-87562-01-19-05h42m31s-0-testuser1-1705643043-tjafu - 59553b7a-26c2-4b64-be84-a280b43f939a [2024-01-19T05:45:53.684Z] 2024-01-19 05:45:50 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705643045 - d98d4a0f-f890-42db-80af-7cb6f11e6ecf [2024-01-19T05:45:53.684Z] 2024-01-19 05:45:50 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Verifying shared backups of the user testuser1-1705643043 from the admin [2024-01-19T05:45:53.684Z] 2024-01-19 05:45:50 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Sharing user testuser1-1705643043 backups [cluster-share] with user testuser1-1705643048 with ViewOnlyAccess [2024-01-19T05:45:53.684Z] 2024-01-19 05:45:51 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705643048 - 4bf73ecc-57b6-473b-ac64-ece52b1597cf [2024-01-19T05:45:53.684Z] 2024/01/19 05:45:51 DoRetryWithTimeout - Error: {cluster backup share status for cluster source-cluster is still InProgress}, Next try in [10s], timeout [1m0s] [2024-01-19T05:46:01.810Z] 2024-01-19 05:46:01 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.ClusterUpdateBackupShare.func1:#856] - Cluster source-cluster has status - [3] [2024-01-19T05:46:01.810Z] 2024-01-19 05:46:01 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.ClusterUpdateBackupShare:#863] - Cluster backup share check complete [2024-01-19T05:46:01.810Z] 2024-01-19 05:46:01 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Verifying shared backups of the user testuser1-1705643043 from the admin [2024-01-19T05:46:01.811Z] 2024-01-19 05:46:01 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Verifying shared backups of the user testuser1-1705643043 from the admin [2024-01-19T05:46:02.372Z] 2024-01-19 05:46:02 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Deleting the owner [testuser1-1705643043] of the backups [2024-01-19T05:46:03.736Z] 2024-01-19 05:46:03 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705643043 - d5e6c621-e3e7-4f40-a7e9-38162428157a [2024-01-19T05:46:03.994Z] 2024-01-19 05:46:03 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.DeleteUser:#755] - Response for user [testuser1-1705643043] deletion - [2024-01-19T05:46:03.994Z] 2024-01-19 05:46:03 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.DeleteUser:#759] - Deleted User - testuser1-1705643043 [2024-01-19T05:46:05.357Z] 2024-01-19 05:46:05 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Deleting user testuser1-1705643043 backups from the admin [2024-01-19T05:46:05.357Z] 2024/01/19 05:46:05 DoRetryWithTimeout - Error: {backupObject [tp-backup-mysql-ibm-pxb-0-87562-01-19-05h42m31s-0-testuser1-1705643043-tjafu] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:46:37.384Z] 2024-01-19 05:46:35 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Destroying the scheduled applications [2024-01-19T05:46:37.384Z] 2024-01-19 05:46:35 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Destroying apps [2024-01-19T05:46:37.384Z] 2024-01-19 05:46:35 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T05:46:37.384Z] 2024-01-19 05:46:35 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T05:46:37.384Z] 2024-01-19 05:46:35 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] start destroying mysql-ibm app [2024-01-19T05:46:37.384Z] 2024-01-19 05:46:35 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T05:46:37.384Z] 2024-01-19 05:46:35 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T05:46:37.384Z] 2024-01-19 05:46:35 +0000:[WARNING] [{DeleteSharedBackupOfUserFromAdmin}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T05:46:37.384Z] INFO[2024-01-19 05:46:35] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T05:46:37.384Z] INFO[2024-01-19 05:46:35] Actual:true, Expected: true [2024-01-19T05:46:37.395Z] 2024-01-19 05:46:35 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Cleaning backup locations in map [map[9dc3496a-7b1a-4d78-9cb1-35735adc7d6b:autogenerated-backup-location-testuser1-1705643043-ieaps]], cloud credential [autogenerated-cred-testuser1-1705643043-dvtop], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T05:46:37.395Z] INFO[2024-01-19 05:46:36] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-testuser1-1705643043-ieaps] [2024-01-19T05:46:37.395Z] INFO[2024-01-19 05:46:36] Actual:true, Expected: true [2024-01-19T05:46:37.395Z] 2024-01-19 05:46:36 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-testuser1-1705643043-ieaps] is present [2024-01-19T05:46:37.395Z] 2024/01/19 05:46:36 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-testuser1-1705643043-ieaps is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:47:09.433Z] 2024-01-19 05:47:06 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-testuser1-1705643043-ieaps] is present [2024-01-19T05:47:09.433Z] 2024/01/19 05:47:06 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-testuser1-1705643043-ieaps is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T05:47:41.514Z] 2024-01-19 05:47:36 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 aws-global-aws-px-backup-system-test-4cac76dc-bl autogenerated-backup-location-1705628002 autogenerated-backup-location-1705634221 autogenerated-backup-location-1705635982 autogenerated-backup-location-wkyfv] [2024-01-19T05:47:41.514Z] INFO[2024-01-19 05:47:36] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-testuser1-1705643043-ieaps [2024-01-19T05:47:41.514Z] INFO[2024-01-19 05:47:36] Actual:true, Expected: true [2024-01-19T05:47:41.515Z] INFO[2024-01-19 05:47:36] Verifying : Description : Verifying if cloud cred [autogenerated-cred-testuser1-1705643043-dvtop] is present [2024-01-19T05:47:41.515Z] INFO[2024-01-19 05:47:36] Actual:true, Expected: true [2024-01-19T05:47:41.515Z] INFO[2024-01-19 05:47:36] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T05:47:41.515Z] INFO[2024-01-19 05:47:36] Actual:true, Expected: true [2024-01-19T05:47:41.515Z] INFO[2024-01-19 05:47:36] Verifying : Description : Deleting cluster source-cluster [2024-01-19T05:47:41.515Z] ERRO[2024-01-19 05:47:36] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl], cannot be deleted, Expected:  [2024-01-19T05:47:41.515Z] INFO[2024-01-19 05:47:36] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-hrfiy] [2024-01-19T05:47:41.515Z] ERRO[2024-01-19 05:47:36] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-hrfiy:72b1b409-7bb1-4f22-92df-0ba5e77c11d3], Expected:  [2024-01-19T05:47:41.515Z] INFO[2024-01-19 05:47:36] Verifying : Description : Deleting cluster source-cluster [2024-01-19T05:47:41.515Z] ERRO[2024-01-19 05:47:36] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh], cannot be deleted, Expected:  [2024-01-19T05:47:41.515Z] INFO[2024-01-19 05:47:37] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-fbvvo] [2024-01-19T05:47:41.515Z] ERRO[2024-01-19 05:47:37] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-fbvvo:7cddb5aa-c7ff-4044-9d27-458be2bd5cde], Expected:  [2024-01-19T05:47:41.515Z] INFO[2024-01-19 05:47:37] Verifying : Description : Deleting cluster source-cluster [2024-01-19T05:47:41.515Z] ERRO[2024-01-19 05:47:37] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo], cannot be deleted, Expected:  [2024-01-19T05:47:41.515Z] INFO[2024-01-19 05:47:38] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-enzij] [2024-01-19T05:47:41.515Z] ERRO[2024-01-19 05:47:38] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-enzij:4e0ddfc7-8811-4ed5-8e87-ed0841582111], Expected:  [2024-01-19T05:47:41.515Z] INFO[2024-01-19 05:47:38] Verifying : Description : Deleting cluster source-cluster [2024-01-19T05:47:41.515Z] ERRO[2024-01-19 05:47:38] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo], cannot be deleted, Expected:  [2024-01-19T05:47:41.516Z] INFO[2024-01-19 05:47:40] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-wcpfm] [2024-01-19T05:47:41.516Z] ERRO[2024-01-19 05:47:40] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-wcpfm:c5ff084a-a913-4412-99fb-e2b0a003a014], Expected:  [2024-01-19T05:47:41.516Z] INFO[2024-01-19 05:47:40] Verifying : Description : Deleting cluster source-cluster [2024-01-19T05:47:41.516Z] ERRO[2024-01-19 05:47:40] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx], cannot be deleted, Expected:  [2024-01-19T05:47:41.773Z] INFO[2024-01-19 05:47:41] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-vlbwu] [2024-01-19T05:47:41.773Z] ERRO[2024-01-19 05:47:41] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-vlbwu:f0465bb5-2565-4948-bb27-7e45a4b883ff], Expected:  [2024-01-19T05:47:41.773Z] INFO[2024-01-19 05:47:41] Verifying : Description : Deleting cluster source-cluster [2024-01-19T05:47:41.773Z] ERRO[2024-01-19 05:47:41] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa], cannot be deleted, Expected:  [2024-01-19T05:47:43.140Z] INFO[2024-01-19 05:47:42] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-mhxzi] [2024-01-19T05:47:43.140Z] ERRO[2024-01-19 05:47:42] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-mhxzi:f551fb81-1a78-4d7f-b804-759c6e6b562e], Expected:  [2024-01-19T05:47:43.397Z] INFO[2024-01-19 05:47:43] Verifying : Description : Deleting cluster source-cluster [2024-01-19T05:47:43.397Z] INFO[2024-01-19 05:47:43] Actual:true, Expected: true [2024-01-19T05:47:44.761Z] INFO[2024-01-19 05:47:44] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-nahtg] [2024-01-19T05:47:44.761Z] INFO[2024-01-19 05:47:44] Actual:true, Expected: true [2024-01-19T05:47:44.761Z] INFO[2024-01-19 05:47:44] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T05:47:44.761Z] INFO[2024-01-19 05:47:44] Actual:true, Expected: true [2024-01-19T05:47:46.125Z] INFO[2024-01-19 05:47:45] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-uuakz] [2024-01-19T05:47:46.125Z] INFO[2024-01-19 05:47:45] Actual:true, Expected: true [2024-01-19T05:47:47.052Z] 2024-01-19 05:47:46 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705643045 - d98d4a0f-f890-42db-80af-7cb6f11e6ecf [2024-01-19T05:47:47.613Z] 2024-01-19 05:47:47 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.DeleteUser:#755] - Response for user [testuser1-1705643045] deletion - [2024-01-19T05:47:47.613Z] 2024-01-19 05:47:47 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.DeleteUser:#759] - Deleted User - testuser1-1705643045 [2024-01-19T05:47:48.545Z] 2024-01-19 05:47:48 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705643048 - 4bf73ecc-57b6-473b-ac64-ece52b1597cf [2024-01-19T05:47:49.107Z] 2024-01-19 05:47:49 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.DeleteUser:#755] - Response for user [testuser1-1705643048] deletion - [2024-01-19T05:47:49.107Z] 2024-01-19 05:47:49 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [backup.DeleteUser:#759] - Deleted User - testuser1-1705643048 [2024-01-19T05:47:49.107Z] INFO[2024-01-19 05:47:49] --------Test End------ [2024-01-19T05:47:49.107Z] INFO[2024-01-19 05:47:49] #Test: [2024-01-19T05:47:49.107Z] INFO[2024-01-19 05:47:49] #Description: [2024-01-19T05:47:49.107Z] INFO[2024-01-19 05:47:49] ------------------------ [2024-01-19T05:47:49.366Z] 2024-01-19 05:47:49 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-87562-01-19-05h42m31s] [2024-01-19T05:47:49.366Z] 2024-01-19 05:47:49 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T05:47:49.366Z] 2024-01-19 05:47:49 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T05:47:49.928Z] 2024-01-19 05:47:49 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-87562-01-19-05h42m31s] [2024-01-19T05:47:49.929Z] 2024-01-19 05:47:49 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {DeleteSharedBackupOfUserFromAdmin} Deletes backups shared by the user from the admin [2024-01-19T05:47:49.929Z] 2024-01-19 05:47:49 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T05:47:49.929Z] 2024/01/19 05:47:49 DoRetryWithTimeout - Error: {dial tcp 10.240.64.27:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:47:55.184Z] 2024/01/19 05:47:54 DoRetryWithTimeout - Error: {dial tcp 10.240.64.27:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:48:00.428Z] 2024/01/19 05:47:59 DoRetryWithTimeout - Error: {dial tcp 10.240.64.27:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:48:05.683Z] 2024/01/19 05:48:04 DoRetryWithTimeout - Error: {dial tcp 10.240.64.27:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:48:09.849Z] 2024/01/19 05:48:09 DoRetryWithTimeout - Error: {dial tcp 10.240.64.27:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:48:15.128Z] 2024/01/19 05:48:14 DoRetryWithTimeout - Error: {dial tcp 10.240.64.27:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:48:20.387Z] 2024/01/19 05:48:19 DoRetryWithTimeout - Error: {dial tcp 10.240.64.27:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:48:25.634Z] 2024/01/19 05:48:24 DoRetryWithTimeout - Error: {dial tcp 10.240.64.27:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T05:48:29.799Z] 2024/01/19 05:48:29 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.64.27. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.27 10.240.64.27]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:48:29.799Z] {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:48:29.799Z] {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:48:29.799Z] {dial tcp 10.240.64.27:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T05:48:35.041Z] 2024-01-19 05:48:34 +0000:[WARNING] [{DeleteSharedBackupOfUserFromAdmin}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.64.27. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.27 10.240.64.27]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:48:35.041Z] {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:48:35.041Z] {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:48:35.041Z] {dial tcp 10.240.64.27:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T05:48:35.041Z] 2024-01-19 05:48:34 +0000:[ERROR] [{DeleteSharedBackupOfUserFromAdmin}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.64.27. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.64.27 10.240.64.27]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:48:35.041Z] {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:48:35.041Z] {dial tcp 10.240.64.27:22: connect: connection refused} [2024-01-19T05:48:35.041Z] {dial tcp 10.240.64.27:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T05:48:35.041Z] 2024-01-19 05:48:34 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T05:48:35.041Z] 2024-01-19 05:48:34 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] Switching context to [/tmp/source-config] [2024-01-19T05:48:35.041Z] 2024-01-19 05:48:34 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T05:48:35.041Z] 2024-01-19 05:48:34 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T05:48:35.041Z] 2024-01-19 05:48:34 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T05:48:35.041Z] 2024-01-19 05:48:34 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T05:48:35.297Z] 2024-01-19 05:48:35 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T05:48:35.297Z] 2024-01-19 05:48:35 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T05:48:35.859Z] 2024-01-19 05:48:35 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T05:48:36.115Z] 2024-01-19 05:48:35 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T05:48:36.372Z] 2024-01-19 05:48:36 +0000:[INFO] [{DeleteSharedBackupOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T05:48:36.628Z] 2024/01/19 05:48:36 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:48:48.792Z] 2024/01/19 05:48:46 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:48:56.886Z] 2024/01/19 05:48:56 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:49:06.826Z] 2024/01/19 05:49:06 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:49:16.782Z] 2024/01/19 05:49:16 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:49:26.745Z] 2024/01/19 05:49:26 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:49:38.911Z] 2024/01/19 05:49:36 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:49:47.011Z] 2024/01/19 05:49:46 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:49:56.953Z] 2024/01/19 05:49:56 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:50:06.896Z] 2024/01/19 05:50:06 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:50:19.071Z] 2024/01/19 05:50:16 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:50:27.161Z] 2024/01/19 05:50:26 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:50:37.110Z] 2024/01/19 05:50:36 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:50:47.049Z] 2024/01/19 05:50:46 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:50:58.613Z] 2024/01/19 05:50:56 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:51:08.559Z] 2024/01/19 05:51:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:51:18.504Z] 2024/01/19 05:51:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:51:28.466Z] 2024/01/19 05:51:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:51:38.408Z] 2024/01/19 05:51:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:51:48.349Z] 2024/01/19 05:51:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:51:58.288Z] 2024/01/19 05:51:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:52:08.230Z] 2024/01/19 05:52:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:52:18.188Z] 2024/01/19 05:52:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:52:28.137Z] 2024/01/19 05:52:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:52:38.104Z] 2024/01/19 05:52:37 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:52:48.054Z] 2024/01/19 05:52:47 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:52:57.994Z] 2024/01/19 05:52:57 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:53:07.936Z] 2024/01/19 05:53:07 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:53:17.932Z] 2024/01/19 05:53:17 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:53:27.879Z] 2024/01/19 05:53:27 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T05:53:37.829Z] 2024-01-19 05:53:37 +0000:[ERROR] [{DeleteSharedBackupOfUserFromAdmin}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.829Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.830Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.830Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.830Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.830Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.830Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.830Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.830Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.830Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.830Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.830Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.830Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.830Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.830Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.830Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.830Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.830Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.830Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T05:53:37.830Z] • Failure [666.435 seconds] [2024-01-19T05:53:37.830Z] {DeleteSharedBackupOfUserFromAdmin} [2024-01-19T05:53:37.830Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_orphaned_test.go:1560 [2024-01-19T05:53:37.830Z] Deletes backups shared by the user from the admin [] [2024-01-19T05:53:37.830Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_orphaned_test.go:1594 [2024-01-19T05:53:37.830Z] [2024-01-19T05:53:37.830Z] Unexpected error: [2024-01-19T05:53:37.830Z] <*errors.errorString | 0xc00181b250>: { [2024-01-19T05:53:37.832Z] s: "failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{No... [2024-01-19T05:53:37.832Z] [2024-01-19T05:53:37.832Z] Gomega truncated this representation as it exceeds 'format.MaxLength'. [2024-01-19T05:53:37.832Z] Consider having the object provide a custom 'GomegaStringer' representation [2024-01-19T05:53:37.832Z] or adjust the parameters in Gomega's 'format' package. [2024-01-19T05:53:37.832Z] [2024-01-19T05:53:37.832Z] Learn more here: https://onsi.github.io/gomega/#adjusting-output [2024-01-19T05:53:37.832Z] [2024-01-19T05:53:37.832Z] failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.832Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.832Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.832Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.832Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.832Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.832Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.832Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.832Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.832Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.832Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.832Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.832Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.832Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.832Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.833Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.833Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.833Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.833Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.833Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.833Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.833Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.833Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.833Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.833Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.833Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.833Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.833Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.833Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T05:53:37.833Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T05:53:37.833Z] occurred [2024-01-19T05:53:37.833Z] [2024-01-19T05:53:37.833Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T05:53:37.833Z] [2024-01-19T05:53:37.833Z] Full Stack Trace [2024-01-19T05:53:37.833Z] github.com/portworx/torpedo/pkg/log.FailOnError({0x88a5060, 0xc00181b250}, {0x7f74bfd?, 0x6cc0320?}, {0x0?, 0x6cc0320?, 0xc001356320?}) [2024-01-19T05:53:37.833Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 +0x290 [2024-01-19T05:53:37.833Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest.func1() [2024-01-19T05:53:37.833Z] /go/src/github.com/portworx/torpedo/tests/common.go:7283 +0x32 [2024-01-19T05:53:37.833Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest({0xc000ed8180, 0x1, 0x1}) [2024-01-19T05:53:37.833Z] /go/src/github.com/portworx/torpedo/tests/common.go:7299 +0x778 [2024-01-19T05:53:37.833Z] github.com/portworx/torpedo/tests/backup.glob..func30.3() [2024-01-19T05:53:37.833Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_orphaned_test.go:1784 +0x377 [2024-01-19T05:53:37.833Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T05:53:37.833Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T05:53:37.833Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T05:53:37.834Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T05:53:37.834Z] created by testing.(*T).Run [2024-01-19T05:53:37.834Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T05:53:37.834Z] ------------------------------ [2024-01-19T05:53:37.834Z] STEP: schedule applications [2024-01-19T05:53:37.834Z] STEP: Validate applications [2024-01-19T05:53:37.834Z] STEP: validate applications [2024-01-19T05:53:37.834Z] STEP: validate mysql-ibm app's volumes [2024-01-19T05:53:37.834Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T05:53:37.834Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T05:53:37.834Z] STEP: get mysql-ibm app's volume: pvc-c0b1d91d-4874-4bd8-8a75-55b7c1857201 inspected by the volume driver [2024-01-19T05:53:37.834Z] STEP: wait for mysql-ibm app to start running [2024-01-19T05:53:37.834Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T05:53:37.834Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T05:53:37.834Z] STEP: Validate Px pod restart count [2024-01-19T05:53:37.834Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T05:53:37.834Z] STEP: Create 3 users with px-backup-infra.admin role [2024-01-19T05:53:37.834Z] STEP: Create cloud credential and backup location from the user testuser1-1705643043 [2024-01-19T05:53:37.834Z] STEP: Create source and destination cluster from the user testuser1-1705643043 [2024-01-19T05:53:37.834Z] STEP: Take backup of applications from the user testuser1-1705643043 [2024-01-19T05:53:37.834Z] STEP: Share user testuser1-1705643043 backups [backup-share] with user testuser1-1705643045 with ViewOnlyAccess [2024-01-19T05:53:37.834Z] STEP: Verify shared backups of the user testuser1-1705643043 from the user testuser1-1705643045 [2024-01-19T05:53:37.834Z] STEP: Share user testuser1-1705643043 backups [cluster-share] with user testuser1-1705643048 with ViewOnlyAccess [2024-01-19T05:53:37.834Z] STEP: Verify shared backups of the user testuser1-1705643043 from the user testuser1-1705643048 [2024-01-19T05:53:37.834Z] STEP: Verify shared backups of the user testuser1-1705643043 from the admin [2024-01-19T05:53:37.834Z] STEP: Delete the owner [testuser1-1705643043] of the backups [2024-01-19T05:53:37.834Z] STEP: Delete user testuser1-1705643043 backups from the admin [2024-01-19T05:53:37.834Z] STEP: destroy apps [2024-01-19T05:53:37.834Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T05:53:37.834Z] STEP: start destroying mysql-ibm app [2024-01-19T05:53:37.834Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T05:53:37.834Z] 2024-01-19 05:53:37 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Log Dir: /testresults//ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress.log [2024-01-19T05:53:37.834Z] INFO[2024-01-19 05:53:37] --------Test Start------ [2024-01-19T05:53:37.834Z] INFO[2024-01-19 05:53:37] #Test: ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress [2024-01-19T05:53:37.834Z] INFO[2024-01-19 05:53:37] #Description: Scale down px-backup deployment to 0 when backups and restores are in progress [2024-01-19T05:53:37.834Z] INFO[2024-01-19 05:53:37] ------------------------ [2024-01-19T05:53:37.834Z] 2024-01-19 05:53:37 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Deploying applications required for the testcase [2024-01-19T05:53:37.834Z] 2024-01-19 05:53:37 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T05:53:37.834Z] 2024-01-19 05:53:37 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T05:53:37.834Z] 2024-01-19 05:53:37 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T05:53:37.834Z] 2024-01-19 05:53:37 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T05:53:37.835Z] 2024-01-19 05:53:37 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T05:53:37.835Z] 2024-01-19 05:53:37 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T05:53:37.835Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T05:53:37.835Z] 2024-01-19 05:53:37 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T05:53:37.836Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T05:53:37.836Z] 2024-01-19 05:53:37 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T05:53:37.836Z] 2024-01-19 05:53:37 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Validating the deployed applications [2024-01-19T05:53:37.836Z] 2024-01-19 05:53:37 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Validate applications [2024-01-19T05:53:37.836Z] 2024-01-19 05:53:37 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Validating mysql-ibm app [2024-01-19T05:53:37.836Z] 2024-01-19 05:53:37 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Validating mysql-ibm app's volumes [2024-01-19T05:53:38.092Z] 2024-01-19 05:53:37 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T05:53:38.092Z] 2024/01/19 05:53:37 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T05:53:48.033Z] 2024/01/19 05:53:47 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T05:53:57.988Z] 2024-01-19 05:53:57 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-58074-01-19-05h53m37s [2024-01-19T05:53:57.988Z] 2024-01-19 05:53:57 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T05:53:58.244Z] 2024-01-19 05:53:58 +0000:[WARNING] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T05:53:58.244Z] 2024-01-19 05:53:58 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] wait for mysql-ibm app to start running [2024-01-19T05:53:58.244Z] 2024-01-19 05:53:58 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T05:53:58.244Z] 2024/01/19 05:53:58 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:53:58.244Z] pod name:mysql-7d766cd65c-zfkxc namespace:mysql-ibm-pxb-0-58074-01-19-05h53m37s running:false ready:false node:10.240.64.8 [2024-01-19T05:53:58.244Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:54:10.413Z] 2024/01/19 05:54:08 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:54:10.413Z] pod name:mysql-7d766cd65c-zfkxc namespace:mysql-ibm-pxb-0-58074-01-19-05h53m37s running:true ready:false node:10.240.64.8 [2024-01-19T05:54:10.413Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:54:18.542Z] 2024/01/19 05:54:18 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:54:18.542Z] pod name:mysql-7d766cd65c-zfkxc namespace:mysql-ibm-pxb-0-58074-01-19-05h53m37s running:true ready:false node:10.240.64.8 [2024-01-19T05:54:18.542Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:54:28.491Z] 2024/01/19 05:54:28 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:54:28.491Z] pod name:mysql-7d766cd65c-zfkxc namespace:mysql-ibm-pxb-0-58074-01-19-05h53m37s running:true ready:false node:10.240.64.8 [2024-01-19T05:54:28.491Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:54:40.662Z] 2024/01/19 05:54:38 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:54:40.662Z] pod name:mysql-7d766cd65c-zfkxc namespace:mysql-ibm-pxb-0-58074-01-19-05h53m37s running:true ready:false node:10.240.64.8 [2024-01-19T05:54:40.662Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:54:48.752Z] 2024/01/19 05:54:48 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:54:48.752Z] pod name:mysql-7d766cd65c-zfkxc namespace:mysql-ibm-pxb-0-58074-01-19-05h53m37s running:true ready:false node:10.240.64.8 [2024-01-19T05:54:48.752Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:54:58.693Z] 2024/01/19 05:54:58 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:54:58.693Z] pod name:mysql-7d766cd65c-zfkxc namespace:mysql-ibm-pxb-0-58074-01-19-05h53m37s running:true ready:false node:10.240.64.8 [2024-01-19T05:54:58.693Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:55:00.065Z] INFO[2024-01-19 05:54:59] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T05:55:00.065Z] INFO[2024-01-19 05:54:59] sched stats: tick interval: 51659 < 1.2s (100%) [2024-01-19T05:55:00.065Z] INFO[2024-01-19 05:54:59] sched stats: tick process durations: 51658 < 500ms (100%) [2024-01-19T05:55:10.004Z] 2024/01/19 05:55:08 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T05:55:10.004Z] pod name:mysql-7d766cd65c-zfkxc namespace:mysql-ibm-pxb-0-58074-01-19-05h53m37s running:true ready:false node:10.240.64.8 [2024-01-19T05:55:10.004Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T05:55:20.012Z] 2024-01-19 05:55:18 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T05:55:20.012Z] 2024-01-19 05:55:18 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T05:55:20.013Z] 2024-01-19 05:55:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] validate if mysql-ibm app's volumes are setup [2024-01-19T05:55:20.013Z] 2024-01-19 05:55:19 +0000:[WARNING] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T05:55:20.013Z] 2024-01-19 05:55:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T05:55:20.013Z] 2024-01-19 05:55:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T05:55:20.013Z] 2024-01-19 05:55:19 +0000:[WARNING] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T05:55:20.013Z] 2024-01-19 05:55:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Adding cloud credential and backup location [2024-01-19T05:55:20.013Z] 2024-01-19 05:55:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [cloudcred-aws-1705643719] for org [default] with [aws] as provider [2024-01-19T05:55:20.013Z] 2024-01-19 05:55:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T05:55:20.013Z] 2024-01-19 05:55:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cloudcred-aws-1705643719] is present [2024-01-19T05:55:20.013Z] INFO[2024-01-19 05:55:19] Verifying : Description : Verifying creation of cloud credential named [cloudcred-aws-1705643719] for org [default] with [aws] as provider [2024-01-19T05:55:20.013Z] INFO[2024-01-19 05:55:19] Actual:true, Expected: true [2024-01-19T05:55:23.290Z] INFO[2024-01-19 05:55:22] Verifying : Description : Creating backup location aws-global-aws-px-backup-system-test-4cac76dc-1705643719-bl [2024-01-19T05:55:23.290Z] INFO[2024-01-19 05:55:22] Actual:true, Expected: true [2024-01-19T05:55:23.291Z] 2024-01-19 05:55:22 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Registering source and destination clusters for backup [2024-01-19T05:55:23.291Z] INFO[2024-01-19 05:55:22] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T05:55:23.291Z] INFO[2024-01-19 05:55:22] Actual:true, Expected: true [2024-01-19T05:55:23.291Z] 2024-01-19 05:55:22 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T05:55:23.291Z] 2024-01-19 05:55:22 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T05:55:23.291Z] 2024-01-19 05:55:22 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T05:55:23.291Z] 2024-01-19 05:55:22 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T05:55:23.291Z] 2024-01-19 05:55:22 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Create cluster [source-cluster] in org [default] [2024-01-19T05:55:23.291Z] 2024-01-19 05:55:22 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T05:55:23.291Z] 2024-01-19 05:55:22 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T05:55:23.291Z] 2024-01-19 05:55:22 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Create cluster [destination-cluster] in org [default] [2024-01-19T05:55:23.291Z] 2024-01-19 05:55:22 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T05:55:23.291Z] 2024-01-19 05:55:22 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T05:55:23.291Z] 2024-01-19 05:55:22 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-hajtm] for IBM [2024-01-19T05:55:23.291Z] 2024-01-19 05:55:22 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-hajtm] for org [default] with [ibm] as provider [2024-01-19T05:55:23.291Z] 2024-01-19 05:55:22 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T05:55:23.291Z] 2024-01-19 05:55:22 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T05:55:23.291Z] 2024-01-19 05:55:23 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-hajtm] is present [2024-01-19T05:55:23.291Z] 2024-01-19 05:55:23 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Create cluster [source-cluster] in org [default] [2024-01-19T05:55:27.465Z] 2024-01-19 05:55:26 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-uzmhh] for IBM [2024-01-19T05:55:27.465Z] 2024-01-19 05:55:26 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-uzmhh] for org [default] with [ibm] as provider [2024-01-19T05:55:27.465Z] 2024-01-19 05:55:26 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T05:55:27.465Z] 2024-01-19 05:55:26 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T05:55:27.465Z] 2024-01-19 05:55:26 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-uzmhh] is present [2024-01-19T05:55:27.465Z] 2024-01-19 05:55:26 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Create cluster [destination-cluster] in org [default] [2024-01-19T05:55:27.723Z] INFO[2024-01-19 05:55:27] Verifying : Description : Creating source cluster source-cluster and destination cluster destination-cluster [2024-01-19T05:55:27.723Z] INFO[2024-01-19 05:55:27] Actual:true, Expected: true [2024-01-19T05:55:27.723Z] INFO[2024-01-19 05:55:27] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T05:55:27.723Z] INFO[2024-01-19 05:55:27] Actual:Online, Expected: Online [2024-01-19T05:55:27.723Z] INFO[2024-01-19 05:55:27] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T05:55:27.723Z] INFO[2024-01-19 05:55:27] Actual:true, Expected: true [2024-01-19T05:55:27.723Z] INFO[2024-01-19 05:55:27] Verifying : Description : Verifying if [destination-cluster] cluster is online [2024-01-19T05:55:27.723Z] INFO[2024-01-19 05:55:27] Actual:Online, Expected: Online [2024-01-19T05:55:27.723Z] 2024-01-19 05:55:27 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.glob..func53.2.4:#1403] - Getting the replica factor of px-backup deployment in backup namespace [px-backup] before taking backup [2024-01-19T05:55:27.723Z] 2024-01-19 05:55:27 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.glob..func53.2.4:#1407] - Replica count for px-backup pod before taking backup is 1 [2024-01-19T05:55:27.723Z] INFO[2024-01-19 05:55:27] Verifying : Description : Verifying px_backup deployment replica before taking backup [2024-01-19T05:55:27.723Z] INFO[2024-01-19 05:55:27] Actual:true, Expected: true [2024-01-19T05:55:27.724Z] 2024-01-19 05:55:27 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Taking backup of applications [2024-01-19T05:55:32.969Z] 2024-01-19 05:55:32 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T05:55:33.225Z] 2024-01-19 05:55:32 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Volume snapshot classes found in the cluster - [2024-01-19T05:55:33.225Z] 2024-01-19 05:55:32 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] ibmc-vpcblock-snapshot [2024-01-19T05:55:33.225Z] 2024-01-19 05:55:32 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T05:55:33.489Z] INFO[2024-01-19 05:55:33] Verifying : Description : Taking backup [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-0-1705643732] of application [mysql-ibm-pxb-0-58074-01-19-05h53m37s] [2024-01-19T05:55:33.489Z] INFO[2024-01-19 05:55:33] Actual:true, Expected: true [2024-01-19T05:55:38.799Z] 2024-01-19 05:55:37 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T05:55:38.799Z] 2024-01-19 05:55:37 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Volume snapshot classes found in the cluster - [2024-01-19T05:55:38.799Z] 2024-01-19 05:55:37 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] ibmc-vpcblock-snapshot [2024-01-19T05:55:38.799Z] 2024-01-19 05:55:37 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T05:55:38.799Z] INFO[2024-01-19 05:55:38] Verifying : Description : Taking backup [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737] of application [mysql-ibm-pxb-0-58074-01-19-05h53m37s] [2024-01-19T05:55:38.799Z] INFO[2024-01-19 05:55:38] Actual:true, Expected: true [2024-01-19T05:55:42.962Z] 2024-01-19 05:55:42 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T05:55:43.217Z] 2024-01-19 05:55:42 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Volume snapshot classes found in the cluster - [2024-01-19T05:55:43.217Z] 2024-01-19 05:55:42 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] ibmc-vpcblock-snapshot [2024-01-19T05:55:43.217Z] 2024-01-19 05:55:42 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T05:55:43.778Z] INFO[2024-01-19 05:55:43] Verifying : Description : Taking backup [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-2-1705643742] of application [mysql-ibm-pxb-0-58074-01-19-05h53m37s] [2024-01-19T05:55:43.778Z] INFO[2024-01-19 05:55:43] Actual:true, Expected: true [2024-01-19T05:55:47.949Z] 2024-01-19 05:55:47 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T05:55:48.207Z] 2024-01-19 05:55:47 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Volume snapshot classes found in the cluster - [2024-01-19T05:55:48.207Z] 2024-01-19 05:55:47 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] ibmc-vpcblock-snapshot [2024-01-19T05:55:48.207Z] 2024-01-19 05:55:47 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T05:55:48.462Z] INFO[2024-01-19 05:55:48] Verifying : Description : Taking backup [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-3-1705643747] of application [mysql-ibm-pxb-0-58074-01-19-05h53m37s] [2024-01-19T05:55:48.462Z] INFO[2024-01-19 05:55:48] Actual:true, Expected: true [2024-01-19T05:55:48.463Z] 2024-01-19 05:55:48 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] The list of backups taken are: [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-0-1705643732 tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737 tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-2-1705643742 tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-3-1705643747] [2024-01-19T05:55:48.463Z] 2024-01-19 05:55:48 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Scaling px-backup deployment replica count to 0 while backup is in progress [2024-01-19T05:55:48.463Z] 2024-01-19 05:55:48 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.glob..func53.2.6:#1440] - px-backup replica count after scaling to 0 is 0 [2024-01-19T05:55:48.463Z] INFO[2024-01-19 05:55:48] Verifying : Description : Verify px-backup deployment replica after scaling down [2024-01-19T05:55:48.463Z] INFO[2024-01-19 05:55:48] Actual:true, Expected: true [2024-01-19T05:55:48.463Z] 2024-01-19 05:55:48 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Sleeping for 10 minute while backup is in progress and px-backup deployment's replica count is 0 [2024-01-19T06:05:10.030Z] INFO[2024-01-19 06:05:00] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T06:05:10.030Z] INFO[2024-01-19 06:05:00] sched stats: tick interval: 52260 < 1.2s (100%) [2024-01-19T06:05:10.030Z] INFO[2024-01-19 06:05:00] sched stats: tick process durations: 52259 < 500ms (100%) [2024-01-19T06:05:48.700Z] 2024-01-19 06:05:48 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Scaling px-backup deployment to original replica while backup is in progress [2024-01-19T06:05:48.700Z] INFO[2024-01-19 06:05:48] Verifying : Description : Verify px-backup deployment replica after scaling back to original [2024-01-19T06:05:48.700Z] INFO[2024-01-19 06:05:48] Actual:1, Expected: 1 [2024-01-19T06:05:48.700Z] 2024-01-19 06:05:48 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.glob..func53.2.6:#1451] - Verify px-backup pod status after scaling the replica count to original [2024-01-19T06:05:48.700Z] 2024/01/19 06:05:48 DoRetryWithTimeout - Error: {px-backup pod is not ready yet}, Next try in [30s], timeout [20m0s] [2024-01-19T06:06:20.751Z] 2024/01/19 06:06:18 DoRetryWithTimeout - Error: {px-backup pod is not ready yet}, Next try in [30s], timeout [20m0s] [2024-01-19T06:06:52.813Z] 2024/01/19 06:06:48 DoRetryWithTimeout - Error: {px-backup pod is not ready yet}, Next try in [30s], timeout [20m0s] [2024-01-19T06:07:19.332Z] 2024-01-19 06:07:18 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.glob..func53.2.6:#1464] - Number of px-backup pods in Ready state are 1 [2024-01-19T06:07:19.332Z] INFO[2024-01-19 06:07:18] Verifying : Description : Verifying if the px-backup pod is in Ready state [2024-01-19T06:07:19.332Z] INFO[2024-01-19 06:07:18] Actual:true, Expected: true [2024-01-19T06:07:19.332Z] 2024-01-19 06:07:18 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Check if backup is successful after px-backup deployment is scaled back to original replica [2024-01-19T06:07:19.332Z] 2024-01-19 06:07:18 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Validating backup [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-0-1705643732] in org [default] [2024-01-19T06:07:19.332Z] 2024-01-19 06:07:18 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-0-1705643732] [2024-01-19T06:07:19.332Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58074-01-19-05h53m37s] in backup [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-0-1705643732] [2024-01-19T06:07:19.332Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58074-01-19-05h53m37s] in backup [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-0-1705643732] [2024-01-19T06:07:19.333Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58074-01-19-05h53m37s] [2024-01-19T06:07:19.333Z] 2024-01-19 06:07:19 +0000:[WARNING] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T06:07:19.333Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T06:07:19.333Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-58074-01-19-05h53m37s] are [[mysql-data]] [2024-01-19T06:07:19.333Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T06:07:19.333Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Validating CR cleanup [2024-01-19T06:07:19.333Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T06:07:19.333Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] All CRs in [mysql-ibm-pxb-0-58074-01-19-05h53m37s] are [[]] [2024-01-19T06:07:19.333Z] INFO[2024-01-19 06:07:19] Verifying : Description : Verification of success and Validation of the backup [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-0-1705643732] [2024-01-19T06:07:19.333Z] INFO[2024-01-19 06:07:19] Actual:true, Expected: true [2024-01-19T06:07:19.590Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Validating backup [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737] in org [default] [2024-01-19T06:07:19.591Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737] [2024-01-19T06:07:19.591Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58074-01-19-05h53m37s] in backup [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737] [2024-01-19T06:07:19.591Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58074-01-19-05h53m37s] in backup [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737] [2024-01-19T06:07:19.591Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58074-01-19-05h53m37s] [2024-01-19T06:07:19.591Z] 2024-01-19 06:07:19 +0000:[WARNING] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T06:07:19.591Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T06:07:19.591Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-58074-01-19-05h53m37s] are [[mysql-data]] [2024-01-19T06:07:19.591Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T06:07:19.591Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Validating CR cleanup [2024-01-19T06:07:19.846Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T06:07:19.846Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] All CRs in [mysql-ibm-pxb-0-58074-01-19-05h53m37s] are [[]] [2024-01-19T06:07:19.846Z] INFO[2024-01-19 06:07:19] Verifying : Description : Verification of success and Validation of the backup [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737] [2024-01-19T06:07:19.846Z] INFO[2024-01-19 06:07:19] Actual:true, Expected: true [2024-01-19T06:07:19.846Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Validating backup [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-2-1705643742] in org [default] [2024-01-19T06:07:19.847Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-2-1705643742] [2024-01-19T06:07:19.847Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58074-01-19-05h53m37s] in backup [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-2-1705643742] [2024-01-19T06:07:19.847Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58074-01-19-05h53m37s] in backup [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-2-1705643742] [2024-01-19T06:07:19.847Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58074-01-19-05h53m37s] [2024-01-19T06:07:20.138Z] 2024-01-19 06:07:19 +0000:[WARNING] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T06:07:20.138Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T06:07:20.138Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-58074-01-19-05h53m37s] are [[mysql-data]] [2024-01-19T06:07:20.138Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T06:07:20.138Z] 2024-01-19 06:07:19 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Validating CR cleanup [2024-01-19T06:07:20.138Z] 2024-01-19 06:07:20 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T06:07:20.138Z] 2024-01-19 06:07:20 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] All CRs in [mysql-ibm-pxb-0-58074-01-19-05h53m37s] are [[]] [2024-01-19T06:07:20.138Z] INFO[2024-01-19 06:07:20] Verifying : Description : Verification of success and Validation of the backup [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-2-1705643742] [2024-01-19T06:07:20.138Z] INFO[2024-01-19 06:07:20] Actual:true, Expected: true [2024-01-19T06:07:20.395Z] 2024-01-19 06:07:20 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Validating backup [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-3-1705643747] in org [default] [2024-01-19T06:07:20.395Z] 2024-01-19 06:07:20 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-3-1705643747] [2024-01-19T06:07:20.395Z] 2024-01-19 06:07:20 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58074-01-19-05h53m37s] in backup [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-3-1705643747] [2024-01-19T06:07:20.395Z] 2024-01-19 06:07:20 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58074-01-19-05h53m37s] in backup [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-3-1705643747] [2024-01-19T06:07:20.395Z] 2024-01-19 06:07:20 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-58074-01-19-05h53m37s] [2024-01-19T06:07:20.395Z] 2024-01-19 06:07:20 +0000:[WARNING] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T06:07:20.395Z] 2024-01-19 06:07:20 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T06:07:20.396Z] 2024-01-19 06:07:20 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-58074-01-19-05h53m37s] are [[mysql-data]] [2024-01-19T06:07:20.396Z] 2024-01-19 06:07:20 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T06:07:20.396Z] 2024-01-19 06:07:20 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Validating CR cleanup [2024-01-19T06:07:20.655Z] 2024-01-19 06:07:20 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T06:07:20.655Z] 2024-01-19 06:07:20 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] All CRs in [mysql-ibm-pxb-0-58074-01-19-05h53m37s] are [[]] [2024-01-19T06:07:20.655Z] INFO[2024-01-19 06:07:20] Verifying : Description : Verification of success and Validation of the backup [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-3-1705643747] [2024-01-19T06:07:20.655Z] INFO[2024-01-19 06:07:20] Actual:true, Expected: true [2024-01-19T06:07:20.655Z] 2024-01-19 06:07:20 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Restoring the backups taken [2024-01-19T06:07:22.023Z] 2024-01-19 06:07:21 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.CreateRestoreWithoutCheck:#1089] - Getting the UID of the backup needed to be restored [2024-01-19T06:07:22.024Z] 2024-01-19 06:07:21 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.CreateRestoreWithoutCheck:#1089] - Getting the UID of the backup needed to be restored [2024-01-19T06:07:22.024Z] 2024-01-19 06:07:21 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.CreateRestoreWithoutCheck:#1089] - Getting the UID of the backup needed to be restored [2024-01-19T06:07:22.024Z] 2024-01-19 06:07:21 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.CreateRestoreWithoutCheck:#1089] - Getting the UID of the backup needed to be restored [2024-01-19T06:07:22.024Z] 2024-01-19 06:07:21 +0000:[DEBUG] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.CreateRestoreWithoutCheck:#1093] - Enumerate backup response - [2024-01-19T06:07:22.054Z] backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"91bbc09a-215b-439b-ab0c-92ce6a19ce34" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"6b37bd8f-0745-426f-8c4c-dd38e36b91aa" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"3a02d016-11cb-4c7f-8788-4ceaf8415cfc" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705643748 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"f5e5d306-256a-4fb2-bc8d-3d14a8395806" > backup_path:"mysql-ibm-pxb-0-58074-01-19-05h53m37s/tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-3-1705643747-ec8288b/cd79a492-95c2-458e-b8ec-6707e6ed5156" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-3-1705643747-ec8288b" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"cd79a492-95c2-458e-b8ec-6707e6ed5156" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58074-01-19-05h53m37s" > > backups: last_update_time: create_time_in_sec:1705643743 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"f5e5d306-256a-4fb2-bc8d-3d14a8395806" > backup_path:"mysql-ibm-pxb-0-58074-01-19-05h53m37s/tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-2-1705643742-fd339fb/c61992de-a28b-4d9f-8cc0-6b31a008833b" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-2-1705643742-fd339fb" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"c61992de-a28b-4d9f-8cc0-6b31a008833b" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58074-01-19-05h53m37s" > > backups: last_update_time: create_time_in_sec:1705643737 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"f5e5d306-256a-4fb2-bc8d-3d14a8395806" > backup_path:"mysql-ibm-pxb-0-58074-01-19-05h53m37s/tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-0346f69/ebdcc4b7-0530-4b28-adda-afb34eddfc3f" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-0346f69" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"ebdcc4b7-0530-4b28-adda-afb34eddfc3f" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58074-01-19-05h53m37s" > > backups: last_update_time: create_time_in_sec:1705643733 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"f5e5d306-256a-4fb2-bc8d-3d14a8395806" > backup_path:"mysql-ibm-pxb-0-58074-01-19-05h53m37s/tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-0-1705643732-fc2b4a0/7fe9a8df-72a6-4d6b-bd49-5d25bb840685" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-0-1705643732-fc2b4a0" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"7fe9a8df-72a6-4d6b-bd49-5d25bb840685" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58074-01-19-05h53m37s" > > backups: last_update_time: create_time_in_sec:1705642899 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-054139/3aa4643e-9295-46cf-a06b-a4c07518d9d8" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-054139" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"3aa4643e-9295-46cf-a06b-a4c07518d9d8" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705642889 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-054129/0b7672cd-25fc-43e6-aa86-840056c9daba" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-054129" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"0b7672cd-25fc-43e6-aa86-840056c9daba" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705642869 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-054109/f8982d89-a20f-4f17-88e2-0a945c67016e" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-054109" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"f8982d89-a20f-4f17-88e2-0a945c67016e" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705641996 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-052636/32e1598a-433b-4ba5-8c3e-77739af62ee1" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-052636" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"32e1598a-433b-4ba5-8c3e-77739af62ee1" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705641986 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-052626/e184a208-93ff-48f5-b489-ebf277965b70" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-052626" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"e184a208-93ff-48f5-b489-ebf277965b70" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705641965 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-052605/91af1c68-b944-4724-8ca0-bb83bed5f953" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-052605" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"91af1c68-b944-4724-8ca0-bb83bed5f953" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705641092 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-051131/4aab834b-760d-4478-9c81-54ff3ace0342" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-051131" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"4aab834b-760d-4478-9c81-54ff3ace0342" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705641082 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-051122/48a7f22f-f5ee-46a5-ad19-9491d4784295" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-051122" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"48a7f22f-f5ee-46a5-ad19-9491d4784295" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705641061 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-051101/5118f5ae-42fe-4e48-b072-db0d9927765a" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-051101" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"5118f5ae-42fe-4e48-b072-db0d9927765a" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705640188 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-045628/eb64c9e4-c28c-44e5-a15e-b82d3b4577f3" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-045628" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"eb64c9e4-c28c-44e5-a15e-b82d3b4577f3" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705640179 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-045619/c62e5464-4874-4f57-9fd3-81a984109ce5" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-045619" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"c62e5464-4874-4f57-9fd3-81a984109ce5" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705640158 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-045558/43891ca6-f557-4f2d-b642-c85e10bc26be" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-045558" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"43891ca6-f557-4f2d-b642-c85e10bc26be" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705639285 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-044125/7812287c-c3c7-4989-b581-c0ad67883daf" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-044125" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"7812287c-c3c7-4989-b581-c0ad67883daf" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705639276 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-044116/81a3f029-df95-4b19-8849-cdfe37f34949" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-044116" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"81a3f029-df95-4b19-8849-cdfe37f34949" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705639255 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-044055/96f6cbb3-fcf1-4f05-9410-a485b469ed2d" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-044055" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"96f6cbb3-fcf1-4f05-9410-a485b469ed2d" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705638401 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641/2f8c5f33-efe2-4fc8-b4a5-9198fa7a68ea" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"2f8c5f33-efe2-4fc8-b4a5-9198fa7a68ea" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705638391 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631/330ff754-ceae-4d57-a14f-382049ef3ad7" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"330ff754-ceae-4d57-a14f-382049ef3ad7" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705638381 ownership: > > backup_info: stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-042621" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"2ba4999e-c75e-41c9-99b6-16f0bd0d46f2" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705638372 ownership: > > backup_info: stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-042612" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"9de817ec-cbbc-4237-811c-69d4566add84" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705638362 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602/976bcfaa-b5db-4c0a-8ebf-e315258d4d79" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"976bcfaa-b5db-4c0a-8ebf-e315258d4d79" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705638351 ownership: > > backup_info: stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-042551" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"ff44892e-46f8-41bf-8c46-6eb5ca681d32" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637964 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923/06b259d8-4651-423e-9dcf-96116213128a" stage:Final backup_schedule: cr_name:"tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"06b259d8-4651-423e-9dcf-96116213128a" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637954 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913/49d38a79-b3b8-4efd-8576-d7638e4e613d" stage:Final backup_schedule: cr_name:"tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"49d38a79-b3b8-4efd-8576-d7638e4e613d" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637944 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903/c3806637-8570-4e04-8f9b-6f5a92a4c797" stage:Final backup_schedule: cr_name:"tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"c3806637-8570-4e04-8f9b-6f5a92a4c797" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637934 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853/37f14588-139d-41ee-89fd-db97bc346c2a" stage:Final backup_schedule: cr_name:"tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"37f14588-139d-41ee-89fd-db97bc346c2a" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637924 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843/3db43d77-4d00-4f0a-ad6c-845b7bd27c58" stage:Final backup_schedule: cr_name:"tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"3db43d77-4d00-4f0a-ad6c-845b7bd27c58" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637913 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833/38593f1b-e04b-4019-98f3-24dcba9402cd" stage:Final backup_schedule: cr_name:"tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"38593f1b-e04b-4019-98f3-24dcba9402cd" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637622 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > backup_path:"mysql-ibm-pxb-0-87560-01-19-04h09m51s/tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342/d9bf8bec-a042-46f8-afad-4601de5ded66" stage:Final backup_schedule: cr_name:"tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342" total_size:10737418240 resource_count:5 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"d9bf8bec-a042-46f8-afad-4601de5ded66" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-87560-01-19-04h09m51s" > > backups: last_update_time: create_time_in_sec:1705637612 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > backup_path:"mysql-ibm-pxb-0-87560-01-19-04h09m51s/tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332/1ffe6426-de78-4aef-a334-e0f671c795c2" stage:Final backup_schedule: cr_name:"tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332" total_size:10737418240 resource_count:5 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"1ffe6426-de78-4aef-a334-e0f671c795c2" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-87560-01-19-04h09m51s" > > backups: last_update_time: create_time_in_sec:1705637602 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > backup_path:"mysql-ibm-pxb-0-87560-01-19-04h09m51s/tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322/f826b260-75e6-4c35-988a-eabfe09cff23" stage:Final backup_schedule: cr_name:"tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322" total_size:10737418240 resource_count:5 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"f826b260-75e6-4c35-988a-eabfe09cff23" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-87560-01-19-04h09m51s" > > backups: last_update_time: create_time_in_sec:1705637592 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > backup_path:"mysql-ibm-pxb-0-87560-01-19-04h09m51s/tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312/930b83b8-2896-404d-bbf4-6163b31fbb86" stage:Final backup_schedule: cr_name:"tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312" total_size:10737418240 resource_count:5 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"930b83b8-2896-404d-bbf4-6163b31fbb86" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-87560-01-19-04h09m51s" > > backups: last_update_time: create_time_in_sec:1705637582 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > backup_path:"mysql-ibm-pxb-0-87560-01-19-04h09m51s/tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302/9cc88818-757b-4e20-a579-6ef043a0a9d6" stage:Final backup_schedule: cr_name:"tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302" total_size:10737418240 resource_count:5 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"9cc88818-757b-4e20-a579-6ef043a0a9d6" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-87560-01-19-04h09m51s" > > backups: last_update_time: create_time_in_sec:1705637572 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > backup_path:"mysql-ibm-pxb-0-87560-01-19-04h09m51s/tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl-7c3a711-interval-2024-01-19-041252/d3d05800-4a96-4445-8181-b06b9abb74c0" stage:Final backup_schedule: cr_name:"tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl-7c3a711-interval-2024-01-19-041252" total_size:10737418240 resource_count:5 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"d3d05800-4a96-4445-8181-b06b9abb74c0" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-87560-01-19-04h09m51s" > > backups: last_update_time: create_time_in_sec:1705636048 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"90e3a732-2ebc-496e-a729-97557156d692" > backup_path:"mysql-ibm-pxb-0-82935-01-19-03h43m53s/tp-backup-1705636048-1fc8542/cd3e9917-51f1-4555-91bb-4fc7495b7ed2" stage:Final cr_name:"tp-backup-1705636048-1fc8542" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"cd3e9917-51f1-4555-91bb-4fc7495b7ed2" backup_share: collaborators: > user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-82935-01-19-03h43m53s" > > backups: last_update_time: create_time_in_sec:1705636038 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"90e3a732-2ebc-496e-a729-97557156d692" > backup_path:"mysql-ibm-pxb-0-82935-01-19-03h43m53s/tp-backup-1705636038-e1406d0/92b8fa99-cdca-4982-a69a-5d2310e01425" stage:Final cr_name:"tp-backup-1705636038-e1406d0" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"92b8fa99-cdca-4982-a69a-5d2310e01425" backup_share: collaborators: > user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-82935-01-19-03h43m53s" > > backups: last_update_time: create_time_in_sec:1705634227 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bf614768-3741-4b91-8db1-9e09d3532ce9" > backup_path:"mysql-ibm-pxb-0-82950-01-19-03h15m28s/tp-backup-1705634226-42d91ac/94232aaf-8783-4d71-9c10-55927a592a8c" stage:Final cr_name:"tp-backup-1705634226-42d91ac" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"94232aaf-8783-4d71-9c10-55927a592a8c" backup_share: collaborators: collaborators: > user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-82950-01-19-03h15m28s" > > total_count:64 [2024-01-19T06:07:22.054Z] 2024-01-19 06:07:21 +0000:[DEBUG] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.CreateRestoreWithoutCheck:#1093] - Enumerate backup response - [2024-01-19T06:07:22.348Z] backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"91bbc09a-215b-439b-ab0c-92ce6a19ce34" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"6b37bd8f-0745-426f-8c4c-dd38e36b91aa" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"3a02d016-11cb-4c7f-8788-4ceaf8415cfc" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705643748 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"f5e5d306-256a-4fb2-bc8d-3d14a8395806" > backup_path:"mysql-ibm-pxb-0-58074-01-19-05h53m37s/tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-3-1705643747-ec8288b/cd79a492-95c2-458e-b8ec-6707e6ed5156" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-3-1705643747-ec8288b" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"cd79a492-95c2-458e-b8ec-6707e6ed5156" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58074-01-19-05h53m37s" > > backups: last_update_time: create_time_in_sec:1705643743 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"f5e5d306-256a-4fb2-bc8d-3d14a8395806" > backup_path:"mysql-ibm-pxb-0-58074-01-19-05h53m37s/tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-2-1705643742-fd339fb/c61992de-a28b-4d9f-8cc0-6b31a008833b" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-2-1705643742-fd339fb" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"c61992de-a28b-4d9f-8cc0-6b31a008833b" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58074-01-19-05h53m37s" > > backups: last_update_time: create_time_in_sec:1705643737 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"f5e5d306-256a-4fb2-bc8d-3d14a8395806" > backup_path:"mysql-ibm-pxb-0-58074-01-19-05h53m37s/tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-0346f69/ebdcc4b7-0530-4b28-adda-afb34eddfc3f" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-0346f69" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"ebdcc4b7-0530-4b28-adda-afb34eddfc3f" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58074-01-19-05h53m37s" > > backups: last_update_time: create_time_in_sec:1705643733 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"f5e5d306-256a-4fb2-bc8d-3d14a8395806" > backup_path:"mysql-ibm-pxb-0-58074-01-19-05h53m37s/tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-0-1705643732-fc2b4a0/7fe9a8df-72a6-4d6b-bd49-5d25bb840685" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-0-1705643732-fc2b4a0" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"7fe9a8df-72a6-4d6b-bd49-5d25bb840685" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58074-01-19-05h53m37s" > > backups: last_update_time: create_time_in_sec:1705642899 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-054139/3aa4643e-9295-46cf-a06b-a4c07518d9d8" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-054139" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"3aa4643e-9295-46cf-a06b-a4c07518d9d8" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705642889 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-054129/0b7672cd-25fc-43e6-aa86-840056c9daba" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-054129" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"0b7672cd-25fc-43e6-aa86-840056c9daba" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705642869 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-054109/f8982d89-a20f-4f17-88e2-0a945c67016e" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-054109" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"f8982d89-a20f-4f17-88e2-0a945c67016e" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705641996 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-052636/32e1598a-433b-4ba5-8c3e-77739af62ee1" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-052636" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"32e1598a-433b-4ba5-8c3e-77739af62ee1" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705641986 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-052626/e184a208-93ff-48f5-b489-ebf277965b70" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-052626" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"e184a208-93ff-48f5-b489-ebf277965b70" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705641965 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-052605/91af1c68-b944-4724-8ca0-bb83bed5f953" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-052605" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"91af1c68-b944-4724-8ca0-bb83bed5f953" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705641092 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-051131/4aab834b-760d-4478-9c81-54ff3ace0342" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-051131" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"4aab834b-760d-4478-9c81-54ff3ace0342" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705641082 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-051122/48a7f22f-f5ee-46a5-ad19-9491d4784295" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-051122" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"48a7f22f-f5ee-46a5-ad19-9491d4784295" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705641061 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-051101/5118f5ae-42fe-4e48-b072-db0d9927765a" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-051101" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"5118f5ae-42fe-4e48-b072-db0d9927765a" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705640188 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-045628/eb64c9e4-c28c-44e5-a15e-b82d3b4577f3" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-045628" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"eb64c9e4-c28c-44e5-a15e-b82d3b4577f3" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705640179 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-045619/c62e5464-4874-4f57-9fd3-81a984109ce5" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-045619" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"c62e5464-4874-4f57-9fd3-81a984109ce5" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705640158 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-045558/43891ca6-f557-4f2d-b642-c85e10bc26be" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-045558" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"43891ca6-f557-4f2d-b642-c85e10bc26be" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705639285 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-044125/7812287c-c3c7-4989-b581-c0ad67883daf" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-044125" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"7812287c-c3c7-4989-b581-c0ad67883daf" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705639276 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-044116/81a3f029-df95-4b19-8849-cdfe37f34949" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-044116" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"81a3f029-df95-4b19-8849-cdfe37f34949" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705639255 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-044055/96f6cbb3-fcf1-4f05-9410-a485b469ed2d" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-044055" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"96f6cbb3-fcf1-4f05-9410-a485b469ed2d" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705638401 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641/2f8c5f33-efe2-4fc8-b4a5-9198fa7a68ea" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"2f8c5f33-efe2-4fc8-b4a5-9198fa7a68ea" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705638391 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631/330ff754-ceae-4d57-a14f-382049ef3ad7" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"330ff754-ceae-4d57-a14f-382049ef3ad7" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705638381 ownership: > > backup_info: stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-042621" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"2ba4999e-c75e-41c9-99b6-16f0bd0d46f2" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705638372 ownership: > > backup_info: stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-042612" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"9de817ec-cbbc-4237-811c-69d4566add84" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705638362 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602/976bcfaa-b5db-4c0a-8ebf-e315258d4d79" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"976bcfaa-b5db-4c0a-8ebf-e315258d4d79" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705638351 ownership: > > backup_info: stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-042551" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"ff44892e-46f8-41bf-8c46-6eb5ca681d32" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637964 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923/06b259d8-4651-423e-9dcf-96116213128a" stage:Final backup_schedule: cr_name:"tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"06b259d8-4651-423e-9dcf-96116213128a" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637954 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913/49d38a79-b3b8-4efd-8576-d7638e4e613d" stage:Final backup_schedule: cr_name:"tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"49d38a79-b3b8-4efd-8576-d7638e4e613d" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637944 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903/c3806637-8570-4e04-8f9b-6f5a92a4c797" stage:Final backup_schedule: cr_name:"tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"c3806637-8570-4e04-8f9b-6f5a92a4c797" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637934 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853/37f14588-139d-41ee-89fd-db97bc346c2a" stage:Final backup_schedule: cr_name:"tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"37f14588-139d-41ee-89fd-db97bc346c2a" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637924 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843/3db43d77-4d00-4f0a-ad6c-845b7bd27c58" stage:Final backup_schedule: cr_name:"tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"3db43d77-4d00-4f0a-ad6c-845b7bd27c58" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637913 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833/38593f1b-e04b-4019-98f3-24dcba9402cd" stage:Final backup_schedule: cr_name:"tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"38593f1b-e04b-4019-98f3-24dcba9402cd" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637622 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > backup_path:"mysql-ibm-pxb-0-87560-01-19-04h09m51s/tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342/d9bf8bec-a042-46f8-afad-4601de5ded66" stage:Final backup_schedule: cr_name:"tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342" total_size:10737418240 resource_count:5 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"d9bf8bec-a042-46f8-afad-4601de5ded66" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-87560-01-19-04h09m51s" > > backups: last_update_time: create_time_in_sec:1705637612 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > backup_path:"mysql-ibm-pxb-0-87560-01-19-04h09m51s/tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332/1ffe6426-de78-4aef-a334-e0f671c795c2" stage:Final backup_schedule: cr_name:"tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332" total_size:10737418240 resource_count:5 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"1ffe6426-de78-4aef-a334-e0f671c795c2" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-87560-01-19-04h09m51s" > > backups: last_update_time: create_time_in_sec:1705637602 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > backup_path:"mysql-ibm-pxb-0-87560-01-19-04h09m51s/tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322/f826b260-75e6-4c35-988a-eabfe09cff23" stage:Final backup_schedule: cr_name:"tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322" total_size:10737418240 resource_count:5 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"f826b260-75e6-4c35-988a-eabfe09cff23" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-87560-01-19-04h09m51s" > > backups: last_update_time: create_time_in_sec:1705637592 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > backup_path:"mysql-ibm-pxb-0-87560-01-19-04h09m51s/tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312/930b83b8-2896-404d-bbf4-6163b31fbb86" stage:Final backup_schedule: cr_name:"tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312" total_size:10737418240 resource_count:5 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"930b83b8-2896-404d-bbf4-6163b31fbb86" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-87560-01-19-04h09m51s" > > backups: last_update_time: create_time_in_sec:1705637582 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > backup_path:"mysql-ibm-pxb-0-87560-01-19-04h09m51s/tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302/9cc88818-757b-4e20-a579-6ef043a0a9d6" stage:Final backup_schedule: cr_name:"tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302" total_size:10737418240 resource_count:5 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"9cc88818-757b-4e20-a579-6ef043a0a9d6" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-87560-01-19-04h09m51s" > > backups: last_update_time: create_time_in_sec:1705637572 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > backup_path:"mysql-ibm-pxb-0-87560-01-19-04h09m51s/tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl-7c3a711-interval-2024-01-19-041252/d3d05800-4a96-4445-8181-b06b9abb74c0" stage:Final backup_schedule: cr_name:"tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl-7c3a711-interval-2024-01-19-041252" total_size:10737418240 resource_count:5 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"d3d05800-4a96-4445-8181-b06b9abb74c0" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-87560-01-19-04h09m51s" > > backups: last_update_time: create_time_in_sec:1705636048 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"90e3a732-2ebc-496e-a729-97557156d692" > backup_path:"mysql-ibm-pxb-0-82935-01-19-03h43m53s/tp-backup-1705636048-1fc8542/cd3e9917-51f1-4555-91bb-4fc7495b7ed2" stage:Final cr_name:"tp-backup-1705636048-1fc8542" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"cd3e9917-51f1-4555-91bb-4fc7495b7ed2" backup_share: collaborators: > user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-82935-01-19-03h43m53s" > > backups: last_update_time: create_time_in_sec:1705636038 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"90e3a732-2ebc-496e-a729-97557156d692" > backup_path:"mysql-ibm-pxb-0-82935-01-19-03h43m53s/tp-backup-1705636038-e1406d0/92b8fa99-cdca-4982-a69a-5d2310e01425" stage:Final cr_name:"tp-backup-1705636038-e1406d0" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"92b8fa99-cdca-4982-a69a-5d2310e01425" backup_share: collaborators: > user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-82935-01-19-03h43m53s" > > backups: last_update_time: create_time_in_sec:1705634227 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bf614768-3741-4b91-8db1-9e09d3532ce9" > backup_path:"mysql-ibm-pxb-0-82950-01-19-03h15m28s/tp-backup-1705634226-42d91ac/94232aaf-8783-4d71-9c10-55927a592a8c" stage:Final cr_name:"tp-backup-1705634226-42d91ac" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"94232aaf-8783-4d71-9c10-55927a592a8c" backup_share: collaborators: collaborators: > user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-82950-01-19-03h15m28s" > > total_count:64 [2024-01-19T06:07:22.348Z] 2024-01-19 06:07:21 +0000:[DEBUG] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.CreateRestoreWithoutCheck:#1093] - Enumerate backup response - [2024-01-19T06:07:22.379Z] backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"91bbc09a-215b-439b-ab0c-92ce6a19ce34" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"6b37bd8f-0745-426f-8c4c-dd38e36b91aa" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"3a02d016-11cb-4c7f-8788-4ceaf8415cfc" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705643748 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"f5e5d306-256a-4fb2-bc8d-3d14a8395806" > backup_path:"mysql-ibm-pxb-0-58074-01-19-05h53m37s/tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-3-1705643747-ec8288b/cd79a492-95c2-458e-b8ec-6707e6ed5156" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-3-1705643747-ec8288b" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"cd79a492-95c2-458e-b8ec-6707e6ed5156" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58074-01-19-05h53m37s" > > backups: last_update_time: create_time_in_sec:1705643743 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"f5e5d306-256a-4fb2-bc8d-3d14a8395806" > backup_path:"mysql-ibm-pxb-0-58074-01-19-05h53m37s/tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-2-1705643742-fd339fb/c61992de-a28b-4d9f-8cc0-6b31a008833b" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-2-1705643742-fd339fb" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"c61992de-a28b-4d9f-8cc0-6b31a008833b" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58074-01-19-05h53m37s" > > backups: last_update_time: create_time_in_sec:1705643737 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"f5e5d306-256a-4fb2-bc8d-3d14a8395806" > backup_path:"mysql-ibm-pxb-0-58074-01-19-05h53m37s/tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-0346f69/ebdcc4b7-0530-4b28-adda-afb34eddfc3f" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-0346f69" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"ebdcc4b7-0530-4b28-adda-afb34eddfc3f" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58074-01-19-05h53m37s" > > backups: last_update_time: create_time_in_sec:1705643733 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"f5e5d306-256a-4fb2-bc8d-3d14a8395806" > backup_path:"mysql-ibm-pxb-0-58074-01-19-05h53m37s/tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-0-1705643732-fc2b4a0/7fe9a8df-72a6-4d6b-bd49-5d25bb840685" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-0-1705643732-fc2b4a0" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"7fe9a8df-72a6-4d6b-bd49-5d25bb840685" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58074-01-19-05h53m37s" > > backups: last_update_time: create_time_in_sec:1705642899 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-054139/3aa4643e-9295-46cf-a06b-a4c07518d9d8" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-054139" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"3aa4643e-9295-46cf-a06b-a4c07518d9d8" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705642889 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-054129/0b7672cd-25fc-43e6-aa86-840056c9daba" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-054129" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"0b7672cd-25fc-43e6-aa86-840056c9daba" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705642869 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-054109/f8982d89-a20f-4f17-88e2-0a945c67016e" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-054109" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"f8982d89-a20f-4f17-88e2-0a945c67016e" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705641996 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-052636/32e1598a-433b-4ba5-8c3e-77739af62ee1" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-052636" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"32e1598a-433b-4ba5-8c3e-77739af62ee1" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705641986 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-052626/e184a208-93ff-48f5-b489-ebf277965b70" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-052626" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"e184a208-93ff-48f5-b489-ebf277965b70" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705641965 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-052605/91af1c68-b944-4724-8ca0-bb83bed5f953" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-052605" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"91af1c68-b944-4724-8ca0-bb83bed5f953" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705641092 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-051131/4aab834b-760d-4478-9c81-54ff3ace0342" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-051131" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"4aab834b-760d-4478-9c81-54ff3ace0342" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705641082 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-051122/48a7f22f-f5ee-46a5-ad19-9491d4784295" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-051122" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"48a7f22f-f5ee-46a5-ad19-9491d4784295" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705641061 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-051101/5118f5ae-42fe-4e48-b072-db0d9927765a" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-051101" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"5118f5ae-42fe-4e48-b072-db0d9927765a" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705640188 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-045628/eb64c9e4-c28c-44e5-a15e-b82d3b4577f3" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-045628" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"eb64c9e4-c28c-44e5-a15e-b82d3b4577f3" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705640179 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-045619/c62e5464-4874-4f57-9fd3-81a984109ce5" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-045619" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"c62e5464-4874-4f57-9fd3-81a984109ce5" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705640158 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-045558/43891ca6-f557-4f2d-b642-c85e10bc26be" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-045558" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"43891ca6-f557-4f2d-b642-c85e10bc26be" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705639285 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-044125/7812287c-c3c7-4989-b581-c0ad67883daf" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-044125" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"7812287c-c3c7-4989-b581-c0ad67883daf" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705639276 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-044116/81a3f029-df95-4b19-8849-cdfe37f34949" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-044116" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"81a3f029-df95-4b19-8849-cdfe37f34949" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705639255 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-044055/96f6cbb3-fcf1-4f05-9410-a485b469ed2d" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-044055" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"96f6cbb3-fcf1-4f05-9410-a485b469ed2d" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705638401 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641/2f8c5f33-efe2-4fc8-b4a5-9198fa7a68ea" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"2f8c5f33-efe2-4fc8-b4a5-9198fa7a68ea" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705638391 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631/330ff754-ceae-4d57-a14f-382049ef3ad7" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"330ff754-ceae-4d57-a14f-382049ef3ad7" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705638381 ownership: > > backup_info: stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-042621" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"2ba4999e-c75e-41c9-99b6-16f0bd0d46f2" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705638372 ownership: > > backup_info: stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-042612" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"9de817ec-cbbc-4237-811c-69d4566add84" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705638362 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602/976bcfaa-b5db-4c0a-8ebf-e315258d4d79" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"976bcfaa-b5db-4c0a-8ebf-e315258d4d79" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705638351 ownership: > > backup_info: stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-042551" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"ff44892e-46f8-41bf-8c46-6eb5ca681d32" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637964 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923/06b259d8-4651-423e-9dcf-96116213128a" stage:Final backup_schedule: cr_name:"tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"06b259d8-4651-423e-9dcf-96116213128a" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637954 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913/49d38a79-b3b8-4efd-8576-d7638e4e613d" stage:Final backup_schedule: cr_name:"tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"49d38a79-b3b8-4efd-8576-d7638e4e613d" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637944 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903/c3806637-8570-4e04-8f9b-6f5a92a4c797" stage:Final backup_schedule: cr_name:"tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"c3806637-8570-4e04-8f9b-6f5a92a4c797" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637934 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853/37f14588-139d-41ee-89fd-db97bc346c2a" stage:Final backup_schedule: cr_name:"tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"37f14588-139d-41ee-89fd-db97bc346c2a" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637924 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843/3db43d77-4d00-4f0a-ad6c-845b7bd27c58" stage:Final backup_schedule: cr_name:"tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"3db43d77-4d00-4f0a-ad6c-845b7bd27c58" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637913 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833/38593f1b-e04b-4019-98f3-24dcba9402cd" stage:Final backup_schedule: cr_name:"tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"38593f1b-e04b-4019-98f3-24dcba9402cd" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637622 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > backup_path:"mysql-ibm-pxb-0-87560-01-19-04h09m51s/tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342/d9bf8bec-a042-46f8-afad-4601de5ded66" stage:Final backup_schedule: cr_name:"tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342" total_size:10737418240 resource_count:5 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"d9bf8bec-a042-46f8-afad-4601de5ded66" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-87560-01-19-04h09m51s" > > backups: last_update_time: create_time_in_sec:1705637612 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > backup_path:"mysql-ibm-pxb-0-87560-01-19-04h09m51s/tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332/1ffe6426-de78-4aef-a334-e0f671c795c2" stage:Final backup_schedule: cr_name:"tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332" total_size:10737418240 resource_count:5 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"1ffe6426-de78-4aef-a334-e0f671c795c2" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-87560-01-19-04h09m51s" > > backups: last_update_time: create_time_in_sec:1705637602 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > backup_path:"mysql-ibm-pxb-0-87560-01-19-04h09m51s/tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322/f826b260-75e6-4c35-988a-eabfe09cff23" stage:Final backup_schedule: cr_name:"tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322" total_size:10737418240 resource_count:5 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"f826b260-75e6-4c35-988a-eabfe09cff23" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-87560-01-19-04h09m51s" > > backups: last_update_time: create_time_in_sec:1705637592 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > backup_path:"mysql-ibm-pxb-0-87560-01-19-04h09m51s/tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312/930b83b8-2896-404d-bbf4-6163b31fbb86" stage:Final backup_schedule: cr_name:"tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312" total_size:10737418240 resource_count:5 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"930b83b8-2896-404d-bbf4-6163b31fbb86" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-87560-01-19-04h09m51s" > > backups: last_update_time: create_time_in_sec:1705637582 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > backup_path:"mysql-ibm-pxb-0-87560-01-19-04h09m51s/tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302/9cc88818-757b-4e20-a579-6ef043a0a9d6" stage:Final backup_schedule: cr_name:"tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302" total_size:10737418240 resource_count:5 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"9cc88818-757b-4e20-a579-6ef043a0a9d6" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-87560-01-19-04h09m51s" > > backups: last_update_time: create_time_in_sec:1705637572 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > backup_path:"mysql-ibm-pxb-0-87560-01-19-04h09m51s/tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl-7c3a711-interval-2024-01-19-041252/d3d05800-4a96-4445-8181-b06b9abb74c0" stage:Final backup_schedule: cr_name:"tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl-7c3a711-interval-2024-01-19-041252" total_size:10737418240 resource_count:5 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"d3d05800-4a96-4445-8181-b06b9abb74c0" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-87560-01-19-04h09m51s" > > backups: last_update_time: create_time_in_sec:1705636048 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"90e3a732-2ebc-496e-a729-97557156d692" > backup_path:"mysql-ibm-pxb-0-82935-01-19-03h43m53s/tp-backup-1705636048-1fc8542/cd3e9917-51f1-4555-91bb-4fc7495b7ed2" stage:Final cr_name:"tp-backup-1705636048-1fc8542" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"cd3e9917-51f1-4555-91bb-4fc7495b7ed2" backup_share: collaborators: > user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-82935-01-19-03h43m53s" > > backups: last_update_time: create_time_in_sec:1705636038 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"90e3a732-2ebc-496e-a729-97557156d692" > backup_path:"mysql-ibm-pxb-0-82935-01-19-03h43m53s/tp-backup-1705636038-e1406d0/92b8fa99-cdca-4982-a69a-5d2310e01425" stage:Final cr_name:"tp-backup-1705636038-e1406d0" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"92b8fa99-cdca-4982-a69a-5d2310e01425" backup_share: collaborators: > user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-82935-01-19-03h43m53s" > > backups: last_update_time: create_time_in_sec:1705634227 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bf614768-3741-4b91-8db1-9e09d3532ce9" > backup_path:"mysql-ibm-pxb-0-82950-01-19-03h15m28s/tp-backup-1705634226-42d91ac/94232aaf-8783-4d71-9c10-55927a592a8c" stage:Final cr_name:"tp-backup-1705634226-42d91ac" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"94232aaf-8783-4d71-9c10-55927a592a8c" backup_share: collaborators: collaborators: > user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-82950-01-19-03h15m28s" > > total_count:64 [2024-01-19T06:07:22.379Z] 2024-01-19 06:07:21 +0000:[DEBUG] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.CreateRestoreWithoutCheck:#1093] - Enumerate backup response - [2024-01-19T06:07:22.411Z] backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643803 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643/91bbc09a-215b-439b-ab0c-92ce6a19ce34" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"91bbc09a-215b-439b-ab0c-92ce6a19ce34" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643793 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633/6b37bd8f-0745-426f-8c4c-dd38e36b91aa" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"6b37bd8f-0745-426f-8c4c-dd38e36b91aa" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: user_backupshare_access:FullAccess ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" > > backups: last_update_time: create_time_in_sec:1705643772 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612/3a02d016-11cb-4c7f-8788-4ceaf8415cfc" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"3a02d016-11cb-4c7f-8788-4ceaf8415cfc" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705643748 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"f5e5d306-256a-4fb2-bc8d-3d14a8395806" > backup_path:"mysql-ibm-pxb-0-58074-01-19-05h53m37s/tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-3-1705643747-ec8288b/cd79a492-95c2-458e-b8ec-6707e6ed5156" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-3-1705643747-ec8288b" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"cd79a492-95c2-458e-b8ec-6707e6ed5156" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58074-01-19-05h53m37s" > > backups: last_update_time: create_time_in_sec:1705643743 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"f5e5d306-256a-4fb2-bc8d-3d14a8395806" > backup_path:"mysql-ibm-pxb-0-58074-01-19-05h53m37s/tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-2-1705643742-fd339fb/c61992de-a28b-4d9f-8cc0-6b31a008833b" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-2-1705643742-fd339fb" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"c61992de-a28b-4d9f-8cc0-6b31a008833b" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58074-01-19-05h53m37s" > > backups: last_update_time: create_time_in_sec:1705643737 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"f5e5d306-256a-4fb2-bc8d-3d14a8395806" > backup_path:"mysql-ibm-pxb-0-58074-01-19-05h53m37s/tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-0346f69/ebdcc4b7-0530-4b28-adda-afb34eddfc3f" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-0346f69" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"ebdcc4b7-0530-4b28-adda-afb34eddfc3f" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58074-01-19-05h53m37s" > > backups: last_update_time: create_time_in_sec:1705643733 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"f5e5d306-256a-4fb2-bc8d-3d14a8395806" > backup_path:"mysql-ibm-pxb-0-58074-01-19-05h53m37s/tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-0-1705643732-fc2b4a0/7fe9a8df-72a6-4d6b-bd49-5d25bb840685" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-0-1705643732-fc2b4a0" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"7fe9a8df-72a6-4d6b-bd49-5d25bb840685" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-58074-01-19-05h53m37s" > > backups: last_update_time: create_time_in_sec:1705642899 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-054139/3aa4643e-9295-46cf-a06b-a4c07518d9d8" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-054139" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"3aa4643e-9295-46cf-a06b-a4c07518d9d8" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705642889 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-054129/0b7672cd-25fc-43e6-aa86-840056c9daba" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-054129" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"0b7672cd-25fc-43e6-aa86-840056c9daba" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705642869 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-054109/f8982d89-a20f-4f17-88e2-0a945c67016e" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-054109" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"f8982d89-a20f-4f17-88e2-0a945c67016e" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705641996 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-052636/32e1598a-433b-4ba5-8c3e-77739af62ee1" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-052636" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"32e1598a-433b-4ba5-8c3e-77739af62ee1" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705641986 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-052626/e184a208-93ff-48f5-b489-ebf277965b70" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-052626" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"e184a208-93ff-48f5-b489-ebf277965b70" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705641965 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-052605/91af1c68-b944-4724-8ca0-bb83bed5f953" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-052605" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"91af1c68-b944-4724-8ca0-bb83bed5f953" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705641092 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-051131/4aab834b-760d-4478-9c81-54ff3ace0342" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-051131" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"4aab834b-760d-4478-9c81-54ff3ace0342" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705641082 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-051122/48a7f22f-f5ee-46a5-ad19-9491d4784295" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-051122" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"48a7f22f-f5ee-46a5-ad19-9491d4784295" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705641061 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-051101/5118f5ae-42fe-4e48-b072-db0d9927765a" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-051101" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"5118f5ae-42fe-4e48-b072-db0d9927765a" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705640188 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-045628/eb64c9e4-c28c-44e5-a15e-b82d3b4577f3" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-045628" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"eb64c9e4-c28c-44e5-a15e-b82d3b4577f3" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705640179 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-045619/c62e5464-4874-4f57-9fd3-81a984109ce5" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-045619" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"c62e5464-4874-4f57-9fd3-81a984109ce5" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705640158 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-045558/43891ca6-f557-4f2d-b642-c85e10bc26be" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-045558" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"43891ca6-f557-4f2d-b642-c85e10bc26be" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705639285 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-044125/7812287c-c3c7-4989-b581-c0ad67883daf" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-044125" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"7812287c-c3c7-4989-b581-c0ad67883daf" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705639276 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-044116/81a3f029-df95-4b19-8849-cdfe37f34949" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-044116" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"81a3f029-df95-4b19-8849-cdfe37f34949" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705639255 ownership: > > backup_info: backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-044055/96f6cbb3-fcf1-4f05-9410-a485b469ed2d" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-044055" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"96f6cbb3-fcf1-4f05-9410-a485b469ed2d" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705638401 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641/2f8c5f33-efe2-4fc8-b4a5-9198fa7a68ea" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"2f8c5f33-efe2-4fc8-b4a5-9198fa7a68ea" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705638391 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631/330ff754-ceae-4d57-a14f-382049ef3ad7" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"330ff754-ceae-4d57-a14f-382049ef3ad7" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705638381 ownership: > > backup_info: stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-042621" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"2ba4999e-c75e-41c9-99b6-16f0bd0d46f2" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705638372 ownership: > > backup_info: stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-042612" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"9de817ec-cbbc-4237-811c-69d4566add84" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705638362 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602/976bcfaa-b5db-4c0a-8ebf-e315258d4d79" stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"976bcfaa-b5db-4c0a-8ebf-e315258d4d79" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705638351 ownership: > > backup_info: stage:Final backup_schedule: cr_name:"tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-042551" backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"ff44892e-46f8-41bf-8c46-6eb5ca681d32" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-4172c3ae-23ff-46c1-8d53-9fbf67478e7a=4172c3ae-23ff-46c1-8d53-9fbf67478e7a" target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637964 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923/06b259d8-4651-423e-9dcf-96116213128a" stage:Final backup_schedule: cr_name:"tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"06b259d8-4651-423e-9dcf-96116213128a" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637954 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913/49d38a79-b3b8-4efd-8576-d7638e4e613d" stage:Final backup_schedule: cr_name:"tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"49d38a79-b3b8-4efd-8576-d7638e4e613d" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637944 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903/c3806637-8570-4e04-8f9b-6f5a92a4c797" stage:Final backup_schedule: cr_name:"tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"c3806637-8570-4e04-8f9b-6f5a92a4c797" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637934 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853/37f14588-139d-41ee-89fd-db97bc346c2a" stage:Final backup_schedule: cr_name:"tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"37f14588-139d-41ee-89fd-db97bc346c2a" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637924 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843/3db43d77-4d00-4f0a-ad6c-845b7bd27c58" stage:Final backup_schedule: cr_name:"tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"3db43d77-4d00-4f0a-ad6c-845b7bd27c58" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637913 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"92a4a8b6-9bb3-4d44-8dab-f56b6602acd7" > backup_path:"kube-system/tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833/38593f1b-e04b-4019-98f3-24dcba9402cd" stage:Final backup_schedule: cr_name:"tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"38593f1b-e04b-4019-98f3-24dcba9402cd" user_backupshare_access:FullAccess cluster_ref: target_namespace:"kube-system" > > backups: last_update_time: create_time_in_sec:1705637622 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > backup_path:"mysql-ibm-pxb-0-87560-01-19-04h09m51s/tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342/d9bf8bec-a042-46f8-afad-4601de5ded66" stage:Final backup_schedule: cr_name:"tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342" total_size:10737418240 resource_count:5 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"d9bf8bec-a042-46f8-afad-4601de5ded66" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-87560-01-19-04h09m51s" > > backups: last_update_time: create_time_in_sec:1705637612 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > backup_path:"mysql-ibm-pxb-0-87560-01-19-04h09m51s/tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332/1ffe6426-de78-4aef-a334-e0f671c795c2" stage:Final backup_schedule: cr_name:"tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332" total_size:10737418240 resource_count:5 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"1ffe6426-de78-4aef-a334-e0f671c795c2" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-87560-01-19-04h09m51s" > > backups: last_update_time: create_time_in_sec:1705637602 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > backup_path:"mysql-ibm-pxb-0-87560-01-19-04h09m51s/tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322/f826b260-75e6-4c35-988a-eabfe09cff23" stage:Final backup_schedule: cr_name:"tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322" total_size:10737418240 resource_count:5 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"f826b260-75e6-4c35-988a-eabfe09cff23" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-87560-01-19-04h09m51s" > > backups: last_update_time: create_time_in_sec:1705637592 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > backup_path:"mysql-ibm-pxb-0-87560-01-19-04h09m51s/tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312/930b83b8-2896-404d-bbf4-6163b31fbb86" stage:Final backup_schedule: cr_name:"tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312" total_size:10737418240 resource_count:5 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"930b83b8-2896-404d-bbf4-6163b31fbb86" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-87560-01-19-04h09m51s" > > backups: last_update_time: create_time_in_sec:1705637582 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > backup_path:"mysql-ibm-pxb-0-87560-01-19-04h09m51s/tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302/9cc88818-757b-4e20-a579-6ef043a0a9d6" stage:Final backup_schedule: cr_name:"tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302" total_size:10737418240 resource_count:5 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"9cc88818-757b-4e20-a579-6ef043a0a9d6" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-87560-01-19-04h09m51s" > > backups: last_update_time: create_time_in_sec:1705637572 ownership: > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"03378f3a-b457-4bd9-a6df-8a06e43b2a80" > backup_path:"mysql-ibm-pxb-0-87560-01-19-04h09m51s/tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl-7c3a711-interval-2024-01-19-041252/d3d05800-4a96-4445-8181-b06b9abb74c0" stage:Final backup_schedule: cr_name:"tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl-7c3a711-interval-2024-01-19-041252" total_size:10737418240 resource_count:5 backup_location_ref: cloud_credential_ref: backup_type: cr_uid:"d3d05800-4a96-4445-8181-b06b9abb74c0" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-87560-01-19-04h09m51s" > > backups: last_update_time: create_time_in_sec:1705636048 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"90e3a732-2ebc-496e-a729-97557156d692" > backup_path:"mysql-ibm-pxb-0-82935-01-19-03h43m53s/tp-backup-1705636048-1fc8542/cd3e9917-51f1-4555-91bb-4fc7495b7ed2" stage:Final cr_name:"tp-backup-1705636048-1fc8542" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"cd3e9917-51f1-4555-91bb-4fc7495b7ed2" backup_share: collaborators: > user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-82935-01-19-03h43m53s" > > backups: last_update_time: create_time_in_sec:1705636038 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"90e3a732-2ebc-496e-a729-97557156d692" > backup_path:"mysql-ibm-pxb-0-82935-01-19-03h43m53s/tp-backup-1705636038-e1406d0/92b8fa99-cdca-4982-a69a-5d2310e01425" stage:Final cr_name:"tp-backup-1705636038-e1406d0" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"92b8fa99-cdca-4982-a69a-5d2310e01425" backup_share: collaborators: > user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-82935-01-19-03h43m53s" > > backups: last_update_time: create_time_in_sec:1705634227 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"bf614768-3741-4b91-8db1-9e09d3532ce9" > backup_path:"mysql-ibm-pxb-0-82950-01-19-03h15m28s/tp-backup-1705634226-42d91ac/94232aaf-8783-4d71-9c10-55927a592a8c" stage:Final cr_name:"tp-backup-1705634226-42d91ac" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"94232aaf-8783-4d71-9c10-55927a592a8c" backup_share: collaborators: collaborators: > user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-82950-01-19-03h15m28s" > > total_count:64 [2024-01-19T06:07:22.411Z] INFO[2024-01-19 06:07:22] Verifying : Description : Restoring the backup tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-0-1705643732 with name tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-0-1705643732-restore-1705644441 [2024-01-19T06:07:22.411Z] INFO[2024-01-19 06:07:22] Actual:true, Expected: true [2024-01-19T06:07:22.411Z] INFO[2024-01-19 06:07:22] Verifying : Description : Restoring the backup tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-3-1705643747 with name tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-3-1705643747-restore-1705644441 [2024-01-19T06:07:22.411Z] INFO[2024-01-19 06:07:22] Actual:true, Expected: true [2024-01-19T06:07:22.412Z] INFO[2024-01-19 06:07:22] Verifying : Description : Restoring the backup tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737 with name tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441 [2024-01-19T06:07:22.412Z] INFO[2024-01-19 06:07:22] Actual:true, Expected: true [2024-01-19T06:07:22.412Z] INFO[2024-01-19 06:07:22] Verifying : Description : Restoring the backup tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-2-1705643742 with name tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-2-1705643742-restore-1705644441 [2024-01-19T06:07:22.412Z] INFO[2024-01-19 06:07:22] Actual:true, Expected: true [2024-01-19T06:07:22.412Z] 2024-01-19 06:07:22 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] The list of restores are: [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-0-1705643732-restore-1705644441 tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441 tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-2-1705643742-restore-1705644441 tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-3-1705643747-restore-1705644441] [2024-01-19T06:07:22.412Z] 2024-01-19 06:07:22 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Scaling px-backup deployment replica to 0 while restore is in progress [2024-01-19T06:07:22.668Z] 2024-01-19 06:07:22 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.glob..func53.2.9:#1501] - px-backup replica after scaling to 0 is 0 [2024-01-19T06:07:22.668Z] INFO[2024-01-19 06:07:22] Verifying : Description : Verifying px-backup deployment replica after scaling down [2024-01-19T06:07:22.668Z] INFO[2024-01-19 06:07:22] Actual:true, Expected: true [2024-01-19T06:07:22.668Z] 2024-01-19 06:07:22 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Sleeping for 10 minute while restore is in progress and px-backup deployment's replica count is 0 [2024-01-19T06:15:14.207Z] INFO[2024-01-19 06:15:01] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T06:15:14.207Z] INFO[2024-01-19 06:15:01] sched stats: tick interval: 52861 < 1.2s (100%) [2024-01-19T06:15:14.207Z] INFO[2024-01-19 06:15:01] sched stats: tick process durations: 52860 < 500ms (100%) [2024-01-19T06:17:35.632Z] 2024-01-19 06:17:22 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Scaling px-backup deployment to original replica while restore is in progress [2024-01-19T06:17:35.632Z] INFO[2024-01-19 06:17:22] Verifying : Description : Verify px-backup deployment replica after scaling back to original replica count [2024-01-19T06:17:35.632Z] INFO[2024-01-19 06:17:22] Actual:1, Expected: 1 [2024-01-19T06:17:35.632Z] 2024-01-19 06:17:22 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.glob..func53.2.9:#1512] - Verify that px-backup pod is in Ready state [2024-01-19T06:17:35.632Z] 2024/01/19 06:17:22 DoRetryWithTimeout - Error: {px-backup pod is not ready yet}, Next try in [30s], timeout [20m0s] [2024-01-19T06:17:53.696Z] 2024/01/19 06:17:52 DoRetryWithTimeout - Error: {px-backup pod is not ready yet}, Next try in [30s], timeout [20m0s] [2024-01-19T06:18:25.753Z] 2024/01/19 06:18:22 DoRetryWithTimeout - Error: {px-backup pod is not ready yet}, Next try in [30s], timeout [20m0s] [2024-01-19T06:18:57.774Z] 2024-01-19 06:18:52 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.glob..func53.2.9:#1525] - Number of px_backup pod in Ready state are 1 [2024-01-19T06:18:57.774Z] INFO[2024-01-19 06:18:52] Verifying : Description : Verifying that px_backup pod is in Ready state [2024-01-19T06:18:57.774Z] INFO[2024-01-19 06:18:52] Actual:true, Expected: true [2024-01-19T06:18:57.774Z] 2024-01-19 06:18:52 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Check if restore is successful after px_backup deployment is scaled back to original replica [2024-01-19T06:18:57.774Z] 2024/01/19 06:18:52 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:19:25.863Z] 2024/01/19 06:19:22 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:19:57.905Z] 2024/01/19 06:19:52 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:20:24.406Z] 2024/01/19 06:20:22 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:20:56.472Z] 2024/01/19 06:20:52 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:21:22.975Z] 2024/01/19 06:21:22 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:21:55.031Z] 2024/01/19 06:21:52 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:22:27.073Z] 2024/01/19 06:22:22 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:22:53.569Z] 2024/01/19 06:22:52 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:23:25.600Z] 2024/01/19 06:23:22 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:23:57.635Z] 2024/01/19 06:23:52 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:24:24.173Z] 2024/01/19 06:24:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:24:56.232Z] 2024/01/19 06:24:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:25:02.806Z] INFO[2024-01-19 06:25:01] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T06:25:02.806Z] INFO[2024-01-19 06:25:01] sched stats: tick interval: 53461 < 1.2s (100%) [2024-01-19T06:25:02.806Z] INFO[2024-01-19 06:25:01] sched stats: tick process durations: 53460 < 500ms (100%) [2024-01-19T06:25:24.693Z] 2024/01/19 06:25:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:25:57.603Z] 2024/01/19 06:25:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:26:24.103Z] 2024/01/19 06:26:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:26:56.138Z] 2024/01/19 06:26:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:27:28.167Z] 2024/01/19 06:27:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:27:54.668Z] 2024/01/19 06:27:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:28:26.696Z] 2024/01/19 06:28:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:28:53.196Z] 2024/01/19 06:28:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:29:25.255Z] 2024/01/19 06:29:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:29:57.287Z] 2024/01/19 06:29:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:30:23.782Z] 2024/01/19 06:30:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:30:55.804Z] 2024/01/19 06:30:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:31:27.872Z] 2024/01/19 06:31:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:31:54.371Z] 2024/01/19 06:31:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:32:26.396Z] 2024/01/19 06:32:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:32:58.430Z] 2024/01/19 06:32:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:33:24.922Z] 2024/01/19 06:33:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:33:56.948Z] 2024/01/19 06:33:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:34:23.449Z] 2024/01/19 06:34:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:34:55.498Z] 2024/01/19 06:34:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:35:03.590Z] INFO[2024-01-19 06:35:02] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T06:35:03.590Z] INFO[2024-01-19 06:35:02] sched stats: tick interval: 54062 < 1.2s (100%) [2024-01-19T06:35:03.590Z] INFO[2024-01-19 06:35:02] sched stats: tick process durations: 54061 < 500ms (100%) [2024-01-19T06:35:25.478Z] 2024/01/19 06:35:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:35:57.520Z] 2024/01/19 06:35:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:36:24.131Z] 2024/01/19 06:36:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:36:56.182Z] 2024/01/19 06:36:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:37:28.201Z] 2024/01/19 06:37:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:37:54.697Z] 2024/01/19 06:37:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:38:26.741Z] 2024/01/19 06:38:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:38:58.759Z] 2024/01/19 06:38:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:39:25.258Z] 2024/01/19 06:39:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:39:57.279Z] 2024/01/19 06:39:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:40:23.778Z] 2024/01/19 06:40:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:40:55.803Z] 2024/01/19 06:40:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:41:27.835Z] 2024/01/19 06:41:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:41:54.348Z] 2024/01/19 06:41:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:42:26.368Z] 2024/01/19 06:42:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:42:58.403Z] 2024/01/19 06:42:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:43:24.966Z] 2024/01/19 06:43:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:43:56.987Z] 2024/01/19 06:43:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:44:29.020Z] 2024/01/19 06:44:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:44:55.509Z] 2024/01/19 06:44:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:45:03.597Z] INFO[2024-01-19 06:45:02] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T06:45:03.597Z] INFO[2024-01-19 06:45:02] sched stats: tick interval: 54662 < 1.2s (100%) [2024-01-19T06:45:03.597Z] INFO[2024-01-19 06:45:02] sched stats: tick process durations: 54661 < 500ms (100%) [2024-01-19T06:45:25.482Z] 2024/01/19 06:45:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:45:57.518Z] 2024/01/19 06:45:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:46:24.021Z] 2024/01/19 06:46:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:46:56.089Z] 2024/01/19 06:46:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:47:28.104Z] 2024/01/19 06:47:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:47:54.631Z] 2024/01/19 06:47:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:48:26.688Z] 2024/01/19 06:48:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:48:58.710Z] 2024/01/19 06:48:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:49:25.202Z] 2024/01/19 06:49:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:49:57.230Z] 2024/01/19 06:49:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:50:29.256Z] 2024/01/19 06:50:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:50:55.769Z] 2024/01/19 06:50:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:51:27.820Z] 2024/01/19 06:51:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:51:54.384Z] 2024/01/19 06:51:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:52:26.435Z] 2024/01/19 06:52:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:52:58.493Z] 2024/01/19 06:52:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:53:24.983Z] 2024/01/19 06:53:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:53:57.017Z] 2024/01/19 06:53:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:54:29.044Z] 2024/01/19 06:54:23 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:54:55.545Z] 2024/01/19 06:54:53 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:55:03.634Z] INFO[2024-01-19 06:55:02] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T06:55:03.634Z] INFO[2024-01-19 06:55:02] sched stats: tick interval: 55262 < 1.2s (100%) [2024-01-19T06:55:03.634Z] INFO[2024-01-19 06:55:02] sched stats: tick process durations: 55261 < 500ms (100%) [2024-01-19T06:55:25.560Z] 2024/01/19 06:55:24 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:55:57.619Z] 2024/01/19 06:55:54 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:56:24.124Z] 2024/01/19 06:56:24 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:56:56.180Z] 2024/01/19 06:56:54 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:57:28.199Z] 2024/01/19 06:57:24 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:57:54.709Z] 2024/01/19 06:57:54 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:58:26.731Z] 2024/01/19 06:58:24 DoRetryWithTimeout - Error: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T06:58:58.758Z] 2024-01-19 06:58:54 +0000:[ERROR] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.glob..func53.2.10:#1534] - Getting the status for restore- tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.758Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.758Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.758Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.758Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.758Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.758Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.758Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.758Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.758Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.758Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.758Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.758Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.758Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.758Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.758Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.759Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.759Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.759Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.759Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.759Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.759Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.759Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.759Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.759Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.759Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.759Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.759Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.759Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.759Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.759Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.759Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.759Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.759Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.760Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.760Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.760Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.760Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.760Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.760Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.760Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.760Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.760Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.760Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.760Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.760Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.760Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.760Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.760Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.760Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.760Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.760Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.761Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.761Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.761Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.761Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.761Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.761Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.761Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.761Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.761Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.761Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.761Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.761Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.761Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.761Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.761Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.761Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.761Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.761Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.762Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.762Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.762Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.762Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.762Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.762Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.762Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.762Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.762Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.762Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T06:58:58.762Z] 2024-01-19 06:58:54 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Updating the px_backup deployment replica count as it was at the start of this testcase [2024-01-19T06:58:58.762Z] 2024-01-19 06:58:54 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.glob..func53.3:#1550] - Verify that all the px_backup deployment pod are in Ready state at the end of the testcase [2024-01-19T06:58:58.762Z] INFO[2024-01-19 06:58:54] Verifying : Description : Validating if the px_backup pod is ready [2024-01-19T06:58:58.762Z] INFO[2024-01-19 06:58:54] Actual:true, Expected: true [2024-01-19T06:58:58.762Z] 2024-01-19 06:58:54 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.glob..func53.3:#1563] - Number of px_backup pods in Ready state are 1 [2024-01-19T06:58:58.762Z] INFO[2024-01-19 06:58:54] Verifying : Description : Verifying that px_backup pod is in Ready state at the end of the testcase [2024-01-19T06:58:58.762Z] INFO[2024-01-19 06:58:54] Actual:true, Expected: true [2024-01-19T06:58:58.762Z] 2024-01-19 06:58:54 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.glob..func53.3:#1567] - Deleting the deployed applications [2024-01-19T06:58:58.762Z] 2024-01-19 06:58:54 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Destroying apps [2024-01-19T06:58:58.762Z] 2024-01-19 06:58:54 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T06:58:58.762Z] 2024-01-19 06:58:54 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T06:58:58.762Z] 2024-01-19 06:58:54 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] start destroying mysql-ibm app [2024-01-19T06:58:58.763Z] 2024-01-19 06:58:54 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T06:58:58.763Z] 2024-01-19 06:58:54 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T06:58:58.763Z] 2024-01-19 06:58:54 +0000:[WARNING] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T06:58:58.763Z] INFO[2024-01-19 06:58:54] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T06:58:58.763Z] INFO[2024-01-19 06:58:54] Actual:true, Expected: true [2024-01-19T06:58:58.763Z] 2024-01-19 06:58:54 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Deleting the restores taken [2024-01-19T06:58:58.763Z] INFO[2024-01-19 06:58:54] Verifying : Description : Getting the backup driver [2024-01-19T06:58:58.763Z] INFO[2024-01-19 06:58:54] Actual:true, Expected: true [2024-01-19T06:58:58.763Z] INFO[2024-01-19 06:58:54] Verifying : Description : Getting the backup driver [2024-01-19T06:58:58.763Z] INFO[2024-01-19 06:58:54] Actual:true, Expected: true [2024-01-19T06:58:58.763Z] INFO[2024-01-19 06:58:54] Verifying : Description : Getting the backup driver [2024-01-19T06:58:58.763Z] INFO[2024-01-19 06:58:54] Actual:true, Expected: true [2024-01-19T06:58:58.763Z] INFO[2024-01-19 06:58:54] Verifying : Description : Getting the backup driver [2024-01-19T06:58:58.763Z] INFO[2024-01-19 06:58:54] Actual:true, Expected: true [2024-01-19T06:58:58.763Z] INFO[2024-01-19 06:58:54] Verifying : Description : Deleting Restore tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-0-1705643732-restore-1705644441 [2024-01-19T06:58:58.763Z] INFO[2024-01-19 06:58:54] Actual:true, Expected: true [2024-01-19T06:58:58.763Z] INFO[2024-01-19 06:58:55] Verifying : Description : Deleting Restore tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441 [2024-01-19T06:58:58.763Z] INFO[2024-01-19 06:58:55] Actual:true, Expected: true [2024-01-19T06:58:58.763Z] INFO[2024-01-19 06:58:55] Verifying : Description : Deleting Restore tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-2-1705643742-restore-1705644441 [2024-01-19T06:58:58.763Z] INFO[2024-01-19 06:58:55] Actual:true, Expected: true [2024-01-19T06:58:58.763Z] INFO[2024-01-19 06:58:55] Verifying : Description : Deleting Restore tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-3-1705643747-restore-1705644441 [2024-01-19T06:58:58.763Z] INFO[2024-01-19 06:58:55] Actual:true, Expected: true [2024-01-19T06:58:58.763Z] 2024-01-19 06:58:55 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Cleaning backup locations in map [map[e1fad01f-6a50-4c2c-9e46-4f6df9365ff8:aws-global-aws-px-backup-system-test-4cac76dc-1705643719-bl]], cloud credential [cloudcred-aws-1705643719], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T06:58:58.763Z] INFO[2024-01-19 06:58:55] Verifying : Description : Verifying deletion of backup location [aws-global-aws-px-backup-system-test-4cac76dc-1705643719-bl] [2024-01-19T06:58:58.763Z] INFO[2024-01-19 06:58:55] Actual:true, Expected: true [2024-01-19T06:58:58.763Z] 2024-01-19 06:58:55 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.IsBackupLocationPresent:#2442] - Backup location [aws-global-aws-px-backup-system-test-4cac76dc-1705643719-bl] is present [2024-01-19T06:58:58.763Z] 2024/01/19 06:58:55 DoRetryWithTimeout - Error: {backup location aws-global-aws-px-backup-system-test-4cac76dc-1705643719-bl is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T06:59:30.791Z] 2024-01-19 06:59:25 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.IsBackupLocationPresent:#2442] - Backup location [aws-global-aws-px-backup-system-test-4cac76dc-1705643719-bl] is present [2024-01-19T06:59:30.791Z] 2024/01/19 06:59:25 DoRetryWithTimeout - Error: {backup location aws-global-aws-px-backup-system-test-4cac76dc-1705643719-bl is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T06:59:57.283Z] 2024-01-19 06:59:55 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.IsBackupLocationPresent:#2442] - Backup location [aws-global-aws-px-backup-system-test-4cac76dc-1705643719-bl] is present [2024-01-19T06:59:57.283Z] 2024/01/19 06:59:55 DoRetryWithTimeout - Error: {backup location aws-global-aws-px-backup-system-test-4cac76dc-1705643719-bl is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:00:29.307Z] 2024-01-19 07:00:25 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.IsBackupLocationPresent:#2442] - Backup location [aws-global-aws-px-backup-system-test-4cac76dc-1705643719-bl] is present [2024-01-19T07:00:29.307Z] 2024/01/19 07:00:25 DoRetryWithTimeout - Error: {backup location aws-global-aws-px-backup-system-test-4cac76dc-1705643719-bl is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:00:55.801Z] 2024-01-19 07:00:55 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 aws-global-aws-px-backup-system-test-4cac76dc-bl autogenerated-backup-location-1705628002 autogenerated-backup-location-1705634221 autogenerated-backup-location-1705635982 autogenerated-backup-location-wkyfv] [2024-01-19T07:00:55.801Z] INFO[2024-01-19 07:00:55] Verifying : Description : Verifying backup location deletion status aws-global-aws-px-backup-system-test-4cac76dc-1705643719-bl [2024-01-19T07:00:55.801Z] INFO[2024-01-19 07:00:55] Actual:true, Expected: true [2024-01-19T07:00:56.056Z] 2024-01-19 07:00:55 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cloudcred-aws-1705643719] is present [2024-01-19T07:00:56.056Z] INFO[2024-01-19 07:00:55] Verifying : Description : Verifying if cloud cred [cloudcred-aws-1705643719] is present [2024-01-19T07:00:56.056Z] INFO[2024-01-19 07:00:55] Actual:true, Expected: true [2024-01-19T07:00:56.056Z] INFO[2024-01-19 07:00:55] Verifying : Description : Verifying deletion of cloud cred [cloudcred-aws-1705643719] [2024-01-19T07:00:56.056Z] INFO[2024-01-19 07:00:55] Actual:true, Expected: true [2024-01-19T07:00:56.312Z] INFO[2024-01-19 07:00:56] Verifying : Description : Deleting cloud cred cloudcred-aws-1705643719 [2024-01-19T07:00:56.312Z] INFO[2024-01-19 07:00:56] Actual:true, Expected: true [2024-01-19T07:00:56.312Z] INFO[2024-01-19 07:00:56] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T07:00:56.312Z] INFO[2024-01-19 07:00:56] Actual:true, Expected: true [2024-01-19T07:00:56.312Z] INFO[2024-01-19 07:00:56] Verifying : Description : Deleting cluster source-cluster [2024-01-19T07:00:56.312Z] ERRO[2024-01-19 07:00:56] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl], cannot be deleted, Expected:  [2024-01-19T07:00:56.873Z] INFO[2024-01-19 07:00:56] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-hrfiy] [2024-01-19T07:00:56.873Z] ERRO[2024-01-19 07:00:56] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-hrfiy:72b1b409-7bb1-4f22-92df-0ba5e77c11d3], Expected:  [2024-01-19T07:00:56.874Z] INFO[2024-01-19 07:00:56] Verifying : Description : Deleting cluster source-cluster [2024-01-19T07:00:56.874Z] ERRO[2024-01-19 07:00:56] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh], cannot be deleted, Expected:  [2024-01-19T07:00:57.164Z] INFO[2024-01-19 07:00:57] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-fbvvo] [2024-01-19T07:00:57.164Z] ERRO[2024-01-19 07:00:57] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-fbvvo:7cddb5aa-c7ff-4044-9d27-458be2bd5cde], Expected:  [2024-01-19T07:00:57.164Z] INFO[2024-01-19 07:00:57] Verifying : Description : Deleting cluster source-cluster [2024-01-19T07:00:57.164Z] ERRO[2024-01-19 07:00:57] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo], cannot be deleted, Expected:  [2024-01-19T07:00:58.541Z] INFO[2024-01-19 07:00:58] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-enzij] [2024-01-19T07:00:58.541Z] ERRO[2024-01-19 07:00:58] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-enzij:4e0ddfc7-8811-4ed5-8e87-ed0841582111], Expected:  [2024-01-19T07:00:58.797Z] INFO[2024-01-19 07:00:58] Verifying : Description : Deleting cluster source-cluster [2024-01-19T07:00:58.797Z] ERRO[2024-01-19 07:00:58] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo], cannot be deleted, Expected:  [2024-01-19T07:01:00.167Z] INFO[2024-01-19 07:00:59] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-wcpfm] [2024-01-19T07:01:00.167Z] ERRO[2024-01-19 07:00:59] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-wcpfm:c5ff084a-a913-4412-99fb-e2b0a003a014], Expected:  [2024-01-19T07:01:00.167Z] INFO[2024-01-19 07:00:59] Verifying : Description : Deleting cluster source-cluster [2024-01-19T07:01:00.167Z] ERRO[2024-01-19 07:00:59] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx], cannot be deleted, Expected:  [2024-01-19T07:01:01.534Z] INFO[2024-01-19 07:01:01] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-vlbwu] [2024-01-19T07:01:01.534Z] ERRO[2024-01-19 07:01:01] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-vlbwu:f0465bb5-2565-4948-bb27-7e45a4b883ff], Expected:  [2024-01-19T07:01:01.534Z] INFO[2024-01-19 07:01:01] Verifying : Description : Deleting cluster source-cluster [2024-01-19T07:01:01.534Z] ERRO[2024-01-19 07:01:01] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa], cannot be deleted, Expected:  [2024-01-19T07:01:02.941Z] INFO[2024-01-19 07:01:02] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-mhxzi] [2024-01-19T07:01:02.942Z] ERRO[2024-01-19 07:01:02] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-mhxzi:f551fb81-1a78-4d7f-b804-759c6e6b562e], Expected:  [2024-01-19T07:01:03.200Z] INFO[2024-01-19 07:01:02] Verifying : Description : Deleting cluster source-cluster [2024-01-19T07:01:03.200Z] INFO[2024-01-19 07:01:02] Actual:true, Expected: true [2024-01-19T07:01:04.130Z] INFO[2024-01-19 07:01:04] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-hajtm] [2024-01-19T07:01:04.130Z] INFO[2024-01-19 07:01:04] Actual:true, Expected: true [2024-01-19T07:01:04.387Z] INFO[2024-01-19 07:01:04] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T07:01:04.387Z] INFO[2024-01-19 07:01:04] Actual:true, Expected: true [2024-01-19T07:01:05.758Z] INFO[2024-01-19 07:01:05] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-uzmhh] [2024-01-19T07:01:05.758Z] INFO[2024-01-19 07:01:05] Actual:true, Expected: true [2024-01-19T07:01:05.758Z] INFO[2024-01-19 07:01:05] --------Test End------ [2024-01-19T07:01:05.758Z] INFO[2024-01-19 07:01:05] #Test: [2024-01-19T07:01:05.758Z] INFO[2024-01-19 07:01:05] #Description: [2024-01-19T07:01:05.758Z] INFO[2024-01-19 07:01:05] ------------------------ [2024-01-19T07:01:05.758Z] 2024-01-19 07:01:05 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.EndPxBackupTorpedoTest:#7261] - >>>> FAILED TEST: {ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress} Scale down px-backup deployment when backups/restores are in progress and validate [2024-01-19T07:01:05.758Z] 2024-01-19 07:01:05 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-58074-01-19-05h53m37s] [2024-01-19T07:01:06.014Z] 2024-01-19 07:01:05 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T07:01:06.014Z] 2024-01-19 07:01:05 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T07:01:06.271Z] 2024-01-19 07:01:06 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-58074-01-19-05h53m37s] [2024-01-19T07:01:06.271Z] 2024-01-19 07:01:06 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress} Scale down px-backup deployment when backups/restores are in progress and validate [2024-01-19T07:01:06.271Z] 2024-01-19 07:01:06 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T07:01:06.271Z] 2024/01/19 07:01:06 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T07:01:11.521Z] 2024/01/19 07:01:11 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T07:01:16.765Z] 2024/01/19 07:01:16 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T07:01:22.009Z] 2024/01/19 07:01:21 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T07:01:27.251Z] 2024/01/19 07:01:26 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T07:01:31.419Z] 2024/01/19 07:01:31 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T07:01:36.666Z] 2024/01/19 07:01:36 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T07:01:41.905Z] 2024/01/19 07:01:41 DoRetryWithTimeout - Error: {dial tcp 10.240.0.8:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T07:01:47.175Z] 2024/01/19 07:01:46 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.0.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.8 10.240.0.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T07:01:47.175Z] {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T07:01:47.175Z] {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T07:01:47.175Z] {dial tcp 10.240.0.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T07:01:51.337Z] 2024-01-19 07:01:51 +0000:[WARNING] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.8 10.240.0.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T07:01:51.337Z] {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T07:01:51.337Z] {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T07:01:51.337Z] {dial tcp 10.240.0.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T07:01:51.337Z] 2024-01-19 07:01:51 +0000:[ERROR] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.8. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.8 10.240.0.8]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T07:01:51.337Z] {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T07:01:51.337Z] {dial tcp 10.240.0.8:22: connect: connection refused} [2024-01-19T07:01:51.337Z] {dial tcp 10.240.0.8:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T07:01:51.337Z] 2024-01-19 07:01:51 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T07:01:51.337Z] 2024-01-19 07:01:51 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] Switching context to [/tmp/source-config] [2024-01-19T07:01:51.337Z] 2024-01-19 07:01:51 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T07:01:51.337Z] 2024-01-19 07:01:51 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T07:01:51.594Z] 2024-01-19 07:01:51 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T07:01:51.594Z] 2024-01-19 07:01:51 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T07:01:51.594Z] 2024-01-19 07:01:51 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T07:01:51.849Z] 2024-01-19 07:01:51 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T07:01:52.104Z] 2024-01-19 07:01:52 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T07:01:52.742Z] 2024-01-19 07:01:52 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T07:01:53.002Z] 2024-01-19 07:01:52 +0000:[INFO] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T07:01:53.260Z] 2024/01/19 07:01:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:02:03.202Z] 2024/01/19 07:02:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:02:13.171Z] 2024/01/19 07:02:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:02:25.339Z] 2024/01/19 07:02:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:02:33.426Z] 2024/01/19 07:02:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:02:43.367Z] 2024/01/19 07:02:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:02:53.320Z] 2024/01/19 07:02:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:03:03.257Z] 2024/01/19 07:03:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:03:15.457Z] 2024/01/19 07:03:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:03:23.548Z] 2024/01/19 07:03:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:03:33.604Z] 2024/01/19 07:03:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:03:43.541Z] 2024/01/19 07:03:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:03:53.493Z] 2024/01/19 07:03:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:04:03.456Z] 2024/01/19 07:04:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:04:15.630Z] 2024/01/19 07:04:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:04:23.717Z] 2024/01/19 07:04:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:04:33.659Z] 2024/01/19 07:04:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:04:43.601Z] 2024/01/19 07:04:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:04:53.552Z] 2024/01/19 07:04:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:05:05.734Z] 2024/01/19 07:05:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:05:05.734Z] INFO[2024-01-19 07:05:03] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T07:05:05.734Z] INFO[2024-01-19 07:05:03] sched stats: tick interval: 55863 < 1.2s (100%) [2024-01-19T07:05:05.734Z] INFO[2024-01-19 07:05:03] sched stats: tick process durations: 55862 < 500ms (100%) [2024-01-19T07:05:13.857Z] 2024/01/19 07:05:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:05:23.806Z] 2024/01/19 07:05:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:05:33.757Z] 2024/01/19 07:05:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:05:43.724Z] 2024/01/19 07:05:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:05:53.707Z] 2024/01/19 07:05:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:06:05.877Z] 2024/01/19 07:06:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:06:13.961Z] 2024/01/19 07:06:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:06:24.150Z] 2024/01/19 07:06:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:06:34.114Z] 2024/01/19 07:06:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:06:44.067Z] 2024/01/19 07:06:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:06:54.004Z] 2024-01-19 07:06:53 +0000:[ERROR] [{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.004Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.004Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.004Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.004Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.004Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.004Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.005Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.005Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.005Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.005Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.005Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.005Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.005Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.005Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.005Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.005Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.005Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.005Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.005Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.005Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.005Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.005Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.005Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.005Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.005Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.005Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.005Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.005Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:06:54.005Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T07:06:54.006Z] • Failure [4396.273 seconds] [2024-01-19T07:06:54.006Z] {ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress} [2024-01-19T07:06:54.006Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_resiliency_test.go:1319 [2024-01-19T07:06:54.006Z] Scale down px-backup deployment when backups/restores are in progress and validate [It] [2024-01-19T07:06:54.006Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_resiliency_test.go:1361 [2024-01-19T07:06:54.006Z] [2024-01-19T07:06:54.006Z] Unexpected error: [2024-01-19T07:06:54.006Z] <*task.ErrTimedOut | 0xc000ea2510>: { [2024-01-19T07:06:54.007Z] Reason: "DoRetryWithTimeout timed out. Errors generated in retries: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}\n{restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}\n{restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}\n{restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}\n{restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}\n{restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}\n{restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}\n{restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}\n{restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}\n{restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}\n{restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}\n{restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}\n{restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}\n{restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}\n{restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}\n{restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}\n{restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]}\n{restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but go... [2024-01-19T07:06:54.007Z] [2024-01-19T07:06:54.007Z] Gomega truncated this representation as it exceeds 'format.MaxLength'. [2024-01-19T07:06:54.007Z] Consider having the object provide a custom 'GomegaStringer' representation [2024-01-19T07:06:54.007Z] or adjust the parameters in Gomega's 'format' package. [2024-01-19T07:06:54.007Z] [2024-01-19T07:06:54.007Z] Learn more here: https://onsi.github.io/gomega/#adjusting-output [2024-01-19T07:06:54.007Z] [2024-01-19T07:06:54.007Z] timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.007Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.007Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.007Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.007Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.007Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.007Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.007Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.007Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.007Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.007Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.007Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.007Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.008Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.008Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.008Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.008Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.008Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.008Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.008Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.008Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.008Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.008Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.008Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.008Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.008Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.008Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.008Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.008Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.008Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.009Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.009Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.009Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.009Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.009Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.009Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.009Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.009Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.009Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.009Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.009Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.009Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.009Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.009Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.009Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.009Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.009Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.010Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.010Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.010Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.010Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.010Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.010Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.010Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.010Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.010Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.010Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.010Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.010Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.010Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.010Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.010Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.010Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.010Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.010Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.011Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.011Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.011Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.011Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.011Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.011Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.011Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.011Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.011Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.011Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.011Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.011Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.011Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.011Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.011Z] {restore status for [tp-backup-mysql-ibm-pxb-0-58074-01-19-05h53m37s-1-1705643737-restore-1705644441] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume or Resource restores are in progress]} [2024-01-19T07:06:54.011Z] occurred [2024-01-19T07:06:54.011Z] [2024-01-19T07:06:54.011Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T07:06:54.011Z] [2024-01-19T07:06:54.011Z] Full Stack Trace [2024-01-19T07:06:54.011Z] github.com/portworx/torpedo/pkg/log.FailOnError({0x88adf80, 0xc000ea2510}, {0xc001998150?, 0xc001b920f0?}, {0x0?, 0x7d22b40?, 0x88eb700?}) [2024-01-19T07:06:54.011Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 +0x290 [2024-01-19T07:06:54.011Z] github.com/portworx/torpedo/tests/backup.glob..func53.2.10() [2024-01-19T07:06:54.011Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_resiliency_test.go:1534 +0x1ce [2024-01-19T07:06:54.011Z] github.com/portworx/torpedo/tests/backup.glob..func53.2() [2024-01-19T07:06:54.011Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_resiliency_test.go:1528 +0xb29 [2024-01-19T07:06:54.012Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T07:06:54.012Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T07:06:54.012Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T07:06:54.012Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T07:06:54.012Z] created by testing.(*T).Run [2024-01-19T07:06:54.012Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T07:06:54.012Z] ------------------------------ [2024-01-19T07:06:54.012Z] STEP: schedule applications [2024-01-19T07:06:54.012Z] STEP: Validating the deployed applications [2024-01-19T07:06:54.012Z] STEP: validate applications [2024-01-19T07:06:54.012Z] STEP: validate mysql-ibm app's volumes [2024-01-19T07:06:54.012Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T07:06:54.012Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T07:06:54.012Z] STEP: get mysql-ibm app's volume: pvc-f5e5d306-256a-4fb2-bc8d-3d14a8395806 inspected by the volume driver [2024-01-19T07:06:54.012Z] STEP: wait for mysql-ibm app to start running [2024-01-19T07:06:54.012Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T07:06:54.012Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T07:06:54.012Z] STEP: Validate Px pod restart count [2024-01-19T07:06:54.012Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T07:06:54.012Z] STEP: Adding cloud credential and backup location [2024-01-19T07:06:54.012Z] STEP: Registering source and destination clusters for backup [2024-01-19T07:06:54.012Z] STEP: Getting the replica factor of px-backup deployment in backup namespace before taking backup [2024-01-19T07:06:54.012Z] STEP: Taking backup of applications [2024-01-19T07:06:54.012Z] STEP: Scaling px-backup deployment replica count to 0 and back to original replica while backup is in progress [2024-01-19T07:06:54.012Z] STEP: Check if backup is successful after px-backup deployment is scaled back to original replica [2024-01-19T07:06:54.012Z] STEP: Restoring the backups taken [2024-01-19T07:06:54.012Z] STEP: Scaling px-backup deployment replica to 0 and back to original replica while restore is in progress [2024-01-19T07:06:54.012Z] STEP: Check if restore is successful after px_backup deployment is scaled back to original replica [2024-01-19T07:06:54.012Z] STEP: destroy apps [2024-01-19T07:06:54.012Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T07:06:54.012Z] STEP: start destroying mysql-ibm app [2024-01-19T07:06:54.012Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T07:06:54.012Z] 2024-01-19 07:06:53 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] Log Dir: /testresults//LicensingCountBeforeAndAfterBackupPodRestart.log [2024-01-19T07:06:54.012Z] INFO[2024-01-19 07:06:53] --------Test Start------ [2024-01-19T07:06:54.012Z] INFO[2024-01-19 07:06:53] #Test: LicensingCountBeforeAndAfterBackupPodRestart [2024-01-19T07:06:54.012Z] INFO[2024-01-19 07:06:53] #Description: Verifies the license count before and after the backup pod restarts [2024-01-19T07:06:54.012Z] INFO[2024-01-19 07:06:53] ------------------------ [2024-01-19T07:06:54.269Z] 2024-01-19 07:06:54 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] Adding source and destination clusters for backup [2024-01-19T07:06:54.269Z] INFO[2024-01-19 07:06:54] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T07:06:54.269Z] INFO[2024-01-19 07:06:54] Actual:true, Expected: true [2024-01-19T07:06:54.269Z] 2024-01-19 07:06:54 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T07:06:54.269Z] 2024-01-19 07:06:54 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T07:06:54.269Z] 2024-01-19 07:06:54 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T07:06:54.269Z] 2024-01-19 07:06:54 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T07:06:54.269Z] 2024-01-19 07:06:54 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] Create cluster [source-cluster] in org [default] [2024-01-19T07:06:54.269Z] 2024-01-19 07:06:54 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T07:06:54.269Z] 2024-01-19 07:06:54 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T07:06:54.270Z] 2024-01-19 07:06:54 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] Create cluster [destination-cluster] in org [default] [2024-01-19T07:06:54.270Z] 2024-01-19 07:06:54 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T07:06:54.270Z] 2024-01-19 07:06:54 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T07:06:54.270Z] 2024-01-19 07:06:54 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-qditt] for IBM [2024-01-19T07:06:54.270Z] 2024-01-19 07:06:54 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-qditt] for org [default] with [ibm] as provider [2024-01-19T07:06:54.270Z] 2024-01-19 07:06:54 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T07:06:54.270Z] 2024-01-19 07:06:54 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T07:06:54.270Z] 2024-01-19 07:06:54 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-qditt] is present [2024-01-19T07:06:54.270Z] 2024-01-19 07:06:54 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] Create cluster [source-cluster] in org [default] [2024-01-19T07:06:58.435Z] 2024-01-19 07:06:57 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-tfcyl] for IBM [2024-01-19T07:06:58.436Z] 2024-01-19 07:06:57 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-tfcyl] for org [default] with [ibm] as provider [2024-01-19T07:06:58.436Z] 2024-01-19 07:06:57 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T07:06:58.436Z] 2024-01-19 07:06:57 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T07:06:58.436Z] 2024-01-19 07:06:57 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-tfcyl] is present [2024-01-19T07:06:58.436Z] 2024-01-19 07:06:57 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] Create cluster [destination-cluster] in org [default] [2024-01-19T07:06:58.692Z] 2024-01-19 07:06:58 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] Getting the total number of worker nodes in source and destination cluster [2024-01-19T07:06:58.692Z] 2024-01-19 07:06:58 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] Total number of worker nodes in source cluster are 0 [2024-01-19T07:06:58.692Z] 2024-01-19 07:06:58 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] Switching cluster context to destination cluster [2024-01-19T07:06:58.692Z] 2024-01-19 07:06:58 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T07:06:58.692Z] 2024-01-19 07:06:58 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T07:06:58.692Z] 2024-01-19 07:06:58 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] Total number of worker nodes in destination cluster are 0 [2024-01-19T07:06:58.692Z] 2024-01-19 07:06:58 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] Total number of worker nodes in source and destination cluster are 0 [2024-01-19T07:06:58.692Z] 2024-01-19 07:06:58 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] Switching cluster context back to source cluster [2024-01-19T07:06:58.692Z] 2024-01-19 07:06:58 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T07:06:58.692Z] 2024-01-19 07:06:58 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] Switching context to [/tmp/source-config] [2024-01-19T07:06:58.692Z] 2024-01-19 07:06:58 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T07:06:58.692Z] 2024-01-19 07:06:58 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T07:06:58.692Z] 2024-01-19 07:06:58 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T07:06:58.692Z] 2024-01-19 07:06:58 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T07:06:58.692Z] 2024-01-19 07:06:58 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T07:06:58.948Z] 2024-01-19 07:06:58 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T07:06:59.509Z] 2024-01-19 07:06:59 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T07:06:59.765Z] 2024-01-19 07:06:59 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T07:07:00.326Z] 2024-01-19 07:07:00 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T07:07:00.326Z] 2024/01/19 07:07:00 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:07:10.305Z] 2024/01/19 07:07:10 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:07:22.466Z] 2024/01/19 07:07:20 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:07:30.564Z] 2024/01/19 07:07:30 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:07:40.500Z] 2024/01/19 07:07:40 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:07:50.445Z] 2024/01/19 07:07:50 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:08:02.621Z] 2024/01/19 07:08:00 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:08:10.703Z] 2024/01/19 07:08:10 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:08:20.642Z] 2024/01/19 07:08:20 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:08:30.578Z] 2024/01/19 07:08:30 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:08:42.751Z] 2024/01/19 07:08:40 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:08:50.873Z] 2024/01/19 07:08:50 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:09:00.843Z] 2024/01/19 07:09:00 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:09:10.783Z] 2024/01/19 07:09:10 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:09:20.719Z] 2024/01/19 07:09:20 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:09:30.661Z] 2024/01/19 07:09:30 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:09:42.827Z] 2024/01/19 07:09:40 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:09:50.911Z] 2024/01/19 07:09:50 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:10:00.850Z] 2024/01/19 07:10:00 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:10:10.819Z] 2024/01/19 07:10:10 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:10:20.771Z] 2024/01/19 07:10:20 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:10:32.942Z] 2024/01/19 07:10:30 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:10:41.035Z] 2024/01/19 07:10:40 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:10:50.980Z] 2024/01/19 07:10:50 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:11:00.949Z] 2024/01/19 07:11:00 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:11:10.889Z] 2024/01/19 07:11:10 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:11:23.057Z] 2024/01/19 07:11:20 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:11:31.165Z] 2024/01/19 07:11:30 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:11:41.107Z] 2024/01/19 07:11:40 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:11:51.055Z] 2024/01/19 07:11:50 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:12:00.992Z] 2024-01-19 07:12:00 +0000:[ERROR] [{LicensingCountBeforeAndAfterBackupPodRestart}] [backup.glob..func13.2.2:#354] - Switching context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.993Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.993Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.993Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.993Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.993Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.993Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.993Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.993Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.993Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.993Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.993Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.993Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.993Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.993Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.993Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.993Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.993Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.993Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.993Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.993Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.993Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.994Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.994Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.994Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.994Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.994Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.994Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.994Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:12:00.994Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T07:12:00.994Z] 2024-01-19 07:12:00 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T07:12:00.994Z] 2024-01-19 07:12:00 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T07:12:00.994Z] INFO[2024-01-19 07:12:00] Verifying : Description : Switching context to destination cluster [2024-01-19T07:12:00.994Z] INFO[2024-01-19 07:12:00] Actual:true, Expected: true [2024-01-19T07:12:00.994Z] 2024-01-19 07:12:00 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] Removing label portworx.io/nobackup=true from all worker nodes on destination cluster if present [2024-01-19T07:12:00.994Z] 2024-01-19 07:12:00 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] Switching cluster context back to source cluster [2024-01-19T07:12:00.994Z] 2024-01-19 07:12:00 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T07:12:00.994Z] 2024-01-19 07:12:00 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] Switching context to [/tmp/source-config] [2024-01-19T07:12:00.994Z] 2024-01-19 07:12:00 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T07:12:01.258Z] 2024-01-19 07:12:01 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T07:12:01.258Z] 2024-01-19 07:12:01 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T07:12:01.258Z] 2024-01-19 07:12:01 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T07:12:01.258Z] 2024-01-19 07:12:01 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T07:12:01.515Z] 2024-01-19 07:12:01 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T07:12:02.079Z] 2024-01-19 07:12:01 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T07:12:02.335Z] 2024-01-19 07:12:02 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T07:12:02.590Z] 2024-01-19 07:12:02 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T07:12:02.846Z] 2024/01/19 07:12:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:12:12.818Z] 2024/01/19 07:12:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:12:25.005Z] 2024/01/19 07:12:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:12:33.082Z] 2024/01/19 07:12:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:12:43.019Z] 2024/01/19 07:12:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:12:52.961Z] 2024/01/19 07:12:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:13:05.128Z] 2024/01/19 07:13:02 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:13:13.225Z] 2024/01/19 07:13:12 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:13:23.215Z] 2024/01/19 07:13:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:13:33.169Z] 2024/01/19 07:13:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:13:43.105Z] 2024/01/19 07:13:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:13:53.052Z] 2024/01/19 07:13:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:14:05.240Z] 2024/01/19 07:14:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:14:13.341Z] 2024/01/19 07:14:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:14:23.283Z] 2024/01/19 07:14:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:14:33.231Z] 2024/01/19 07:14:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:14:43.182Z] 2024/01/19 07:14:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:14:55.352Z] 2024/01/19 07:14:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:15:03.453Z] 2024/01/19 07:15:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:15:04.821Z] INFO[2024-01-19 07:15:04] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T07:15:04.821Z] INFO[2024-01-19 07:15:04] sched stats: tick interval: 56464 < 1.2s (100%) [2024-01-19T07:15:04.821Z] INFO[2024-01-19 07:15:04] sched stats: tick process durations: 56463 < 500ms (100%) [2024-01-19T07:15:14.762Z] 2024/01/19 07:15:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:15:24.707Z] 2024/01/19 07:15:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:15:34.544Z] 2024/01/19 07:15:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:15:44.487Z] 2024/01/19 07:15:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:15:54.430Z] 2024/01/19 07:15:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:16:04.383Z] 2024/01/19 07:16:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:16:14.341Z] 2024/01/19 07:16:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:16:24.284Z] 2024/01/19 07:16:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:16:34.258Z] 2024/01/19 07:16:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:16:44.206Z] 2024/01/19 07:16:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:16:54.151Z] 2024/01/19 07:16:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:17:04.092Z] INFO[2024-01-19 07:17:03] Verifying : Description : Switching context to source cluster [2024-01-19T07:17:04.092Z] ERRO[2024-01-19 07:17:03] failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.092Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.092Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.092Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.092Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.092Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.092Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.092Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.092Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.093Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.093Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.093Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.093Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.093Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.093Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.093Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.093Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.093Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.093Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.093Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.093Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.093Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.093Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.093Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.093Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.093Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.093Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.093Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.093Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:17:04.093Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T07:17:04.093Z] 2024-01-19 07:17:03 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] Removing label portworx.io/nobackup=true from all worker nodes on source cluster if present [2024-01-19T07:17:04.093Z] 2024-01-19 07:17:03 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] Cleaning backup locations in map [map[]], cloud credential [], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T07:17:04.094Z] INFO[2024-01-19 07:17:03] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T07:17:04.094Z] INFO[2024-01-19 07:17:03] Actual:true, Expected: true [2024-01-19T07:17:04.094Z] INFO[2024-01-19 07:17:03] Verifying : Description : Deleting cluster source-cluster [2024-01-19T07:17:04.094Z] ERRO[2024-01-19 07:17:03] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl], cannot be deleted, Expected:  [2024-01-19T07:17:04.656Z] INFO[2024-01-19 07:17:04] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-hrfiy] [2024-01-19T07:17:04.657Z] ERRO[2024-01-19 07:17:04] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-hrfiy:72b1b409-7bb1-4f22-92df-0ba5e77c11d3], Expected:  [2024-01-19T07:17:04.657Z] INFO[2024-01-19 07:17:04] Verifying : Description : Deleting cluster source-cluster [2024-01-19T07:17:04.657Z] ERRO[2024-01-19 07:17:04] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh], cannot be deleted, Expected:  [2024-01-19T07:17:06.021Z] INFO[2024-01-19 07:17:05] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-fbvvo] [2024-01-19T07:17:06.021Z] ERRO[2024-01-19 07:17:05] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-fbvvo:7cddb5aa-c7ff-4044-9d27-458be2bd5cde], Expected:  [2024-01-19T07:17:06.021Z] INFO[2024-01-19 07:17:05] Verifying : Description : Deleting cluster source-cluster [2024-01-19T07:17:06.021Z] ERRO[2024-01-19 07:17:05] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo], cannot be deleted, Expected:  [2024-01-19T07:17:07.391Z] INFO[2024-01-19 07:17:07] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-enzij] [2024-01-19T07:17:07.391Z] ERRO[2024-01-19 07:17:07] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-enzij:4e0ddfc7-8811-4ed5-8e87-ed0841582111], Expected:  [2024-01-19T07:17:07.391Z] INFO[2024-01-19 07:17:07] Verifying : Description : Deleting cluster source-cluster [2024-01-19T07:17:07.391Z] ERRO[2024-01-19 07:17:07] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo], cannot be deleted, Expected:  [2024-01-19T07:17:08.761Z] INFO[2024-01-19 07:17:08] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-wcpfm] [2024-01-19T07:17:08.761Z] ERRO[2024-01-19 07:17:08] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-wcpfm:c5ff084a-a913-4412-99fb-e2b0a003a014], Expected:  [2024-01-19T07:17:08.761Z] INFO[2024-01-19 07:17:08] Verifying : Description : Deleting cluster source-cluster [2024-01-19T07:17:08.761Z] ERRO[2024-01-19 07:17:08] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx], cannot be deleted, Expected:  [2024-01-19T07:17:10.129Z] INFO[2024-01-19 07:17:09] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-vlbwu] [2024-01-19T07:17:10.129Z] ERRO[2024-01-19 07:17:09] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-vlbwu:f0465bb5-2565-4948-bb27-7e45a4b883ff], Expected:  [2024-01-19T07:17:10.129Z] INFO[2024-01-19 07:17:10] Verifying : Description : Deleting cluster source-cluster [2024-01-19T07:17:10.129Z] ERRO[2024-01-19 07:17:10] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa], cannot be deleted, Expected:  [2024-01-19T07:17:11.496Z] INFO[2024-01-19 07:17:11] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-mhxzi] [2024-01-19T07:17:11.496Z] ERRO[2024-01-19 07:17:11] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-mhxzi:f551fb81-1a78-4d7f-b804-759c6e6b562e], Expected:  [2024-01-19T07:17:11.752Z] INFO[2024-01-19 07:17:11] Verifying : Description : Deleting cluster source-cluster [2024-01-19T07:17:11.752Z] INFO[2024-01-19 07:17:11] Actual:true, Expected: true [2024-01-19T07:17:13.124Z] INFO[2024-01-19 07:17:12] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-qditt] [2024-01-19T07:17:13.124Z] INFO[2024-01-19 07:17:12] Actual:true, Expected: true [2024-01-19T07:17:13.124Z] INFO[2024-01-19 07:17:13] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T07:17:13.124Z] INFO[2024-01-19 07:17:13] Actual:true, Expected: true [2024-01-19T07:17:14.492Z] INFO[2024-01-19 07:17:14] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-tfcyl] [2024-01-19T07:17:14.492Z] INFO[2024-01-19 07:17:14] Actual:true, Expected: true [2024-01-19T07:17:14.492Z] INFO[2024-01-19 07:17:14] --------Test End------ [2024-01-19T07:17:14.492Z] INFO[2024-01-19 07:17:14] #Test: [2024-01-19T07:17:14.492Z] INFO[2024-01-19 07:17:14] #Description: [2024-01-19T07:17:14.492Z] INFO[2024-01-19 07:17:14] ------------------------ [2024-01-19T07:17:14.492Z] 2024-01-19 07:17:14 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.EndPxBackupTorpedoTest:#7261] - >>>> FAILED TEST: {LicensingCountBeforeAndAfterBackupPodRestart} Verify the license count before and after the backup pod restarts [2024-01-19T07:17:14.492Z] 2024-01-19 07:17:14 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T07:17:14.492Z] 2024-01-19 07:17:14 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T07:17:14.749Z] 2024-01-19 07:17:14 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {LicensingCountBeforeAndAfterBackupPodRestart} Verify the license count before and after the backup pod restarts [2024-01-19T07:17:14.749Z] 2024-01-19 07:17:14 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T07:17:14.749Z] 2024/01/19 07:17:14 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T07:17:19.998Z] 2024/01/19 07:17:19 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T07:17:25.286Z] 2024/01/19 07:17:24 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T07:17:30.575Z] 2024/01/19 07:17:29 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T07:17:34.743Z] 2024/01/19 07:17:34 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T07:17:40.091Z] 2024/01/19 07:17:39 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T07:17:45.343Z] 2024/01/19 07:17:44 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T07:17:50.593Z] 2024/01/19 07:17:49 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T07:17:54.760Z] 2024/01/19 07:17:54 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.128.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.7 10.240.128.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T07:17:54.760Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T07:17:54.760Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T07:17:54.760Z] {dial tcp 10.240.128.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T07:18:00.006Z] 2024-01-19 07:17:59 +0000:[WARNING] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.7 10.240.128.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T07:18:00.006Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T07:18:00.006Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T07:18:00.006Z] {dial tcp 10.240.128.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T07:18:00.006Z] 2024-01-19 07:17:59 +0000:[ERROR] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.7 10.240.128.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T07:18:00.006Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T07:18:00.006Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T07:18:00.006Z] {dial tcp 10.240.128.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T07:18:00.006Z] 2024-01-19 07:17:59 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T07:18:00.006Z] 2024-01-19 07:17:59 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] Switching context to [/tmp/source-config] [2024-01-19T07:18:00.006Z] 2024-01-19 07:17:59 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T07:18:00.006Z] 2024-01-19 07:17:59 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T07:18:00.006Z] 2024-01-19 07:17:59 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T07:18:00.006Z] 2024-01-19 07:17:59 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T07:18:00.006Z] 2024-01-19 07:17:59 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T07:18:00.263Z] 2024-01-19 07:18:00 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T07:18:00.520Z] 2024-01-19 07:18:00 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T07:18:01.083Z] 2024-01-19 07:18:00 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T07:18:01.339Z] 2024-01-19 07:18:01 +0000:[INFO] [{LicensingCountBeforeAndAfterBackupPodRestart}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T07:18:01.596Z] 2024/01/19 07:18:01 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:18:11.550Z] 2024/01/19 07:18:11 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:18:21.497Z] 2024/01/19 07:18:21 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:18:33.669Z] 2024/01/19 07:18:31 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:18:41.753Z] 2024/01/19 07:18:41 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:18:51.694Z] 2024/01/19 07:18:51 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:19:01.633Z] 2024/01/19 07:19:01 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:19:11.592Z] 2024/01/19 07:19:11 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:19:23.804Z] 2024/01/19 07:19:21 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:19:31.909Z] 2024/01/19 07:19:31 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:19:41.848Z] 2024/01/19 07:19:41 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:19:51.810Z] 2024/01/19 07:19:51 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:20:01.767Z] 2024/01/19 07:20:01 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:20:13.941Z] 2024/01/19 07:20:11 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:20:22.049Z] 2024/01/19 07:20:21 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:20:31.990Z] 2024/01/19 07:20:31 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:20:43.005Z] 2024/01/19 07:20:41 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:20:53.025Z] 2024/01/19 07:20:51 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:21:03.024Z] 2024/01/19 07:21:01 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:21:12.967Z] 2024/01/19 07:21:11 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:21:22.911Z] 2024/01/19 07:21:21 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:21:32.864Z] 2024/01/19 07:21:31 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:21:42.824Z] 2024/01/19 07:21:41 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:21:52.767Z] 2024/01/19 07:21:51 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:22:02.717Z] 2024/01/19 07:22:01 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:22:12.656Z] 2024/01/19 07:22:11 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:22:22.593Z] 2024/01/19 07:22:22 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:22:32.570Z] 2024/01/19 07:22:32 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:22:42.513Z] 2024/01/19 07:22:42 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:22:52.453Z] 2024/01/19 07:22:52 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:23:02.395Z] 2024-01-19 07:23:02 +0000:[ERROR] [{LicensingCountBeforeAndAfterBackupPodRestart}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.395Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.395Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.395Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.395Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.396Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.396Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.396Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.396Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.396Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.396Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.396Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.396Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.396Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.396Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.396Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.396Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.396Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.396Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.396Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.396Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.396Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.396Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.396Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.396Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.396Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.396Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.397Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.397Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.397Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T07:23:02.397Z] • Failure [968.296 seconds] [2024-01-19T07:23:02.397Z] {LicensingCountBeforeAndAfterBackupPodRestart} [2024-01-19T07:23:02.397Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_license_test.go:319 [2024-01-19T07:23:02.397Z] Verify the license count before and after the backup pod restarts [It] [2024-01-19T07:23:02.397Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_license_test.go:332 [2024-01-19T07:23:02.397Z] [2024-01-19T07:23:02.397Z] Unexpected error: [2024-01-19T07:23:02.397Z] <*errors.errorString | 0xc001423a60>: { [2024-01-19T07:23:02.398Z] s: "failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{No... [2024-01-19T07:23:02.398Z] [2024-01-19T07:23:02.398Z] Gomega truncated this representation as it exceeds 'format.MaxLength'. [2024-01-19T07:23:02.398Z] Consider having the object provide a custom 'GomegaStringer' representation [2024-01-19T07:23:02.398Z] or adjust the parameters in Gomega's 'format' package. [2024-01-19T07:23:02.398Z] [2024-01-19T07:23:02.398Z] Learn more here: https://onsi.github.io/gomega/#adjusting-output [2024-01-19T07:23:02.398Z] [2024-01-19T07:23:02.398Z] failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.398Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.398Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.398Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.398Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.398Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.398Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.398Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.398Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.398Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.398Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.398Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.398Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.399Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.399Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.399Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.399Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.399Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.399Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.399Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.399Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.399Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.399Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.399Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.399Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.399Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.399Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.399Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.399Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:23:02.399Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T07:23:02.399Z] occurred [2024-01-19T07:23:02.399Z] [2024-01-19T07:23:02.399Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T07:23:02.399Z] [2024-01-19T07:23:02.399Z] Full Stack Trace [2024-01-19T07:23:02.399Z] github.com/portworx/torpedo/pkg/log.FailOnError({0x88a5060, 0xc001423a60}, {0x7f42292?, 0x0?}, {0x0?, 0x2?, 0x12?}) [2024-01-19T07:23:02.399Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 +0x290 [2024-01-19T07:23:02.399Z] github.com/portworx/torpedo/tests/backup.glob..func13.2.2() [2024-01-19T07:23:02.399Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_license_test.go:354 +0x773 [2024-01-19T07:23:02.399Z] github.com/portworx/torpedo/tests/backup.glob..func13.2() [2024-01-19T07:23:02.399Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_license_test.go:340 +0x249 [2024-01-19T07:23:02.399Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T07:23:02.399Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T07:23:02.399Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T07:23:02.399Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T07:23:02.399Z] created by testing.(*T).Run [2024-01-19T07:23:02.399Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T07:23:02.400Z] ------------------------------ [2024-01-19T07:23:02.400Z] STEP: Adding source and destination clusters for backup [2024-01-19T07:23:02.400Z] STEP: Getting the total number of worker nodes in source and destination cluster [2024-01-19T07:23:02.400Z] 2024-01-19 07:23:02 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Log Dir: /testresults//ShareBackupsAndClusterWithUser.log [2024-01-19T07:23:02.400Z] INFO[2024-01-19 07:23:02] --------Test Start------ [2024-01-19T07:23:02.400Z] INFO[2024-01-19 07:23:02] #Test: ShareBackupsAndClusterWithUser [2024-01-19T07:23:02.400Z] INFO[2024-01-19 07:23:02] #Description: Share backup to user with full access and try to duplicate the backup from the shared user [2024-01-19T07:23:02.400Z] INFO[2024-01-19 07:23:02] ------------------------ [2024-01-19T07:23:02.400Z] 2024-01-19 07:23:02 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Deploy applications need for taking backup [2024-01-19T07:23:02.400Z] 2024-01-19 07:23:02 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T07:23:02.400Z] 2024-01-19 07:23:02 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T07:23:02.400Z] 2024-01-19 07:23:02 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T07:23:02.400Z] 2024-01-19 07:23:02 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T07:23:02.400Z] 2024-01-19 07:23:02 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T07:23:02.400Z] 2024-01-19 07:23:02 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T07:23:02.401Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T07:23:02.401Z] 2024-01-19 07:23:02 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T07:23:02.401Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T07:23:02.658Z] 2024-01-19 07:23:02 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T07:23:02.914Z] 2024-01-19 07:23:02 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Validate applications [2024-01-19T07:23:02.914Z] 2024-01-19 07:23:02 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Validate applications [2024-01-19T07:23:02.914Z] 2024-01-19 07:23:02 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Validating mysql-ibm app [2024-01-19T07:23:02.914Z] 2024-01-19 07:23:02 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Validating mysql-ibm app's volumes [2024-01-19T07:23:02.914Z] 2024-01-19 07:23:02 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T07:23:02.914Z] 2024/01/19 07:23:02 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T07:23:12.900Z] 2024/01/19 07:23:12 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T07:23:22.863Z] 2024-01-19 07:23:22 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-82943-01-19-07h23m02s [2024-01-19T07:23:22.863Z] 2024-01-19 07:23:22 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T07:23:23.120Z] 2024-01-19 07:23:22 +0000:[WARNING] [{ShareBackupsAndClusterWithUser}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T07:23:23.120Z] 2024-01-19 07:23:22 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] wait for mysql-ibm app to start running [2024-01-19T07:23:23.121Z] 2024-01-19 07:23:22 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T07:23:23.121Z] 2024/01/19 07:23:22 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T07:23:23.121Z] pod name:mysql-7d766cd65c-95z76 namespace:mysql-ibm-pxb-0-82943-01-19-07h23m02s running:false ready:false node:10.240.64.8 [2024-01-19T07:23:23.121Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T07:23:35.293Z] 2024/01/19 07:23:33 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T07:23:35.293Z] pod name:mysql-7d766cd65c-95z76 namespace:mysql-ibm-pxb-0-82943-01-19-07h23m02s running:true ready:false node:10.240.64.8 [2024-01-19T07:23:35.293Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T07:23:43.386Z] 2024/01/19 07:23:43 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T07:23:43.387Z] pod name:mysql-7d766cd65c-95z76 namespace:mysql-ibm-pxb-0-82943-01-19-07h23m02s running:true ready:false node:10.240.64.8 [2024-01-19T07:23:43.387Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T07:23:53.366Z] 2024/01/19 07:23:53 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T07:23:53.366Z] pod name:mysql-7d766cd65c-95z76 namespace:mysql-ibm-pxb-0-82943-01-19-07h23m02s running:true ready:false node:10.240.64.8 [2024-01-19T07:23:53.366Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T07:24:05.587Z] 2024/01/19 07:24:03 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T07:24:05.587Z] pod name:mysql-7d766cd65c-95z76 namespace:mysql-ibm-pxb-0-82943-01-19-07h23m02s running:true ready:false node:10.240.64.8 [2024-01-19T07:24:05.587Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T07:24:13.677Z] 2024/01/19 07:24:13 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T07:24:13.677Z] pod name:mysql-7d766cd65c-95z76 namespace:mysql-ibm-pxb-0-82943-01-19-07h23m02s running:true ready:false node:10.240.64.8 [2024-01-19T07:24:13.677Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T07:24:25.845Z] 2024/01/19 07:24:23 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T07:24:25.845Z] pod name:mysql-7d766cd65c-95z76 namespace:mysql-ibm-pxb-0-82943-01-19-07h23m02s running:true ready:false node:10.240.64.8 [2024-01-19T07:24:25.845Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T07:24:33.925Z] 2024/01/19 07:24:33 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T07:24:33.925Z] pod name:mysql-7d766cd65c-95z76 namespace:mysql-ibm-pxb-0-82943-01-19-07h23m02s running:true ready:false node:10.240.64.8 [2024-01-19T07:24:33.925Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T07:24:43.884Z] 2024-01-19 07:24:43 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T07:24:43.884Z] 2024-01-19 07:24:43 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T07:24:44.144Z] 2024-01-19 07:24:43 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] validate if mysql-ibm app's volumes are setup [2024-01-19T07:24:44.144Z] 2024-01-19 07:24:43 +0000:[WARNING] [{ShareBackupsAndClusterWithUser}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T07:24:44.144Z] 2024-01-19 07:24:43 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T07:24:44.144Z] 2024-01-19 07:24:43 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T07:24:44.144Z] 2024-01-19 07:24:43 +0000:[WARNING] [{ShareBackupsAndClusterWithUser}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T07:24:44.144Z] 2024-01-19 07:24:43 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Creating 1 users [2024-01-19T07:24:44.400Z] 2024-01-19 07:24:44 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [backup.AddUser:#728] - Response for user [testuser1-1705649083] creation - [2024-01-19T07:24:44.400Z] 2024-01-19 07:24:44 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [backup.AddUser:#733] - User [testuser1-1705649083] created [2024-01-19T07:24:44.400Z] INFO[2024-01-19 07:24:44] Verifying : Description : Creating user - testuser1-1705649083 [2024-01-19T07:24:44.400Z] INFO[2024-01-19 07:24:44] Actual:true, Expected: true [2024-01-19T07:24:44.400Z] 2024-01-19 07:24:44 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [backup.glob..func89.2.2:#2458] - Created 1 users and users list is [testuser1-1705649083] [2024-01-19T07:24:44.400Z] 2024-01-19 07:24:44 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Creating backup location and cloud setting [2024-01-19T07:24:44.400Z] 2024-01-19 07:24:44 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [cloudcred-aws-1705649084] for org [default] with [aws] as provider [2024-01-19T07:24:44.401Z] 2024-01-19 07:24:44 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T07:24:44.401Z] 2024-01-19 07:24:44 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cloudcred-aws-1705649084] is present [2024-01-19T07:24:44.401Z] INFO[2024-01-19 07:24:44] Verifying : Description : Verifying creation of cloud credential named [cloudcred-aws-1705649084] for org [default] with [aws] as provider [2024-01-19T07:24:44.401Z] INFO[2024-01-19 07:24:44] Actual:true, Expected: true [2024-01-19T07:24:46.918Z] INFO[2024-01-19 07:24:46] Verifying : Description : Creating backup location aws-global-aws-px-backup-system-test-4cac76dc-1705649084-bl [2024-01-19T07:24:46.918Z] INFO[2024-01-19 07:24:46] Actual:true, Expected: true [2024-01-19T07:24:46.918Z] INFO[2024-01-19 07:24:46] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T07:24:46.918Z] INFO[2024-01-19 07:24:46] Actual:true, Expected: true [2024-01-19T07:24:46.918Z] 2024-01-19 07:24:46 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T07:24:46.918Z] 2024-01-19 07:24:46 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T07:24:46.918Z] 2024-01-19 07:24:46 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T07:24:46.918Z] 2024-01-19 07:24:46 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T07:24:46.918Z] 2024-01-19 07:24:46 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Create cluster [source-cluster] in org [default] [2024-01-19T07:24:46.918Z] 2024-01-19 07:24:46 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T07:24:46.918Z] 2024-01-19 07:24:46 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T07:24:46.918Z] 2024-01-19 07:24:46 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Create cluster [destination-cluster] in org [default] [2024-01-19T07:24:46.918Z] 2024-01-19 07:24:46 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T07:24:46.918Z] 2024-01-19 07:24:46 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T07:24:46.918Z] 2024-01-19 07:24:46 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-gfjkf] for IBM [2024-01-19T07:24:46.918Z] 2024-01-19 07:24:46 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-gfjkf] for org [default] with [ibm] as provider [2024-01-19T07:24:46.919Z] 2024-01-19 07:24:46 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T07:24:46.919Z] 2024-01-19 07:24:46 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T07:24:46.919Z] 2024-01-19 07:24:46 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-gfjkf] is present [2024-01-19T07:24:46.919Z] 2024-01-19 07:24:46 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Create cluster [source-cluster] in org [default] [2024-01-19T07:24:51.087Z] 2024-01-19 07:24:50 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-cnzjj] for IBM [2024-01-19T07:24:51.087Z] 2024-01-19 07:24:50 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-cnzjj] for org [default] with [ibm] as provider [2024-01-19T07:24:51.087Z] 2024-01-19 07:24:50 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T07:24:51.087Z] 2024-01-19 07:24:50 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T07:24:51.087Z] 2024-01-19 07:24:50 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-cnzjj] is present [2024-01-19T07:24:51.087Z] 2024-01-19 07:24:50 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Create cluster [destination-cluster] in org [default] [2024-01-19T07:24:51.343Z] INFO[2024-01-19 07:24:51] Verifying : Description : Creating source and destination cluster [2024-01-19T07:24:51.343Z] INFO[2024-01-19 07:24:51] Actual:true, Expected: true [2024-01-19T07:24:51.343Z] INFO[2024-01-19 07:24:51] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T07:24:51.343Z] INFO[2024-01-19 07:24:51] Actual:Online, Expected: Online [2024-01-19T07:24:51.343Z] INFO[2024-01-19 07:24:51] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T07:24:51.343Z] INFO[2024-01-19 07:24:51] Actual:true, Expected: true [2024-01-19T07:24:51.343Z] 2024-01-19 07:24:51 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Uid of [source-cluster] cluster is 7b86bd8e-2834-4d0c-a1ea-bf29fb79eb9d [2024-01-19T07:24:51.343Z] 2024-01-19 07:24:51 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T07:24:51.343Z] 2024-01-19 07:24:51 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Volume snapshot classes found in the cluster - [2024-01-19T07:24:51.343Z] 2024-01-19 07:24:51 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] ibmc-vpcblock-snapshot [2024-01-19T07:24:51.343Z] 2024-01-19 07:24:51 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T07:24:51.904Z] 2024/01/19 07:24:51 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T07:25:06.752Z] INFO[2024-01-19 07:25:05] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T07:25:06.752Z] INFO[2024-01-19 07:25:05] sched stats: tick interval: 57065 < 1.2s (100%) [2024-01-19T07:25:06.752Z] INFO[2024-01-19 07:25:05] sched stats: tick process durations: 57064 < 500ms (100%) [2024-01-19T07:25:24.801Z] 2024/01/19 07:25:21 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T07:25:56.829Z] 2024/01/19 07:25:51 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T07:26:23.335Z] 2024-01-19 07:26:21 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [backup.CreateBackup:#313] - Backup [tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] created successfully [2024-01-19T07:26:23.335Z] 2024-01-19 07:26:21 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Validating backup [tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] in org [default] [2024-01-19T07:26:23.335Z] 2024-01-19 07:26:21 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] [2024-01-19T07:26:23.335Z] 2024-01-19 07:26:21 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82943-01-19-07h23m02s] in backup [tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] [2024-01-19T07:26:23.335Z] 2024-01-19 07:26:21 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82943-01-19-07h23m02s] in backup [tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] [2024-01-19T07:26:23.335Z] 2024-01-19 07:26:21 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82943-01-19-07h23m02s] [2024-01-19T07:26:23.335Z] 2024-01-19 07:26:21 +0000:[WARNING] [{ShareBackupsAndClusterWithUser}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T07:26:23.335Z] 2024-01-19 07:26:21 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T07:26:23.335Z] 2024-01-19 07:26:21 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82943-01-19-07h23m02s] are [[mysql-data]] [2024-01-19T07:26:23.335Z] 2024-01-19 07:26:21 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T07:26:23.335Z] 2024-01-19 07:26:21 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Validating CR cleanup [2024-01-19T07:26:23.335Z] 2024-01-19 07:26:22 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T07:26:23.335Z] 2024-01-19 07:26:22 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] All CRs in [mysql-ibm-pxb-0-82943-01-19-07h23m02s] are [[]] [2024-01-19T07:26:23.335Z] INFO[2024-01-19 07:26:22] Verifying : Description : Creation and Validation of backup [tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] [2024-01-19T07:26:23.335Z] INFO[2024-01-19 07:26:22] Actual:true, Expected: true [2024-01-19T07:26:23.335Z] 2024-01-19 07:26:22 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Share backup with user having full access [2024-01-19T07:26:23.335Z] 2024-01-19 07:26:22 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [backup.ShareBackup:#715] - Backup UID for tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s - 4956d578-80f1-45ec-8a29-fff567585c11 [2024-01-19T07:26:23.335Z] 2024-01-19 07:26:22 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705649083 - 7a8f5e17-45e5-4587-bc88-876237f707b2 [2024-01-19T07:26:23.335Z] 2024-01-19 07:26:22 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Validating if user with FullAccess cannot duplicate backup shared but can create new backup [2024-01-19T07:26:23.335Z] 2024-01-19 07:26:22 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Registering Source and Destination clusters from user context [2024-01-19T07:26:23.335Z] INFO[2024-01-19 07:26:22] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T07:26:23.335Z] INFO[2024-01-19 07:26:22] Actual:true, Expected: true [2024-01-19T07:26:23.336Z] 2024-01-19 07:26:22 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T07:26:23.336Z] 2024-01-19 07:26:22 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T07:26:23.336Z] 2024-01-19 07:26:22 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T07:26:23.336Z] 2024-01-19 07:26:22 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T07:26:23.336Z] 2024-01-19 07:26:22 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Create cluster [source-cluster] in org [default] [2024-01-19T07:26:23.336Z] 2024-01-19 07:26:22 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T07:26:23.336Z] 2024-01-19 07:26:22 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T07:26:23.336Z] 2024-01-19 07:26:22 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Create cluster [destination-cluster] in org [default] [2024-01-19T07:26:23.336Z] 2024-01-19 07:26:22 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T07:26:23.336Z] 2024-01-19 07:26:22 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T07:26:23.336Z] 2024-01-19 07:26:22 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-giori] for IBM [2024-01-19T07:26:23.336Z] 2024-01-19 07:26:22 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-giori] for org [default] with [ibm] as provider [2024-01-19T07:26:23.336Z] 2024-01-19 07:26:22 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T07:26:23.336Z] 2024-01-19 07:26:22 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T07:26:23.336Z] 2024-01-19 07:26:22 +0000:[WARNING] [{ShareBackupsAndClusterWithUser}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-source-config-cloud-cred-giori] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user]] [2024-01-19T07:26:23.336Z] 2024-01-19 07:26:22 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user] [2024-01-19T07:26:23.336Z] 2024-01-19 07:26:23 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-source-config-cloud-cred-giori from admin context and sharing with all the users [2024-01-19T07:26:23.336Z] 2024-01-19 07:26:23 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-giori] for org [default] with [ibm] as provider [2024-01-19T07:26:23.336Z] 2024-01-19 07:26:23 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T07:26:23.336Z] 2024-01-19 07:26:23 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T07:26:23.593Z] 2024-01-19 07:26:23 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-giori] is present [2024-01-19T07:26:23.593Z] 2024-01-19 07:26:23 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Create cluster [source-cluster] in org [default] [2024-01-19T07:26:27.759Z] 2024-01-19 07:26:27 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-jeilk] for IBM [2024-01-19T07:26:27.759Z] 2024-01-19 07:26:27 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-jeilk] for org [default] with [ibm] as provider [2024-01-19T07:26:27.759Z] 2024-01-19 07:26:27 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T07:26:27.759Z] 2024-01-19 07:26:27 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T07:26:27.759Z] 2024-01-19 07:26:27 +0000:[WARNING] [{ShareBackupsAndClusterWithUser}] [tests.CreateCloudCredential:#4506] - failed to create cloud credential with name [ibm-destination-config-cloud-cred-jeilk] in org [default] with [ibm] as provider with error [rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user]] [2024-01-19T07:26:27.759Z] 2024-01-19 07:26:27 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateApplicationClusters:#4235] - The error is - rpc error: code = PermissionDenied desc = Access denied for [Resource: cloudcredential] [API: create] for user FirstName1 LastName1 with roles: [px-backup-app.user] [2024-01-19T07:26:27.759Z] 2024-01-19 07:26:27 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateApplicationClusters:#4240] - Creating cloud credential ibm-destination-config-cloud-cred-jeilk from admin context and sharing with all the users [2024-01-19T07:26:27.760Z] 2024-01-19 07:26:27 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-jeilk] for org [default] with [ibm] as provider [2024-01-19T07:26:27.760Z] 2024-01-19 07:26:27 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T07:26:27.760Z] 2024-01-19 07:26:27 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T07:26:27.760Z] 2024-01-19 07:26:27 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-jeilk] is present [2024-01-19T07:26:27.760Z] 2024-01-19 07:26:27 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Create cluster [destination-cluster] in org [default] [2024-01-19T07:26:28.321Z] INFO[2024-01-19 07:26:28] Verifying : Description : Creating source and destination cluster [2024-01-19T07:26:28.322Z] INFO[2024-01-19 07:26:28] Actual:true, Expected: true [2024-01-19T07:26:28.322Z] INFO[2024-01-19 07:26:28] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T07:26:28.322Z] INFO[2024-01-19 07:26:28] Actual:true, Expected: true [2024-01-19T07:26:28.322Z] 2024-01-19 07:26:28 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Uid of [source-cluster] cluster by user [testuser1-1705649083] is fe37e02a-d9a2-4948-9c61-74b9a1922076 [2024-01-19T07:26:28.322Z] 2024-01-19 07:26:28 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T07:26:28.583Z] 2024-01-19 07:26:28 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Volume snapshot classes found in the cluster - [2024-01-19T07:26:28.583Z] 2024-01-19 07:26:28 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] ibmc-vpcblock-snapshot [2024-01-19T07:26:28.583Z] 2024-01-19 07:26:28 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T07:26:28.839Z] 2024/01/19 07:26:28 DoRetryWithTimeout - Error: {backup status for [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T07:27:00.866Z] 2024/01/19 07:26:58 DoRetryWithTimeout - Error: {backup status for [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T07:27:32.905Z] 2024/01/19 07:27:28 DoRetryWithTimeout - Error: {backup status for [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T07:27:59.433Z] 2024/01/19 07:27:58 DoRetryWithTimeout - Error: {backup status for [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T07:28:31.474Z] 2024/01/19 07:28:28 DoRetryWithTimeout - Error: {backup status for [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T07:29:03.496Z] 2024-01-19 07:28:58 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [backup.CreateBackup:#313] - Backup [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] created successfully [2024-01-19T07:29:03.496Z] 2024-01-19 07:28:58 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Validating backup [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] in org [default] [2024-01-19T07:29:03.496Z] 2024-01-19 07:28:58 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] [2024-01-19T07:29:03.497Z] 2024-01-19 07:28:58 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82943-01-19-07h23m02s] in backup [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] [2024-01-19T07:29:03.497Z] 2024-01-19 07:28:58 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82943-01-19-07h23m02s] in backup [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] [2024-01-19T07:29:03.497Z] 2024-01-19 07:28:58 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-82943-01-19-07h23m02s] [2024-01-19T07:29:03.497Z] 2024-01-19 07:28:58 +0000:[WARNING] [{ShareBackupsAndClusterWithUser}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T07:29:03.497Z] 2024-01-19 07:28:58 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T07:29:03.497Z] 2024-01-19 07:28:58 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-82943-01-19-07h23m02s] are [[mysql-data]] [2024-01-19T07:29:03.497Z] 2024-01-19 07:28:58 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T07:29:03.497Z] 2024-01-19 07:28:58 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Validating CR cleanup [2024-01-19T07:29:03.497Z] 2024-01-19 07:28:58 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T07:29:03.497Z] 2024-01-19 07:28:59 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] All CRs in [mysql-ibm-pxb-0-82943-01-19-07h23m02s] are [[]] [2024-01-19T07:29:03.497Z] INFO[2024-01-19 07:28:59] Verifying : Description : Creation and Validation of backup [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] [2024-01-19T07:29:03.497Z] INFO[2024-01-19 07:28:59] Actual:true, Expected: true [2024-01-19T07:29:03.497Z] 2024-01-19 07:28:59 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Destroying apps [2024-01-19T07:29:03.497Z] 2024-01-19 07:28:59 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T07:29:03.497Z] 2024-01-19 07:28:59 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T07:29:03.497Z] 2024-01-19 07:28:59 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] start destroying mysql-ibm app [2024-01-19T07:29:03.497Z] 2024-01-19 07:28:59 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T07:29:03.497Z] 2024-01-19 07:28:59 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T07:29:03.497Z] 2024-01-19 07:28:59 +0000:[WARNING] [{ShareBackupsAndClusterWithUser}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T07:29:03.497Z] INFO[2024-01-19 07:28:59] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T07:29:03.497Z] INFO[2024-01-19 07:28:59] Actual:true, Expected: true [2024-01-19T07:29:03.497Z] 2024-01-19 07:28:59 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [backup.glob..func89.3:#2525] - Deleting backup created by user - testuser1-1705649083 [2024-01-19T07:29:03.497Z] INFO[2024-01-19 07:28:59] Verifying : Description : Getting backup UID of user for backup user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s [2024-01-19T07:29:03.497Z] INFO[2024-01-19 07:28:59] Actual:true, Expected: true [2024-01-19T07:29:03.497Z] INFO[2024-01-19 07:28:59] Verifying : Description : Deleting backup user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s created by user [2024-01-19T07:29:03.498Z] INFO[2024-01-19 07:28:59] Actual:true, Expected: true [2024-01-19T07:29:03.498Z] 2024/01/19 07:28:59 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:29:31.605Z] 2024/01/19 07:29:29 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:30:03.633Z] 2024/01/19 07:29:59 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:30:30.152Z] 2024/01/19 07:30:29 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:31:02.185Z] 2024/01/19 07:30:59 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:31:34.217Z] 2024/01/19 07:31:29 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:32:00.722Z] 2024/01/19 07:31:59 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:32:32.753Z] 2024/01/19 07:32:29 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:33:04.784Z] 2024/01/19 07:32:59 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:33:31.290Z] 2024/01/19 07:33:29 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:34:03.313Z] 2024/01/19 07:33:59 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:34:35.337Z] 2024/01/19 07:34:29 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:35:01.842Z] 2024/01/19 07:34:59 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:35:07.089Z] INFO[2024-01-19 07:35:06] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T07:35:07.089Z] INFO[2024-01-19 07:35:06] sched stats: tick interval: 57666 < 1.2s (100%) [2024-01-19T07:35:07.089Z] INFO[2024-01-19 07:35:06] sched stats: tick process durations: 57665 < 500ms (100%) [2024-01-19T07:35:33.591Z] 2024/01/19 07:35:29 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:36:00.106Z] 2024/01/19 07:35:59 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:36:32.136Z] 2024/01/19 07:36:29 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:37:04.190Z] 2024/01/19 07:36:59 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:37:30.744Z] 2024/01/19 07:37:30 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:38:02.770Z] 2024/01/19 07:38:00 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:38:34.828Z] 2024/01/19 07:38:30 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:39:01.353Z] 2024/01/19 07:39:00 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:39:33.379Z] 2024/01/19 07:39:30 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:40:05.402Z] 2024/01/19 07:40:00 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:40:31.899Z] 2024/01/19 07:40:30 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:41:03.929Z] 2024/01/19 07:41:00 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:41:30.482Z] 2024/01/19 07:41:30 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:42:02.515Z] 2024/01/19 07:42:00 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:42:34.602Z] 2024/01/19 07:42:30 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:43:01.158Z] 2024/01/19 07:43:00 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:43:33.905Z] 2024/01/19 07:43:30 DoRetryWithTimeout - Error: {backupObject [user-tp-backup-mysql-ibm-pxb-0-82943-01-19-07h23m02s] is not yet deleted . status:[status:Deleting reason:"Volumes and resources are being deleted" ] }, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:44:00.448Z] 2024-01-19 07:44:00 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [backup.glob..func89.3:#2533] - Deleting registered clusters for non-admin context [2024-01-19T07:44:00.448Z] INFO[2024-01-19 07:44:00] Verifying : Description : Deleting cluster source-cluster [2024-01-19T07:44:00.448Z] INFO[2024-01-19 07:44:00] Actual:true, Expected: true [2024-01-19T07:44:00.448Z] INFO[2024-01-19 07:44:00] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T07:44:00.448Z] INFO[2024-01-19 07:44:00] Actual:true, Expected: true [2024-01-19T07:44:00.448Z] 2024-01-19 07:44:00 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Cleaning backup locations in map [map[75e4ae32-04e3-4c9e-89b9-10ac5e10f350:aws-global-aws-px-backup-system-test-4cac76dc-1705649084-bl]], cloud credential [cloudcred-aws-1705649084], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T07:44:00.704Z] INFO[2024-01-19 07:44:00] Verifying : Description : Verifying deletion of backup location [aws-global-aws-px-backup-system-test-4cac76dc-1705649084-bl] [2024-01-19T07:44:00.704Z] INFO[2024-01-19 07:44:00] Actual:true, Expected: true [2024-01-19T07:44:00.704Z] 2024-01-19 07:44:00 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [backup.IsBackupLocationPresent:#2442] - Backup location [aws-global-aws-px-backup-system-test-4cac76dc-1705649084-bl] is present [2024-01-19T07:44:00.704Z] 2024/01/19 07:44:00 DoRetryWithTimeout - Error: {backup location aws-global-aws-px-backup-system-test-4cac76dc-1705649084-bl is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:44:32.725Z] 2024-01-19 07:44:30 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [backup.IsBackupLocationPresent:#2442] - Backup location [aws-global-aws-px-backup-system-test-4cac76dc-1705649084-bl] is present [2024-01-19T07:44:32.725Z] 2024/01/19 07:44:30 DoRetryWithTimeout - Error: {backup location aws-global-aws-px-backup-system-test-4cac76dc-1705649084-bl is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:45:04.760Z] 2024-01-19 07:45:00 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [backup.IsBackupLocationPresent:#2442] - Backup location [aws-global-aws-px-backup-system-test-4cac76dc-1705649084-bl] is present [2024-01-19T07:45:04.760Z] 2024/01/19 07:45:00 DoRetryWithTimeout - Error: {backup location aws-global-aws-px-backup-system-test-4cac76dc-1705649084-bl is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:45:06.650Z] INFO[2024-01-19 07:45:06] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T07:45:06.650Z] INFO[2024-01-19 07:45:06] sched stats: tick interval: 58266 < 1.2s (100%) [2024-01-19T07:45:06.650Z] INFO[2024-01-19 07:45:06] sched stats: tick process durations: 58265 < 500ms (100%) [2024-01-19T07:45:33.149Z] 2024-01-19 07:45:30 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [backup.IsBackupLocationPresent:#2442] - Backup location [aws-global-aws-px-backup-system-test-4cac76dc-1705649084-bl] is present [2024-01-19T07:45:33.149Z] 2024/01/19 07:45:31 DoRetryWithTimeout - Error: {backup location aws-global-aws-px-backup-system-test-4cac76dc-1705649084-bl is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T07:46:05.185Z] 2024-01-19 07:46:01 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 aws-global-aws-px-backup-system-test-4cac76dc-bl autogenerated-backup-location-1705628002 autogenerated-backup-location-1705634221 autogenerated-backup-location-1705635982 autogenerated-backup-location-wkyfv] [2024-01-19T07:46:05.185Z] INFO[2024-01-19 07:46:01] Verifying : Description : Verifying backup location deletion status aws-global-aws-px-backup-system-test-4cac76dc-1705649084-bl [2024-01-19T07:46:05.185Z] INFO[2024-01-19 07:46:01] Actual:true, Expected: true [2024-01-19T07:46:05.185Z] 2024-01-19 07:46:01 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cloudcred-aws-1705649084] is present [2024-01-19T07:46:05.185Z] INFO[2024-01-19 07:46:01] Verifying : Description : Verifying if cloud cred [cloudcred-aws-1705649084] is present [2024-01-19T07:46:05.185Z] INFO[2024-01-19 07:46:01] Actual:true, Expected: true [2024-01-19T07:46:05.186Z] INFO[2024-01-19 07:46:01] Verifying : Description : Verifying deletion of cloud cred [cloudcred-aws-1705649084] [2024-01-19T07:46:05.186Z] INFO[2024-01-19 07:46:01] Actual:true, Expected: true [2024-01-19T07:46:05.186Z] INFO[2024-01-19 07:46:01] Verifying : Description : Deleting cloud cred cloudcred-aws-1705649084 [2024-01-19T07:46:05.186Z] INFO[2024-01-19 07:46:01] Actual:true, Expected: true [2024-01-19T07:46:05.186Z] INFO[2024-01-19 07:46:01] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T07:46:05.186Z] INFO[2024-01-19 07:46:01] Actual:true, Expected: true [2024-01-19T07:46:05.186Z] INFO[2024-01-19 07:46:01] Verifying : Description : Deleting cluster source-cluster [2024-01-19T07:46:05.186Z] ERRO[2024-01-19 07:46:01] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl], cannot be deleted, Expected:  [2024-01-19T07:46:05.186Z] INFO[2024-01-19 07:46:01] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-hrfiy] [2024-01-19T07:46:05.186Z] ERRO[2024-01-19 07:46:01] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-hrfiy:72b1b409-7bb1-4f22-92df-0ba5e77c11d3], Expected:  [2024-01-19T07:46:05.186Z] INFO[2024-01-19 07:46:02] Verifying : Description : Deleting cluster source-cluster [2024-01-19T07:46:05.186Z] ERRO[2024-01-19 07:46:02] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh], cannot be deleted, Expected:  [2024-01-19T07:46:05.186Z] INFO[2024-01-19 07:46:02] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-fbvvo] [2024-01-19T07:46:05.186Z] ERRO[2024-01-19 07:46:02] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-fbvvo:7cddb5aa-c7ff-4044-9d27-458be2bd5cde], Expected:  [2024-01-19T07:46:05.186Z] INFO[2024-01-19 07:46:02] Verifying : Description : Deleting cluster source-cluster [2024-01-19T07:46:05.186Z] ERRO[2024-01-19 07:46:02] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo], cannot be deleted, Expected:  [2024-01-19T07:46:05.186Z] INFO[2024-01-19 07:46:03] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-enzij] [2024-01-19T07:46:05.186Z] ERRO[2024-01-19 07:46:03] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-enzij:4e0ddfc7-8811-4ed5-8e87-ed0841582111], Expected:  [2024-01-19T07:46:05.186Z] INFO[2024-01-19 07:46:03] Verifying : Description : Deleting cluster source-cluster [2024-01-19T07:46:05.186Z] ERRO[2024-01-19 07:46:03] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo], cannot be deleted, Expected:  [2024-01-19T07:46:05.443Z] INFO[2024-01-19 07:46:05] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-wcpfm] [2024-01-19T07:46:05.443Z] ERRO[2024-01-19 07:46:05] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-wcpfm:c5ff084a-a913-4412-99fb-e2b0a003a014], Expected:  [2024-01-19T07:46:05.443Z] INFO[2024-01-19 07:46:05] Verifying : Description : Deleting cluster source-cluster [2024-01-19T07:46:05.443Z] ERRO[2024-01-19 07:46:05] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx], cannot be deleted, Expected:  [2024-01-19T07:46:06.807Z] INFO[2024-01-19 07:46:06] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-vlbwu] [2024-01-19T07:46:06.807Z] ERRO[2024-01-19 07:46:06] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-vlbwu:f0465bb5-2565-4948-bb27-7e45a4b883ff], Expected:  [2024-01-19T07:46:06.807Z] INFO[2024-01-19 07:46:06] Verifying : Description : Deleting cluster source-cluster [2024-01-19T07:46:06.807Z] ERRO[2024-01-19 07:46:06] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa], cannot be deleted, Expected:  [2024-01-19T07:46:08.191Z] INFO[2024-01-19 07:46:07] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-mhxzi] [2024-01-19T07:46:08.191Z] ERRO[2024-01-19 07:46:07] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-mhxzi:f551fb81-1a78-4d7f-b804-759c6e6b562e], Expected:  [2024-01-19T07:46:08.530Z] INFO[2024-01-19 07:46:08] Verifying : Description : Deleting cluster source-cluster [2024-01-19T07:46:08.530Z] INFO[2024-01-19 07:46:08] Actual:true, Expected: true [2024-01-19T07:46:09.900Z] INFO[2024-01-19 07:46:09] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-gfjkf] [2024-01-19T07:46:09.900Z] INFO[2024-01-19 07:46:09] Actual:true, Expected: true [2024-01-19T07:46:09.900Z] INFO[2024-01-19 07:46:09] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T07:46:09.900Z] INFO[2024-01-19 07:46:09] Actual:true, Expected: true [2024-01-19T07:46:11.263Z] INFO[2024-01-19 07:46:10] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-cnzjj] [2024-01-19T07:46:11.263Z] INFO[2024-01-19 07:46:10] Actual:true, Expected: true [2024-01-19T07:46:11.263Z] INFO[2024-01-19 07:46:10] --------Test End------ [2024-01-19T07:46:11.263Z] INFO[2024-01-19 07:46:10] #Test: [2024-01-19T07:46:11.263Z] INFO[2024-01-19 07:46:10] #Description: [2024-01-19T07:46:11.263Z] INFO[2024-01-19 07:46:10] ------------------------ [2024-01-19T07:46:11.263Z] 2024-01-19 07:46:11 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-82943-01-19-07h23m02s] [2024-01-19T07:46:11.263Z] 2024-01-19 07:46:11 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T07:46:11.263Z] 2024-01-19 07:46:11 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T07:46:11.521Z] 2024-01-19 07:46:11 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-82943-01-19-07h23m02s] [2024-01-19T07:46:11.778Z] 2024-01-19 07:46:11 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {ShareBackupsAndClusterWithUser} Share Backup With Full Access Users and try to duplicate the backup [2024-01-19T07:46:11.778Z] 2024-01-19 07:46:11 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T07:46:11.778Z] 2024/01/19 07:46:11 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T07:46:17.021Z] 2024/01/19 07:46:16 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T07:46:22.278Z] 2024/01/19 07:46:21 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T07:46:27.525Z] 2024/01/19 07:46:26 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T07:46:31.700Z] 2024/01/19 07:46:31 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T07:46:36.958Z] 2024/01/19 07:46:36 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T07:46:42.228Z] 2024/01/19 07:46:41 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T07:46:47.473Z] 2024/01/19 07:46:46 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T07:46:51.639Z] 2024/01/19 07:46:51 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.128.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.7 10.240.128.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T07:46:51.639Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T07:46:51.640Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T07:46:51.640Z] {dial tcp 10.240.128.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T07:46:56.888Z] 2024-01-19 07:46:56 +0000:[WARNING] [{ShareBackupsAndClusterWithUser}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.7 10.240.128.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T07:46:56.888Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T07:46:56.888Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T07:46:56.888Z] {dial tcp 10.240.128.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T07:46:56.888Z] 2024-01-19 07:46:56 +0000:[ERROR] [{ShareBackupsAndClusterWithUser}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.7 10.240.128.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T07:46:56.888Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T07:46:56.888Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T07:46:56.888Z] {dial tcp 10.240.128.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T07:46:56.888Z] 2024-01-19 07:46:56 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T07:46:56.888Z] 2024-01-19 07:46:56 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] Switching context to [/tmp/source-config] [2024-01-19T07:46:56.888Z] 2024-01-19 07:46:56 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T07:46:56.888Z] 2024-01-19 07:46:56 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T07:46:56.888Z] 2024-01-19 07:46:56 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T07:46:56.888Z] 2024-01-19 07:46:56 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T07:46:57.145Z] 2024-01-19 07:46:56 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T07:46:57.145Z] 2024-01-19 07:46:56 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T07:46:57.400Z] 2024-01-19 07:46:57 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T07:46:58.033Z] 2024-01-19 07:46:57 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T07:46:58.296Z] 2024-01-19 07:46:58 +0000:[INFO] [{ShareBackupsAndClusterWithUser}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T07:46:58.552Z] 2024/01/19 07:46:58 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:47:08.500Z] 2024/01/19 07:47:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:47:18.450Z] 2024/01/19 07:47:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:47:30.612Z] 2024/01/19 07:47:28 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:47:38.733Z] 2024/01/19 07:47:38 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:47:48.672Z] 2024/01/19 07:47:48 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:47:58.621Z] 2024/01/19 07:47:58 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:48:10.811Z] 2024/01/19 07:48:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:48:18.895Z] 2024/01/19 07:48:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:48:28.831Z] 2024/01/19 07:48:28 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:48:38.773Z] 2024/01/19 07:48:38 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:48:48.712Z] 2024/01/19 07:48:48 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:49:00.878Z] 2024/01/19 07:48:58 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:49:08.987Z] 2024/01/19 07:49:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:49:18.932Z] 2024/01/19 07:49:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:49:28.879Z] 2024/01/19 07:49:28 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:49:41.043Z] 2024/01/19 07:49:38 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:49:49.132Z] 2024/01/19 07:49:48 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:49:59.087Z] 2024/01/19 07:49:58 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:50:09.066Z] 2024/01/19 07:50:08 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:50:19.009Z] 2024/01/19 07:50:18 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:50:31.182Z] 2024/01/19 07:50:28 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:50:39.342Z] 2024/01/19 07:50:39 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:50:49.316Z] 2024/01/19 07:50:49 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:50:59.256Z] 2024/01/19 07:50:59 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:51:09.213Z] 2024/01/19 07:51:09 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:51:19.165Z] 2024/01/19 07:51:19 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:51:31.341Z] 2024/01/19 07:51:29 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:51:39.434Z] 2024/01/19 07:51:39 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:51:49.380Z] 2024/01/19 07:51:49 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:51:59.342Z] 2024-01-19 07:51:59 +0000:[ERROR] [{ShareBackupsAndClusterWithUser}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.342Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.343Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.343Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.343Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.343Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.343Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.343Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.343Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.343Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.343Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.343Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.343Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.343Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.343Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.343Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.343Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.343Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.344Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.344Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.344Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.344Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.344Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.344Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.344Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.344Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.344Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.344Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.344Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.344Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T07:51:59.344Z] • Failure [1737.117 seconds] [2024-01-19T07:51:59.344Z] {ShareBackupsAndClusterWithUser} [2024-01-19T07:51:59.344Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:2414 [2024-01-19T07:51:59.344Z] Share Backup With Full Access Users and try to duplicate the backup [] [2024-01-19T07:51:59.344Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:2449 [2024-01-19T07:51:59.344Z] [2024-01-19T07:51:59.344Z] Unexpected error: [2024-01-19T07:51:59.344Z] <*errors.errorString | 0xc0010a2ef0>: { [2024-01-19T07:51:59.345Z] s: "failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{No... [2024-01-19T07:51:59.345Z] [2024-01-19T07:51:59.345Z] Gomega truncated this representation as it exceeds 'format.MaxLength'. [2024-01-19T07:51:59.346Z] Consider having the object provide a custom 'GomegaStringer' representation [2024-01-19T07:51:59.346Z] or adjust the parameters in Gomega's 'format' package. [2024-01-19T07:51:59.346Z] [2024-01-19T07:51:59.346Z] Learn more here: https://onsi.github.io/gomega/#adjusting-output [2024-01-19T07:51:59.346Z] [2024-01-19T07:51:59.346Z] failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.346Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.346Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.346Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.346Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.346Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.346Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.346Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.346Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.346Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.346Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.346Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.346Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.346Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.346Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.346Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.346Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.346Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.346Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.346Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.347Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.347Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.347Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.347Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.347Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.347Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.347Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.347Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.347Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T07:51:59.347Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T07:51:59.347Z] occurred [2024-01-19T07:51:59.347Z] [2024-01-19T07:51:59.347Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T07:51:59.347Z] [2024-01-19T07:51:59.347Z] Full Stack Trace [2024-01-19T07:51:59.347Z] github.com/portworx/torpedo/pkg/log.FailOnError({0x88a5060, 0xc0010a2ef0}, {0x7f74bfd?, 0x6cc0320?}, {0x0?, 0x6cc0320?, 0xc0018ee1f0?}) [2024-01-19T07:51:59.347Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 +0x290 [2024-01-19T07:51:59.347Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest.func1() [2024-01-19T07:51:59.347Z] /go/src/github.com/portworx/torpedo/tests/common.go:7283 +0x32 [2024-01-19T07:51:59.347Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest({0xc00048ce80, 0x1, 0x1}) [2024-01-19T07:51:59.347Z] /go/src/github.com/portworx/torpedo/tests/common.go:7299 +0x778 [2024-01-19T07:51:59.347Z] github.com/portworx/torpedo/tests/backup.glob..func89.3() [2024-01-19T07:51:59.347Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_share_test.go:2540 +0x638 [2024-01-19T07:51:59.347Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T07:51:59.347Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T07:51:59.347Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T07:51:59.347Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T07:51:59.347Z] created by testing.(*T).Run [2024-01-19T07:51:59.347Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T07:51:59.347Z] ------------------------------ [2024-01-19T07:51:59.347Z] STEP: schedule applications [2024-01-19T07:51:59.347Z] STEP: Validate applications [2024-01-19T07:51:59.347Z] STEP: validate applications [2024-01-19T07:51:59.347Z] STEP: validate mysql-ibm app's volumes [2024-01-19T07:51:59.347Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T07:51:59.347Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T07:51:59.347Z] STEP: get mysql-ibm app's volume: pvc-ef403025-537b-43fa-a971-0d3a92b4b4d0 inspected by the volume driver [2024-01-19T07:51:59.347Z] STEP: wait for mysql-ibm app to start running [2024-01-19T07:51:59.347Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T07:51:59.347Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T07:51:59.347Z] STEP: Validate Px pod restart count [2024-01-19T07:51:59.347Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T07:51:59.347Z] STEP: Create Users [2024-01-19T07:51:59.347Z] STEP: Creating backup location and cloud setting [2024-01-19T07:51:59.347Z] STEP: Register cluster for backup [2024-01-19T07:51:59.347Z] STEP: Taking backup of applications [2024-01-19T07:51:59.347Z] STEP: Share backup with user having full access [2024-01-19T07:51:59.347Z] STEP: Create backup from the shared user with FullAccess [2024-01-19T07:51:59.348Z] STEP: destroy apps [2024-01-19T07:51:59.348Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T07:51:59.348Z] STEP: start destroying mysql-ibm app [2024-01-19T07:51:59.348Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T07:51:59.348Z] 2024-01-19 07:51:59 +0000:[INFO] [{CustomResourceBackupAndRestore}] Log Dir: /testresults//CustomResourceBackupAndRestore.log [2024-01-19T07:51:59.348Z] INFO[2024-01-19 07:51:59] --------Test Start------ [2024-01-19T07:51:59.348Z] INFO[2024-01-19 07:51:59] #Test: CustomResourceBackupAndRestore [2024-01-19T07:51:59.348Z] INFO[2024-01-19 07:51:59] #Description: Create custom resource backup and restore [2024-01-19T07:51:59.348Z] INFO[2024-01-19 07:51:59] ------------------------ [2024-01-19T07:51:59.348Z] 2024-01-19 07:51:59 +0000:[INFO] [{CustomResourceBackupAndRestore}] Deploy applications [2024-01-19T07:51:59.348Z] 2024-01-19 07:51:59 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T07:51:59.348Z] 2024-01-19 07:51:59 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T07:51:59.348Z] 2024-01-19 07:51:59 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T07:51:59.348Z] 2024-01-19 07:51:59 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T07:51:59.609Z] 2024-01-19 07:51:59 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T07:51:59.609Z] 2024-01-19 07:51:59 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T07:51:59.610Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T07:51:59.610Z] 2024-01-19 07:51:59 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T07:51:59.611Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T07:51:59.611Z] 2024-01-19 07:51:59 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T07:51:59.611Z] 2024-01-19 07:51:59 +0000:[INFO] [{CustomResourceBackupAndRestore}] Validate applications [2024-01-19T07:51:59.611Z] 2024-01-19 07:51:59 +0000:[INFO] [{CustomResourceBackupAndRestore}] Validating mysql-ibm app [2024-01-19T07:51:59.611Z] 2024-01-19 07:51:59 +0000:[INFO] [{CustomResourceBackupAndRestore}] Validating mysql-ibm app's volumes [2024-01-19T07:51:59.611Z] 2024-01-19 07:51:59 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T07:51:59.611Z] 2024/01/19 07:51:59 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T07:52:11.777Z] 2024/01/19 07:52:09 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T07:52:20.955Z] 2024-01-19 07:52:19 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-83720-01-19-07h51m59s [2024-01-19T07:52:20.955Z] 2024-01-19 07:52:19 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T07:52:20.955Z] 2024-01-19 07:52:19 +0000:[WARNING] [{CustomResourceBackupAndRestore}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T07:52:20.955Z] 2024-01-19 07:52:19 +0000:[INFO] [{CustomResourceBackupAndRestore}] wait for mysql-ibm app to start running [2024-01-19T07:52:20.955Z] 2024-01-19 07:52:19 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T07:52:20.955Z] 2024/01/19 07:52:19 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T07:52:20.955Z] pod name:mysql-7d766cd65c-qn4dc namespace:mysql-ibm-pxb-0-83720-01-19-07h51m59s running:false ready:false node:10.240.64.8 [2024-01-19T07:52:20.955Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T07:52:30.915Z] 2024/01/19 07:52:30 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T07:52:30.915Z] pod name:mysql-7d766cd65c-qn4dc namespace:mysql-ibm-pxb-0-83720-01-19-07h51m59s running:true ready:false node:10.240.64.8 [2024-01-19T07:52:30.915Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T07:52:40.873Z] 2024/01/19 07:52:40 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T07:52:40.873Z] pod name:mysql-7d766cd65c-qn4dc namespace:mysql-ibm-pxb-0-83720-01-19-07h51m59s running:true ready:false node:10.240.64.8 [2024-01-19T07:52:40.873Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T07:52:50.825Z] 2024/01/19 07:52:50 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T07:52:50.825Z] pod name:mysql-7d766cd65c-qn4dc namespace:mysql-ibm-pxb-0-83720-01-19-07h51m59s running:true ready:false node:10.240.64.8 [2024-01-19T07:52:50.825Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T07:53:00.765Z] 2024/01/19 07:53:00 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T07:53:00.765Z] pod name:mysql-7d766cd65c-qn4dc namespace:mysql-ibm-pxb-0-83720-01-19-07h51m59s running:true ready:false node:10.240.64.8 [2024-01-19T07:53:00.765Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T07:53:10.717Z] 2024/01/19 07:53:10 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T07:53:10.717Z] pod name:mysql-7d766cd65c-qn4dc namespace:mysql-ibm-pxb-0-83720-01-19-07h51m59s running:true ready:false node:10.240.64.8 [2024-01-19T07:53:10.717Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T07:53:20.684Z] 2024/01/19 07:53:20 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T07:53:20.684Z] pod name:mysql-7d766cd65c-qn4dc namespace:mysql-ibm-pxb-0-83720-01-19-07h51m59s running:true ready:false node:10.240.64.8 [2024-01-19T07:53:20.684Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T07:53:32.862Z] 2024/01/19 07:53:30 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T07:53:32.862Z] pod name:mysql-7d766cd65c-qn4dc namespace:mysql-ibm-pxb-0-83720-01-19-07h51m59s running:true ready:false node:10.240.64.8 [2024-01-19T07:53:32.862Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T07:53:41.014Z] 2024-01-19 07:53:40 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T07:53:41.014Z] 2024-01-19 07:53:40 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T07:53:41.014Z] 2024-01-19 07:53:40 +0000:[INFO] [{CustomResourceBackupAndRestore}] validate if mysql-ibm app's volumes are setup [2024-01-19T07:53:41.014Z] 2024-01-19 07:53:40 +0000:[WARNING] [{CustomResourceBackupAndRestore}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T07:53:41.014Z] 2024-01-19 07:53:40 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T07:53:41.014Z] 2024-01-19 07:53:40 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T07:53:41.014Z] 2024-01-19 07:53:40 +0000:[WARNING] [{CustomResourceBackupAndRestore}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T07:53:41.275Z] 2024-01-19 07:53:41 +0000:[INFO] [{CustomResourceBackupAndRestore}] Creating cloud credentials [2024-01-19T07:53:41.531Z] 2024-01-19 07:53:41 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [cred-aws-1705650821] for org [default] with [aws] as provider [2024-01-19T07:53:41.531Z] 2024-01-19 07:53:41 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T07:53:41.531Z] 2024-01-19 07:53:41 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cred-aws-1705650821] is present [2024-01-19T07:53:41.531Z] INFO[2024-01-19 07:53:41] Verifying : Description : Verifying creation of cloud credential named [cred-aws-1705650821] for org [default] with [aws] as provider [2024-01-19T07:53:41.531Z] INFO[2024-01-19 07:53:41] Actual:true, Expected: true [2024-01-19T07:53:42.460Z] INFO[2024-01-19 07:53:42] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T07:53:42.461Z] INFO[2024-01-19 07:53:42] Actual:true, Expected: true [2024-01-19T07:53:42.461Z] 2024-01-19 07:53:42 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T07:53:42.717Z] 2024-01-19 07:53:42 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T07:53:42.717Z] 2024-01-19 07:53:42 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T07:53:42.717Z] 2024-01-19 07:53:42 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T07:53:42.717Z] 2024-01-19 07:53:42 +0000:[INFO] [{CustomResourceBackupAndRestore}] Create cluster [source-cluster] in org [default] [2024-01-19T07:53:42.717Z] 2024-01-19 07:53:42 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T07:53:42.717Z] 2024-01-19 07:53:42 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T07:53:42.717Z] 2024-01-19 07:53:42 +0000:[INFO] [{CustomResourceBackupAndRestore}] Create cluster [destination-cluster] in org [default] [2024-01-19T07:53:42.717Z] 2024-01-19 07:53:42 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T07:53:42.717Z] 2024-01-19 07:53:42 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T07:53:42.717Z] 2024-01-19 07:53:42 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-upjpy] for IBM [2024-01-19T07:53:42.717Z] 2024-01-19 07:53:42 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-upjpy] for org [default] with [ibm] as provider [2024-01-19T07:53:42.717Z] 2024-01-19 07:53:42 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T07:53:42.717Z] 2024-01-19 07:53:42 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T07:53:42.717Z] 2024-01-19 07:53:42 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-upjpy] is present [2024-01-19T07:53:42.717Z] 2024-01-19 07:53:42 +0000:[INFO] [{CustomResourceBackupAndRestore}] Create cluster [source-cluster] in org [default] [2024-01-19T07:53:46.884Z] 2024-01-19 07:53:46 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-dkvyw] for IBM [2024-01-19T07:53:46.884Z] 2024-01-19 07:53:46 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-dkvyw] for org [default] with [ibm] as provider [2024-01-19T07:53:46.884Z] 2024-01-19 07:53:46 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T07:53:46.884Z] 2024-01-19 07:53:46 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T07:53:46.884Z] 2024-01-19 07:53:46 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-dkvyw] is present [2024-01-19T07:53:46.884Z] 2024-01-19 07:53:46 +0000:[INFO] [{CustomResourceBackupAndRestore}] Create cluster [destination-cluster] in org [default] [2024-01-19T07:53:47.139Z] INFO[2024-01-19 07:53:47] Verifying : Description : Creating source and destination cluster [2024-01-19T07:53:47.139Z] INFO[2024-01-19 07:53:47] Actual:true, Expected: true [2024-01-19T07:53:47.139Z] INFO[2024-01-19 07:53:47] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T07:53:47.140Z] INFO[2024-01-19 07:53:47] Actual:Online, Expected: Online [2024-01-19T07:53:47.140Z] INFO[2024-01-19 07:53:47] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T07:53:47.140Z] INFO[2024-01-19 07:53:47] Actual:true, Expected: true [2024-01-19T07:53:47.140Z] 2024-01-19 07:53:47 +0000:[INFO] [{CustomResourceBackupAndRestore}] Creating backup location [2024-01-19T07:53:49.655Z] INFO[2024-01-19 07:53:49] Verifying : Description : Creating backup location autogenerated-backup-location-1705650827 [2024-01-19T07:53:49.655Z] INFO[2024-01-19 07:53:49] Actual:true, Expected: true [2024-01-19T07:53:49.655Z] 2024-01-19 07:53:49 +0000:[INFO] [{CustomResourceBackupAndRestore}] Taking backup of applications [2024-01-19T07:53:49.910Z] 2024-01-19 07:53:49 +0000:[INFO] [{CustomResourceBackupAndRestore}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T07:53:49.910Z] 2024-01-19 07:53:49 +0000:[INFO] [{CustomResourceBackupAndRestore}] Volume snapshot classes found in the cluster - [2024-01-19T07:53:49.911Z] 2024-01-19 07:53:49 +0000:[INFO] [{CustomResourceBackupAndRestore}] ibmc-vpcblock-snapshot [2024-01-19T07:53:49.911Z] 2024-01-19 07:53:49 +0000:[INFO] [{CustomResourceBackupAndRestore}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T07:53:50.167Z] 2024-01-19 07:53:49 +0000:[INFO] [{CustomResourceBackupAndRestore}] [backup.CreateBackupWithCustomResourceTypeWithoutValidation:#471] - Backup [tp-backup-mysql-ibm-pxb-0-83720-01-19-07h51m59s-fohkz] created successfully [2024-01-19T07:53:50.167Z] 2024-01-19 07:53:49 +0000:[INFO] [{CustomResourceBackupAndRestore}] Validating backup [tp-backup-mysql-ibm-pxb-0-83720-01-19-07h51m59s-fohkz] in org [default] [2024-01-19T07:53:50.167Z] 2024-01-19 07:53:49 +0000:[INFO] [{CustomResourceBackupAndRestore}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-mysql-ibm-pxb-0-83720-01-19-07h51m59s-fohkz] [2024-01-19T07:53:50.167Z] 2024/01/19 07:53:50 DoRetryWithTimeout - Error: {ValidateBackup requires backup [tp-backup-mysql-ibm-pxb-0-83720-01-19-07h51m59s-fohkz] to have a status of Success or PartialSuccess,got -Pending}, Next try in [30s], timeout [40m0s] [2024-01-19T07:54:22.197Z] 2024/01/19 07:54:20 DoRetryWithTimeout - Error: {ValidateBackup requires backup [tp-backup-mysql-ibm-pxb-0-83720-01-19-07h51m59s-fohkz] to have a status of Success or PartialSuccess,got -InProgress}, Next try in [30s], timeout [40m0s] [2024-01-19T07:54:54.229Z] 2024/01/19 07:54:50 DoRetryWithTimeout - Error: {ValidateBackup requires backup [tp-backup-mysql-ibm-pxb-0-83720-01-19-07h51m59s-fohkz] to have a status of Success or PartialSuccess,got -InProgress}, Next try in [30s], timeout [40m0s] [2024-01-19T07:55:09.086Z] INFO[2024-01-19 07:55:07] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T07:55:09.086Z] INFO[2024-01-19 07:55:07] sched stats: tick interval: 58867 < 1.2s (100%) [2024-01-19T07:55:09.086Z] INFO[2024-01-19 07:55:07] sched stats: tick process durations: 58866 < 500ms (100%) [2024-01-19T07:55:21.279Z] 2024-01-19 07:55:20 +0000:[INFO] [{CustomResourceBackupAndRestore}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-83720-01-19-07h51m59s] in backup [tp-backup-mysql-ibm-pxb-0-83720-01-19-07h51m59s-fohkz] [2024-01-19T07:55:21.279Z] 2024-01-19 07:55:20 +0000:[INFO] [{CustomResourceBackupAndRestore}] [backup.ValidateBackup:#1865] - kind: [Service] is not in resourceTypes [[PersistentVolumeClaim]], so spec (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-83720-01-19-07h51m59s]) in scheduledAppContext [mysql-ibm-pxb-0-83720-01-19-07h51m59s] will not be checked for in backup [tp-backup-mysql-ibm-pxb-0-83720-01-19-07h51m59s-fohkz] [2024-01-19T07:55:21.279Z] 2024-01-19 07:55:20 +0000:[INFO] [{CustomResourceBackupAndRestore}] [backup.ValidateBackup:#1865] - kind: [Deployment] is not in resourceTypes [[PersistentVolumeClaim]], so spec (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-83720-01-19-07h51m59s]) in scheduledAppContext [mysql-ibm-pxb-0-83720-01-19-07h51m59s] will not be checked for in backup [tp-backup-mysql-ibm-pxb-0-83720-01-19-07h51m59s-fohkz] [2024-01-19T07:55:21.279Z] 2024-01-19 07:55:20 +0000:[INFO] [{CustomResourceBackupAndRestore}] [backup.ValidateBackup:#1865] - kind: [Deployment] is not in resourceTypes [[PersistentVolumeClaim]], so spec (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-83720-01-19-07h51m59s]) in scheduledAppContext [mysql-ibm-pxb-0-83720-01-19-07h51m59s] will not be checked for in backup [tp-backup-mysql-ibm-pxb-0-83720-01-19-07h51m59s-fohkz] [2024-01-19T07:55:21.279Z] 2024-01-19 07:55:20 +0000:[INFO] [{CustomResourceBackupAndRestore}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-83720-01-19-07h51m59s] in backup [tp-backup-mysql-ibm-pxb-0-83720-01-19-07h51m59s-fohkz] [2024-01-19T07:55:21.279Z] 2024-01-19 07:55:20 +0000:[INFO] [{CustomResourceBackupAndRestore}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-83720-01-19-07h51m59s] [2024-01-19T07:55:21.279Z] 2024-01-19 07:55:20 +0000:[WARNING] [{CustomResourceBackupAndRestore}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T07:55:21.279Z] 2024-01-19 07:55:20 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T07:55:21.279Z] 2024-01-19 07:55:20 +0000:[INFO] [{CustomResourceBackupAndRestore}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-83720-01-19-07h51m59s] are [[mysql-data]] [2024-01-19T07:55:21.279Z] 2024-01-19 07:55:20 +0000:[INFO] [{CustomResourceBackupAndRestore}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T07:55:21.279Z] 2024-01-19 07:55:20 +0000:[INFO] [{CustomResourceBackupAndRestore}] Validating CR cleanup [2024-01-19T07:55:21.279Z] 2024-01-19 07:55:20 +0000:[INFO] [{CustomResourceBackupAndRestore}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T07:55:21.279Z] 2024-01-19 07:55:20 +0000:[INFO] [{CustomResourceBackupAndRestore}] All CRs in [mysql-ibm-pxb-0-83720-01-19-07h51m59s] are [[]] [2024-01-19T07:55:21.279Z] INFO[2024-01-19 07:55:20] Verifying : Description : Creation and Validation of backup [tp-backup-mysql-ibm-pxb-0-83720-01-19-07h51m59s-fohkz] with custom resources [2024-01-19T07:55:21.279Z] INFO[2024-01-19 07:55:20] Actual:true, Expected: true [2024-01-19T07:55:21.280Z] 2024-01-19 07:55:20 +0000:[INFO] [{CustomResourceBackupAndRestore}] Restoring the backed up application [2024-01-19T07:55:21.280Z] 2024-01-19 07:55:20 +0000:[INFO] [{CustomResourceBackupAndRestore}] Namespace mapping is map[mysql-ibm-pxb-0-83720-01-19-07h51m59s:lvxhbvwycf-restored]: [2024-01-19T07:55:21.540Z] 2024/01/19 07:55:21 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-mysql-ibm-pxb-0-83720-01-19-07h51m59s-fohkz] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T07:55:53.584Z] 2024-01-19 07:55:51 +0000:[INFO] [{CustomResourceBackupAndRestore}] [backup.CreateRestore:#936] - Restore [tp-restore-tp-backup-mysql-ibm-pxb-0-83720-01-19-07h51m59s-fohkz] created successfully [2024-01-19T07:55:53.584Z] INFO[2024-01-19 07:55:51] Verifying : Description : Creating restore: tp-restore-tp-backup-mysql-ibm-pxb-0-83720-01-19-07h51m59s-fohkz from backup: tp-backup-mysql-ibm-pxb-0-83720-01-19-07h51m59s-fohkz [2024-01-19T07:55:53.584Z] INFO[2024-01-19 07:55:51] Actual:true, Expected: true [2024-01-19T07:55:53.584Z] 2024-01-19 07:55:51 +0000:[INFO] [{CustomResourceBackupAndRestore}] Validating custom resource backup restores [2024-01-19T07:55:53.584Z] 2024-01-19 07:55:51 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T07:55:53.584Z] 2024-01-19 07:55:51 +0000:[INFO] [{CustomResourceBackupAndRestore}] Switching context to [/tmp/source-config] [2024-01-19T07:55:53.584Z] 2024-01-19 07:55:51 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T07:55:53.584Z] 2024-01-19 07:55:51 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T07:55:53.584Z] 2024-01-19 07:55:51 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T07:55:53.584Z] 2024-01-19 07:55:51 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T07:55:53.584Z] 2024-01-19 07:55:51 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T07:55:53.584Z] 2024-01-19 07:55:52 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T07:55:53.584Z] 2024-01-19 07:55:52 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T07:55:53.584Z] 2024-01-19 07:55:52 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T07:55:53.584Z] 2024-01-19 07:55:53 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T07:55:53.585Z] 2024/01/19 07:55:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:56:05.783Z] 2024/01/19 07:56:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:56:13.874Z] 2024/01/19 07:56:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:56:23.833Z] 2024/01/19 07:56:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:56:33.807Z] 2024/01/19 07:56:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:56:43.753Z] 2024/01/19 07:56:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:56:55.918Z] 2024/01/19 07:56:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:57:04.060Z] 2024/01/19 07:57:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:57:14.005Z] 2024/01/19 07:57:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:57:23.972Z] 2024/01/19 07:57:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:57:36.138Z] 2024/01/19 07:57:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:57:44.221Z] 2024/01/19 07:57:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:57:54.170Z] 2024/01/19 07:57:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:58:04.114Z] 2024/01/19 07:58:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:58:14.057Z] 2024/01/19 07:58:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:58:26.229Z] 2024/01/19 07:58:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:58:34.316Z] 2024/01/19 07:58:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:58:44.292Z] 2024/01/19 07:58:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:58:54.252Z] 2024/01/19 07:58:54 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:59:06.418Z] 2024/01/19 07:59:04 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:59:14.528Z] 2024/01/19 07:59:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:59:24.472Z] 2024/01/19 07:59:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:59:34.422Z] 2024/01/19 07:59:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:59:46.595Z] 2024/01/19 07:59:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T07:59:54.721Z] 2024/01/19 07:59:54 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:00:04.691Z] 2024/01/19 08:00:04 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:00:14.643Z] 2024/01/19 08:00:14 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:00:24.617Z] 2024/01/19 08:00:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:00:36.870Z] 2024/01/19 08:00:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:00:44.969Z] 2024/01/19 08:00:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:00:54.924Z] INFO[2024-01-19 08:00:54] Verifying : Description : Validating custom resource restore - [tp-restore-tp-backup-mysql-ibm-pxb-0-83720-01-19-07h51m59s-fohkz] [2024-01-19T08:00:54.924Z] ERRO[2024-01-19 08:00:54] failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.925Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.925Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.925Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.925Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.925Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.925Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.925Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.925Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.925Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.925Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.925Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.925Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.925Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.925Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.925Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.925Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.925Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.925Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.925Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.925Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.925Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.925Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.925Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.926Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.926Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.926Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.926Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.926Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:00:54.926Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T08:00:55.181Z] 2024-01-19 08:00:54 +0000:[INFO] [{CustomResourceBackupAndRestore}] Deleting the deployed apps after the testcase [2024-01-19T08:00:55.181Z] 2024-01-19 08:00:54 +0000:[INFO] [{CustomResourceBackupAndRestore}] Destroying apps [2024-01-19T08:00:55.181Z] 2024-01-19 08:00:54 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T08:00:55.181Z] 2024-01-19 08:00:55 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T08:00:55.181Z] 2024-01-19 08:00:55 +0000:[INFO] [{CustomResourceBackupAndRestore}] start destroying mysql-ibm app [2024-01-19T08:00:55.181Z] 2024-01-19 08:00:55 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T08:00:55.181Z] 2024-01-19 08:00:55 +0000:[INFO] [{CustomResourceBackupAndRestore}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T08:00:55.181Z] 2024-01-19 08:00:55 +0000:[WARNING] [{CustomResourceBackupAndRestore}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T08:00:55.181Z] INFO[2024-01-19 08:00:55] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T08:00:55.181Z] INFO[2024-01-19 08:00:55] Actual:true, Expected: true [2024-01-19T08:00:55.182Z] INFO[2024-01-19 08:00:55] Verifying : Description : Getting the backup driver [2024-01-19T08:00:55.182Z] INFO[2024-01-19 08:00:55] Actual:true, Expected: true [2024-01-19T08:00:55.437Z] INFO[2024-01-19 08:00:55] Verifying : Description : Deleting Restore tp-restore-tp-backup-mysql-ibm-pxb-0-83720-01-19-07h51m59s-fohkz [2024-01-19T08:00:55.437Z] INFO[2024-01-19 08:00:55] Actual:true, Expected: true [2024-01-19T08:00:55.437Z] INFO[2024-01-19 08:00:55] Verifying : Description : Getting backup UID for backup tp-backup-mysql-ibm-pxb-0-83720-01-19-07h51m59s-fohkz [2024-01-19T08:00:55.437Z] INFO[2024-01-19 08:00:55] Actual:true, Expected: true [2024-01-19T08:00:55.437Z] INFO[2024-01-19 08:00:55] Verifying : Description : Deleting backup - tp-backup-mysql-ibm-pxb-0-83720-01-19-07h51m59s-fohkz [2024-01-19T08:00:55.437Z] INFO[2024-01-19 08:00:55] Actual:true, Expected: true [2024-01-19T08:00:55.437Z] 2024-01-19 08:00:55 +0000:[INFO] [{CustomResourceBackupAndRestore}] Cleaning backup locations in map [map[335631db-f562-4c7d-9fca-b4d84eb37c8e:autogenerated-backup-location-1705650827]], cloud credential [cred-aws-1705650821], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T08:00:55.693Z] INFO[2024-01-19 08:00:55] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-1705650827] [2024-01-19T08:00:55.693Z] INFO[2024-01-19 08:00:55] Actual:true, Expected: true [2024-01-19T08:00:55.693Z] 2024-01-19 08:00:55 +0000:[INFO] [{CustomResourceBackupAndRestore}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705650827] is present [2024-01-19T08:00:55.693Z] 2024/01/19 08:00:55 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705650827 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T08:01:27.741Z] 2024-01-19 08:01:25 +0000:[INFO] [{CustomResourceBackupAndRestore}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705650827] is present [2024-01-19T08:01:27.741Z] 2024/01/19 08:01:25 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705650827 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T08:01:59.832Z] 2024-01-19 08:01:55 +0000:[INFO] [{CustomResourceBackupAndRestore}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 aws-global-aws-px-backup-system-test-4cac76dc-bl autogenerated-backup-location-1705628002 autogenerated-backup-location-1705634221 autogenerated-backup-location-1705635982 autogenerated-backup-location-wkyfv] [2024-01-19T08:01:59.832Z] INFO[2024-01-19 08:01:55] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-1705650827 [2024-01-19T08:01:59.832Z] INFO[2024-01-19 08:01:55] Actual:true, Expected: true [2024-01-19T08:01:59.832Z] 2024-01-19 08:01:55 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [cred-aws-1705650821] is present [2024-01-19T08:01:59.832Z] INFO[2024-01-19 08:01:55] Verifying : Description : Verifying if cloud cred [cred-aws-1705650821] is present [2024-01-19T08:01:59.832Z] INFO[2024-01-19 08:01:55] Actual:true, Expected: true [2024-01-19T08:01:59.832Z] INFO[2024-01-19 08:01:55] Verifying : Description : Verifying deletion of cloud cred [cred-aws-1705650821] [2024-01-19T08:01:59.832Z] INFO[2024-01-19 08:01:55] Actual:true, Expected: true [2024-01-19T08:01:59.832Z] INFO[2024-01-19 08:01:56] Verifying : Description : Deleting cloud cred cred-aws-1705650821 [2024-01-19T08:01:59.832Z] INFO[2024-01-19 08:01:56] Actual:true, Expected: true [2024-01-19T08:01:59.832Z] INFO[2024-01-19 08:01:56] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T08:01:59.832Z] INFO[2024-01-19 08:01:56] Actual:true, Expected: true [2024-01-19T08:01:59.832Z] INFO[2024-01-19 08:01:56] Verifying : Description : Deleting cluster source-cluster [2024-01-19T08:01:59.832Z] ERRO[2024-01-19 08:01:56] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl], cannot be deleted, Expected:  [2024-01-19T08:01:59.833Z] INFO[2024-01-19 08:01:56] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-hrfiy] [2024-01-19T08:01:59.833Z] ERRO[2024-01-19 08:01:56] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-hrfiy:72b1b409-7bb1-4f22-92df-0ba5e77c11d3], Expected:  [2024-01-19T08:01:59.833Z] INFO[2024-01-19 08:01:56] Verifying : Description : Deleting cluster source-cluster [2024-01-19T08:01:59.833Z] ERRO[2024-01-19 08:01:56] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh], cannot be deleted, Expected:  [2024-01-19T08:01:59.833Z] INFO[2024-01-19 08:01:57] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-fbvvo] [2024-01-19T08:01:59.833Z] ERRO[2024-01-19 08:01:57] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-fbvvo:7cddb5aa-c7ff-4044-9d27-458be2bd5cde], Expected:  [2024-01-19T08:01:59.833Z] INFO[2024-01-19 08:01:57] Verifying : Description : Deleting cluster source-cluster [2024-01-19T08:01:59.833Z] ERRO[2024-01-19 08:01:57] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo], cannot be deleted, Expected:  [2024-01-19T08:01:59.833Z] INFO[2024-01-19 08:01:58] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-enzij] [2024-01-19T08:01:59.833Z] ERRO[2024-01-19 08:01:58] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-enzij:4e0ddfc7-8811-4ed5-8e87-ed0841582111], Expected:  [2024-01-19T08:01:59.833Z] INFO[2024-01-19 08:01:58] Verifying : Description : Deleting cluster source-cluster [2024-01-19T08:01:59.833Z] ERRO[2024-01-19 08:01:58] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo], cannot be deleted, Expected:  [2024-01-19T08:02:00.093Z] INFO[2024-01-19 08:01:59] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-wcpfm] [2024-01-19T08:02:00.093Z] ERRO[2024-01-19 08:01:59] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-wcpfm:c5ff084a-a913-4412-99fb-e2b0a003a014], Expected:  [2024-01-19T08:02:00.093Z] INFO[2024-01-19 08:01:59] Verifying : Description : Deleting cluster source-cluster [2024-01-19T08:02:00.093Z] ERRO[2024-01-19 08:01:59] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx], cannot be deleted, Expected:  [2024-01-19T08:02:01.457Z] INFO[2024-01-19 08:02:01] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-vlbwu] [2024-01-19T08:02:01.457Z] ERRO[2024-01-19 08:02:01] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-vlbwu:f0465bb5-2565-4948-bb27-7e45a4b883ff], Expected:  [2024-01-19T08:02:01.457Z] INFO[2024-01-19 08:02:01] Verifying : Description : Deleting cluster source-cluster [2024-01-19T08:02:01.457Z] ERRO[2024-01-19 08:02:01] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa], cannot be deleted, Expected:  [2024-01-19T08:02:02.825Z] INFO[2024-01-19 08:02:02] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-mhxzi] [2024-01-19T08:02:02.825Z] ERRO[2024-01-19 08:02:02] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-mhxzi:f551fb81-1a78-4d7f-b804-759c6e6b562e], Expected:  [2024-01-19T08:02:03.081Z] INFO[2024-01-19 08:02:02] Verifying : Description : Deleting cluster source-cluster [2024-01-19T08:02:03.081Z] INFO[2024-01-19 08:02:02] Actual:true, Expected: true [2024-01-19T08:02:04.444Z] INFO[2024-01-19 08:02:04] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-upjpy] [2024-01-19T08:02:04.444Z] INFO[2024-01-19 08:02:04] Actual:true, Expected: true [2024-01-19T08:02:04.444Z] INFO[2024-01-19 08:02:04] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T08:02:04.444Z] INFO[2024-01-19 08:02:04] Actual:true, Expected: true [2024-01-19T08:02:05.813Z] INFO[2024-01-19 08:02:05] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-dkvyw] [2024-01-19T08:02:05.813Z] INFO[2024-01-19 08:02:05] Actual:true, Expected: true [2024-01-19T08:02:05.814Z] INFO[2024-01-19 08:02:05] --------Test End------ [2024-01-19T08:02:05.814Z] INFO[2024-01-19 08:02:05] #Test: [2024-01-19T08:02:05.814Z] INFO[2024-01-19 08:02:05] #Description: [2024-01-19T08:02:05.814Z] INFO[2024-01-19 08:02:05] ------------------------ [2024-01-19T08:02:05.814Z] 2024-01-19 08:02:05 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.EndPxBackupTorpedoTest:#7261] - >>>> FAILED TEST: {CustomResourceBackupAndRestore} Create custom resource backup and restore [2024-01-19T08:02:05.814Z] 2024-01-19 08:02:05 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-83720-01-19-07h51m59s] [2024-01-19T08:02:06.071Z] 2024-01-19 08:02:05 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T08:02:06.071Z] 2024-01-19 08:02:05 +0000:[INFO] [{CustomResourceBackupAndRestore}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T08:02:06.327Z] 2024-01-19 08:02:06 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-83720-01-19-07h51m59s] [2024-01-19T08:02:06.327Z] 2024-01-19 08:02:06 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {CustomResourceBackupAndRestore} Create custom resource backup and restore [2024-01-19T08:02:06.327Z] 2024-01-19 08:02:06 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T08:02:06.327Z] 2024/01/19 08:02:06 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T08:02:11.577Z] 2024/01/19 08:02:11 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T08:02:16.827Z] 2024/01/19 08:02:16 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T08:02:22.073Z] 2024/01/19 08:02:21 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T08:02:26.247Z] 2024/01/19 08:02:26 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T08:02:31.491Z] 2024/01/19 08:02:31 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T08:02:36.737Z] 2024/01/19 08:02:36 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T08:02:41.981Z] 2024/01/19 08:02:41 DoRetryWithTimeout - Error: {dial tcp 10.240.128.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T08:02:47.222Z] 2024/01/19 08:02:46 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.128.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.7 10.240.128.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T08:02:47.222Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T08:02:47.222Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T08:02:47.222Z] {dial tcp 10.240.128.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T08:02:51.387Z] 2024-01-19 08:02:51 +0000:[WARNING] [{CustomResourceBackupAndRestore}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.7 10.240.128.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T08:02:51.387Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T08:02:51.387Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T08:02:51.387Z] {dial tcp 10.240.128.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T08:02:51.387Z] 2024-01-19 08:02:51 +0000:[ERROR] [{CustomResourceBackupAndRestore}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.7 10.240.128.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T08:02:51.387Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T08:02:51.387Z] {dial tcp 10.240.128.7:22: connect: connection refused} [2024-01-19T08:02:51.387Z] {dial tcp 10.240.128.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T08:02:51.387Z] 2024-01-19 08:02:51 +0000:[INFO] [{CustomResourceBackupAndRestore}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T08:02:51.387Z] 2024-01-19 08:02:51 +0000:[INFO] [{CustomResourceBackupAndRestore}] Switching context to [/tmp/source-config] [2024-01-19T08:02:51.387Z] 2024-01-19 08:02:51 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T08:02:51.387Z] 2024-01-19 08:02:51 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T08:02:51.643Z] 2024-01-19 08:02:51 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T08:02:51.643Z] 2024-01-19 08:02:51 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T08:02:51.643Z] 2024-01-19 08:02:51 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T08:02:51.643Z] 2024-01-19 08:02:51 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T08:02:52.204Z] 2024-01-19 08:02:52 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T08:02:52.460Z] 2024-01-19 08:02:52 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T08:02:53.021Z] 2024-01-19 08:02:52 +0000:[INFO] [{CustomResourceBackupAndRestore}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T08:02:53.277Z] 2024/01/19 08:02:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:03:03.280Z] 2024/01/19 08:03:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:03:13.225Z] 2024/01/19 08:03:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:03:23.181Z] 2024/01/19 08:03:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:03:35.357Z] 2024/01/19 08:03:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:03:43.442Z] 2024/01/19 08:03:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:03:53.406Z] 2024/01/19 08:03:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:04:03.374Z] 2024/01/19 08:04:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:04:15.539Z] 2024/01/19 08:04:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:04:23.669Z] 2024/01/19 08:04:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:04:33.614Z] 2024/01/19 08:04:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:04:43.623Z] 2024/01/19 08:04:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:04:53.564Z] 2024/01/19 08:04:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:05:03.619Z] 2024/01/19 08:05:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:05:08.873Z] INFO[2024-01-19 08:05:08] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T08:05:08.873Z] INFO[2024-01-19 08:05:08] sched stats: tick interval: 59468 < 1.2s (100%) [2024-01-19T08:05:08.873Z] INFO[2024-01-19 08:05:08] sched stats: tick process durations: 59467 < 500ms (100%) [2024-01-19T08:05:14.115Z] 2024/01/19 08:05:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:05:24.080Z] 2024/01/19 08:05:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:05:34.030Z] 2024/01/19 08:05:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:05:43.967Z] 2024/01/19 08:05:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:05:53.904Z] 2024/01/19 08:05:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:06:03.854Z] 2024/01/19 08:06:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:06:16.024Z] 2024/01/19 08:06:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:06:24.108Z] 2024/01/19 08:06:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:06:34.065Z] 2024/01/19 08:06:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:06:44.014Z] 2024/01/19 08:06:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:06:53.977Z] 2024/01/19 08:06:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:07:06.158Z] 2024/01/19 08:07:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:07:14.243Z] 2024/01/19 08:07:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:07:24.190Z] 2024/01/19 08:07:24 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:07:34.130Z] 2024/01/19 08:07:34 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:07:46.299Z] 2024/01/19 08:07:44 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:07:54.383Z] 2024-01-19 08:07:54 +0000:[ERROR] [{CustomResourceBackupAndRestore}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.384Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.384Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.384Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.384Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.384Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.384Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.384Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.384Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.384Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.384Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.384Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.384Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.384Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.384Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.384Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.384Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.384Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.384Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.384Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.384Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.384Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.385Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.385Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.385Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.385Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.385Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.385Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.385Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.385Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T08:07:54.385Z] • Failure [954.903 seconds] [2024-01-19T08:07:54.385Z] {CustomResourceBackupAndRestore} [2024-01-19T08:07:54.385Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:158 [2024-01-19T08:07:54.385Z] Create custom resource backup and restore [It] [2024-01-19T08:07:54.385Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:194 [2024-01-19T08:07:54.385Z] [2024-01-19T08:07:54.385Z] Unexpected error: [2024-01-19T08:07:54.385Z] <*errors.errorString | 0xc001890e00>: { [2024-01-19T08:07:54.386Z] s: "error: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknow... [2024-01-19T08:07:54.386Z] [2024-01-19T08:07:54.386Z] Gomega truncated this representation as it exceeds 'format.MaxLength'. [2024-01-19T08:07:54.386Z] Consider having the object provide a custom 'GomegaStringer' representation [2024-01-19T08:07:54.386Z] or adjust the parameters in Gomega's 'format' package. [2024-01-19T08:07:54.386Z] [2024-01-19T08:07:54.386Z] Learn more here: https://onsi.github.io/gomega/#adjusting-output [2024-01-19T08:07:54.386Z] [2024-01-19T08:07:54.386Z] error: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.386Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.386Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.386Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.386Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.386Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.386Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.386Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.386Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:07:54.387Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}], Description: Validating custom resource restore - [tp-restore-tp-backup-mysql-ibm-pxb-0-83720-01-19-07h51m59s-fohkz] [2024-01-19T08:07:54.387Z] occurred [2024-01-19T08:07:54.387Z] [2024-01-19T08:07:54.387Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-19T08:07:54.387Z] [2024-01-19T08:07:54.388Z] Full Stack Trace [2024-01-19T08:07:54.388Z] github.com/portworx/torpedo/pkg/aetosutil.(*Dashboard).VerifyFatal(0x7f5e601?, {0x702af20?, 0xc001890ce0}, {0x0?, 0x0}, {0xc002112f50, 0x67}) [2024-01-19T08:07:54.388Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 +0x143 [2024-01-19T08:07:54.388Z] github.com/portworx/torpedo/tests/backup.glob..func56.2.7() [2024-01-19T08:07:54.388Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:275 +0x174 [2024-01-19T08:07:54.388Z] github.com/portworx/torpedo/tests/backup.glob..func56.2() [2024-01-19T08:07:54.388Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_restore_basic_test.go:270 +0x6c9 [2024-01-19T08:07:54.388Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T08:07:54.388Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T08:07:54.388Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T08:07:54.388Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T08:07:54.388Z] created by testing.(*T).Run [2024-01-19T08:07:54.388Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T08:07:54.388Z] ------------------------------ [2024-01-19T08:07:54.388Z] STEP: schedule applications [2024-01-19T08:07:54.388Z] STEP: Validate applications [2024-01-19T08:07:54.388Z] STEP: validate applications [2024-01-19T08:07:54.388Z] STEP: validate mysql-ibm app's volumes [2024-01-19T08:07:54.388Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T08:07:54.388Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T08:07:54.388Z] STEP: get mysql-ibm app's volume: pvc-10ad1d3b-c030-42e4-b8d3-b845c9743aaa inspected by the volume driver [2024-01-19T08:07:54.388Z] STEP: wait for mysql-ibm app to start running [2024-01-19T08:07:54.388Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T08:07:54.388Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T08:07:54.388Z] STEP: Validate Px pod restart count [2024-01-19T08:07:54.388Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T08:07:54.388Z] STEP: Creating cloud credentials [2024-01-19T08:07:54.388Z] STEP: Register cluster for backup [2024-01-19T08:07:54.388Z] STEP: Creating backup location [2024-01-19T08:07:54.388Z] STEP: Taking backup of applications [2024-01-19T08:07:54.388Z] STEP: Restoring the backed up application [2024-01-19T08:07:54.388Z] STEP: Validating custom resource backup restores [2024-01-19T08:07:54.388Z] STEP: destroy apps [2024-01-19T08:07:54.388Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T08:07:54.388Z] STEP: start destroying mysql-ibm app [2024-01-19T08:07:54.388Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T08:07:54.388Z] 2024-01-19 08:07:54 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Log Dir: /testresults//DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin.log [2024-01-19T08:07:54.388Z] INFO[2024-01-19 08:07:54] --------Test Start------ [2024-01-19T08:07:54.388Z] INFO[2024-01-19 08:07:54] #Test: DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin  [2024-01-19T08:07:54.388Z] INFO[2024-01-19 08:07:54] #Description: Delete failed and in-progress backups and restores of user from the admin side  [2024-01-19T08:07:54.388Z] INFO[2024-01-19 08:07:54] ------------------------ [2024-01-19T08:07:54.388Z] 2024-01-19 08:07:54 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Scheduling applications [2024-01-19T08:07:54.388Z] 2024-01-19 08:07:54 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T08:07:54.388Z] 2024-01-19 08:07:54 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T08:07:54.388Z] 2024-01-19 08:07:54 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T08:07:54.388Z] 2024-01-19 08:07:54 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T08:07:54.388Z] 2024-01-19 08:07:54 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T08:07:54.389Z] 2024-01-19 08:07:54 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T08:07:54.389Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T08:07:54.646Z] 2024-01-19 08:07:54 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T08:07:54.646Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T08:07:54.646Z] 2024-01-19 08:07:54 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T08:07:54.646Z] 2024-01-19 08:07:54 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Scheduled application namespaces: [mysql-ibm-pxb-0-87564-01-19-08h07m54s] [2024-01-19T08:07:54.646Z] 2024-01-19 08:07:54 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Validating applications [2024-01-19T08:07:54.646Z] 2024-01-19 08:07:54 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Validate applications [2024-01-19T08:07:54.646Z] 2024-01-19 08:07:54 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Validating mysql-ibm app [2024-01-19T08:07:54.646Z] 2024-01-19 08:07:54 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Validating mysql-ibm app's volumes [2024-01-19T08:07:54.646Z] 2024-01-19 08:07:54 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T08:07:54.646Z] 2024/01/19 08:07:54 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T08:08:04.597Z] 2024/01/19 08:08:04 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T08:08:16.773Z] 2024-01-19 08:08:14 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-87564-01-19-08h07m54s [2024-01-19T08:08:16.773Z] 2024-01-19 08:08:14 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T08:08:16.773Z] 2024-01-19 08:08:14 +0000:[WARNING] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T08:08:16.773Z] 2024-01-19 08:08:14 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] wait for mysql-ibm app to start running [2024-01-19T08:08:16.773Z] 2024-01-19 08:08:14 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T08:08:16.773Z] 2024/01/19 08:08:14 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T08:08:16.773Z] pod name:mysql-7d766cd65c-4fnzl namespace:mysql-ibm-pxb-0-87564-01-19-08h07m54s running:false ready:false node:10.240.64.8 [2024-01-19T08:08:16.773Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T08:08:26.731Z] 2024/01/19 08:08:24 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T08:08:26.731Z] pod name:mysql-7d766cd65c-4fnzl namespace:mysql-ibm-pxb-0-87564-01-19-08h07m54s running:true ready:false node:10.240.64.8 [2024-01-19T08:08:26.731Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T08:08:36.673Z] 2024/01/19 08:08:35 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T08:08:36.673Z] pod name:mysql-7d766cd65c-4fnzl namespace:mysql-ibm-pxb-0-87564-01-19-08h07m54s running:true ready:false node:10.240.64.8 [2024-01-19T08:08:36.673Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T08:08:46.621Z] 2024/01/19 08:08:45 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T08:08:46.621Z] pod name:mysql-7d766cd65c-4fnzl namespace:mysql-ibm-pxb-0-87564-01-19-08h07m54s running:true ready:false node:10.240.64.8 [2024-01-19T08:08:46.621Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T08:08:56.563Z] 2024/01/19 08:08:55 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T08:08:56.563Z] pod name:mysql-7d766cd65c-4fnzl namespace:mysql-ibm-pxb-0-87564-01-19-08h07m54s running:true ready:false node:10.240.64.8 [2024-01-19T08:08:56.563Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T08:09:06.526Z] 2024/01/19 08:09:05 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T08:09:06.526Z] pod name:mysql-7d766cd65c-4fnzl namespace:mysql-ibm-pxb-0-87564-01-19-08h07m54s running:true ready:false node:10.240.64.8 [2024-01-19T08:09:06.526Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T08:09:16.464Z] 2024/01/19 08:09:15 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T08:09:16.464Z] pod name:mysql-7d766cd65c-4fnzl namespace:mysql-ibm-pxb-0-87564-01-19-08h07m54s running:true ready:false node:10.240.64.8 [2024-01-19T08:09:16.464Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T08:09:26.418Z] 2024/01/19 08:09:25 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T08:09:26.418Z] pod name:mysql-7d766cd65c-4fnzl namespace:mysql-ibm-pxb-0-87564-01-19-08h07m54s running:true ready:false node:10.240.64.8 [2024-01-19T08:09:26.418Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T08:09:36.363Z] 2024-01-19 08:09:35 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T08:09:36.363Z] 2024-01-19 08:09:35 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T08:09:36.363Z] 2024-01-19 08:09:35 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] validate if mysql-ibm app's volumes are setup [2024-01-19T08:09:36.363Z] 2024-01-19 08:09:35 +0000:[WARNING] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T08:09:36.363Z] 2024-01-19 08:09:35 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T08:09:36.364Z] 2024-01-19 08:09:35 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T08:09:36.364Z] 2024-01-19 08:09:35 +0000:[WARNING] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T08:09:36.364Z] 2024-01-19 08:09:36 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Creating 1 users [2024-01-19T08:09:36.364Z] 2024-01-19 08:09:36 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.AddUser:#728] - Response for user [testuser1-1705651776] creation - [2024-01-19T08:09:36.364Z] 2024-01-19 08:09:36 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.AddUser:#733] - User [testuser1-1705651776] created [2024-01-19T08:09:36.364Z] INFO[2024-01-19 08:09:36] Verifying : Description : Creating user - testuser1-1705651776 [2024-01-19T08:09:36.364Z] INFO[2024-01-19 08:09:36] Actual:true, Expected: true [2024-01-19T08:09:36.621Z] 2024-01-19 08:09:36 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705651776 - 851f3c49-8151-41c2-9478-ca825d4ffc81 [2024-01-19T08:09:37.985Z] 2024-01-19 08:09:37 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.glob..func29.2.2:#1120] - username testuser1-1705651776 common password K1osV#93fgcmdm [2024-01-19T08:09:37.985Z] 2024-01-19 08:09:37 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Creating cloud credential and backup location from the user [2024-01-19T08:09:38.243Z] 2024-01-19 08:09:38 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-testuser1-1705651776-jzudn] for org [default] with [aws] as provider [2024-01-19T08:09:38.243Z] 2024-01-19 08:09:38 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T08:09:38.499Z] 2024-01-19 08:09:38 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-testuser1-1705651776-jzudn] is present [2024-01-19T08:09:40.386Z] 2024-01-19 08:09:40 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Creating source and destination cluster from the user testuser1-1705651776 [2024-01-19T08:09:40.641Z] INFO[2024-01-19 08:09:40] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T08:09:40.641Z] INFO[2024-01-19 08:09:40] Actual:true, Expected: true [2024-01-19T08:09:40.641Z] 2024-01-19 08:09:40 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T08:09:40.641Z] 2024-01-19 08:09:40 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T08:09:40.641Z] 2024-01-19 08:09:40 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T08:09:40.641Z] 2024-01-19 08:09:40 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T08:09:40.641Z] 2024-01-19 08:09:40 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Create cluster [source-cluster] in org [default] [2024-01-19T08:09:40.641Z] 2024-01-19 08:09:40 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T08:09:40.641Z] 2024-01-19 08:09:40 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T08:09:40.641Z] 2024-01-19 08:09:40 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-19T08:09:40.641Z] 2024-01-19 08:09:40 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T08:09:40.641Z] 2024-01-19 08:09:40 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T08:09:40.642Z] 2024-01-19 08:09:40 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-pygnl] for IBM [2024-01-19T08:09:40.642Z] 2024-01-19 08:09:40 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-pygnl] for org [default] with [ibm] as provider [2024-01-19T08:09:40.642Z] 2024-01-19 08:09:40 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T08:09:40.642Z] 2024-01-19 08:09:40 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T08:09:40.642Z] 2024-01-19 08:09:40 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-pygnl] is present [2024-01-19T08:09:40.642Z] 2024-01-19 08:09:40 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Create cluster [source-cluster] in org [default] [2024-01-19T08:09:44.806Z] 2024-01-19 08:09:44 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-hiacj] for IBM [2024-01-19T08:09:44.806Z] 2024-01-19 08:09:44 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-hiacj] for org [default] with [ibm] as provider [2024-01-19T08:09:44.806Z] 2024-01-19 08:09:44 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T08:09:44.806Z] 2024-01-19 08:09:44 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T08:09:44.806Z] 2024-01-19 08:09:44 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-hiacj] is present [2024-01-19T08:09:44.806Z] 2024-01-19 08:09:44 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Create cluster [destination-cluster] in org [default] [2024-01-19T08:09:45.061Z] INFO[2024-01-19 08:09:44] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T08:09:45.061Z] INFO[2024-01-19 08:09:44] Actual:Online, Expected: Online [2024-01-19T08:09:45.062Z] INFO[2024-01-19 08:09:44] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T08:09:45.062Z] INFO[2024-01-19 08:09:44] Actual:true, Expected: true [2024-01-19T08:09:45.062Z] 2024-01-19 08:09:44 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Taking in progress backup of applications from the user testuser1-1705651776 [2024-01-19T08:09:45.062Z] 2024-01-19 08:09:44 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T08:09:45.062Z] 2024-01-19 08:09:45 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Volume snapshot classes found in the cluster - [2024-01-19T08:09:45.062Z] 2024-01-19 08:09:45 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] ibmc-vpcblock-snapshot [2024-01-19T08:09:45.062Z] 2024-01-19 08:09:45 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T08:09:45.317Z] INFO[2024-01-19 08:09:45] Verifying : Description : Verifying creation of backup tp-backup-mysql-ibm-pxb-0-87564-01-19-08h07m54s-0-testuser1-1705651776-zqwyh of namespace mysql-ibm-pxb-0-87564-01-19-08h07m54s [2024-01-19T08:09:45.317Z] INFO[2024-01-19 08:09:45] Actual:true, Expected: true [2024-01-19T08:09:45.317Z] 2024-01-19 08:09:45 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.glob..func29.2.5:#1176] - The list of in progress user backups taken are: [tp-backup-mysql-ibm-pxb-0-87564-01-19-08h07m54s-0-testuser1-1705651776-zqwyh] [2024-01-19T08:09:45.317Z] 2024-01-19 08:09:45 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Verifying in progress backups of the user testuser1-1705651776 from the admin [2024-01-19T08:09:45.880Z] 2024-01-19 08:09:45 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Deleting in progress backups taken by user testuser1-1705651776 from the admin [2024-01-19T08:09:47.246Z] INFO[2024-01-19 08:09:47] Verifying : Description : Attempting to delete in progress backups [2024-01-19T08:09:47.246Z] INFO[2024-01-19 08:09:47] Actual:true, Expected: true [2024-01-19T08:09:47.246Z] INFO[2024-01-19 08:09:47] Verifying : Description : Verifying if there is at least one in progress backup [2024-01-19T08:09:47.246Z] INFO[2024-01-19 08:09:47] Actual:true, Expected: true [2024-01-19T08:09:47.246Z] 2024-01-19 08:09:47 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Taking failed backup of applications from the user testuser1-1705651776 [2024-01-19T08:09:47.502Z] 2024-01-19 08:09:47 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T08:09:47.502Z] 2024-01-19 08:09:47 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Volume snapshot classes found in the cluster - [2024-01-19T08:09:47.502Z] 2024-01-19 08:09:47 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] ibmc-vpcblock-snapshot [2024-01-19T08:09:47.502Z] 2024-01-19 08:09:47 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T08:09:47.757Z] 2024-01-19 08:09:47 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.glob..func29.2.8:#1298] - The list of failed user backups taken are: [tp-backup-mysql-ibm-pxb-0-87564-01-19-08h07m54s-0-testuser1-1705651776-iqrrt] [2024-01-19T08:09:47.757Z] 2024-01-19 08:09:47 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Verifying failed backups of the user testuser1-1705651776 from the admin [2024-01-19T08:09:49.149Z] 2024-01-19 08:09:49 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Deleting failed backups taken by user testuser1-1705651776 from the admin [2024-01-19T08:09:51.035Z] 2024-01-19 08:09:50 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Taking successful backup of applications from the user testuser1-1705651776 [2024-01-19T08:09:51.035Z] 2024-01-19 08:09:50 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.AdditionalBackupRequestParams:#3981] - Detected backup type - native_csi [2024-01-19T08:09:51.036Z] 2024-01-19 08:09:50 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Volume snapshot classes found in the cluster - [2024-01-19T08:09:51.036Z] 2024-01-19 08:09:50 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] ibmc-vpcblock-snapshot [2024-01-19T08:09:51.036Z] 2024-01-19 08:09:50 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T08:09:51.604Z] 2024/01/19 08:09:51 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87564-01-19-08h07m54s-0-testuser1-1705651776-qjgax] expected was [[Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T08:10:23.618Z] 2024/01/19 08:10:21 DoRetryWithTimeout - Error: {backup status for [tp-backup-mysql-ibm-pxb-0-87564-01-19-08h07m54s-0-testuser1-1705651776-qjgax] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T08:10:55.678Z] 2024-01-19 08:10:51 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.CreateBackup:#313] - Backup [tp-backup-mysql-ibm-pxb-0-87564-01-19-08h07m54s-0-testuser1-1705651776-qjgax] created successfully [2024-01-19T08:10:55.678Z] 2024-01-19 08:10:51 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Validating backup [tp-backup-mysql-ibm-pxb-0-87564-01-19-08h07m54s-0-testuser1-1705651776-qjgax] in org [default] [2024-01-19T08:10:55.678Z] 2024-01-19 08:10:51 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-mysql-ibm-pxb-0-87564-01-19-08h07m54s-0-testuser1-1705651776-qjgax] [2024-01-19T08:10:55.678Z] 2024-01-19 08:10:51 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87564-01-19-08h07m54s] in backup [tp-backup-mysql-ibm-pxb-0-87564-01-19-08h07m54s-0-testuser1-1705651776-qjgax] [2024-01-19T08:10:55.678Z] 2024-01-19 08:10:51 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87564-01-19-08h07m54s] in backup [tp-backup-mysql-ibm-pxb-0-87564-01-19-08h07m54s-0-testuser1-1705651776-qjgax] [2024-01-19T08:10:55.678Z] 2024-01-19 08:10:51 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-87564-01-19-08h07m54s] [2024-01-19T08:10:55.678Z] 2024-01-19 08:10:51 +0000:[WARNING] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T08:10:55.678Z] 2024-01-19 08:10:51 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T08:10:55.678Z] 2024-01-19 08:10:51 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-87564-01-19-08h07m54s] are [[mysql-data]] [2024-01-19T08:10:55.679Z] 2024-01-19 08:10:51 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T08:10:55.679Z] 2024-01-19 08:10:51 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Validating CR cleanup [2024-01-19T08:10:55.679Z] 2024-01-19 08:10:51 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T08:10:55.679Z] 2024-01-19 08:10:51 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] All CRs in [mysql-ibm-pxb-0-87564-01-19-08h07m54s] are [[]] [2024-01-19T08:10:55.679Z] INFO[2024-01-19 08:10:51] Verifying : Description : Verifying creation and validation of backup [tp-backup-mysql-ibm-pxb-0-87564-01-19-08h07m54s-0-testuser1-1705651776-qjgax] of namespace (scheduled Context) [mysql-ibm-pxb-0-87564-01-19-08h07m54s] [2024-01-19T08:10:55.679Z] INFO[2024-01-19 08:10:51] Actual:true, Expected: true [2024-01-19T08:10:55.679Z] 2024-01-19 08:10:51 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.glob..func29.2.11:#1355] - The list of successful user backups taken are: map[tp-backup-mysql-ibm-pxb-0-87564-01-19-08h07m54s-0-testuser1-1705651776-qjgax:mysql-ibm-pxb-0-87564-01-19-08h07m54s] [2024-01-19T08:10:55.679Z] 2024-01-19 08:10:51 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Verifying successful backups of the user testuser1-1705651776 from the admin [2024-01-19T08:10:55.679Z] 2024-01-19 08:10:52 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Taking in progress restore of backups from the user testuser1-1705651776 [2024-01-19T08:10:55.679Z] 2024-01-19 08:10:52 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.CreateRestoreWithoutCheck:#1089] - Getting the UID of the backup needed to be restored [2024-01-19T08:10:55.679Z] 2024-01-19 08:10:52 +0000:[DEBUG] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.CreateRestoreWithoutCheck:#1093] - Enumerate backup response - [2024-01-19T08:10:55.682Z] backups: last_update_time: create_time_in_sec:1705651791 ownership: public:<> > > backup_info: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"fecbc04a-2cc5-40ba-bdf0-51efe24d003b" > backup_path:"mysql-ibm-pxb-0-87564-01-19-08h07m54s/tp-backup-mysql-ibm-pxb-0-87564-01-19-08h07m54s-0-testuser1-1705651776-qjgax-a2836b9/12a9d7e6-dc11-4120-9375-c367bae3e6c2" stage:Final cr_name:"tp-backup-mysql-ibm-pxb-0-87564-01-19-08h07m54s-0-testuser1-1705651776-qjgax-a2836b9" total_size:10737418240 resource_count:5 csi_snapshot_class_name:"ibmc-vpcblock-snapshot" backup_location_ref: pre_exec_rule_ref:<> post_exec_rule_ref:<> cloud_credential_ref: backup_type: cr_uid:"12a9d7e6-dc11-4120-9375-c367bae3e6c2" user_backupshare_access:FullAccess cluster_ref: target_namespace:"mysql-ibm-pxb-0-87564-01-19-08h07m54s" > > total_count:1 [2024-01-19T08:10:55.683Z] INFO[2024-01-19 08:10:52] Verifying : Description : Verifying creation of restore tp-restore-tp-backup-mysql-ibm-pxb-0-87564-01-19-08h07m54s-0-testuser1-1705651776-qjgax of backup tp-backup-mysql-ibm-pxb-0-87564-01-19-08h07m54s-0-testuser1-1705651776-qjgax [2024-01-19T08:10:55.683Z] INFO[2024-01-19 08:10:52] Actual:true, Expected: true [2024-01-19T08:10:55.683Z] 2024-01-19 08:10:52 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.glob..func29.2.13:#1396] - The list of in progress user restores taken are: [tp-restore-tp-backup-mysql-ibm-pxb-0-87564-01-19-08h07m54s-0-testuser1-1705651776-qjgax] [2024-01-19T08:10:55.683Z] 2024-01-19 08:10:52 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Verifying in progress restores of the user testuser1-1705651776 from the admin [2024-01-19T08:10:55.683Z] 2024-01-19 08:10:54 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Deleting in progress restores taken by user testuser1-1705651776 from the admin [2024-01-19T08:10:55.683Z] 2024-01-19 08:10:55 +0000:[WARNING] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.glob..func29.2.15.1:#1456] - restore testuser1-1705637494-test-restore-single-ns-pbdt with uid default may not have been created by this testcase [2024-01-19T08:10:55.683Z] 2024-01-19 08:10:55 +0000:[WARNING] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.glob..func29.2.15.1:#1456] - restore tp-restore-tp-backup-manual-single-ns-with-rules-bfdr with uid default may not have been created by this testcase [2024-01-19T08:10:55.683Z] 2024-01-19 08:10:55 +0000:[WARNING] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.glob..func29.2.15.1:#1456] - restore tp-restore-1705636305 with uid default may not have been created by this testcase [2024-01-19T08:10:55.683Z] 2024-01-19 08:10:55 +0000:[WARNING] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.glob..func29.2.15.1:#1456] - restore tp-restore-1705634322 with uid default may not have been created by this testcase [2024-01-19T08:10:55.683Z] 2024-01-19 08:10:55 +0000:[WARNING] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.glob..func29.2.15.1:#1456] - restore tp-restore-backup-1705633224 with uid default may not have been created by this testcase [2024-01-19T08:10:55.683Z] 2024-01-19 08:10:55 +0000:[WARNING] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.glob..func29.2.15.1:#1456] - restore tp-restore-1705628372 with uid default may not have been created by this testcase [2024-01-19T08:10:55.683Z] 2024-01-19 08:10:55 +0000:[WARNING] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.glob..func29.2.15.1:#1456] - restore tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855 with uid default may not have been created by this testcase [2024-01-19T08:10:55.967Z] INFO[2024-01-19 08:10:55] Verifying : Description : Attempting to delete in progress restores [2024-01-19T08:10:55.967Z] INFO[2024-01-19 08:10:55] Actual:true, Expected: true [2024-01-19T08:10:55.967Z] INFO[2024-01-19 08:10:55] Verifying : Description : Verifying if there is at least one in progress restores [2024-01-19T08:10:55.967Z] INFO[2024-01-19 08:10:55] Actual:true, Expected: true [2024-01-19T08:10:55.967Z] 2024-01-19 08:10:55 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Taking failed restore of backups from the user testuser1-1705651776 [2024-01-19T08:10:55.967Z] 2024-01-19 08:10:55 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.CreateRestoreWithUID:#1054] - Getting the UID of the backup needed to be restored [2024-01-19T08:10:56.231Z] 2024-01-19 08:10:56 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.glob..func29.2.16:#1511] - The list of failed user restores taken are: [tp-restore-tp-backup-mysql-ibm-pxb-0-87564-01-19-08h07m54s-0-testuser1-1705651776-qjgax] [2024-01-19T08:10:56.231Z] 2024-01-19 08:10:56 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Verifying failed restores of the user testuser1-1705651776 from the admin [2024-01-19T08:10:58.142Z] 2024-01-19 08:10:57 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Deleting failed restores taken by user testuser1-1705651776 from the admin [2024-01-19T08:10:59.507Z] 2024-01-19 08:10:59 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Deleting the deployed apps after the testcase [2024-01-19T08:10:59.507Z] 2024-01-19 08:10:59 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Destroying apps [2024-01-19T08:10:59.507Z] 2024-01-19 08:10:59 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T08:10:59.507Z] 2024-01-19 08:10:59 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T08:10:59.507Z] 2024-01-19 08:10:59 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] start destroying mysql-ibm app [2024-01-19T08:10:59.762Z] 2024-01-19 08:10:59 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T08:10:59.762Z] 2024-01-19 08:10:59 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T08:10:59.762Z] 2024-01-19 08:10:59 +0000:[WARNING] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T08:10:59.762Z] INFO[2024-01-19 08:10:59] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T08:10:59.762Z] INFO[2024-01-19 08:10:59] Actual:true, Expected: true [2024-01-19T08:11:00.325Z] 2024-01-19 08:11:00 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Cleaning backup locations in map [map[e0dac898-8f7b-447e-9afc-cdadf3b0276c:autogenerated-backup-location-testuser1-1705651776-aetfb]], cloud credential [autogenerated-cred-testuser1-1705651776-jzudn], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T08:11:00.325Z] INFO[2024-01-19 08:11:00] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-testuser1-1705651776-aetfb] [2024-01-19T08:11:00.325Z] INFO[2024-01-19 08:11:00] Actual:true, Expected: true [2024-01-19T08:11:00.325Z] 2024-01-19 08:11:00 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-testuser1-1705651776-aetfb] is present [2024-01-19T08:11:00.325Z] 2024/01/19 08:11:00 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-testuser1-1705651776-aetfb is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T08:11:33.898Z] 2024-01-19 08:11:30 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-testuser1-1705651776-aetfb] is present [2024-01-19T08:11:33.898Z] 2024/01/19 08:11:30 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-testuser1-1705651776-aetfb is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T08:12:00.387Z] 2024-01-19 08:12:00 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-testuser1-1705651776-aetfb] is present [2024-01-19T08:12:00.644Z] 2024/01/19 08:12:00 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-testuser1-1705651776-aetfb is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T08:12:32.667Z] 2024-01-19 08:12:30 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-testuser1-1705651776-aetfb] is present [2024-01-19T08:12:32.667Z] 2024/01/19 08:12:30 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-testuser1-1705651776-aetfb is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T08:13:04.704Z] 2024-01-19 08:13:00 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [] [2024-01-19T08:13:04.704Z] INFO[2024-01-19 08:13:00] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-testuser1-1705651776-aetfb [2024-01-19T08:13:04.704Z] INFO[2024-01-19 08:13:00] Actual:true, Expected: true [2024-01-19T08:13:04.704Z] 2024-01-19 08:13:00 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-testuser1-1705651776-jzudn] is present [2024-01-19T08:13:04.704Z] INFO[2024-01-19 08:13:00] Verifying : Description : Verifying if cloud cred [autogenerated-cred-testuser1-1705651776-jzudn] is present [2024-01-19T08:13:04.704Z] INFO[2024-01-19 08:13:00] Actual:true, Expected: true [2024-01-19T08:13:04.704Z] INFO[2024-01-19 08:13:00] Verifying : Description : Verifying deletion of cloud cred [autogenerated-cred-testuser1-1705651776-jzudn] [2024-01-19T08:13:04.704Z] INFO[2024-01-19 08:13:00] Actual:true, Expected: true [2024-01-19T08:13:04.704Z] INFO[2024-01-19 08:13:00] Verifying : Description : Deleting cloud cred autogenerated-cred-testuser1-1705651776-jzudn [2024-01-19T08:13:04.704Z] INFO[2024-01-19 08:13:00] Actual:true, Expected: true [2024-01-19T08:13:04.704Z] INFO[2024-01-19 08:13:00] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T08:13:04.704Z] INFO[2024-01-19 08:13:00] Actual:true, Expected: true [2024-01-19T08:13:04.704Z] INFO[2024-01-19 08:13:01] Verifying : Description : Deleting cluster source-cluster [2024-01-19T08:13:04.704Z] INFO[2024-01-19 08:13:01] Actual:true, Expected: true [2024-01-19T08:13:04.704Z] INFO[2024-01-19 08:13:01] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-pygnl] [2024-01-19T08:13:04.704Z] INFO[2024-01-19 08:13:01] Actual:true, Expected: true [2024-01-19T08:13:04.704Z] INFO[2024-01-19 08:13:01] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T08:13:04.704Z] INFO[2024-01-19 08:13:01] Actual:true, Expected: true [2024-01-19T08:13:04.704Z] INFO[2024-01-19 08:13:02] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-hiacj] [2024-01-19T08:13:04.704Z] INFO[2024-01-19 08:13:02] Actual:true, Expected: true [2024-01-19T08:13:04.704Z] 2024-01-19 08:13:03 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705651776 - 851f3c49-8151-41c2-9478-ca825d4ffc81 [2024-01-19T08:13:04.704Z] 2024-01-19 08:13:03 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.DeleteUser:#755] - Response for user [testuser1-1705651776] deletion - [2024-01-19T08:13:04.704Z] 2024-01-19 08:13:03 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [backup.DeleteUser:#759] - Deleted User - testuser1-1705651776 [2024-01-19T08:13:04.704Z] INFO[2024-01-19 08:13:03] --------Test End------ [2024-01-19T08:13:04.704Z] INFO[2024-01-19 08:13:03] #Test: [2024-01-19T08:13:04.705Z] INFO[2024-01-19 08:13:03] #Description: [2024-01-19T08:13:04.705Z] INFO[2024-01-19 08:13:03] ------------------------ [2024-01-19T08:13:04.705Z] 2024-01-19 08:13:03 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-87564-01-19-08h07m54s] [2024-01-19T08:13:04.705Z] 2024-01-19 08:13:03 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T08:13:04.705Z] 2024-01-19 08:13:03 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T08:13:04.705Z] 2024-01-19 08:13:04 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-87564-01-19-08h07m54s] [2024-01-19T08:13:04.705Z] 2024-01-19 08:13:04 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin} Delete failed and in-progress backups and restores of user from the admin [2024-01-19T08:13:04.705Z] 2024-01-19 08:13:04 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T08:13:04.705Z] 2024/01/19 08:13:04 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T08:13:09.948Z] 2024/01/19 08:13:09 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T08:13:15.238Z] 2024/01/19 08:13:14 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T08:13:19.412Z] 2024/01/19 08:13:19 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T08:13:24.657Z] 2024/01/19 08:13:24 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T08:13:29.905Z] 2024/01/19 08:13:29 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T08:13:35.149Z] 2024/01/19 08:13:34 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T08:13:39.315Z] 2024/01/19 08:13:39 DoRetryWithTimeout - Error: {dial tcp 10.240.128.20:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T08:13:44.561Z] 2024/01/19 08:13:44 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.128.20. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.20 10.240.128.20]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T08:13:44.561Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T08:13:44.561Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T08:13:44.561Z] {dial tcp 10.240.128.20:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T08:13:49.806Z] 2024-01-19 08:13:49 +0000:[WARNING] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.20. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.20 10.240.128.20]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T08:13:49.806Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T08:13:49.806Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T08:13:49.806Z] {dial tcp 10.240.128.20:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T08:13:49.806Z] 2024-01-19 08:13:49 +0000:[ERROR] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.20. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.20 10.240.128.20]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T08:13:49.806Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T08:13:49.806Z] {dial tcp 10.240.128.20:22: connect: connection refused} [2024-01-19T08:13:49.806Z] {dial tcp 10.240.128.20:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T08:13:49.806Z] 2024-01-19 08:13:49 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T08:13:49.806Z] 2024-01-19 08:13:49 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] Switching context to [/tmp/source-config] [2024-01-19T08:13:49.806Z] 2024-01-19 08:13:49 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T08:13:49.806Z] 2024-01-19 08:13:49 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T08:13:49.806Z] 2024-01-19 08:13:49 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T08:13:49.806Z] 2024-01-19 08:13:49 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T08:13:49.807Z] 2024-01-19 08:13:49 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T08:13:49.807Z] 2024-01-19 08:13:49 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T08:13:50.367Z] 2024-01-19 08:13:50 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T08:13:50.623Z] 2024-01-19 08:13:50 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T08:13:51.217Z] 2024-01-19 08:13:50 +0000:[INFO] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T08:13:51.217Z] 2024/01/19 08:13:51 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:14:01.172Z] 2024/01/19 08:14:01 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:14:13.349Z] 2024/01/19 08:14:11 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:14:21.434Z] 2024/01/19 08:14:21 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:14:31.372Z] 2024/01/19 08:14:31 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:14:41.315Z] 2024/01/19 08:14:41 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:14:51.316Z] 2024/01/19 08:14:51 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:15:03.486Z] 2024/01/19 08:15:01 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:15:10.025Z] INFO[2024-01-19 08:15:09] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T08:15:10.025Z] INFO[2024-01-19 08:15:09] sched stats: tick interval: 60069 < 1.2s (100%) [2024-01-19T08:15:10.025Z] INFO[2024-01-19 08:15:09] sched stats: tick process durations: 60068 < 500ms (100%) [2024-01-19T08:15:11.388Z] 2024/01/19 08:15:11 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:15:21.329Z] 2024/01/19 08:15:21 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:15:33.514Z] 2024/01/19 08:15:31 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:15:41.628Z] 2024/01/19 08:15:41 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:15:51.563Z] 2024/01/19 08:15:51 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:16:01.510Z] 2024/01/19 08:16:01 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:16:11.445Z] 2024/01/19 08:16:11 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:16:23.610Z] 2024/01/19 08:16:21 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:16:31.697Z] 2024/01/19 08:16:31 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:16:41.646Z] 2024/01/19 08:16:41 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:16:51.588Z] 2024/01/19 08:16:51 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:17:03.768Z] 2024/01/19 08:17:01 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:17:11.864Z] 2024/01/19 08:17:11 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:17:21.811Z] 2024/01/19 08:17:21 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:17:31.753Z] 2024/01/19 08:17:31 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:17:41.704Z] 2024/01/19 08:17:41 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:17:53.874Z] 2024/01/19 08:17:51 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:18:01.968Z] 2024/01/19 08:18:01 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:18:11.927Z] 2024/01/19 08:18:11 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:18:21.877Z] 2024/01/19 08:18:21 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:18:31.832Z] 2024/01/19 08:18:31 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:18:44.005Z] 2024/01/19 08:18:41 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T08:18:52.105Z] 2024-01-19 08:18:51 +0000:[ERROR] [{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.105Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.105Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.105Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.105Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.105Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.105Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.105Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.105Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.105Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.105Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.105Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.106Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.106Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.106Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.106Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.106Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.106Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.106Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.106Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.106Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.106Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.106Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.106Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.106Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.106Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.106Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.106Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.106Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.106Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T08:18:52.106Z] • Failure [657.680 seconds] [2024-01-19T08:18:52.106Z] {DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin} [2024-01-19T08:18:52.106Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_orphaned_test.go:1071 [2024-01-19T08:18:52.106Z] Delete failed and in-progress backups and restores of user from the admin [] [2024-01-19T08:18:52.106Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_orphaned_test.go:1111 [2024-01-19T08:18:52.106Z] [2024-01-19T08:18:52.106Z] Unexpected error: [2024-01-19T08:18:52.106Z] <*errors.errorString | 0xc00100d270>: { [2024-01-19T08:18:52.108Z] s: "failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{No... [2024-01-19T08:18:52.108Z] [2024-01-19T08:18:52.108Z] Gomega truncated this representation as it exceeds 'format.MaxLength'. [2024-01-19T08:18:52.108Z] Consider having the object provide a custom 'GomegaStringer' representation [2024-01-19T08:18:52.108Z] or adjust the parameters in Gomega's 'format' package. [2024-01-19T08:18:52.108Z] [2024-01-19T08:18:52.108Z] Learn more here: https://onsi.github.io/gomega/#adjusting-output [2024-01-19T08:18:52.108Z] [2024-01-19T08:18:52.108Z] failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.108Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.108Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.108Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.108Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.108Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.108Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.108Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.108Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.108Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.108Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.108Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.108Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.108Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.108Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.109Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.109Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.109Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.109Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.109Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.109Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.109Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.109Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.109Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.109Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.109Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.109Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.109Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.109Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T08:18:52.109Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T08:18:52.109Z] occurred [2024-01-19T08:18:52.109Z] [2024-01-19T08:18:52.109Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T08:18:52.109Z] [2024-01-19T08:18:52.109Z] Full Stack Trace [2024-01-19T08:18:52.109Z] github.com/portworx/torpedo/pkg/log.FailOnError({0x88a5060, 0xc00100d270}, {0x7f74bfd?, 0x6cc0320?}, {0x0?, 0x6cc0320?, 0xc001861e40?}) [2024-01-19T08:18:52.109Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 +0x290 [2024-01-19T08:18:52.109Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest.func1() [2024-01-19T08:18:52.109Z] /go/src/github.com/portworx/torpedo/tests/common.go:7283 +0x32 [2024-01-19T08:18:52.109Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest({0xc000582f70, 0x1, 0x1}) [2024-01-19T08:18:52.109Z] /go/src/github.com/portworx/torpedo/tests/common.go:7299 +0x778 [2024-01-19T08:18:52.109Z] github.com/portworx/torpedo/tests/backup.glob..func29.3() [2024-01-19T08:18:52.109Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_orphaned_test.go:1556 +0x2bb [2024-01-19T08:18:52.109Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T08:18:52.109Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T08:18:52.109Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T08:18:52.109Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T08:18:52.109Z] created by testing.(*T).Run [2024-01-19T08:18:52.109Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T08:18:52.109Z] ------------------------------ [2024-01-19T08:18:52.109Z] STEP: schedule applications [2024-01-19T08:18:52.109Z] STEP: Validate applications [2024-01-19T08:18:52.109Z] STEP: validate applications [2024-01-19T08:18:52.109Z] STEP: validate mysql-ibm app's volumes [2024-01-19T08:18:52.110Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T08:18:52.110Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T08:18:52.110Z] STEP: get mysql-ibm app's volume: pvc-fecbc04a-2cc5-40ba-bdf0-51efe24d003b inspected by the volume driver [2024-01-19T08:18:52.110Z] STEP: wait for mysql-ibm app to start running [2024-01-19T08:18:52.110Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T08:18:52.110Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T08:18:52.110Z] STEP: Validate Px pod restart count [2024-01-19T08:18:52.110Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T08:18:52.110Z] STEP: Create user with px-backup-infra.admin role [2024-01-19T08:18:52.110Z] STEP: Create cloud credential and backup location from the user testuser1-1705651776 [2024-01-19T08:18:52.110Z] STEP: Create source and destination cluster from the user testuser1-1705651776 [2024-01-19T08:18:52.110Z] STEP: Take in progress backup of applications from the user testuser1-1705651776 [2024-01-19T08:18:52.110Z] STEP: Verify in progress backups of the user testuser1-1705651776 from the admin [2024-01-19T08:18:52.110Z] STEP: Delete in progress backups taken by user testuser1-1705651776 from the admin [2024-01-19T08:18:52.110Z] STEP: Take failed backup of applications from the user testuser1-1705651776 [2024-01-19T08:18:52.110Z] STEP: Verify failed backups of the user testuser1-1705651776 from the admin [2024-01-19T08:18:52.110Z] STEP: Delete failed backups taken by user testuser1-1705651776 from the admin [2024-01-19T08:18:52.110Z] STEP: Take successful backup of applications from the user testuser1-1705651776 [2024-01-19T08:18:52.110Z] STEP: Verify successful backups of the user testuser1-1705651776 from the admin [2024-01-19T08:18:52.110Z] STEP: Take in progress restore of backups from the user testuser1-1705651776 [2024-01-19T08:18:52.110Z] STEP: Verify in progress restores of the user testuser1-1705651776 from the admin [2024-01-19T08:18:52.110Z] STEP: Delete in progress restores taken by user testuser1-1705651776 from the admin [2024-01-19T08:18:52.110Z] STEP: Take failed restore of backups from the user testuser1-1705651776 [2024-01-19T08:18:52.110Z] STEP: Verify failed restores of the user testuser1-1705651776 from the admin [2024-01-19T08:18:52.110Z] STEP: Delete failed restores taken by user testuser1-1705651776 from the admin [2024-01-19T08:18:52.110Z] STEP: destroy apps [2024-01-19T08:18:52.110Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T08:18:52.110Z] STEP: start destroying mysql-ibm app [2024-01-19T08:18:52.110Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T08:18:52.110Z] SSS2024-01-19 08:18:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Log Dir: /testresults//ScheduleBackupWithAdditionAndRemovalOfNS.log [2024-01-19T08:18:52.110Z] INFO[2024-01-19 08:18:51] --------Test Start------ [2024-01-19T08:18:52.110Z] INFO[2024-01-19 08:18:51] #Test: ScheduleBackupWithAdditionAndRemovalOfNS [2024-01-19T08:18:52.110Z] INFO[2024-01-19 08:18:51] #Description: Perform schedule backup during which remove and add namespace and verify restoration of removed namespace [2024-01-19T08:18:52.110Z] INFO[2024-01-19 08:18:51] ------------------------ [2024-01-19T08:18:52.110Z] 2024-01-19 08:18:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Deploy applications [2024-01-19T08:18:52.110Z] 2024-01-19 08:18:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T08:18:52.110Z] 2024-01-19 08:18:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T08:18:52.110Z] 2024-01-19 08:18:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T08:18:52.110Z] 2024-01-19 08:18:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T08:18:52.111Z] 2024-01-19 08:18:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T08:18:52.111Z] 2024-01-19 08:18:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T08:18:52.111Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T08:18:52.111Z] 2024-01-19 08:18:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T08:18:52.112Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T08:18:52.112Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T08:18:52.112Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T08:18:52.112Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T08:18:52.112Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T08:18:52.370Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T08:18:52.370Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T08:18:52.370Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T08:18:52.371Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T08:18:52.371Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T08:18:52.372Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T08:18:52.372Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T08:18:52.372Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T08:18:52.372Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T08:18:52.372Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T08:18:52.629Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T08:18:52.629Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T08:18:52.629Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T08:18:52.630Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T08:18:52.886Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T08:18:52.886Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T08:18:52.886Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T08:18:52.886Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T08:18:52.886Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T08:18:52.886Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T08:18:52.886Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T08:18:52.886Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T08:18:53.143Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T08:18:53.144Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T08:18:53.144Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T08:18:53.145Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T08:18:53.145Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T08:18:53.145Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T08:18:53.145Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T08:18:53.145Z] 2024-01-19 08:18:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T08:18:53.145Z] 2024-01-19 08:18:53 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T08:18:53.401Z] 2024-01-19 08:18:53 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T08:18:53.401Z] 2024-01-19 08:18:53 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T08:18:53.402Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T08:18:53.402Z] 2024-01-19 08:18:53 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T08:18:53.402Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T08:18:53.403Z] 2024-01-19 08:18:53 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T08:18:53.403Z] 2024-01-19 08:18:53 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Created namespaces : [mysql-ibm-pxb-0-84848-01-19-08h18m51s mysql-ibm-pxb-1-84848-01-19-08h18m51s mysql-ibm-pxb-2-84848-01-19-08h18m51s mysql-ibm-pxb-3-84848-01-19-08h18m51s mysql-ibm-pxb-4-84848-01-19-08h18m51s] [2024-01-19T08:18:53.403Z] 2024-01-19 08:18:53 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validate applications [2024-01-19T08:18:53.403Z] 2024-01-19 08:18:53 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validate applications [2024-01-19T08:18:53.403Z] 2024-01-19 08:18:53 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating mysql-ibm app [2024-01-19T08:18:53.403Z] 2024-01-19 08:18:53 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating mysql-ibm app's volumes [2024-01-19T08:18:53.403Z] 2024-01-19 08:18:53 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T08:18:53.403Z] 2024/01/19 08:18:53 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T08:19:04.928Z] 2024-01-19 08:19:03 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-84848-01-19-08h18m51s [2024-01-19T08:19:04.928Z] 2024-01-19 08:19:03 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T08:19:04.928Z] 2024-01-19 08:19:03 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T08:19:04.928Z] 2024-01-19 08:19:03 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] wait for mysql-ibm app to start running [2024-01-19T08:19:04.928Z] 2024-01-19 08:19:03 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T08:19:04.929Z] 2024/01/19 08:19:03 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T08:19:04.929Z] pod name:mysql-7d766cd65c-9pd4j namespace:mysql-ibm-pxb-0-84848-01-19-08h18m51s running:true ready:false node: [2024-01-19T08:19:04.929Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T08:19:14.870Z] 2024/01/19 08:19:13 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T08:19:14.870Z] pod name:mysql-7d766cd65c-9pd4j namespace:mysql-ibm-pxb-0-84848-01-19-08h18m51s running:false ready:false node:10.240.64.8 [2024-01-19T08:19:14.870Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T08:19:24.815Z] 2024/01/19 08:19:23 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T08:19:24.815Z] pod name:mysql-7d766cd65c-9pd4j namespace:mysql-ibm-pxb-0-84848-01-19-08h18m51s running:false ready:false node:10.240.64.8 [2024-01-19T08:19:24.815Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T08:19:34.760Z] 2024/01/19 08:19:33 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T08:19:34.760Z] pod name:mysql-7d766cd65c-9pd4j namespace:mysql-ibm-pxb-0-84848-01-19-08h18m51s running:true ready:false node:10.240.64.8 [2024-01-19T08:19:34.760Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T08:19:44.719Z] 2024/01/19 08:19:43 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T08:19:44.719Z] pod name:mysql-7d766cd65c-9pd4j namespace:mysql-ibm-pxb-0-84848-01-19-08h18m51s running:true ready:false node:10.240.64.8 [2024-01-19T08:19:44.719Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T08:19:54.682Z] 2024/01/19 08:19:54 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T08:19:54.682Z] pod name:mysql-7d766cd65c-9pd4j namespace:mysql-ibm-pxb-0-84848-01-19-08h18m51s running:true ready:false node:10.240.64.8 [2024-01-19T08:19:54.682Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T08:20:04.651Z] 2024/01/19 08:20:04 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T08:20:04.651Z] pod name:mysql-7d766cd65c-9pd4j namespace:mysql-ibm-pxb-0-84848-01-19-08h18m51s running:true ready:false node:10.240.64.8 [2024-01-19T08:20:04.651Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T08:20:14.707Z] 2024/01/19 08:20:14 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T08:20:14.707Z] pod name:mysql-7d766cd65c-9pd4j namespace:mysql-ibm-pxb-0-84848-01-19-08h18m51s running:true ready:false node:10.240.64.8 [2024-01-19T08:20:14.707Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T08:20:24.670Z] 2024/01/19 08:20:24 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T08:20:24.670Z] pod name:mysql-7d766cd65c-9pd4j namespace:mysql-ibm-pxb-0-84848-01-19-08h18m51s running:true ready:false node:10.240.64.8 [2024-01-19T08:20:24.670Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T08:20:34.620Z] 2024-01-19 08:20:34 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T08:20:34.620Z] 2024-01-19 08:20:34 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T08:20:34.878Z] 2024-01-19 08:20:34 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] validate if mysql-ibm app's volumes are setup [2024-01-19T08:20:34.879Z] 2024-01-19 08:20:34 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T08:20:34.879Z] 2024-01-19 08:20:34 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T08:20:34.879Z] 2024-01-19 08:20:34 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T08:20:34.879Z] 2024-01-19 08:20:34 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T08:20:34.879Z] 2024-01-19 08:20:34 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating mysql-ibm app [2024-01-19T08:20:34.879Z] 2024-01-19 08:20:34 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating mysql-ibm app's volumes [2024-01-19T08:20:34.879Z] 2024-01-19 08:20:34 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T08:20:34.879Z] 2024-01-19 08:20:34 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T08:20:34.879Z] 2024-01-19 08:20:34 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-1-84848-01-19-08h18m51s [2024-01-19T08:20:34.879Z] 2024-01-19 08:20:34 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T08:20:35.134Z] 2024-01-19 08:20:34 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T08:20:35.134Z] 2024-01-19 08:20:34 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] wait for mysql-ibm app to start running [2024-01-19T08:20:35.135Z] 2024-01-19 08:20:34 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T08:20:35.135Z] 2024-01-19 08:20:34 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T08:20:35.697Z] 2024-01-19 08:20:35 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T08:20:36.646Z] 2024-01-19 08:20:36 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] validate if mysql-ibm app's volumes are setup [2024-01-19T08:20:36.646Z] 2024-01-19 08:20:36 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T08:20:36.646Z] 2024-01-19 08:20:36 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T08:20:36.646Z] 2024-01-19 08:20:36 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T08:20:36.646Z] 2024-01-19 08:20:36 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T08:20:36.646Z] 2024-01-19 08:20:36 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating mysql-ibm app [2024-01-19T08:20:36.646Z] 2024-01-19 08:20:36 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating mysql-ibm app's volumes [2024-01-19T08:20:36.646Z] 2024-01-19 08:20:36 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T08:20:36.646Z] 2024-01-19 08:20:36 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T08:20:36.646Z] 2024-01-19 08:20:36 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-2-84848-01-19-08h18m51s [2024-01-19T08:20:36.646Z] 2024-01-19 08:20:36 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T08:20:36.646Z] 2024-01-19 08:20:36 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T08:20:36.646Z] 2024-01-19 08:20:36 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] wait for mysql-ibm app to start running [2024-01-19T08:20:36.903Z] 2024-01-19 08:20:36 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T08:20:37.162Z] 2024-01-19 08:20:36 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T08:20:37.730Z] 2024-01-19 08:20:37 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T08:20:38.657Z] 2024-01-19 08:20:38 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] validate if mysql-ibm app's volumes are setup [2024-01-19T08:20:38.657Z] 2024-01-19 08:20:38 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T08:20:38.657Z] 2024-01-19 08:20:38 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T08:20:38.657Z] 2024-01-19 08:20:38 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T08:20:38.657Z] 2024-01-19 08:20:38 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T08:20:38.657Z] 2024-01-19 08:20:38 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating mysql-ibm app [2024-01-19T08:20:38.657Z] 2024-01-19 08:20:38 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating mysql-ibm app's volumes [2024-01-19T08:20:38.657Z] 2024-01-19 08:20:38 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T08:20:38.657Z] 2024-01-19 08:20:38 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T08:20:38.657Z] 2024-01-19 08:20:38 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-3-84848-01-19-08h18m51s [2024-01-19T08:20:38.657Z] 2024-01-19 08:20:38 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T08:20:38.657Z] 2024-01-19 08:20:38 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T08:20:38.657Z] 2024-01-19 08:20:38 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] wait for mysql-ibm app to start running [2024-01-19T08:20:38.914Z] 2024-01-19 08:20:38 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T08:20:39.170Z] 2024-01-19 08:20:38 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T08:20:39.731Z] 2024-01-19 08:20:39 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T08:20:40.658Z] 2024-01-19 08:20:40 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] validate if mysql-ibm app's volumes are setup [2024-01-19T08:20:40.658Z] 2024-01-19 08:20:40 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T08:20:40.658Z] 2024-01-19 08:20:40 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T08:20:40.658Z] 2024-01-19 08:20:40 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T08:20:40.658Z] 2024-01-19 08:20:40 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T08:20:40.658Z] 2024-01-19 08:20:40 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating mysql-ibm app [2024-01-19T08:20:40.658Z] 2024-01-19 08:20:40 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating mysql-ibm app's volumes [2024-01-19T08:20:40.658Z] 2024-01-19 08:20:40 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T08:20:40.658Z] 2024-01-19 08:20:40 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T08:20:40.659Z] 2024-01-19 08:20:40 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-4-84848-01-19-08h18m51s [2024-01-19T08:20:40.659Z] 2024-01-19 08:20:40 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T08:20:40.659Z] 2024-01-19 08:20:40 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T08:20:40.659Z] 2024-01-19 08:20:40 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] wait for mysql-ibm app to start running [2024-01-19T08:20:40.659Z] 2024-01-19 08:20:40 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T08:20:41.219Z] 2024-01-19 08:20:40 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T08:20:41.780Z] 2024-01-19 08:20:41 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T08:20:42.706Z] 2024-01-19 08:20:42 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] validate if mysql-ibm app's volumes are setup [2024-01-19T08:20:42.706Z] 2024-01-19 08:20:42 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T08:20:42.706Z] 2024-01-19 08:20:42 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T08:20:42.706Z] 2024-01-19 08:20:42 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T08:20:42.706Z] 2024-01-19 08:20:42 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T08:20:42.706Z] 2024-01-19 08:20:42 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Generate and add labels to namespaces [2024-01-19T08:20:43.269Z] INFO[2024-01-19 08:20:43] Verifying : Description : Adding labels [map[0-c6da2d74-ce5b-4ce3-8d03-77e5936c0590:c6da2d74-ce5b-4ce3-8d03-77e5936c0590 1-c6da2d74-ce5b-4ce3-8d03-77e5936c0590:c6da2d74-ce5b-4ce3-8d03-77e5936c0590 2-c6da2d74-ce5b-4ce3-8d03-77e5936c0590:c6da2d74-ce5b-4ce3-8d03-77e5936c0590]] to namespaces [[mysql-ibm-pxb-0-84848-01-19-08h18m51s mysql-ibm-pxb-1-84848-01-19-08h18m51s mysql-ibm-pxb-2-84848-01-19-08h18m51s mysql-ibm-pxb-3-84848-01-19-08h18m51s mysql-ibm-pxb-4-84848-01-19-08h18m51s]] [2024-01-19T08:20:43.269Z] INFO[2024-01-19 08:20:43] Actual:true, Expected: true [2024-01-19T08:20:43.269Z] 2024-01-19 08:20:43 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Generating namespace label string from label map [2024-01-19T08:20:43.269Z] 2024-01-19 08:20:43 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.glob..func21.2.3:#688] - Generated labels [0-c6da2d74-ce5b-4ce3-8d03-77e5936c0590=c6da2d74-ce5b-4ce3-8d03-77e5936c0590,1-c6da2d74-ce5b-4ce3-8d03-77e5936c0590=c6da2d74-ce5b-4ce3-8d03-77e5936c0590,2-c6da2d74-ce5b-4ce3-8d03-77e5936c0590=c6da2d74-ce5b-4ce3-8d03-77e5936c0590] [2024-01-19T08:20:43.269Z] 2024-01-19 08:20:43 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Creating cloud credentials and backup location [2024-01-19T08:20:45.161Z] 2024-01-19 08:20:44 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-1705652444] for org [default] with [aws] as provider [2024-01-19T08:20:45.161Z] 2024-01-19 08:20:44 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T08:20:45.161Z] 2024-01-19 08:20:44 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-1705652444] is present [2024-01-19T08:20:45.161Z] INFO[2024-01-19 08:20:44] Verifying : Description : Creating cloud credentials autogenerated-cred-1705652444 with provider aws [2024-01-19T08:20:45.161Z] INFO[2024-01-19 08:20:44] Actual:true, Expected: true [2024-01-19T08:20:45.161Z] 2024-01-19 08:20:44 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Created Cloud Credentials with name - autogenerated-cred-1705652444 [2024-01-19T08:20:49.328Z] INFO[2024-01-19 08:20:48] Verifying : Description : Creating backup location autogenerated-backup-location-1705652444 [2024-01-19T08:20:49.328Z] INFO[2024-01-19 08:20:48] Actual:true, Expected: true [2024-01-19T08:20:49.328Z] 2024-01-19 08:20:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Adding source and destination clusters with px-central-admin ctx [2024-01-19T08:20:49.328Z] INFO[2024-01-19 08:20:48] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T08:20:49.328Z] INFO[2024-01-19 08:20:48] Actual:true, Expected: true [2024-01-19T08:20:49.328Z] 2024-01-19 08:20:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T08:20:49.328Z] 2024-01-19 08:20:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T08:20:49.328Z] 2024-01-19 08:20:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T08:20:49.328Z] 2024-01-19 08:20:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T08:20:49.328Z] 2024-01-19 08:20:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Create cluster [source-cluster] in org [default] [2024-01-19T08:20:49.328Z] 2024-01-19 08:20:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T08:20:49.328Z] 2024-01-19 08:20:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T08:20:49.328Z] 2024-01-19 08:20:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Create cluster [destination-cluster] in org [default] [2024-01-19T08:20:49.328Z] 2024-01-19 08:20:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T08:20:49.328Z] 2024-01-19 08:20:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T08:20:49.328Z] 2024-01-19 08:20:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-llcnj] for IBM [2024-01-19T08:20:49.329Z] 2024-01-19 08:20:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-llcnj] for org [default] with [ibm] as provider [2024-01-19T08:20:49.329Z] 2024-01-19 08:20:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T08:20:49.329Z] 2024-01-19 08:20:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T08:20:49.329Z] 2024-01-19 08:20:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-llcnj] is present [2024-01-19T08:20:49.329Z] 2024-01-19 08:20:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Create cluster [source-cluster] in org [default] [2024-01-19T08:20:52.618Z] 2024-01-19 08:20:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-fiomu] for IBM [2024-01-19T08:20:52.618Z] 2024-01-19 08:20:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-fiomu] for org [default] with [ibm] as provider [2024-01-19T08:20:52.618Z] 2024-01-19 08:20:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T08:20:52.618Z] 2024-01-19 08:20:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T08:20:52.618Z] 2024-01-19 08:20:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-fiomu] is present [2024-01-19T08:20:52.618Z] 2024-01-19 08:20:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Create cluster [destination-cluster] in org [default] [2024-01-19T08:20:53.547Z] INFO[2024-01-19 08:20:53] Verifying : Description : Verifying addition of source [source-cluster] and destination [destination-cluster] clusters [2024-01-19T08:20:53.547Z] INFO[2024-01-19 08:20:53] Actual:true, Expected: true [2024-01-19T08:20:53.547Z] INFO[2024-01-19 08:20:53] Verifying : Description : Verifying if [source-cluster] cluster is online [2024-01-19T08:20:53.547Z] INFO[2024-01-19 08:20:53] Actual:Online, Expected: Online [2024-01-19T08:20:53.547Z] INFO[2024-01-19 08:20:53] Verifying : Description : Fetching [source-cluster] cluster uid [2024-01-19T08:20:53.547Z] INFO[2024-01-19 08:20:53] Actual:true, Expected: true [2024-01-19T08:20:53.547Z] 2024-01-19 08:20:53 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] UID of [source-cluster] cluster is 4782ef2f-0fc3-4450-8dca-1dd5e9359e3c [2024-01-19T08:20:53.547Z] INFO[2024-01-19 08:20:53] Verifying : Description : Verifying if [destination-cluster] cluster is online [2024-01-19T08:20:53.547Z] INFO[2024-01-19 08:20:53] Actual:Online, Expected: Online [2024-01-19T08:20:53.547Z] 2024-01-19 08:20:53 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Creating a schedule policy [2024-01-19T08:20:53.804Z] 2024-01-19 08:20:53 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Create Backup Schedule Policy [2024-01-19T08:20:54.367Z] INFO[2024-01-19 08:20:54] Verifying : Description : Verifying creation of periodic schedule policy of interval [15] minutes named [periodic-1705652453] [2024-01-19T08:20:54.367Z] INFO[2024-01-19 08:20:54] Actual:true, Expected: true [2024-01-19T08:20:54.367Z] INFO[2024-01-19 08:20:54] Verifying : Description : Fetching uid of periodic schedule policy named [periodic-1705652453] [2024-01-19T08:20:54.367Z] INFO[2024-01-19 08:20:54] Actual:true, Expected: true [2024-01-19T08:20:54.367Z] 2024-01-19 08:20:54 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Creating schedule backup with namespace label [2024-01-19T08:20:55.733Z] 2024-01-19 08:20:55 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T08:20:55.733Z] 2024-01-19 08:20:55 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Volume snapshot classes found in the cluster - [2024-01-19T08:20:55.733Z] 2024-01-19 08:20:55 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] ibmc-vpcblock-snapshot [2024-01-19T08:20:55.733Z] 2024-01-19 08:20:55 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T08:22:03.355Z] 2024-01-19 08:21:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] first schedule backup for schedule name [tp-backup-schedule-1705652455] is [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055] [2024-01-19T08:22:03.355Z] 2024/01/19 08:21:56 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T08:22:29.852Z] 2024/01/19 08:22:26 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T08:22:56.403Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating backup [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055] in org [default] [2024-01-19T08:22:56.403Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateBackup:#1774] - Obtaining backup info for backup [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055] [2024-01-19T08:22:56.403Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84848-01-19-08h18m51s] in backup [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055] [2024-01-19T08:22:56.403Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84848-01-19-08h18m51s] in backup [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055] [2024-01-19T08:22:56.403Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-0-84848-01-19-08h18m51s] [2024-01-19T08:22:56.403Z] 2024-01-19 08:22:56 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T08:22:56.403Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T08:22:56.403Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-84848-01-19-08h18m51s] are [[mysql-data]] [2024-01-19T08:22:56.403Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T08:22:56.403Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84848-01-19-08h18m51s] in backup [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055] [2024-01-19T08:22:56.403Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84848-01-19-08h18m51s] in backup [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055] [2024-01-19T08:22:56.403Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-1-84848-01-19-08h18m51s] [2024-01-19T08:22:56.403Z] 2024-01-19 08:22:56 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T08:22:56.403Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T08:22:56.403Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-84848-01-19-08h18m51s] are [[mysql-data]] [2024-01-19T08:22:56.661Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T08:22:56.661Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84848-01-19-08h18m51s] in backup [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055] [2024-01-19T08:22:56.661Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84848-01-19-08h18m51s] in backup [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055] [2024-01-19T08:22:56.661Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-2-84848-01-19-08h18m51s] [2024-01-19T08:22:56.661Z] 2024-01-19 08:22:56 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T08:22:56.661Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T08:22:56.661Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-2-84848-01-19-08h18m51s] are [[mysql-data]] [2024-01-19T08:22:56.661Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T08:22:56.661Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-3-84848-01-19-08h18m51s] in backup [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055] [2024-01-19T08:22:56.662Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-3-84848-01-19-08h18m51s] in backup [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055] [2024-01-19T08:22:56.662Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-3-84848-01-19-08h18m51s] [2024-01-19T08:22:56.662Z] 2024-01-19 08:22:56 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T08:22:56.662Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T08:22:56.662Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-3-84848-01-19-08h18m51s] are [[mysql-data]] [2024-01-19T08:22:56.662Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T08:22:56.662Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating specs for the namespace (scheduledAppContext) [mysql-ibm-pxb-4-84848-01-19-08h18m51s] in backup [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055] [2024-01-19T08:22:56.662Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating backed up volumes for the namespace (scheduledAppContext) [mysql-ibm-pxb-4-84848-01-19-08h18m51s] in backup [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055] [2024-01-19T08:22:56.662Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateBackup:#1900] - getting the volumes bounded to the PVCs in the namespace (scheduledAppContext) [mysql-ibm-pxb-4-84848-01-19-08h18m51s] [2024-01-19T08:22:56.662Z] 2024-01-19 08:22:56 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T08:22:56.662Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T08:22:56.662Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateBackup:#1911] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-4-84848-01-19-08h18m51s] are [[mysql-data]] [2024-01-19T08:22:56.662Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateBackup:#1974] - in case of native CSI backup volumes in backup object is not updated with storage class [2024-01-19T08:22:56.662Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating CR cleanup [2024-01-19T08:22:56.662Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.getCurrentAdminNamespace:#5925] - Current admin namespace - [kube-system] [2024-01-19T08:22:56.662Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]] [2024-01-19T08:22:56.662Z] 2024-01-19 08:22:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]] under [kube-system] namespace [2024-01-19T08:22:56.662Z] 2024/01/19 08:22:56 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]}, Next try in [5s], timeout [5m0s] [2024-01-19T08:23:01.904Z] 2024-01-19 08:23:01 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]] [2024-01-19T08:23:01.904Z] 2024-01-19 08:23:01 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]] under [kube-system] namespace [2024-01-19T08:23:01.904Z] 2024/01/19 08:23:01 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]}, Next try in [5s], timeout [5m0s] [2024-01-19T08:23:07.153Z] 2024-01-19 08:23:06 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]] [2024-01-19T08:23:07.153Z] 2024-01-19 08:23:06 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]] under [kube-system] namespace [2024-01-19T08:23:07.153Z] 2024/01/19 08:23:06 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]}, Next try in [5s], timeout [5m0s] [2024-01-19T08:23:12.399Z] 2024-01-19 08:23:11 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]] [2024-01-19T08:23:12.399Z] 2024-01-19 08:23:11 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]] under [kube-system] namespace [2024-01-19T08:23:12.399Z] 2024/01/19 08:23:11 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]}, Next try in [5s], timeout [5m0s] [2024-01-19T08:23:17.646Z] 2024-01-19 08:23:16 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]] [2024-01-19T08:23:17.646Z] 2024-01-19 08:23:16 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]] under [kube-system] namespace [2024-01-19T08:23:17.646Z] 2024/01/19 08:23:16 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]}, Next try in [5s], timeout [5m0s] [2024-01-19T08:23:22.893Z] 2024-01-19 08:23:21 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]] [2024-01-19T08:23:22.893Z] 2024-01-19 08:23:21 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]] under [kube-system] namespace [2024-01-19T08:23:22.893Z] 2024/01/19 08:23:21 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]}, Next try in [5s], timeout [5m0s] [2024-01-19T08:23:27.064Z] 2024-01-19 08:23:26 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]] [2024-01-19T08:23:27.064Z] 2024-01-19 08:23:26 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]] under [kube-system] namespace [2024-01-19T08:23:27.064Z] 2024/01/19 08:23:26 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]}, Next try in [5s], timeout [5m0s] [2024-01-19T08:23:32.343Z] 2024-01-19 08:23:31 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]] [2024-01-19T08:23:32.343Z] 2024-01-19 08:23:31 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]] under [kube-system] namespace [2024-01-19T08:23:32.343Z] 2024/01/19 08:23:31 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]}, Next try in [5s], timeout [5m0s] [2024-01-19T08:23:37.589Z] 2024-01-19 08:23:37 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]] [2024-01-19T08:23:37.589Z] 2024-01-19 08:23:37 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]] under [kube-system] namespace [2024-01-19T08:23:37.589Z] 2024/01/19 08:23:37 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]}, Next try in [5s], timeout [5m0s] [2024-01-19T08:23:42.833Z] 2024-01-19 08:23:42 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]] [2024-01-19T08:23:42.833Z] 2024-01-19 08:23:42 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] CR found for [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133 tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]] under [kube-system] namespace [2024-01-19T08:23:42.833Z] 2024/01/19 08:23:42 DoRetryWithTimeout - Error: {CR cleanup validation failed for - [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055]}, Next try in [5s], timeout [5m0s] [2024-01-19T08:23:48.077Z] 2024-01-19 08:23:47 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133]] [2024-01-19T08:23:48.077Z] INFO[2024-01-19 08:23:47] Verifying : Description : Verifying creation of first schedule backup [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055] [2024-01-19T08:23:48.077Z] INFO[2024-01-19 08:23:47] Actual:true, Expected: true [2024-01-19T08:23:48.077Z] 2024-01-19 08:23:47 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Waiting for 15 minutes for the next schedule backup to be triggered [2024-01-19T08:23:48.077Z] 2024-01-19 08:23:47 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Waiting for [15] minutes for the next schedule backup to be triggered [2024-01-19T08:24:44.249Z] Iteration: 65 [2024-01-19T08:24:44.249Z] Logs from iteration: 65 [2024-01-19T08:24:44.249Z] 2024-01-19 08:23:47 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] All CRs in [kube-system] are [[tp-backup-1705598306-7f77056-interval-2024-01-18-181839 tp-backup-1705598479-b6d689c-interval-2024-01-18-182133]] [2024-01-19T08:24:44.249Z] INFO[2024-01-19 08:23:47] Verifying : Description : Verifying creation of first schedule backup [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-082055] [2024-01-19T08:24:44.249Z] INFO[2024-01-19 08:23:47] Actual:true, Expected: true [2024-01-19T08:24:44.249Z] 2024-01-19 08:23:47 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Waiting for 15 minutes for the next schedule backup to be triggered [2024-01-19T08:24:44.249Z] 2024-01-19 08:23:47 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Waiting for [15] minutes for the next schedule backup to be triggered [2024-01-19T08:25:10.752Z] INFO[2024-01-19 08:25:10] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T08:25:10.752Z] INFO[2024-01-19 08:25:10] sched stats: tick interval: 60670 < 1.2s (100%) [2024-01-19T08:25:10.752Z] INFO[2024-01-19 08:25:10] sched stats: tick process durations: 60669 < 500ms (100%) [2024-01-19T08:35:17.580Z] INFO[2024-01-19 08:35:11] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T08:35:17.580Z] INFO[2024-01-19 08:35:11] sched stats: tick interval: 61271 < 1.2s (100%) [2024-01-19T08:35:17.580Z] INFO[2024-01-19 08:35:11] sched stats: tick process durations: 61270 < 500ms (100%) [2024-01-19T08:38:54.020Z] 2024-01-19 08:38:47 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] schedule name tp-backup-schedule-1705652455, Next schedule backup name: tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-083600 [2024-01-19T08:38:54.023Z] 2024-01-19 08:38:47 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Inspect obj backup: last_update_time: create_time_in_sec:1705653360 ownership: > > backup_info: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"243cb06a-b1ed-4a50-88cc-dc64336ab9ab" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"95d70ae5-7e16-4c07-9f1a-959ae4ee07f4" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"cdf4d9a1-900b-496d-922a-ed052aea120f" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c1fbe809-b984-4f36-8bf4-e43de3dd4c61" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"3f33e25b-0edd-4fa1-9b0d-76acb9b9fa94" > backup_path:"kube-system/tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-083600/e67fcccb-c74f-460f-b681-3a147f451f5f" stage:Final backup_schedule: cr_name:"tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-083600" total_size:53687091200 resource_count:25 backup_location_ref: cloud_credential_ref: cr_uid:"e67fcccb-c74f-460f-b681-3a147f451f5f" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-c6da2d74-ce5b-4ce3-8d03-77e5936c0590=c6da2d74-ce5b-4ce3-8d03-77e5936c0590,1-c6da2d74-ce5b-4ce3-8d03-77e5936c0590=c6da2d74-ce5b-4ce3-8d03-77e5936c0590,2-c6da2d74-ce5b-4ce3-8d03-77e5936c0590=c6da2d74-ce5b-4ce3-8d03-77e5936c0590" target_namespace:"kube-system" > > [2024-01-19T08:38:54.023Z] INFO[2024-01-19 08:38:47] Verifying : Description : Fetching the name of second schedule backup [tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-083600] [2024-01-19T08:38:54.023Z] INFO[2024-01-19 08:38:47] Actual:true, Expected: true [2024-01-19T08:38:54.023Z] 2024-01-19 08:38:47 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Remove namespace label from namespace and check backup success of next schedule backup [2024-01-19T08:38:54.023Z] INFO[2024-01-19 08:38:47] Verifying : Description : Removing labels [map[0-c6da2d74-ce5b-4ce3-8d03-77e5936c0590:c6da2d74-ce5b-4ce3-8d03-77e5936c0590 1-c6da2d74-ce5b-4ce3-8d03-77e5936c0590:c6da2d74-ce5b-4ce3-8d03-77e5936c0590 2-c6da2d74-ce5b-4ce3-8d03-77e5936c0590:c6da2d74-ce5b-4ce3-8d03-77e5936c0590]] from namespaces [[mysql-ibm-pxb-0-84848-01-19-08h18m51s mysql-ibm-pxb-1-84848-01-19-08h18m51s mysql-ibm-pxb-2-84848-01-19-08h18m51s]] [2024-01-19T08:38:54.023Z] INFO[2024-01-19 08:38:47] Actual:true, Expected: true [2024-01-19T08:38:54.024Z] 2024-01-19 08:38:47 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Waiting for [15] minutes for the next schedule backup to be triggered [2024-01-19T08:45:15.560Z] INFO[2024-01-19 08:45:11] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T08:45:15.560Z] INFO[2024-01-19 08:45:11] sched stats: tick interval: 61871 < 1.2s (100%) [2024-01-19T08:45:15.560Z] INFO[2024-01-19 08:45:11] sched stats: tick process durations: 61870 < 500ms (100%) [2024-01-19T08:53:52.103Z] 2024-01-19 08:53:47 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] schedule name tp-backup-schedule-1705652455, Next schedule backup name: tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-085103 [2024-01-19T08:53:52.105Z] 2024-01-19 08:53:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Inspect obj backup: last_update_time: create_time_in_sec:1705654263 ownership: > > backup_info: resources: resources: resources: resources: resources: resources: resources: resources: resources: resources: volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"c1fbe809-b984-4f36-8bf4-e43de3dd4c61" > volumes: driver_name:"csi" options: options: total_size:10737418240 actual_size:10737418240 pvc_id:"3f33e25b-0edd-4fa1-9b0d-76acb9b9fa94" > backup_path:"kube-system/tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-085103/a11153da-1517-4433-9294-684b6cfa90f5" stage:Final backup_schedule: cr_name:"tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-085103" total_size:21474836480 resource_count:10 backup_location_ref: cloud_credential_ref: cr_uid:"a11153da-1517-4433-9294-684b6cfa90f5" user_backupshare_access:FullAccess cluster_ref: ns_label_selectors:"0-c6da2d74-ce5b-4ce3-8d03-77e5936c0590=c6da2d74-ce5b-4ce3-8d03-77e5936c0590,1-c6da2d74-ce5b-4ce3-8d03-77e5936c0590=c6da2d74-ce5b-4ce3-8d03-77e5936c0590,2-c6da2d74-ce5b-4ce3-8d03-77e5936c0590=c6da2d74-ce5b-4ce3-8d03-77e5936c0590" target_namespace:"kube-system" > > [2024-01-19T08:53:52.105Z] INFO[2024-01-19 08:53:48] Verifying : Description : Verifying backup success for tp-backup-schedule-1705652455-b081a21-interval-2024-01-19-085103 schedule backup after nmaespace removal [mysql-ibm-pxb-0-84848-01-19-08h18m51s mysql-ibm-pxb-1-84848-01-19-08h18m51s mysql-ibm-pxb-2-84848-01-19-08h18m51s] [2024-01-19T08:53:52.105Z] INFO[2024-01-19 08:53:48] Actual:true, Expected: true [2024-01-19T08:53:52.105Z] 2024-01-19 08:53:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Restore the backup which was taken before the namespace removal and the namespace should be recovered [2024-01-19T08:53:52.105Z] 2024/01/19 08:53:48 DoRetryWithTimeout - Error: {restore status for [tp-restore-1705654428] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T08:54:24.132Z] 2024/01/19 08:54:18 DoRetryWithTimeout - Error: {restore status for [tp-restore-1705654428] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T08:54:50.659Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.CreateRestore:#936] - Restore [tp-restore-1705654428] created successfully [2024-01-19T08:54:50.659Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Switching cluster context to cluster [destination-cluster] [2024-01-19T08:54:50.659Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T08:54:50.659Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-0-84848-01-19-08h18m51s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-19T08:54:50.659Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-0-84848-01-19-08h18m51s] [2024-01-19T08:54:50.660Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T08:54:50.660Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T08:54:50.660Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-1-84848-01-19-08h18m51s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-19T08:54:50.660Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-1-84848-01-19-08h18m51s] [2024-01-19T08:54:50.660Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T08:54:50.660Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T08:54:50.660Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-2-84848-01-19-08h18m51s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-19T08:54:50.660Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-2-84848-01-19-08h18m51s] [2024-01-19T08:54:50.660Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T08:54:50.660Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T08:54:50.660Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-3-84848-01-19-08h18m51s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-19T08:54:50.660Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-3-84848-01-19-08h18m51s] [2024-01-19T08:54:50.660Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T08:54:50.660Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T08:54:50.660Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.CloneAppContextAndTransformWithMappings:#2333] - TransformAppContextWithMappings of appContext [mysql-ibm-pxb-4-84848-01-19-08h18m51s] with namespace mapping [map[]] and storage Class Mapping [map[]] [2024-01-19T08:54:50.660Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.CloneAppContextAndTransformWithMappings:#2349] - TransformAppContextWithMappings is for restore contexts, ignoring transformation of 'StorageClass' [mysql-sc] in appContext [mysql-ibm-pxb-4-84848-01-19-08h18m51s] [2024-01-19T08:54:50.660Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T08:54:50.660Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T08:54:50.660Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating restore [tp-restore-1705654428] in org [default] [2024-01-19T08:54:50.661Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2095] - Obtaining restore info for restore [tp-restore-1705654428] [2024-01-19T08:54:50.661Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2110] - Checking status of restore [tp-restore-1705654428] [2024-01-19T08:54:50.661Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-0-84848-01-19-08h18m51s] in restore [tp-restore-1705654428] [2024-01-19T08:54:50.661Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-84848-01-19-08h18m51s]) was found in restore [tp-restore-1705654428], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-84848-01-19-08h18m51s] [2024-01-19T08:54:50.661Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-0-84848-01-19-08h18m51s]) found in the restore [tp-restore-1705654428] was also present on the cluster/namespace [mysql-ibm-pxb-0-84848-01-19-08h18m51s] [2024-01-19T08:54:50.661Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-0-84848-01-19-08h18m51s]) was found in restore [tp-restore-1705654428], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-84848-01-19-08h18m51s] [2024-01-19T08:54:50.661Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-0-84848-01-19-08h18m51s]) found in the restore [tp-restore-1705654428] was also present on the cluster/namespace [mysql-ibm-pxb-0-84848-01-19-08h18m51s] [2024-01-19T08:54:50.661Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-84848-01-19-08h18m51s]) was found in restore [tp-restore-1705654428], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-84848-01-19-08h18m51s] [2024-01-19T08:54:50.661Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-0-84848-01-19-08h18m51s]) found in the restore [tp-restore-1705654428] was also present on the cluster/namespace [mysql-ibm-pxb-0-84848-01-19-08h18m51s] [2024-01-19T08:54:50.661Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-0-84848-01-19-08h18m51s]) was found in restore [tp-restore-1705654428], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-0-84848-01-19-08h18m51s] [2024-01-19T08:54:50.661Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-0-84848-01-19-08h18m51s]) found in the restore [tp-restore-1705654428] was also present on the cluster/namespace [mysql-ibm-pxb-0-84848-01-19-08h18m51s] [2024-01-19T08:54:50.661Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-0-84848-01-19-08h18m51s] in restore [tp-restore-1705654428] [2024-01-19T08:54:50.661Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-0-84848-01-19-08h18m51s] in restore [tp-restore-1705654428] [2024-01-19T08:54:50.661Z] 2024-01-19 08:54:48 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T08:54:50.662Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T08:54:50.662Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-0-84848-01-19-08h18m51s] are [[mysql-data]] [2024-01-19T08:54:50.662Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2240] - in restore [tp-restore-1705654428], the status of the restored volume [pvc-28a6bd8e-b5ac-4847-9b7c-62a661e4c37c] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-19T08:54:50.662Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validate applications in restored namespace [mysql-ibm-pxb-0-84848-01-19-08h18m51s] due to restore [tp-restore-1705654428] [2024-01-19T08:54:50.662Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating mysql-ibm app [2024-01-19T08:54:50.662Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] wait for mysql-ibm app to start running [2024-01-19T08:54:50.662Z] 2024-01-19 08:54:48 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T08:54:50.662Z] 2024-01-19 08:54:49 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T08:54:50.662Z] 2024-01-19 08:54:49 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T08:54:50.662Z] 2024-01-19 08:54:49 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-1-84848-01-19-08h18m51s] in restore [tp-restore-1705654428] [2024-01-19T08:54:50.662Z] 2024-01-19 08:54:49 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-1-84848-01-19-08h18m51s]) was found in restore [tp-restore-1705654428], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-1-84848-01-19-08h18m51s] [2024-01-19T08:54:50.662Z] 2024-01-19 08:54:49 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-1-84848-01-19-08h18m51s]) found in the restore [tp-restore-1705654428] was also present on the cluster/namespace [mysql-ibm-pxb-1-84848-01-19-08h18m51s] [2024-01-19T08:54:50.662Z] 2024-01-19 08:54:49 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-1-84848-01-19-08h18m51s]) was found in restore [tp-restore-1705654428], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-1-84848-01-19-08h18m51s] [2024-01-19T08:54:50.662Z] 2024-01-19 08:54:49 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-1-84848-01-19-08h18m51s]) found in the restore [tp-restore-1705654428] was also present on the cluster/namespace [mysql-ibm-pxb-1-84848-01-19-08h18m51s] [2024-01-19T08:54:50.662Z] 2024-01-19 08:54:49 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-1-84848-01-19-08h18m51s]) was found in restore [tp-restore-1705654428], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-1-84848-01-19-08h18m51s] [2024-01-19T08:54:50.662Z] 2024-01-19 08:54:49 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-1-84848-01-19-08h18m51s]) found in the restore [tp-restore-1705654428] was also present on the cluster/namespace [mysql-ibm-pxb-1-84848-01-19-08h18m51s] [2024-01-19T08:54:50.663Z] 2024-01-19 08:54:49 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-1-84848-01-19-08h18m51s]) was found in restore [tp-restore-1705654428], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-1-84848-01-19-08h18m51s] [2024-01-19T08:54:50.663Z] 2024-01-19 08:54:49 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-1-84848-01-19-08h18m51s]) found in the restore [tp-restore-1705654428] was also present on the cluster/namespace [mysql-ibm-pxb-1-84848-01-19-08h18m51s] [2024-01-19T08:54:50.663Z] 2024-01-19 08:54:49 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-1-84848-01-19-08h18m51s] in restore [tp-restore-1705654428] [2024-01-19T08:54:50.663Z] 2024-01-19 08:54:49 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-1-84848-01-19-08h18m51s] in restore [tp-restore-1705654428] [2024-01-19T08:54:50.663Z] 2024-01-19 08:54:49 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T08:54:50.663Z] 2024-01-19 08:54:49 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T08:54:50.663Z] 2024-01-19 08:54:49 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-1-84848-01-19-08h18m51s] are [[mysql-data]] [2024-01-19T08:54:50.663Z] 2024-01-19 08:54:49 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2240] - in restore [tp-restore-1705654428], the status of the restored volume [pvc-a8c50c0f-b13a-4991-b29b-7998b5e7aa44] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-19T08:54:50.663Z] 2024-01-19 08:54:49 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validate applications in restored namespace [mysql-ibm-pxb-1-84848-01-19-08h18m51s] due to restore [tp-restore-1705654428] [2024-01-19T08:54:50.663Z] 2024-01-19 08:54:49 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating mysql-ibm app [2024-01-19T08:54:50.663Z] 2024-01-19 08:54:49 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] wait for mysql-ibm app to start running [2024-01-19T08:54:50.663Z] 2024-01-19 08:54:49 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T08:54:50.663Z] 2024-01-19 08:54:50 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T08:54:51.226Z] 2024-01-19 08:54:50 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T08:54:52.154Z] 2024-01-19 08:54:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-2-84848-01-19-08h18m51s] in restore [tp-restore-1705654428] [2024-01-19T08:54:52.154Z] 2024-01-19 08:54:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-2-84848-01-19-08h18m51s]) was found in restore [tp-restore-1705654428], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-2-84848-01-19-08h18m51s] [2024-01-19T08:54:52.154Z] 2024-01-19 08:54:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-2-84848-01-19-08h18m51s]) found in the restore [tp-restore-1705654428] was also present on the cluster/namespace [mysql-ibm-pxb-2-84848-01-19-08h18m51s] [2024-01-19T08:54:52.154Z] 2024-01-19 08:54:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-2-84848-01-19-08h18m51s]) was found in restore [tp-restore-1705654428], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-2-84848-01-19-08h18m51s] [2024-01-19T08:54:52.154Z] 2024-01-19 08:54:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-2-84848-01-19-08h18m51s]) found in the restore [tp-restore-1705654428] was also present on the cluster/namespace [mysql-ibm-pxb-2-84848-01-19-08h18m51s] [2024-01-19T08:54:52.154Z] 2024-01-19 08:54:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-2-84848-01-19-08h18m51s]) was found in restore [tp-restore-1705654428], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-2-84848-01-19-08h18m51s] [2024-01-19T08:54:52.154Z] 2024-01-19 08:54:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-2-84848-01-19-08h18m51s]) found in the restore [tp-restore-1705654428] was also present on the cluster/namespace [mysql-ibm-pxb-2-84848-01-19-08h18m51s] [2024-01-19T08:54:52.155Z] 2024-01-19 08:54:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-2-84848-01-19-08h18m51s]) was found in restore [tp-restore-1705654428], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-2-84848-01-19-08h18m51s] [2024-01-19T08:54:52.155Z] 2024-01-19 08:54:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-2-84848-01-19-08h18m51s]) found in the restore [tp-restore-1705654428] was also present on the cluster/namespace [mysql-ibm-pxb-2-84848-01-19-08h18m51s] [2024-01-19T08:54:52.155Z] 2024-01-19 08:54:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-2-84848-01-19-08h18m51s] in restore [tp-restore-1705654428] [2024-01-19T08:54:52.155Z] 2024-01-19 08:54:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-2-84848-01-19-08h18m51s] in restore [tp-restore-1705654428] [2024-01-19T08:54:52.411Z] 2024-01-19 08:54:52 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T08:54:52.411Z] 2024-01-19 08:54:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T08:54:52.411Z] 2024-01-19 08:54:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-2-84848-01-19-08h18m51s] are [[mysql-data]] [2024-01-19T08:54:52.411Z] 2024-01-19 08:54:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2240] - in restore [tp-restore-1705654428], the status of the restored volume [pvc-9be9454d-2943-4a6e-ae56-d5870e600683] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-19T08:54:52.411Z] 2024-01-19 08:54:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validate applications in restored namespace [mysql-ibm-pxb-2-84848-01-19-08h18m51s] due to restore [tp-restore-1705654428] [2024-01-19T08:54:52.411Z] 2024-01-19 08:54:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating mysql-ibm app [2024-01-19T08:54:52.411Z] 2024-01-19 08:54:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] wait for mysql-ibm app to start running [2024-01-19T08:54:52.411Z] 2024-01-19 08:54:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T08:54:52.972Z] 2024-01-19 08:54:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T08:54:53.535Z] 2024-01-19 08:54:53 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T08:54:54.466Z] 2024-01-19 08:54:54 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-3-84848-01-19-08h18m51s] in restore [tp-restore-1705654428] [2024-01-19T08:54:54.466Z] 2024-01-19 08:54:54 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-3-84848-01-19-08h18m51s]) was found in restore [tp-restore-1705654428], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-3-84848-01-19-08h18m51s] [2024-01-19T08:54:54.466Z] 2024-01-19 08:54:54 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-3-84848-01-19-08h18m51s]) found in the restore [tp-restore-1705654428] was also present on the cluster/namespace [mysql-ibm-pxb-3-84848-01-19-08h18m51s] [2024-01-19T08:54:54.466Z] 2024-01-19 08:54:54 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-3-84848-01-19-08h18m51s]) was found in restore [tp-restore-1705654428], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-3-84848-01-19-08h18m51s] [2024-01-19T08:54:54.466Z] 2024-01-19 08:54:54 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-3-84848-01-19-08h18m51s]) found in the restore [tp-restore-1705654428] was also present on the cluster/namespace [mysql-ibm-pxb-3-84848-01-19-08h18m51s] [2024-01-19T08:54:54.466Z] 2024-01-19 08:54:54 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-3-84848-01-19-08h18m51s]) was found in restore [tp-restore-1705654428], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-3-84848-01-19-08h18m51s] [2024-01-19T08:54:54.466Z] 2024-01-19 08:54:54 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-3-84848-01-19-08h18m51s]) found in the restore [tp-restore-1705654428] was also present on the cluster/namespace [mysql-ibm-pxb-3-84848-01-19-08h18m51s] [2024-01-19T08:54:54.466Z] 2024-01-19 08:54:54 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-3-84848-01-19-08h18m51s]) was found in restore [tp-restore-1705654428], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-3-84848-01-19-08h18m51s] [2024-01-19T08:54:54.722Z] 2024-01-19 08:54:54 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-3-84848-01-19-08h18m51s]) found in the restore [tp-restore-1705654428] was also present on the cluster/namespace [mysql-ibm-pxb-3-84848-01-19-08h18m51s] [2024-01-19T08:54:54.722Z] 2024-01-19 08:54:54 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-3-84848-01-19-08h18m51s] in restore [tp-restore-1705654428] [2024-01-19T08:54:54.722Z] 2024-01-19 08:54:54 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-3-84848-01-19-08h18m51s] in restore [tp-restore-1705654428] [2024-01-19T08:54:54.722Z] 2024-01-19 08:54:54 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T08:54:54.722Z] 2024-01-19 08:54:54 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T08:54:54.722Z] 2024-01-19 08:54:54 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-3-84848-01-19-08h18m51s] are [[mysql-data]] [2024-01-19T08:54:54.722Z] 2024-01-19 08:54:54 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2240] - in restore [tp-restore-1705654428], the status of the restored volume [pvc-5238f455-03da-4052-8874-07a94276ea0c] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-19T08:54:54.722Z] 2024-01-19 08:54:54 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validate applications in restored namespace [mysql-ibm-pxb-3-84848-01-19-08h18m51s] due to restore [tp-restore-1705654428] [2024-01-19T08:54:54.722Z] 2024-01-19 08:54:54 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating mysql-ibm app [2024-01-19T08:54:54.722Z] 2024-01-19 08:54:54 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] wait for mysql-ibm app to start running [2024-01-19T08:54:54.722Z] 2024-01-19 08:54:54 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T08:54:55.283Z] 2024-01-19 08:54:55 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T08:54:55.846Z] 2024-01-19 08:54:55 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T08:54:56.774Z] 2024-01-19 08:54:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating specs for the namespace (restoredAppContext) [mysql-ibm-pxb-4-84848-01-19-08h18m51s] in restore [tp-restore-1705654428] [2024-01-19T08:54:56.775Z] 2024-01-19 08:54:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2178] - object (name: [mysql-data], GVK: [core,v1,PersistentVolumeClaim], namespace: [mysql-ibm-pxb-4-84848-01-19-08h18m51s]) was found in restore [tp-restore-1705654428], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-4-84848-01-19-08h18m51s] [2024-01-19T08:54:56.775Z] 2024-01-19 08:54:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2194] - object (name: [mysql-data], kind: [PersistentVolumeClaim], namespace: [mysql-ibm-pxb-4-84848-01-19-08h18m51s]) found in the restore [tp-restore-1705654428] was also present on the cluster/namespace [mysql-ibm-pxb-4-84848-01-19-08h18m51s] [2024-01-19T08:54:56.775Z] 2024-01-19 08:54:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [,v1,Service], namespace: [mysql-ibm-pxb-4-84848-01-19-08h18m51s]) was found in restore [tp-restore-1705654428], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-4-84848-01-19-08h18m51s] [2024-01-19T08:54:56.775Z] 2024-01-19 08:54:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Service], namespace: [mysql-ibm-pxb-4-84848-01-19-08h18m51s]) found in the restore [tp-restore-1705654428] was also present on the cluster/namespace [mysql-ibm-pxb-4-84848-01-19-08h18m51s] [2024-01-19T08:54:56.775Z] 2024-01-19 08:54:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2178] - object (name: [mysql], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-4-84848-01-19-08h18m51s]) was found in restore [tp-restore-1705654428], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-4-84848-01-19-08h18m51s] [2024-01-19T08:54:56.775Z] 2024-01-19 08:54:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2194] - object (name: [mysql], kind: [Deployment], namespace: [mysql-ibm-pxb-4-84848-01-19-08h18m51s]) found in the restore [tp-restore-1705654428] was also present on the cluster/namespace [mysql-ibm-pxb-4-84848-01-19-08h18m51s] [2024-01-19T08:54:56.775Z] 2024-01-19 08:54:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2178] - object (name: [mysqlslap], GVK: [apps,v1,Deployment], namespace: [mysql-ibm-pxb-4-84848-01-19-08h18m51s]) was found in restore [tp-restore-1705654428], as expected by presence in expectedRestoredAppContext [mysql-ibm-pxb-4-84848-01-19-08h18m51s] [2024-01-19T08:54:57.031Z] 2024-01-19 08:54:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2194] - object (name: [mysqlslap], kind: [Deployment], namespace: [mysql-ibm-pxb-4-84848-01-19-08h18m51s]) found in the restore [tp-restore-1705654428] was also present on the cluster/namespace [mysql-ibm-pxb-4-84848-01-19-08h18m51s] [2024-01-19T08:54:57.031Z] 2024-01-19 08:54:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating Restored Volumes for the namespace (restoredAppContext) [mysql-ibm-pxb-4-84848-01-19-08h18m51s] in restore [tp-restore-1705654428] [2024-01-19T08:54:57.031Z] 2024-01-19 08:54:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2222] - getting the volumes bounded to the PVCs in the namespace (restoredAppContext) [mysql-ibm-pxb-4-84848-01-19-08h18m51s] in restore [tp-restore-1705654428] [2024-01-19T08:54:57.031Z] 2024-01-19 08:54:56 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T08:54:57.031Z] 2024-01-19 08:54:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T08:54:57.031Z] 2024-01-19 08:54:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2233] - volumes bounded to the PVCs in the context [mysql-ibm-pxb-4-84848-01-19-08h18m51s] are [[mysql-data]] [2024-01-19T08:54:57.031Z] 2024-01-19 08:54:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.ValidateRestore:#2240] - in restore [tp-restore-1705654428], the status of the restored volume [pvc-6131aa2c-26f7-418f-a727-55f9a80d3f50] was Success. It was [Success] with reason [Volume restore successful: PVC mysql-data is bound] [2024-01-19T08:54:57.031Z] 2024-01-19 08:54:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validate applications in restored namespace [mysql-ibm-pxb-4-84848-01-19-08h18m51s] due to restore [tp-restore-1705654428] [2024-01-19T08:54:57.031Z] 2024-01-19 08:54:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Validating mysql-ibm app [2024-01-19T08:54:57.031Z] 2024-01-19 08:54:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] wait for mysql-ibm app to start running [2024-01-19T08:54:57.031Z] 2024-01-19 08:54:56 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T08:54:57.592Z] 2024-01-19 08:54:57 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T08:54:58.521Z] 2024-01-19 08:54:58 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T08:54:59.083Z] 2024-01-19 08:54:58 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Switching cluster context back to cluster path [/tmp/destination-config] [2024-01-19T08:54:59.083Z] 2024-01-19 08:54:58 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T08:54:59.083Z] INFO[2024-01-19 08:54:58] Verifying : Description : Verifying backup restore for tp-restore-1705654428 [2024-01-19T08:54:59.083Z] ERRO[2024-01-19 08:54:58] ValidateRestore Errors: {in restore [tp-restore-1705654428], said restored volume [pvc-28a6bd8e-b5ac-4847-9b7c-62a661e4c37c] cannot be found in the actual cluster [mysql-ibm-pxb-0-84848-01-19-08h18m51s]} [2024-01-19T08:54:59.083Z] {in restore [tp-restore-1705654428], said restored volume [pvc-a8c50c0f-b13a-4991-b29b-7998b5e7aa44] cannot be found in the actual cluster [mysql-ibm-pxb-1-84848-01-19-08h18m51s]} [2024-01-19T08:54:59.083Z] {in restore [tp-restore-1705654428], said restored volume [pvc-9be9454d-2943-4a6e-ae56-d5870e600683] cannot be found in the actual cluster [mysql-ibm-pxb-2-84848-01-19-08h18m51s]} [2024-01-19T08:54:59.083Z] {in restore [tp-restore-1705654428], said restored volume [pvc-5238f455-03da-4052-8874-07a94276ea0c] cannot be found in the actual cluster [mysql-ibm-pxb-3-84848-01-19-08h18m51s]} [2024-01-19T08:54:59.083Z] {in restore [tp-restore-1705654428], said restored volume [pvc-6131aa2c-26f7-418f-a727-55f9a80d3f50] cannot be found in the actual cluster [mysql-ibm-pxb-4-84848-01-19-08h18m51s]} [2024-01-19T08:54:59.339Z] 2024/01/19 08:54:59 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705652455] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T08:55:11.536Z] 2024/01/19 08:55:09 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705652455] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T08:55:12.902Z] INFO[2024-01-19 08:55:12] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T08:55:12.902Z] INFO[2024-01-19 08:55:12] sched stats: tick interval: 62472 < 1.2s (100%) [2024-01-19T08:55:12.902Z] INFO[2024-01-19 08:55:12] sched stats: tick process durations: 62471 < 500ms (100%) [2024-01-19T08:55:19.450Z] 2024/01/19 08:55:19 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705652455] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T08:55:29.416Z] 2024/01/19 08:55:29 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705652455] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T08:55:39.513Z] 2024/01/19 08:55:39 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705652455] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T08:55:49.469Z] 2024/01/19 08:55:49 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705652455] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T08:55:59.469Z] 2024/01/19 08:55:59 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705652455] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T08:56:11.651Z] 2024/01/19 08:56:09 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705652455] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T08:56:19.734Z] 2024/01/19 08:56:19 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705652455] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T08:56:29.680Z] 2024/01/19 08:56:29 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705652455] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T08:56:39.657Z] 2024/01/19 08:56:39 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705652455] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T08:56:49.605Z] 2024/01/19 08:56:49 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705652455] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T08:56:59.569Z] 2024/01/19 08:56:59 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705652455] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T08:57:11.731Z] 2024/01/19 08:57:09 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705652455] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T08:57:19.819Z] 2024/01/19 08:57:19 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705652455] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T08:57:29.759Z] 2024/01/19 08:57:29 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705652455] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T08:57:39.710Z] 2024/01/19 08:57:39 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705652455] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T08:57:49.645Z] INFO[2024-01-19 08:57:49] Verifying : Description : Verification of deleting backup schedule - tp-backup-schedule-1705652455 [2024-01-19T08:57:49.645Z] INFO[2024-01-19 08:57:49] Actual:true, Expected: true [2024-01-19T08:58:57.277Z] INFO[2024-01-19 08:58:49] Verifying : Description : Deleting backup schedule policies [periodic-1705652453] [2024-01-19T08:58:57.277Z] INFO[2024-01-19 08:58:49] Actual:true, Expected: true [2024-01-19T08:58:57.277Z] 2024-01-19 08:58:49 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Deleting labels from namespaces- [mysql-ibm-pxb-0-84848-01-19-08h18m51s mysql-ibm-pxb-1-84848-01-19-08h18m51s mysql-ibm-pxb-2-84848-01-19-08h18m51s mysql-ibm-pxb-3-84848-01-19-08h18m51s mysql-ibm-pxb-4-84848-01-19-08h18m51s] [2024-01-19T08:58:57.277Z] INFO[2024-01-19 08:58:51] Verifying : Description : Deleting labels [map[0-c6da2d74-ce5b-4ce3-8d03-77e5936c0590:c6da2d74-ce5b-4ce3-8d03-77e5936c0590 1-c6da2d74-ce5b-4ce3-8d03-77e5936c0590:c6da2d74-ce5b-4ce3-8d03-77e5936c0590 2-c6da2d74-ce5b-4ce3-8d03-77e5936c0590:c6da2d74-ce5b-4ce3-8d03-77e5936c0590]] from namespaces [[mysql-ibm-pxb-0-84848-01-19-08h18m51s mysql-ibm-pxb-1-84848-01-19-08h18m51s mysql-ibm-pxb-2-84848-01-19-08h18m51s mysql-ibm-pxb-3-84848-01-19-08h18m51s mysql-ibm-pxb-4-84848-01-19-08h18m51s]] [2024-01-19T08:58:57.277Z] INFO[2024-01-19 08:58:51] Actual:true, Expected: true [2024-01-19T08:58:57.277Z] 2024-01-19 08:58:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Deleting deployed namespaces - [mysql-ibm-pxb-0-84848-01-19-08h18m51s mysql-ibm-pxb-1-84848-01-19-08h18m51s mysql-ibm-pxb-2-84848-01-19-08h18m51s mysql-ibm-pxb-3-84848-01-19-08h18m51s mysql-ibm-pxb-4-84848-01-19-08h18m51s] [2024-01-19T08:58:57.277Z] 2024-01-19 08:58:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Destroying apps [2024-01-19T08:58:57.277Z] 2024-01-19 08:58:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T08:58:57.277Z] 2024-01-19 08:58:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T08:58:57.277Z] 2024-01-19 08:58:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] start destroying mysql-ibm app [2024-01-19T08:58:57.277Z] 2024-01-19 08:58:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T08:58:57.277Z] 2024-01-19 08:58:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T08:58:57.277Z] 2024-01-19 08:58:51 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T08:58:57.277Z] INFO[2024-01-19 08:58:51] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T08:58:57.277Z] INFO[2024-01-19 08:58:51] Actual:true, Expected: true [2024-01-19T08:58:57.278Z] 2024-01-19 08:58:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T08:58:57.278Z] 2024-01-19 08:58:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T08:58:57.278Z] 2024-01-19 08:58:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] start destroying mysql-ibm app [2024-01-19T08:58:57.278Z] 2024-01-19 08:58:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T08:58:57.278Z] 2024-01-19 08:58:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T08:58:57.278Z] 2024-01-19 08:58:52 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T08:58:57.278Z] INFO[2024-01-19 08:58:52] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T08:58:57.278Z] INFO[2024-01-19 08:58:52] Actual:true, Expected: true [2024-01-19T08:58:57.278Z] 2024-01-19 08:58:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T08:58:57.278Z] 2024-01-19 08:58:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T08:58:57.278Z] 2024-01-19 08:58:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] start destroying mysql-ibm app [2024-01-19T08:58:57.278Z] 2024-01-19 08:58:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T08:58:57.278Z] 2024-01-19 08:58:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T08:58:57.278Z] 2024-01-19 08:58:52 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T08:58:57.278Z] INFO[2024-01-19 08:58:52] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T08:58:57.278Z] INFO[2024-01-19 08:58:52] Actual:true, Expected: true [2024-01-19T08:58:57.278Z] 2024-01-19 08:58:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T08:58:57.278Z] 2024-01-19 08:58:53 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T08:58:57.278Z] 2024-01-19 08:58:53 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] start destroying mysql-ibm app [2024-01-19T08:58:57.278Z] 2024-01-19 08:58:53 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T08:58:57.278Z] 2024-01-19 08:58:53 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T08:58:57.278Z] 2024-01-19 08:58:53 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T08:58:57.278Z] INFO[2024-01-19 08:58:53] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T08:58:57.278Z] INFO[2024-01-19 08:58:53] Actual:true, Expected: true [2024-01-19T08:58:57.278Z] 2024-01-19 08:58:53 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T08:58:57.278Z] 2024-01-19 08:58:53 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T08:58:57.278Z] 2024-01-19 08:58:53 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] start destroying mysql-ibm app [2024-01-19T08:58:57.279Z] 2024-01-19 08:58:54 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T08:58:57.279Z] 2024-01-19 08:58:54 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T08:58:57.279Z] 2024-01-19 08:58:54 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T08:58:57.279Z] INFO[2024-01-19 08:58:54] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T08:58:57.279Z] INFO[2024-01-19 08:58:54] Actual:true, Expected: true [2024-01-19T08:58:57.279Z] 2024-01-19 08:58:54 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Cleaning backup locations in map [map[c20cf4aa-9a59-4e6b-befc-077efddfed8f:autogenerated-backup-location-1705652444]], cloud credential [autogenerated-cred-1705652444], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T08:58:57.279Z] INFO[2024-01-19 08:58:54] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-1705652444] [2024-01-19T08:58:57.279Z] INFO[2024-01-19 08:58:54] Actual:true, Expected: true [2024-01-19T08:58:57.279Z] 2024-01-19 08:58:54 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705652444] is present [2024-01-19T08:58:57.279Z] 2024/01/19 08:58:54 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705652444 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T08:59:29.316Z] 2024-01-19 08:59:24 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705652444] is present [2024-01-19T08:59:29.316Z] 2024/01/19 08:59:24 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705652444 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T08:59:55.854Z] 2024-01-19 08:59:54 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 aws-global-aws-px-backup-system-test-4cac76dc-bl autogenerated-backup-location-1705628002 autogenerated-backup-location-1705634221 autogenerated-backup-location-1705635982 autogenerated-backup-location-wkyfv] [2024-01-19T08:59:55.854Z] INFO[2024-01-19 08:59:54] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-1705652444 [2024-01-19T08:59:55.854Z] INFO[2024-01-19 08:59:54] Actual:true, Expected: true [2024-01-19T08:59:55.854Z] 2024-01-19 08:59:54 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-1705652444] is present [2024-01-19T08:59:55.854Z] INFO[2024-01-19 08:59:54] Verifying : Description : Verifying if cloud cred [autogenerated-cred-1705652444] is present [2024-01-19T08:59:55.854Z] INFO[2024-01-19 08:59:54] Actual:true, Expected: true [2024-01-19T08:59:55.854Z] INFO[2024-01-19 08:59:54] Verifying : Description : Verifying deletion of cloud cred [autogenerated-cred-1705652444] [2024-01-19T08:59:55.854Z] INFO[2024-01-19 08:59:54] Actual:true, Expected: true [2024-01-19T08:59:55.854Z] INFO[2024-01-19 08:59:54] Verifying : Description : Deleting cloud cred autogenerated-cred-1705652444 [2024-01-19T08:59:55.854Z] INFO[2024-01-19 08:59:54] Actual:true, Expected: true [2024-01-19T08:59:55.854Z] INFO[2024-01-19 08:59:54] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T08:59:55.854Z] INFO[2024-01-19 08:59:54] Actual:true, Expected: true [2024-01-19T08:59:55.854Z] INFO[2024-01-19 08:59:54] Verifying : Description : Deleting cluster source-cluster [2024-01-19T08:59:55.854Z] ERRO[2024-01-19 08:59:54] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl], cannot be deleted, Expected:  [2024-01-19T08:59:55.854Z] INFO[2024-01-19 08:59:55] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-hrfiy] [2024-01-19T08:59:55.854Z] ERRO[2024-01-19 08:59:55] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-hrfiy:72b1b409-7bb1-4f22-92df-0ba5e77c11d3], Expected:  [2024-01-19T08:59:55.854Z] INFO[2024-01-19 08:59:55] Verifying : Description : Deleting cluster source-cluster [2024-01-19T08:59:55.854Z] ERRO[2024-01-19 08:59:55] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh], cannot be deleted, Expected:  [2024-01-19T08:59:55.854Z] INFO[2024-01-19 08:59:55] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-fbvvo] [2024-01-19T08:59:55.854Z] ERRO[2024-01-19 08:59:55] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-fbvvo:7cddb5aa-c7ff-4044-9d27-458be2bd5cde], Expected:  [2024-01-19T08:59:55.854Z] INFO[2024-01-19 08:59:55] Verifying : Description : Deleting cluster source-cluster [2024-01-19T08:59:55.854Z] ERRO[2024-01-19 08:59:55] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo], cannot be deleted, Expected:  [2024-01-19T08:59:57.219Z] INFO[2024-01-19 08:59:57] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-enzij] [2024-01-19T08:59:57.219Z] ERRO[2024-01-19 08:59:57] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-enzij:4e0ddfc7-8811-4ed5-8e87-ed0841582111], Expected:  [2024-01-19T08:59:57.474Z] INFO[2024-01-19 08:59:57] Verifying : Description : Deleting cluster source-cluster [2024-01-19T08:59:57.474Z] ERRO[2024-01-19 08:59:57] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo], cannot be deleted, Expected:  [2024-01-19T08:59:58.839Z] INFO[2024-01-19 08:59:58] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-wcpfm] [2024-01-19T08:59:58.839Z] ERRO[2024-01-19 08:59:58] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-wcpfm:c5ff084a-a913-4412-99fb-e2b0a003a014], Expected:  [2024-01-19T08:59:58.839Z] INFO[2024-01-19 08:59:58] Verifying : Description : Deleting cluster source-cluster [2024-01-19T08:59:58.839Z] ERRO[2024-01-19 08:59:58] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx], cannot be deleted, Expected:  [2024-01-19T09:00:00.203Z] INFO[2024-01-19 08:59:59] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-vlbwu] [2024-01-19T09:00:00.203Z] ERRO[2024-01-19 08:59:59] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-vlbwu:f0465bb5-2565-4948-bb27-7e45a4b883ff], Expected:  [2024-01-19T09:00:00.203Z] INFO[2024-01-19 09:00:00] Verifying : Description : Deleting cluster source-cluster [2024-01-19T09:00:00.203Z] ERRO[2024-01-19 09:00:00] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa], cannot be deleted, Expected:  [2024-01-19T09:00:01.570Z] INFO[2024-01-19 09:00:01] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-mhxzi] [2024-01-19T09:00:01.570Z] ERRO[2024-01-19 09:00:01] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-mhxzi:f551fb81-1a78-4d7f-b804-759c6e6b562e], Expected:  [2024-01-19T09:00:01.826Z] INFO[2024-01-19 09:00:01] Verifying : Description : Deleting cluster source-cluster [2024-01-19T09:00:01.826Z] INFO[2024-01-19 09:00:01] Actual:true, Expected: true [2024-01-19T09:00:03.190Z] INFO[2024-01-19 09:00:02] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-llcnj] [2024-01-19T09:00:03.190Z] INFO[2024-01-19 09:00:02] Actual:true, Expected: true [2024-01-19T09:00:03.190Z] INFO[2024-01-19 09:00:03] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T09:00:03.190Z] INFO[2024-01-19 09:00:03] Actual:true, Expected: true [2024-01-19T09:00:04.556Z] INFO[2024-01-19 09:00:04] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-fiomu] [2024-01-19T09:00:04.556Z] INFO[2024-01-19 09:00:04] Actual:true, Expected: true [2024-01-19T09:00:04.556Z] INFO[2024-01-19 09:00:04] --------Test End------ [2024-01-19T09:00:04.556Z] INFO[2024-01-19 09:00:04] #Test: [2024-01-19T09:00:04.556Z] INFO[2024-01-19 09:00:04] #Description: [2024-01-19T09:00:04.556Z] INFO[2024-01-19 09:00:04] ------------------------ [2024-01-19T09:00:04.556Z] 2024-01-19 09:00:04 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.EndPxBackupTorpedoTest:#7261] - >>>> FAILED TEST: {ScheduleBackupWithAdditionAndRemovalOfNS} Perform schedule backup during which remove and add namespace and verify restoration of removed namespace [2024-01-19T09:00:04.556Z] 2024-01-19 09:00:04 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-84848-01-19-08h18m51s] [2024-01-19T09:00:04.556Z] 2024-01-19 09:00:04 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-1-84848-01-19-08h18m51s] [2024-01-19T09:00:04.812Z] 2024-01-19 09:00:04 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-2-84848-01-19-08h18m51s] [2024-01-19T09:00:05.067Z] 2024-01-19 09:00:04 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-3-84848-01-19-08h18m51s] [2024-01-19T09:00:05.322Z] 2024-01-19 09:00:05 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-4-84848-01-19-08h18m51s] [2024-01-19T09:00:05.322Z] 2024-01-19 09:00:05 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T09:00:05.322Z] 2024-01-19 09:00:05 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T09:00:05.579Z] 2024-01-19 09:00:05 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-84848-01-19-08h18m51s] [2024-01-19T09:00:05.839Z] 2024-01-19 09:00:05 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-1-84848-01-19-08h18m51s] [2024-01-19T09:00:06.094Z] 2024-01-19 09:00:05 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-2-84848-01-19-08h18m51s] [2024-01-19T09:00:06.095Z] 2024-01-19 09:00:06 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-3-84848-01-19-08h18m51s] [2024-01-19T09:00:06.355Z] 2024-01-19 09:00:06 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-4-84848-01-19-08h18m51s] [2024-01-19T09:00:06.611Z] 2024-01-19 09:00:06 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {ScheduleBackupWithAdditionAndRemovalOfNS} Perform schedule backup during which remove and add namespace and verify restoration of removed namespace [2024-01-19T09:00:06.611Z] 2024-01-19 09:00:06 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T09:00:06.611Z] 2024/01/19 09:00:06 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T09:00:11.854Z] 2024/01/19 09:00:11 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T09:00:17.105Z] 2024/01/19 09:00:16 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T09:00:22.348Z] 2024/01/19 09:00:21 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T09:00:26.514Z] 2024/01/19 09:00:26 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T09:00:31.764Z] 2024/01/19 09:00:31 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T09:00:37.006Z] 2024/01/19 09:00:36 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T09:00:42.254Z] 2024/01/19 09:00:41 DoRetryWithTimeout - Error: {dial tcp 10.240.0.7:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T09:00:47.501Z] 2024/01/19 09:00:46 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.0.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.7 10.240.0.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T09:00:47.501Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T09:00:47.501Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T09:00:47.501Z] {dial tcp 10.240.0.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T09:00:51.667Z] 2024-01-19 09:00:51 +0000:[WARNING] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.7 10.240.0.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T09:00:51.667Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T09:00:51.667Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T09:00:51.667Z] {dial tcp 10.240.0.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T09:00:51.668Z] 2024-01-19 09:00:51 +0000:[ERROR] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.0.7. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.0.7 10.240.0.7]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T09:00:51.668Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T09:00:51.668Z] {dial tcp 10.240.0.7:22: connect: connection refused} [2024-01-19T09:00:51.668Z] {dial tcp 10.240.0.7:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T09:00:51.668Z] 2024-01-19 09:00:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T09:00:51.668Z] 2024-01-19 09:00:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] Switching context to [/tmp/source-config] [2024-01-19T09:00:51.668Z] 2024-01-19 09:00:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T09:00:51.668Z] 2024-01-19 09:00:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T09:00:51.668Z] 2024-01-19 09:00:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T09:00:51.925Z] 2024-01-19 09:00:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T09:00:51.925Z] 2024-01-19 09:00:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T09:00:51.925Z] 2024-01-19 09:00:51 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T09:00:52.487Z] 2024-01-19 09:00:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T09:00:52.745Z] 2024-01-19 09:00:52 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T09:00:53.307Z] 2024-01-19 09:00:53 +0000:[INFO] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T09:00:53.562Z] 2024/01/19 09:00:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:01:03.502Z] 2024/01/19 09:01:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:01:13.460Z] 2024/01/19 09:01:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:01:23.407Z] 2024/01/19 09:01:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:01:35.574Z] 2024/01/19 09:01:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:01:43.664Z] 2024/01/19 09:01:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:01:53.609Z] 2024/01/19 09:01:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:02:03.549Z] 2024/01/19 09:02:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:02:15.721Z] 2024/01/19 09:02:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:02:23.804Z] 2024/01/19 09:02:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:02:33.740Z] 2024/01/19 09:02:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:02:43.843Z] 2024/01/19 09:02:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:02:53.787Z] 2024/01/19 09:02:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:03:03.890Z] 2024/01/19 09:03:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:03:13.832Z] 2024/01/19 09:03:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:03:23.772Z] 2024/01/19 09:03:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:03:33.766Z] 2024/01/19 09:03:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:03:43.721Z] 2024/01/19 09:03:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:03:55.894Z] 2024/01/19 09:03:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:04:03.973Z] 2024/01/19 09:04:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:04:13.921Z] 2024/01/19 09:04:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:04:23.918Z] 2024/01/19 09:04:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:04:33.866Z] 2024/01/19 09:04:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:04:46.039Z] 2024/01/19 09:04:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:04:54.123Z] 2024/01/19 09:04:53 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:05:04.077Z] 2024/01/19 09:05:03 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:05:14.026Z] INFO[2024-01-19 09:05:13] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T09:05:14.026Z] INFO[2024-01-19 09:05:13] sched stats: tick interval: 63073 < 1.2s (100%) [2024-01-19T09:05:14.026Z] INFO[2024-01-19 09:05:13] sched stats: tick process durations: 63072 < 500ms (100%) [2024-01-19T09:05:14.027Z] 2024/01/19 09:05:13 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:05:23.967Z] 2024/01/19 09:05:23 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:05:36.138Z] 2024/01/19 09:05:33 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:05:44.232Z] 2024/01/19 09:05:43 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:05:54.183Z] 2024-01-19 09:05:53 +0000:[ERROR] [{ScheduleBackupWithAdditionAndRemovalOfNS}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.183Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.183Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.183Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.183Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.183Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.183Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.183Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.183Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.183Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.183Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.183Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.183Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.184Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.184Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.184Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.184Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.184Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.184Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.184Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.184Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.184Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.184Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.184Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.184Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.184Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.184Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.184Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.184Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:05:54.184Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T09:05:54.184Z] [2024-01-19T09:05:54.184Z] ------------------------------ [2024-01-19T09:05:54.184Z] • Failure [2822.155 seconds] [2024-01-19T09:05:54.184Z] {ScheduleBackupWithAdditionAndRemovalOfNS} [2024-01-19T09:05:54.185Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_namespace_labelled_test.go:629 [2024-01-19T09:05:54.185Z] Perform schedule backup during which remove and add namespace and verify restoration of removed namespace [It] [2024-01-19T09:05:54.185Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_namespace_labelled_test.go:673 [2024-01-19T09:05:54.185Z] [2024-01-19T09:05:54.185Z] Unexpected error: [2024-01-19T09:05:54.185Z] <*errors.errorString | 0xc00181b1a0>: { [2024-01-19T09:05:54.185Z] s: "error: ValidateRestore Errors: {in restore [tp-restore-1705654428], said restored volume [pvc-28a6bd8e-b5ac-4847-9b7c-62a661e4c37c] cannot be found in the actual cluster [mysql-ibm-pxb-0-84848-01-19-08h18m51s]}\n{in restore [tp-restore-1705654428], said restored volume [pvc-a8c50c0f-b13a-4991-b29b-7998b5e7aa44] cannot be found in the actual cluster [mysql-ibm-pxb-1-84848-01-19-08h18m51s]}\n{in restore [tp-restore-1705654428], said restored volume [pvc-9be9454d-2943-4a6e-ae56-d5870e600683] cannot be found in the actual cluster [mysql-ibm-pxb-2-84848-01-19-08h18m51s]}\n{in restore [tp-restore-1705654428], said restored volume [pvc-5238f455-03da-4052-8874-07a94276ea0c] cannot be found in the actual cluster [mysql-ibm-pxb-3-84848-01-19-08h18m51s]}\n{in restore [tp-restore-1705654428], said restored volume [pvc-6131aa2c-26f7-418f-a727-55f9a80d3f50] cannot be found in the actual cluster [mysql-ibm-pxb-4-84848-01-19-08h18m51s]}, Description: Verifying backup restore for tp-restore-1705654428", [2024-01-19T09:05:54.185Z] } [2024-01-19T09:05:54.185Z] error: ValidateRestore Errors: {in restore [tp-restore-1705654428], said restored volume [pvc-28a6bd8e-b5ac-4847-9b7c-62a661e4c37c] cannot be found in the actual cluster [mysql-ibm-pxb-0-84848-01-19-08h18m51s]} [2024-01-19T09:05:54.185Z] {in restore [tp-restore-1705654428], said restored volume [pvc-a8c50c0f-b13a-4991-b29b-7998b5e7aa44] cannot be found in the actual cluster [mysql-ibm-pxb-1-84848-01-19-08h18m51s]} [2024-01-19T09:05:54.185Z] {in restore [tp-restore-1705654428], said restored volume [pvc-9be9454d-2943-4a6e-ae56-d5870e600683] cannot be found in the actual cluster [mysql-ibm-pxb-2-84848-01-19-08h18m51s]} [2024-01-19T09:05:54.185Z] {in restore [tp-restore-1705654428], said restored volume [pvc-5238f455-03da-4052-8874-07a94276ea0c] cannot be found in the actual cluster [mysql-ibm-pxb-3-84848-01-19-08h18m51s]} [2024-01-19T09:05:54.185Z] {in restore [tp-restore-1705654428], said restored volume [pvc-6131aa2c-26f7-418f-a727-55f9a80d3f50] cannot be found in the actual cluster [mysql-ibm-pxb-4-84848-01-19-08h18m51s]}, Description: Verifying backup restore for tp-restore-1705654428 [2024-01-19T09:05:54.185Z] occurred [2024-01-19T09:05:54.185Z] [2024-01-19T09:05:54.185Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-19T09:05:54.185Z] [2024-01-19T09:05:54.185Z] Full Stack Trace [2024-01-19T09:05:54.185Z] github.com/portworx/torpedo/pkg/aetosutil.(*Dashboard).VerifyFatal(0x7f25b55?, {0x702af20?, 0xc00181af60}, {0x0?, 0x0}, {0xc00188a180, 0x32}) [2024-01-19T09:05:54.185Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 +0x143 [2024-01-19T09:05:54.185Z] github.com/portworx/torpedo/tests/backup.glob..func21.2.9() [2024-01-19T09:05:54.185Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_namespace_labelled_test.go:767 +0x459 [2024-01-19T09:05:54.185Z] github.com/portworx/torpedo/tests/backup.glob..func21.2() [2024-01-19T09:05:54.185Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_namespace_labelled_test.go:760 +0xa29 [2024-01-19T09:05:54.185Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T09:05:54.185Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T09:05:54.185Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T09:05:54.185Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T09:05:54.185Z] created by testing.(*T).Run [2024-01-19T09:05:54.185Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T09:05:54.185Z] ------------------------------ [2024-01-19T09:05:54.185Z] STEP: schedule applications [2024-01-19T09:05:54.185Z] STEP: schedule applications [2024-01-19T09:05:54.185Z] STEP: schedule applications [2024-01-19T09:05:54.185Z] STEP: schedule applications [2024-01-19T09:05:54.186Z] STEP: schedule applications [2024-01-19T09:05:54.186Z] STEP: Validate applications [2024-01-19T09:05:54.186Z] STEP: validate applications [2024-01-19T09:05:54.186Z] STEP: validate mysql-ibm app's volumes [2024-01-19T09:05:54.186Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T09:05:54.186Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T09:05:54.186Z] STEP: get mysql-ibm app's volume: pvc-243cb06a-b1ed-4a50-88cc-dc64336ab9ab inspected by the volume driver [2024-01-19T09:05:54.186Z] STEP: wait for mysql-ibm app to start running [2024-01-19T09:05:54.186Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T09:05:54.186Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T09:05:54.186Z] STEP: Validate Px pod restart count [2024-01-19T09:05:54.186Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T09:05:54.186Z] STEP: validate mysql-ibm app's volumes [2024-01-19T09:05:54.186Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T09:05:54.186Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T09:05:54.186Z] STEP: get mysql-ibm app's volume: pvc-95d70ae5-7e16-4c07-9f1a-959ae4ee07f4 inspected by the volume driver [2024-01-19T09:05:54.186Z] STEP: wait for mysql-ibm app to start running [2024-01-19T09:05:54.186Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T09:05:54.186Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T09:05:54.186Z] STEP: Validate Px pod restart count [2024-01-19T09:05:54.186Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T09:05:54.186Z] STEP: validate mysql-ibm app's volumes [2024-01-19T09:05:54.186Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T09:05:54.186Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T09:05:54.186Z] STEP: get mysql-ibm app's volume: pvc-cdf4d9a1-900b-496d-922a-ed052aea120f inspected by the volume driver [2024-01-19T09:05:54.186Z] STEP: wait for mysql-ibm app to start running [2024-01-19T09:05:54.186Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T09:05:54.186Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T09:05:54.186Z] STEP: Validate Px pod restart count [2024-01-19T09:05:54.186Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T09:05:54.186Z] STEP: validate mysql-ibm app's volumes [2024-01-19T09:05:54.186Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T09:05:54.186Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T09:05:54.186Z] STEP: get mysql-ibm app's volume: pvc-c1fbe809-b984-4f36-8bf4-e43de3dd4c61 inspected by the volume driver [2024-01-19T09:05:54.186Z] STEP: wait for mysql-ibm app to start running [2024-01-19T09:05:54.186Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T09:05:54.186Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T09:05:54.186Z] STEP: Validate Px pod restart count [2024-01-19T09:05:54.186Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T09:05:54.186Z] STEP: validate mysql-ibm app's volumes [2024-01-19T09:05:54.186Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T09:05:54.186Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T09:05:54.186Z] STEP: get mysql-ibm app's volume: pvc-3f33e25b-0edd-4fa1-9b0d-76acb9b9fa94 inspected by the volume driver [2024-01-19T09:05:54.186Z] STEP: wait for mysql-ibm app to start running [2024-01-19T09:05:54.186Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T09:05:54.186Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T09:05:54.186Z] STEP: Validate Px pod restart count [2024-01-19T09:05:54.186Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T09:05:54.186Z] STEP: Generate and add labels to namespaces [2024-01-19T09:05:54.186Z] STEP: Generating namespace label string from label map [2024-01-19T09:05:54.186Z] STEP: Creating cloud credentials and backup location [2024-01-19T09:05:54.186Z] STEP: Configure source and destination clusters with px-central-admin ctx [2024-01-19T09:05:54.186Z] STEP: Create schedule policy [2024-01-19T09:05:54.186Z] STEP: Creating schedule backup with namespace label [2024-01-19T09:05:54.186Z] STEP: Remove namespace label from namespace and check backup success of next schedule backup [2024-01-19T09:05:54.187Z] STEP: Restore the backup which was taken before the namespace removal and the namespace should be recovered [2024-01-19T09:05:54.187Z] STEP: validate mysql-ibm app's volumes [2024-01-19T09:05:54.187Z] STEP: wait for mysql-ibm app to start running [2024-01-19T09:05:54.187Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T09:05:54.187Z] STEP: Validate Px pod restart count [2024-01-19T09:05:54.187Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T09:05:54.187Z] STEP: validate mysql-ibm app's volumes [2024-01-19T09:05:54.187Z] STEP: wait for mysql-ibm app to start running [2024-01-19T09:05:54.187Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T09:05:54.187Z] STEP: Validate Px pod restart count [2024-01-19T09:05:54.187Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T09:05:54.187Z] STEP: validate mysql-ibm app's volumes [2024-01-19T09:05:54.187Z] STEP: wait for mysql-ibm app to start running [2024-01-19T09:05:54.187Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T09:05:54.187Z] STEP: Validate Px pod restart count [2024-01-19T09:05:54.187Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T09:05:54.187Z] STEP: validate mysql-ibm app's volumes [2024-01-19T09:05:54.187Z] STEP: wait for mysql-ibm app to start running [2024-01-19T09:05:54.187Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T09:05:54.187Z] STEP: Validate Px pod restart count [2024-01-19T09:05:54.187Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T09:05:54.187Z] STEP: validate mysql-ibm app's volumes [2024-01-19T09:05:54.187Z] STEP: wait for mysql-ibm app to start running [2024-01-19T09:05:54.187Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T09:05:54.187Z] STEP: Validate Px pod restart count [2024-01-19T09:05:54.187Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T09:05:54.187Z] STEP: destroy apps [2024-01-19T09:05:54.187Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T09:05:54.187Z] STEP: start destroying mysql-ibm app [2024-01-19T09:05:54.187Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T09:05:54.187Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T09:05:54.187Z] STEP: start destroying mysql-ibm app [2024-01-19T09:05:54.187Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T09:05:54.187Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T09:05:54.187Z] STEP: start destroying mysql-ibm app [2024-01-19T09:05:54.187Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T09:05:54.187Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T09:05:54.187Z] STEP: start destroying mysql-ibm app [2024-01-19T09:05:54.187Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T09:05:54.187Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T09:05:54.187Z] STEP: start destroying mysql-ibm app [2024-01-19T09:05:54.187Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T09:05:54.187Z] SS2024-01-19 09:05:53 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Log Dir: /testresults//BackupScheduleForOldAndNewNS.log [2024-01-19T09:05:54.187Z] INFO[2024-01-19 09:05:53] --------Test Start------ [2024-01-19T09:05:54.187Z] INFO[2024-01-19 09:05:53] #Test: BackupScheduleForOldAndNewNS [2024-01-19T09:05:54.187Z] INFO[2024-01-19 09:05:53] #Description: Schedule backup with old and new namespace using namespace label [2024-01-19T09:05:54.187Z] INFO[2024-01-19 09:05:53] ------------------------ [2024-01-19T09:05:54.187Z] 2024-01-19 09:05:53 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Deploy applications [2024-01-19T09:05:54.187Z] 2024-01-19 09:05:53 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T09:05:54.187Z] 2024-01-19 09:05:53 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T09:05:54.187Z] 2024-01-19 09:05:53 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T09:05:54.187Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T09:05:54.187Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T09:05:54.444Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T09:05:54.445Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T09:05:54.445Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T09:05:54.446Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T09:05:54.446Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T09:05:54.446Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T09:05:54.446Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T09:05:54.446Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T09:05:54.704Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T09:05:54.704Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T09:05:54.704Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T09:05:54.705Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T09:05:54.705Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T09:05:54.706Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T09:05:54.706Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T09:05:54.706Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T09:05:54.706Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T09:05:54.962Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T09:05:54.962Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T09:05:54.962Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T09:05:54.962Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T09:05:54.963Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T09:05:54.963Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T09:05:54.964Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T09:05:54.964Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T09:05:54.964Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Created namespaces [mysql-ibm-pxb-0-84852-01-19-09h05m53s mysql-ibm-pxb-1-84852-01-19-09h05m53s mysql-ibm-pxb-2-84852-01-19-09h05m53s] [2024-01-19T09:05:54.964Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Validate applications [2024-01-19T09:05:54.964Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Validate applications [2024-01-19T09:05:54.964Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Validating mysql-ibm app [2024-01-19T09:05:54.964Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Validating mysql-ibm app's volumes [2024-01-19T09:05:55.220Z] 2024-01-19 09:05:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T09:05:55.220Z] 2024/01/19 09:05:55 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T09:06:05.184Z] 2024/01/19 09:06:05 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T09:06:15.185Z] 2024-01-19 09:06:15 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-0-84852-01-19-09h05m53s [2024-01-19T09:06:15.442Z] 2024-01-19 09:06:15 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T09:06:15.442Z] 2024-01-19 09:06:15 +0000:[WARNING] [{BackupScheduleForOldAndNewNS}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T09:06:15.442Z] 2024-01-19 09:06:15 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] wait for mysql-ibm app to start running [2024-01-19T09:06:15.442Z] 2024-01-19 09:06:15 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T09:06:15.442Z] 2024/01/19 09:06:15 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T09:06:15.443Z] pod name:mysql-7d766cd65c-q9s48 namespace:mysql-ibm-pxb-0-84852-01-19-09h05m53s running:false ready:false node:10.240.64.27 [2024-01-19T09:06:15.443Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T09:06:27.629Z] 2024/01/19 09:06:25 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T09:06:27.629Z] pod name:mysql-7d766cd65c-q9s48 namespace:mysql-ibm-pxb-0-84852-01-19-09h05m53s running:true ready:false node:10.240.64.27 [2024-01-19T09:06:27.629Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T09:06:35.718Z] 2024/01/19 09:06:35 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T09:06:35.718Z] pod name:mysql-7d766cd65c-q9s48 namespace:mysql-ibm-pxb-0-84852-01-19-09h05m53s running:true ready:false node:10.240.64.27 [2024-01-19T09:06:35.718Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T09:06:45.664Z] 2024/01/19 09:06:45 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T09:06:45.664Z] pod name:mysql-7d766cd65c-q9s48 namespace:mysql-ibm-pxb-0-84852-01-19-09h05m53s running:true ready:false node:10.240.64.27 [2024-01-19T09:06:45.664Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T09:06:57.836Z] 2024/01/19 09:06:55 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T09:06:57.836Z] pod name:mysql-7d766cd65c-q9s48 namespace:mysql-ibm-pxb-0-84852-01-19-09h05m53s running:true ready:false node:10.240.64.27 [2024-01-19T09:06:57.836Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T09:07:05.929Z] 2024/01/19 09:07:05 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T09:07:05.929Z] pod name:mysql-7d766cd65c-q9s48 namespace:mysql-ibm-pxb-0-84852-01-19-09h05m53s running:true ready:false node:10.240.64.27 [2024-01-19T09:07:05.929Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T09:07:18.095Z] 2024/01/19 09:07:15 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T09:07:18.095Z] pod name:mysql-7d766cd65c-q9s48 namespace:mysql-ibm-pxb-0-84852-01-19-09h05m53s running:true ready:false node:10.240.64.27 [2024-01-19T09:07:18.095Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T09:07:26.180Z] 2024/01/19 09:07:25 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T09:07:26.180Z] pod name:mysql-7d766cd65c-q9s48 namespace:mysql-ibm-pxb-0-84852-01-19-09h05m53s running:true ready:false node:10.240.64.27 [2024-01-19T09:07:26.180Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T09:07:36.127Z] 2024-01-19 09:07:36 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T09:07:36.386Z] 2024-01-19 09:07:36 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T09:07:36.386Z] 2024-01-19 09:07:36 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] validate if mysql-ibm app's volumes are setup [2024-01-19T09:07:36.386Z] 2024-01-19 09:07:36 +0000:[WARNING] [{BackupScheduleForOldAndNewNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T09:07:36.386Z] 2024-01-19 09:07:36 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T09:07:36.386Z] 2024-01-19 09:07:36 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T09:07:36.386Z] 2024-01-19 09:07:36 +0000:[WARNING] [{BackupScheduleForOldAndNewNS}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T09:07:36.386Z] 2024-01-19 09:07:36 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Validating mysql-ibm app [2024-01-19T09:07:36.386Z] 2024-01-19 09:07:36 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Validating mysql-ibm app's volumes [2024-01-19T09:07:36.386Z] 2024-01-19 09:07:36 +0000:[WARNING] [{BackupScheduleForOldAndNewNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T09:07:36.386Z] 2024-01-19 09:07:36 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T09:07:36.644Z] 2024-01-19 09:07:36 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-1-84852-01-19-09h05m53s [2024-01-19T09:07:36.644Z] 2024-01-19 09:07:36 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T09:07:36.644Z] 2024-01-19 09:07:36 +0000:[WARNING] [{BackupScheduleForOldAndNewNS}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T09:07:36.644Z] 2024-01-19 09:07:36 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] wait for mysql-ibm app to start running [2024-01-19T09:07:36.644Z] 2024-01-19 09:07:36 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T09:07:36.644Z] 2024-01-19 09:07:36 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T09:07:37.573Z] 2024-01-19 09:07:37 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T09:07:38.134Z] 2024-01-19 09:07:37 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] validate if mysql-ibm app's volumes are setup [2024-01-19T09:07:38.134Z] 2024-01-19 09:07:38 +0000:[WARNING] [{BackupScheduleForOldAndNewNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T09:07:38.134Z] 2024-01-19 09:07:38 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T09:07:38.134Z] 2024-01-19 09:07:38 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T09:07:38.134Z] 2024-01-19 09:07:38 +0000:[WARNING] [{BackupScheduleForOldAndNewNS}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T09:07:38.134Z] 2024-01-19 09:07:38 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Validating mysql-ibm app [2024-01-19T09:07:38.134Z] 2024-01-19 09:07:38 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Validating mysql-ibm app's volumes [2024-01-19T09:07:38.134Z] 2024-01-19 09:07:38 +0000:[WARNING] [{BackupScheduleForOldAndNewNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T09:07:38.134Z] 2024-01-19 09:07:38 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T09:07:38.390Z] 2024-01-19 09:07:38 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-pxb-2-84852-01-19-09h05m53s [2024-01-19T09:07:38.390Z] 2024-01-19 09:07:38 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T09:07:38.390Z] 2024-01-19 09:07:38 +0000:[WARNING] [{BackupScheduleForOldAndNewNS}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T09:07:38.390Z] 2024-01-19 09:07:38 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] wait for mysql-ibm app to start running [2024-01-19T09:07:38.390Z] 2024-01-19 09:07:38 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T09:07:38.645Z] 2024-01-19 09:07:38 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T09:07:39.572Z] 2024-01-19 09:07:39 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T09:07:40.133Z] 2024-01-19 09:07:39 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] validate if mysql-ibm app's volumes are setup [2024-01-19T09:07:40.133Z] 2024-01-19 09:07:40 +0000:[WARNING] [{BackupScheduleForOldAndNewNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T09:07:40.133Z] 2024-01-19 09:07:40 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T09:07:40.133Z] 2024-01-19 09:07:40 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T09:07:40.133Z] 2024-01-19 09:07:40 +0000:[WARNING] [{BackupScheduleForOldAndNewNS}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T09:07:40.133Z] 2024-01-19 09:07:40 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Adding labels to namespaces [2024-01-19T09:07:40.389Z] INFO[2024-01-19 09:07:40] Verifying : Description : Adding labels [map[0-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 1-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 10-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 11-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 12-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 13-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 14-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 15-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 16-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 17-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 18-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 19-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 2-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 3-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 4-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 5-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 6-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 7-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 8-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 9-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a]] to namespaces [[mysql-ibm-pxb-0-84852-01-19-09h05m53s mysql-ibm-pxb-1-84852-01-19-09h05m53s mysql-ibm-pxb-2-84852-01-19-09h05m53s]] [2024-01-19T09:07:40.389Z] INFO[2024-01-19 09:07:40] Actual:true, Expected: true [2024-01-19T09:07:40.389Z] 2024-01-19 09:07:40 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Generating namespace label string from label map for namespaces [2024-01-19T09:07:40.390Z] 2024-01-19 09:07:40 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [backup.glob..func19.2.3:#272] - label string for namespaces 1-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,5-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,11-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,12-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,13-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,3-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,4-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,6-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,18-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,2-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,7-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,8-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,9-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,10-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,14-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,16-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,19-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,0-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,15-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,17-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a [2024-01-19T09:07:40.390Z] 2024-01-19 09:07:40 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Adding labels to resources [2024-01-19T09:07:40.390Z] INFO[2024-01-19 09:07:40] Verifying : Description : Fetching pvc list [&PersistentVolumeClaimList{ListMeta:{ 1832781 },Items:[]PersistentVolumeClaim{PersistentVolumeClaim{ObjectMeta:{mysql-data mysql-ibm-pxb-0-84852-01-19-09h05m53s ea35ece2-66a9-4466-8e54-1fdf00cca63a 1831639 0 2024-01-19 09:05:54 +0000 UTC map[] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-19 09:06:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-19 09:06:05 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{2147483648 0} {} 2Gi BinarySI},},},VolumeName:pvc-ea35ece2-66a9-4466-8e54-1fdf00cca63a,Selector:nil,StorageClassName:*mysql-sc,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{10737418240 0} {} 10Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},},},}] from namespace [mysql-ibm-pxb-0-84852-01-19-09h05m53s] [2024-01-19T09:07:40.390Z] INFO[2024-01-19 09:07:40] Actual:true, Expected: true [2024-01-19T09:07:40.650Z] INFO[2024-01-19 09:07:40] Verifying : Description : Fetching pvc pointer &PersistentVolumeClaim{ObjectMeta:{mysql-data mysql-ibm-pxb-0-84852-01-19-09h05m53s ea35ece2-66a9-4466-8e54-1fdf00cca63a 1831639 0 2024-01-19 09:05:54 +0000 UTC map[] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-19 09:06:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-19 09:06:05 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{2147483648 0} {} 2Gi BinarySI},},},VolumeName:pvc-ea35ece2-66a9-4466-8e54-1fdf00cca63a,Selector:nil,StorageClassName:*mysql-sc,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{10737418240 0} {} 10Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},} for pvc {{ } {mysql-data mysql-ibm-pxb-0-84852-01-19-09h05m53s ea35ece2-66a9-4466-8e54-1fdf00cca63a 1831639 0 2024-01-19 09:05:54 +0000 UTC map[] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-19 09:06:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-19 09:06:05 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{2147483648 0} {} 2Gi BinarySI}]} pvc-ea35ece2-66a9-4466-8e54-1fdf00cca63a 0xc0016f7be0 0xc0016f7bf0 nil nil} {Bound [ReadWriteOnce] map[storage:{{10737418240 0} {} 10Gi BinarySI}] [] map[] }} [2024-01-19T09:07:40.650Z] INFO[2024-01-19 09:07:40] Actual:true, Expected: true [2024-01-19T09:07:40.650Z] 2024-01-19 09:07:40 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.AddLabelToResource:#5030] - Adding label [059eb132-a858-4c46-8f77-d637339d05d0=f1086e89-2c30-44b2-906d-336a8bfa3a9a] to PVC mysql-data [2024-01-19T09:07:40.909Z] INFO[2024-01-19 09:07:40] Verifying : Description : Adding labels 059eb132-a858-4c46-8f77-d637339d05d0=f1086e89-2c30-44b2-906d-336a8bfa3a9a to resource &PersistentVolumeClaim{ObjectMeta:{mysql-data mysql-ibm-pxb-0-84852-01-19-09h05m53s ea35ece2-66a9-4466-8e54-1fdf00cca63a 1831639 0 2024-01-19 09:05:54 +0000 UTC map[059eb132-a858-4c46-8f77-d637339d05d0:f1086e89-2c30-44b2-906d-336a8bfa3a9a] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-19 09:06:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-19 09:06:05 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{2147483648 0} {} 2Gi BinarySI},},},VolumeName:pvc-ea35ece2-66a9-4466-8e54-1fdf00cca63a,Selector:nil,StorageClassName:*mysql-sc,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{10737418240 0} {} 10Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},} [2024-01-19T09:07:40.909Z] INFO[2024-01-19 09:07:40] Actual:true, Expected: true [2024-01-19T09:07:40.909Z] INFO[2024-01-19 09:07:40] Verifying : Description : Fetching configmap list [&ConfigMapList{ListMeta:{ 1832792 },Items:[]ConfigMap{ConfigMap{ObjectMeta:{kube-****-ca.crt mysql-ibm-pxb-0-84852-01-19-09h05m53s 46f6918b-b534-424e-95cd-7e32f397a1e3 1831269 0 2024-01-19 09:05:53 +0000 UTC map[] map[kubernetes.io/description:Contains a CA bundle that can be used to verify the kube-apiserver when using internal endpoints such as the internal service IP or kubernetes.default.svc. No other usage is guaranteed across distributions of Kubernetes clusters.] [] [] [{kube-controller-manager Update v1 2024-01-19 09:05:53 +0000 UTC FieldsV1 {"f:data":{".":{},"f:ca.crt":{}},"f:metadata":{"f:annotations":{".":{},"f:kubernetes.io/description":{}}}} }]},Data:map[string]string{ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T09:07:40.909Z] MIIDCjCCAfKgAwIBAgIUWkn6rgBaC+VWV1XdJjGA3DwoRRYwDQYJKoZIhvcNAQEL [2024-01-19T09:07:40.909Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:40.909Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:40.909Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:40.909Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:40.909Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:40.909Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:40.909Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:40.909Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:40.909Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:40.909Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAHbD [2024-01-19T09:07:40.909Z] 4nWTgt/8sat7SbbHxZ+JcMGRAIaeLAjMJxkxyqD8xUq6YQcgbjDVHo1Qbh3PXB3T [2024-01-19T09:07:40.909Z] 5BWUsRbDoKi0eRJx/5cyI3846SYEAOpuUfJrg9X2CH7ZoYhxeJ42dqt3MdoMJEM1 [2024-01-19T09:07:40.909Z] k3e2rRqAKuAxwNXugHryxQM+1ehOJjrEuT5DVK6qu8+JYRcT1hBZb6hrebKPp5eG [2024-01-19T09:07:40.909Z] JvBzHP4YOATIAXhfH0Uj5Y+JhwoOdpXmOFpz+6gITPD4aNzzJZe/0ADSFfOyUJaR [2024-01-19T09:07:40.909Z] g5NnJnkeGqMM3SLsur25r+m3y0i3t+RdocZ/4mHTEnF4KWVWvZy4GjENiEBcXEol [2024-01-19T09:07:40.909Z] o4SZVaP4AF7XQJ5vdA0= [2024-01-19T09:07:40.909Z] -----END CERTIFICATE----- [2024-01-19T09:07:40.909Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:40.909Z] MIIDCjCCAfKgAwIBAgIUXhIIFCc/NxZJn6hNEFoZuPQ8N5UwDQYJKoZIhvcNAQEL [2024-01-19T09:07:40.909Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:40.909Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:40.909Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:40.909Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:40.909Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:40.909Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:40.909Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:40.909Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:40.909Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:40.909Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAKG3 [2024-01-19T09:07:40.909Z] nVfv4IkgHVOxgbrNIoYhjvBvVJcLZub0C9yidexLA3A61p1a7b/OpidG5zShPhH2 [2024-01-19T09:07:40.909Z] HRdQcqgp0HIxaS+/oLD3eElOnnC/EKD359K6n3kN66sMgvNbHxNFM84Au0TXqmbT [2024-01-19T09:07:40.909Z] w8afrKjrfe/VglpHC2GfmgHMF4YH+ZF48C1RZA6z2oXj6Aj+uww0PjiClE9qUYqK [2024-01-19T09:07:40.909Z] R9uRa32prExihsi+Bto7n2Axo92qUiK2KrkEdQitnQfH5z/PojS78vbyIKp1O86N [2024-01-19T09:07:40.909Z] qRImvCYY17CPxP+JUQ7dxWDMF4NTgSnabkp+cd5i/pyoAb0guQFlQQUnHAZa4QNC [2024-01-19T09:07:40.909Z] 7fgTo1IUwWyDmwCnN5U= [2024-01-19T09:07:40.909Z] -----END CERTIFICATE----- [2024-01-19T09:07:40.909Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:40.909Z] MIIDCjCCAfKgAwIBAgIUJPzO3O+wmUt4gHCkl3ojsvwZL6owDQYJKoZIhvcNAQEL [2024-01-19T09:07:40.909Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:40.909Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:40.909Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:40.909Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:40.910Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:40.910Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:40.910Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:40.910Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:40.910Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:40.910Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAGVt [2024-01-19T09:07:40.910Z] g9qX+r8kyGYsvFavmnDxgMK1Zkz80irQJ2BJlDHdK1hp5aqpRbT2z2YgLog5ZMIZ [2024-01-19T09:07:40.910Z] uI36uPCOXo8+Gol1jT7D6vsgMWm0Tk9jCNbO4jPkv+wm2q1B9UZo94UGiS0K+G0S [2024-01-19T09:07:40.910Z] nccl4jmSq1gWpBZIVetdDFNjpipKjhEOVQM7x33APhACxSAWRp9iEr6V/CzMOgkf [2024-01-19T09:07:40.910Z] RHyz5D7ljh4a2XVH7++eKsj2QPfKyiKtIG72tCccQ9zGnzveiXXb4Ai8r+DyKH4Z [2024-01-19T09:07:40.910Z] zDt5Hb4ZFl72E3Rl4bIh/VQBVsr9u/SFR1p/Emyq+kGXsn+9YMiM9tK4mLUtjdox [2024-01-19T09:07:40.910Z] LMWUWTelNRGWAiaTVz8= [2024-01-19T09:07:40.910Z] -----END CERTIFICATE----- [2024-01-19T09:07:40.910Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:40.910Z] MIIDCjCCAfKgAwIBAgIUBxbEoan/HnRZ+2deRj+2RvGMrWQwDQYJKoZIhvcNAQEL [2024-01-19T09:07:40.910Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:40.910Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:40.910Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:40.910Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:40.910Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:40.910Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:40.910Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:40.910Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:40.910Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:40.910Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAB65 [2024-01-19T09:07:40.910Z] nHU+98oTP0U2J0/TsL5GFeBTH7YyUXUNRJvPWxkA8riB3swoScFKbavO6XJ/hrI2 [2024-01-19T09:07:40.910Z] ywApOz77Iyq58OLa5aIC5aaPRf3YE8LbJQYIHmtza21LXhtlKVSrLu4W58PauI8Z [2024-01-19T09:07:40.910Z] 7iaL9QPUihHFK7wrZ2ZpabN2rRJvTGWWKlhmgFTjI+71gPrXWmYBHIKmTv55V5m1 [2024-01-19T09:07:40.910Z] LxORTR/gBXeH9gmM7wz1cbaYNazqs8R1LS5SW76p/y9RLjTzU6N39VYefMMGwbsQ [2024-01-19T09:07:40.910Z] DlD11wvMBxTBp3hQYc3pdVLkVEQt8trVIwEUts7zfwUTCu/z8gH7H5XePpEVNana [2024-01-19T09:07:40.910Z] JhlAcXHCO7nN/ddJmoc= [2024-01-19T09:07:40.910Z] -----END CERTIFICATE----- [2024-01-19T09:07:40.910Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:40.910Z] MIIDGDCCAgCgAwIBAgIUEvLaarpXlv1cVvG6nMuxh2DNCKQwDQYJKoZIhvcNAQEL [2024-01-19T09:07:40.910Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:40.910Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:40.910Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:40.910Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:40.910Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:40.910Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:40.910Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:40.910Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:40.910Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:40.910Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:40.910Z] 9w0BAQsFAAOCAQEAOPakzVUmAqHhmQViZZHSQeRh/xYQBrtn6dnEo6X55SXNWcPU [2024-01-19T09:07:40.910Z] 4EEW3DvwBf75uqxKeGoXHTBpqNHG2x1+IncvIKJmi5OWM3aR/vEyJoBOchPBl+A4 [2024-01-19T09:07:40.910Z] 0xZO1vKiS2djxz8XHMkR70cxD8hv/zDJeIQbgBroF4Wa7rim5+y7jgO0O7Rs1zhT [2024-01-19T09:07:40.910Z] LDDStJ8aUKwcVRmOv7NFQxbQHXNx07kaUE2SQJfhryzZmotoBiymmQT2v0H4/4q0 [2024-01-19T09:07:40.910Z] 5dylRhSCWfwN5+It40/C3b1pbC2d9Iv/eO+B0wqg13dLF3xuA1JblEgnqA9mdNnp [2024-01-19T09:07:40.910Z] P6EgKTQBMiO6nPFyYIPqg1eWYkb1eXu9aEkmbw== [2024-01-19T09:07:40.910Z] -----END CERTIFICATE----- [2024-01-19T09:07:40.910Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:40.910Z] MIIDGDCCAgCgAwIBAgIUWf/qoBANFWF/a/yZr7wfLBMfItAwDQYJKoZIhvcNAQEL [2024-01-19T09:07:40.910Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:40.910Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:40.910Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:40.910Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:40.910Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:40.910Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:40.910Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:40.910Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:40.911Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:40.911Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:40.911Z] 9w0BAQsFAAOCAQEAdCguadZU6WJAuuAtWzvYgftPKkDKNBeGVgbqRwWRCB+QqVFl [2024-01-19T09:07:41.166Z] +3ZOQTStpnXJsY3Ig31hmL88BkBPQ47ENEzs7Czhuz59m+dcBhh/buUoIERSnfUY [2024-01-19T09:07:41.166Z] lWIfqxQevwHYZ3Kjx0l3qliYjKBQJo5xDPnukGb4yUnmrCdZ3fxMzlnr0Ais9rUH [2024-01-19T09:07:41.166Z] MjiDsjP/oUa8naQUTZO4DTRZqeEldzNXnxF6YH+3moTkwRLt9xzirNwECmJ9qxrC [2024-01-19T09:07:41.166Z] eCA7zORxbFSftYLnQiGWUS1dnxBuLOvJHjYPC6EkPPrvqnx2fXcTE1V8GizVRrA6 [2024-01-19T09:07:41.166Z] NPxt+e4qZgU/Nk/Jkpsb7h1FVcKtS8CvmtbdqQ== [2024-01-19T09:07:41.166Z] -----END CERTIFICATE----- [2024-01-19T09:07:41.166Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:41.166Z] MIIDGDCCAgCgAwIBAgIUI6D0WS4tl52N+w2lwAS+6V0DPyUwDQYJKoZIhvcNAQEL [2024-01-19T09:07:41.166Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:41.166Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:41.167Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:41.167Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:41.167Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:41.167Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:41.167Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:41.167Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:41.167Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:41.167Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:41.167Z] 9w0BAQsFAAOCAQEAAS4yElh6TMyP18ipzRA0Wenukevjs/vtkAFOsv1u5GNeDIrv [2024-01-19T09:07:41.167Z] lNi6/VjvtS+DoFNzQnTf3QoO6GTuoWHLQWt6Z/V5AwIsnOXduSXLugMwEv3j9l50 [2024-01-19T09:07:41.167Z] aDkA7vJ3OH5cGqIvIzxQN7IUOEvZNhY67GFP7+zAzVNRpJ1FHYbhQGPuFD//b9G3 [2024-01-19T09:07:41.167Z] hr4E3+iUEc+F/iBA+caA8wRx1zR383Jz6wuoFt7n7HpWNcgMvoh91JlUAK7yUp0B [2024-01-19T09:07:41.167Z] ekuG6q7HtVsk6X+aB1RyWfrSJjih75W7b+p+/bV4vNmQS1wSGSOaSIuVRvMhmpwf [2024-01-19T09:07:41.167Z] FpT+4rUpiCxklD69/MET/1cWxqvao/ppf6phzg== [2024-01-19T09:07:41.167Z] -----END CERTIFICATE----- [2024-01-19T09:07:41.167Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:41.167Z] MIIDGDCCAgCgAwIBAgIUIVCCgYV9enDck18n89YsA5b2IdgwDQYJKoZIhvcNAQEL [2024-01-19T09:07:41.167Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:41.167Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:41.167Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:41.167Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:41.167Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:41.167Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:41.167Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:41.167Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:41.167Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:41.167Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:41.167Z] 9w0BAQsFAAOCAQEAa3eFm1ivFf5Vom6a/7E1gDdjl7cjUZ5cqg332Ky1NdzEOiwW [2024-01-19T09:07:41.167Z] mx3yEk35qCd8LAoFMvPvPIuMx1NDS8kwy3FmIOjKnZapHA31ZUpn/gT/seH4GFIU [2024-01-19T09:07:41.167Z] oZw+rODV5h6jmtCVugAn48u/LxOXbY417JWwaeDx5uw+NzOSyebe+td2PbtUjf5X [2024-01-19T09:07:41.167Z] 6xcwYUmaUGsIAgfIMM12hzXOI2/q1qz5dAqwSgnR7dKBReOXnUoFpNOMpxlJ70p/ [2024-01-19T09:07:41.167Z] WUU1aWIWZqKrNI7Pg1T5jljq7+KbT3+A6+ux0FLExr2c1tw3NndXS2Y9fO55m5/q [2024-01-19T09:07:41.167Z] 0EcrTdKx4+QjbWcja3WdX2pGDk0XGNgx+EXn6Q== [2024-01-19T09:07:41.167Z] -----END CERTIFICATE----- [2024-01-19T09:07:41.167Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:41.167Z] MIIFATCCAumgAwIBAgIRAKc9ZKBASymy5TLOEp57N98wDQYJKoZIhvcNAQELBQAw [2024-01-19T09:07:41.167Z] GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDMyMzIyNTM0NloXDTM2 [2024-01-19T09:07:41.167Z] MDMyMzIyNTM0NlowGjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMIICIjANBgkq [2024-01-19T09:07:41.167Z] hkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA+pYHvQw5iU3v2b3iNuYNKYgsWD6KU7aJ [2024-01-19T09:07:41.167Z] diddtZQxSWYzUI3U0I1UsRPTxnhTifs/M9NW4ZlV13ZfB7APwC8oqKOIiwo7IwlP [2024-01-19T09:07:41.167Z] xg0VKgyz+kT8RJfYr66PPIYP0fpTeu42LpMJ+CKo9sbpgVNDZN2z/qiXrRNX/VtG [2024-01-19T09:07:41.167Z] TkPV7a44fZ5bHHVruAxvDnylpQxJobtCBWlJSsbIRGFHMc2z88eUz9NmIOWUKGGj [2024-01-19T09:07:41.167Z] EmP76x8OfRHpIpuxRSCjn0+i9+hR2siIOpcMOGd+40uVJxbRRP5ZXnUFa2fF5FWd [2024-01-19T09:07:41.167Z] O0u0RPI8HON0ovhrwPJY+4eWKkQzyC611oLPYGQ4EbifRsTsCxUZqyUuStGyp8oa [2024-01-19T09:07:41.167Z] aoSKfF6X0+KzGgwwnrjRTUpIl19A92KR0Noo6h622OX+4sZiO/JQdkuX5w/HupK0 [2024-01-19T09:07:41.167Z] A0M0WSMCvU6GOhjGotmh2VTEJwHHY4+TUk0iQYRtv1crONklyZoAQPD76hCrC8Cr [2024-01-19T09:07:41.167Z] IbgsZLfTMC8TWUoMbyUDgvgYkHKMoPm0VGVVuwpRKJxv7+2wXO+pivrrUl2Q9fPe [2024-01-19T09:07:41.167Z] Kk055nJLMV9yPUdig8othUKrRfSxli946AEV1eEOhxddfEwBE3Lt2xn0hhiIedbb [2024-01-19T09:07:41.167Z] Ftf/5kEWFZkXyUmMJK8Ra76Kus2ABueUVEcZ48hrRr1Hf1N9n59VbTUaXgeiZA50 [2024-01-19T09:07:41.167Z] qXf2bymE6F8CAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMB [2024-01-19T09:07:41.167Z] Af8wHQYDVR0OBBYEFMEmdKSKRKDm+iAo2FwjmkWIGHngMA0GCSqGSIb3DQEBCwUA [2024-01-19T09:07:41.167Z] A4ICAQBCPw74M9X/Xx04K1VAES3ypgQYH5bf9FXVDrwhRFSVckria/7dMzoF5wln [2024-01-19T09:07:41.167Z] uq9NGsjkkkDg17AohcQdr8alH4LvPdxpKr3BjpvEcmbqF8xH+MbbeUEnmbSfLI8H [2024-01-19T09:07:41.167Z] sefuhXF9AF/9iYvpVNC8FmJ0OhiVv13VgMQw0CRKkbtjZBf8xaEhq/YqxWVsgOjm [2024-01-19T09:07:41.167Z] dm5CAQ2X0aX7502x8wYRgMnZhA5goC1zVWBVAi8yhhmlhhoDUfg17cXkmaJC5pDd [2024-01-19T09:07:41.167Z] oenZ9NVhW8eDb03MFCrWNvIh89DDeCGWuWfDltDq0n3owyL0IeSn7RfpSclpxVmV [2024-01-19T09:07:41.167Z] /53jkYjwIgxIG7Gsv0LKMbsf6QdBcTjhvfZyMIpBRkTe3zuHd2feKzY9lEkbRvRQ [2024-01-19T09:07:41.168Z] zbh4Ps5YBnG6CKJPTbe2hfi3nhnw/MyEmF3zb0hzvLWNrR9XW3ibb2oL3424XOwc [2024-01-19T09:07:41.168Z] VjrTSCLzO9Rv6s5wi03qoWvKAQQAElqTYRHhynJ3w6wuvKYF5zcZF3MDnrVGLbh1 [2024-01-19T09:07:41.168Z] Q9ePRFBCiXOQ6wPLoUhrrbZ8LpFUFYDXHMtYM7P9sc9IAWoONXREJaO08zgFtMp4 [2024-01-19T09:07:41.168Z] 8iyIYUyQAbsvx8oD2M8kRvrIRSrRJSl6L957b4AFiLIQ/GgV2curs0jje7Edx34c [2024-01-19T09:07:41.168Z] idWw1VrejtwclobqNMVtG3EiPUIpJGpbMcJgbiLSmKkrvQtGng== [2024-01-19T09:07:41.168Z] -----END CERTIFICATE----- [2024-01-19T09:07:41.168Z] ,},BinaryData:map[string][]byte{},Immutable:nil,},ConfigMap{ObjectMeta:{openshift-service-ca.crt mysql-ibm-pxb-0-84852-01-19-09h05m53s 5932922b-5646-47c0-8608-42b990f51a9a 1831273 0 2024-01-19 09:05:53 +0000 UTC map[] map[service.beta.openshift.io/inject-cabundle:true] [] [] [{kube-controller-manager Update v1 2024-01-19 09:05:53 +0000 UTC FieldsV1 {"f:data":{},"f:metadata":{"f:annotations":{".":{},"f:service.beta.openshift.io/inject-cabundle":{}}}} } {service-ca-operator Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:data":{"f:service-ca.crt":{}}} }]},Data:map[string]string{service-ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T09:07:41.168Z] MIIDUTCCAjmgAwIBAgIILvRjno8BjvIwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE [2024-01-19T09:07:41.168Z] Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwNTM5NjA4ODAe [2024-01-19T09:07:41.168Z] Fw0yNDAxMTYwOTA4MDhaFw0yNjAzMTYwOTA4MDlaMDYxNDAyBgNVBAMMK29wZW5z [2024-01-19T09:07:41.168Z] aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDUzOTYwODgwggEiMA0GCSqG [2024-01-19T09:07:41.168Z] SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVr8xmLKsMoemVtfnlzzil6B8CGocEhNRo [2024-01-19T09:07:41.168Z] fkzkD8aSOYgIj5zcZo/M0O7nXDXeg9dqx9/h2qrGk+SS2ikU2Xs4Q5gaQDh9U1lB [2024-01-19T09:07:41.168Z] y8paoOEvw06CmJ/1Qqur5A539/OnimRSTdejTu4lwaDwsJiyoGfhvrBJAdjagtYA [2024-01-19T09:07:41.168Z] 0sNMrCgEAT4NWHS5EDx5dF7nMDHmQ0h0gHYei6vUEtiPyLwcTH1T7uVSj22fCEXX [2024-01-19T09:07:41.168Z] LjgN3OSiquJ+Hf4+csPpwgxAEkfjdBldhnImYkunJbg/x4c90uFL1pygZusrbFAf [2024-01-19T09:07:41.168Z] cERW8GngZuL5JY8MQoIstMu+1qS390ouQDDbEcvWjxC8YFNFHywrAgMBAAGjYzBh [2024-01-19T09:07:41.168Z] MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQS5vRk [2024-01-19T09:07:41.168Z] jkBPISX2qYmk1VEQZIXSLzAfBgNVHSMEGDAWgBQS5vRkjkBPISX2qYmk1VEQZIXS [2024-01-19T09:07:41.168Z] LzANBgkqhkiG9w0BAQsFAAOCAQEAZ/qc2vFGJZgDJCEUaPYuufuLRPSF68ixV9fk [2024-01-19T09:07:41.168Z] e5tBkRN8atCsEBygC/RUcwULlhsdgUMGMcREgNC/FYlwkOsAj3gP8KmcXAZ4gHCa [2024-01-19T09:07:41.168Z] T6PIbIcj1hCX0mdX0tvtaj7PWvRUz9CHh6iFzmPsXK5xjaBA9vZbkUAa7odguUit [2024-01-19T09:07:41.168Z] vz1QnKoV1OaZyccDLeQL50khM1XEBn8L+zeY3k85uo2X514HZW4LDPTQ94cpyNnb [2024-01-19T09:07:41.168Z] zqI6utp7JoPJexsHMWNFKGb//HgjluNg7TsywXIeebk+/GZUEDfNKid0oiTpQvzR [2024-01-19T09:07:41.168Z] TFFF+5KO/sa4+qIh1ERm140JxeaurUMaJzn7cFQbekEgt/N+iA== [2024-01-19T09:07:41.168Z] -----END CERTIFICATE----- [2024-01-19T09:07:41.168Z] ,},BinaryData:map[string][]byte{},Immutable:nil,},},}] from namespace [mysql-ibm-pxb-0-84852-01-19-09h05m53s] [2024-01-19T09:07:41.168Z] INFO[2024-01-19 09:07:40] Actual:true, Expected: true [2024-01-19T09:07:41.168Z] INFO[2024-01-19 09:07:41] Verifying : Description : Fetching configmap pointer &ConfigMap{ObjectMeta:{kube-****-ca.crt mysql-ibm-pxb-0-84852-01-19-09h05m53s 46f6918b-b534-424e-95cd-7e32f397a1e3 1831269 0 2024-01-19 09:05:53 +0000 UTC map[] map[kubernetes.io/description:Contains a CA bundle that can be used to verify the kube-apiserver when using internal endpoints such as the internal service IP or kubernetes.default.svc. No other usage is guaranteed across distributions of Kubernetes clusters.] [] [] [{kube-controller-manager Update v1 2024-01-19 09:05:53 +0000 UTC FieldsV1 {"f:data":{".":{},"f:ca.crt":{}},"f:metadata":{"f:annotations":{".":{},"f:kubernetes.io/description":{}}}} }]},Data:map[string]string{ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T09:07:41.168Z] MIIDCjCCAfKgAwIBAgIUWkn6rgBaC+VWV1XdJjGA3DwoRRYwDQYJKoZIhvcNAQEL [2024-01-19T09:07:41.168Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:41.168Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:41.168Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:41.168Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:41.168Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:41.168Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:41.168Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:41.168Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:41.169Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:41.169Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAHbD [2024-01-19T09:07:41.169Z] 4nWTgt/8sat7SbbHxZ+JcMGRAIaeLAjMJxkxyqD8xUq6YQcgbjDVHo1Qbh3PXB3T [2024-01-19T09:07:41.169Z] 5BWUsRbDoKi0eRJx/5cyI3846SYEAOpuUfJrg9X2CH7ZoYhxeJ42dqt3MdoMJEM1 [2024-01-19T09:07:41.169Z] k3e2rRqAKuAxwNXugHryxQM+1ehOJjrEuT5DVK6qu8+JYRcT1hBZb6hrebKPp5eG [2024-01-19T09:07:41.169Z] JvBzHP4YOATIAXhfH0Uj5Y+JhwoOdpXmOFpz+6gITPD4aNzzJZe/0ADSFfOyUJaR [2024-01-19T09:07:41.169Z] g5NnJnkeGqMM3SLsur25r+m3y0i3t+RdocZ/4mHTEnF4KWVWvZy4GjENiEBcXEol [2024-01-19T09:07:41.169Z] o4SZVaP4AF7XQJ5vdA0= [2024-01-19T09:07:41.169Z] -----END CERTIFICATE----- [2024-01-19T09:07:41.169Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:41.169Z] MIIDCjCCAfKgAwIBAgIUXhIIFCc/NxZJn6hNEFoZuPQ8N5UwDQYJKoZIhvcNAQEL [2024-01-19T09:07:41.169Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:41.169Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:41.169Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:41.169Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:41.169Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:41.169Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:41.169Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:41.169Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:41.169Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:41.169Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAKG3 [2024-01-19T09:07:41.169Z] nVfv4IkgHVOxgbrNIoYhjvBvVJcLZub0C9yidexLA3A61p1a7b/OpidG5zShPhH2 [2024-01-19T09:07:41.169Z] HRdQcqgp0HIxaS+/oLD3eElOnnC/EKD359K6n3kN66sMgvNbHxNFM84Au0TXqmbT [2024-01-19T09:07:41.169Z] w8afrKjrfe/VglpHC2GfmgHMF4YH+ZF48C1RZA6z2oXj6Aj+uww0PjiClE9qUYqK [2024-01-19T09:07:41.169Z] R9uRa32prExihsi+Bto7n2Axo92qUiK2KrkEdQitnQfH5z/PojS78vbyIKp1O86N [2024-01-19T09:07:41.169Z] qRImvCYY17CPxP+JUQ7dxWDMF4NTgSnabkp+cd5i/pyoAb0guQFlQQUnHAZa4QNC [2024-01-19T09:07:41.169Z] 7fgTo1IUwWyDmwCnN5U= [2024-01-19T09:07:41.169Z] -----END CERTIFICATE----- [2024-01-19T09:07:41.169Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:41.169Z] MIIDCjCCAfKgAwIBAgIUJPzO3O+wmUt4gHCkl3ojsvwZL6owDQYJKoZIhvcNAQEL [2024-01-19T09:07:41.169Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:41.169Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:41.169Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:41.169Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:41.169Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:41.169Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:41.169Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:41.169Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:41.169Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:41.169Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAGVt [2024-01-19T09:07:41.169Z] g9qX+r8kyGYsvFavmnDxgMK1Zkz80irQJ2BJlDHdK1hp5aqpRbT2z2YgLog5ZMIZ [2024-01-19T09:07:41.169Z] uI36uPCOXo8+Gol1jT7D6vsgMWm0Tk9jCNbO4jPkv+wm2q1B9UZo94UGiS0K+G0S [2024-01-19T09:07:41.169Z] nccl4jmSq1gWpBZIVetdDFNjpipKjhEOVQM7x33APhACxSAWRp9iEr6V/CzMOgkf [2024-01-19T09:07:41.169Z] RHyz5D7ljh4a2XVH7++eKsj2QPfKyiKtIG72tCccQ9zGnzveiXXb4Ai8r+DyKH4Z [2024-01-19T09:07:41.169Z] zDt5Hb4ZFl72E3Rl4bIh/VQBVsr9u/SFR1p/Emyq+kGXsn+9YMiM9tK4mLUtjdox [2024-01-19T09:07:41.169Z] LMWUWTelNRGWAiaTVz8= [2024-01-19T09:07:41.169Z] -----END CERTIFICATE----- [2024-01-19T09:07:41.169Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:41.169Z] MIIDCjCCAfKgAwIBAgIUBxbEoan/HnRZ+2deRj+2RvGMrWQwDQYJKoZIhvcNAQEL [2024-01-19T09:07:41.169Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:41.169Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:41.169Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:41.169Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:41.169Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:41.169Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:41.169Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:41.169Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:41.169Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:41.169Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAB65 [2024-01-19T09:07:41.169Z] nHU+98oTP0U2J0/TsL5GFeBTH7YyUXUNRJvPWxkA8riB3swoScFKbavO6XJ/hrI2 [2024-01-19T09:07:41.169Z] ywApOz77Iyq58OLa5aIC5aaPRf3YE8LbJQYIHmtza21LXhtlKVSrLu4W58PauI8Z [2024-01-19T09:07:41.169Z] 7iaL9QPUihHFK7wrZ2ZpabN2rRJvTGWWKlhmgFTjI+71gPrXWmYBHIKmTv55V5m1 [2024-01-19T09:07:41.169Z] LxORTR/gBXeH9gmM7wz1cbaYNazqs8R1LS5SW76p/y9RLjTzU6N39VYefMMGwbsQ [2024-01-19T09:07:41.170Z] DlD11wvMBxTBp3hQYc3pdVLkVEQt8trVIwEUts7zfwUTCu/z8gH7H5XePpEVNana [2024-01-19T09:07:41.170Z] JhlAcXHCO7nN/ddJmoc= [2024-01-19T09:07:41.170Z] -----END CERTIFICATE----- [2024-01-19T09:07:41.170Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:41.170Z] MIIDGDCCAgCgAwIBAgIUEvLaarpXlv1cVvG6nMuxh2DNCKQwDQYJKoZIhvcNAQEL [2024-01-19T09:07:41.170Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:41.170Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:41.170Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:41.170Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:41.170Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:41.170Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:41.170Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:41.170Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:41.170Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:41.170Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:41.170Z] 9w0BAQsFAAOCAQEAOPakzVUmAqHhmQViZZHSQeRh/xYQBrtn6dnEo6X55SXNWcPU [2024-01-19T09:07:41.170Z] 4EEW3DvwBf75uqxKeGoXHTBpqNHG2x1+IncvIKJmi5OWM3aR/vEyJoBOchPBl+A4 [2024-01-19T09:07:41.170Z] 0xZO1vKiS2djxz8XHMkR70cxD8hv/zDJeIQbgBroF4Wa7rim5+y7jgO0O7Rs1zhT [2024-01-19T09:07:41.170Z] LDDStJ8aUKwcVRmOv7NFQxbQHXNx07kaUE2SQJfhryzZmotoBiymmQT2v0H4/4q0 [2024-01-19T09:07:41.170Z] 5dylRhSCWfwN5+It40/C3b1pbC2d9Iv/eO+B0wqg13dLF3xuA1JblEgnqA9mdNnp [2024-01-19T09:07:41.170Z] P6EgKTQBMiO6nPFyYIPqg1eWYkb1eXu9aEkmbw== [2024-01-19T09:07:41.170Z] -----END CERTIFICATE----- [2024-01-19T09:07:41.170Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:41.170Z] MIIDGDCCAgCgAwIBAgIUWf/qoBANFWF/a/yZr7wfLBMfItAwDQYJKoZIhvcNAQEL [2024-01-19T09:07:41.170Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:41.170Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:41.170Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:41.170Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:41.170Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:41.170Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:41.170Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:41.170Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:41.170Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:41.170Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:41.170Z] 9w0BAQsFAAOCAQEAdCguadZU6WJAuuAtWzvYgftPKkDKNBeGVgbqRwWRCB+QqVFl [2024-01-19T09:07:41.170Z] +3ZOQTStpnXJsY3Ig31hmL88BkBPQ47ENEzs7Czhuz59m+dcBhh/buUoIERSnfUY [2024-01-19T09:07:41.170Z] lWIfqxQevwHYZ3Kjx0l3qliYjKBQJo5xDPnukGb4yUnmrCdZ3fxMzlnr0Ais9rUH [2024-01-19T09:07:41.170Z] MjiDsjP/oUa8naQUTZO4DTRZqeEldzNXnxF6YH+3moTkwRLt9xzirNwECmJ9qxrC [2024-01-19T09:07:41.170Z] eCA7zORxbFSftYLnQiGWUS1dnxBuLOvJHjYPC6EkPPrvqnx2fXcTE1V8GizVRrA6 [2024-01-19T09:07:41.170Z] NPxt+e4qZgU/Nk/Jkpsb7h1FVcKtS8CvmtbdqQ== [2024-01-19T09:07:41.170Z] -----END CERTIFICATE----- [2024-01-19T09:07:41.170Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:41.170Z] MIIDGDCCAgCgAwIBAgIUI6D0WS4tl52N+w2lwAS+6V0DPyUwDQYJKoZIhvcNAQEL [2024-01-19T09:07:41.170Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:41.170Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:41.170Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:41.170Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:41.170Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:41.170Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:41.170Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:41.170Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:41.170Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:41.170Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:41.170Z] 9w0BAQsFAAOCAQEAAS4yElh6TMyP18ipzRA0Wenukevjs/vtkAFOsv1u5GNeDIrv [2024-01-19T09:07:41.170Z] lNi6/VjvtS+DoFNzQnTf3QoO6GTuoWHLQWt6Z/V5AwIsnOXduSXLugMwEv3j9l50 [2024-01-19T09:07:41.170Z] aDkA7vJ3OH5cGqIvIzxQN7IUOEvZNhY67GFP7+zAzVNRpJ1FHYbhQGPuFD//b9G3 [2024-01-19T09:07:41.170Z] hr4E3+iUEc+F/iBA+caA8wRx1zR383Jz6wuoFt7n7HpWNcgMvoh91JlUAK7yUp0B [2024-01-19T09:07:41.170Z] ekuG6q7HtVsk6X+aB1RyWfrSJjih75W7b+p+/bV4vNmQS1wSGSOaSIuVRvMhmpwf [2024-01-19T09:07:41.170Z] FpT+4rUpiCxklD69/MET/1cWxqvao/ppf6phzg== [2024-01-19T09:07:41.170Z] -----END CERTIFICATE----- [2024-01-19T09:07:41.171Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:41.171Z] MIIDGDCCAgCgAwIBAgIUIVCCgYV9enDck18n89YsA5b2IdgwDQYJKoZIhvcNAQEL [2024-01-19T09:07:41.171Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:41.171Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:41.171Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:41.171Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:41.171Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:41.171Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:41.171Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:41.171Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:41.171Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:41.171Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:41.171Z] 9w0BAQsFAAOCAQEAa3eFm1ivFf5Vom6a/7E1gDdjl7cjUZ5cqg332Ky1NdzEOiwW [2024-01-19T09:07:41.171Z] mx3yEk35qCd8LAoFMvPvPIuMx1NDS8kwy3FmIOjKnZapHA31ZUpn/gT/seH4GFIU [2024-01-19T09:07:41.171Z] oZw+rODV5h6jmtCVugAn48u/LxOXbY417JWwaeDx5uw+NzOSyebe+td2PbtUjf5X [2024-01-19T09:07:41.171Z] 6xcwYUmaUGsIAgfIMM12hzXOI2/q1qz5dAqwSgnR7dKBReOXnUoFpNOMpxlJ70p/ [2024-01-19T09:07:41.171Z] WUU1aWIWZqKrNI7Pg1T5jljq7+KbT3+A6+ux0FLExr2c1tw3NndXS2Y9fO55m5/q [2024-01-19T09:07:41.171Z] 0EcrTdKx4+QjbWcja3WdX2pGDk0XGNgx+EXn6Q== [2024-01-19T09:07:41.171Z] -----END CERTIFICATE----- [2024-01-19T09:07:41.171Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:41.171Z] MIIFATCCAumgAwIBAgIRAKc9ZKBASymy5TLOEp57N98wDQYJKoZIhvcNAQELBQAw [2024-01-19T09:07:41.171Z] GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDMyMzIyNTM0NloXDTM2 [2024-01-19T09:07:41.171Z] MDMyMzIyNTM0NlowGjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMIICIjANBgkq [2024-01-19T09:07:41.171Z] hkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA+pYHvQw5iU3v2b3iNuYNKYgsWD6KU7aJ [2024-01-19T09:07:41.171Z] diddtZQxSWYzUI3U0I1UsRPTxnhTifs/M9NW4ZlV13ZfB7APwC8oqKOIiwo7IwlP [2024-01-19T09:07:41.171Z] xg0VKgyz+kT8RJfYr66PPIYP0fpTeu42LpMJ+CKo9sbpgVNDZN2z/qiXrRNX/VtG [2024-01-19T09:07:41.171Z] TkPV7a44fZ5bHHVruAxvDnylpQxJobtCBWlJSsbIRGFHMc2z88eUz9NmIOWUKGGj [2024-01-19T09:07:41.171Z] EmP76x8OfRHpIpuxRSCjn0+i9+hR2siIOpcMOGd+40uVJxbRRP5ZXnUFa2fF5FWd [2024-01-19T09:07:41.171Z] O0u0RPI8HON0ovhrwPJY+4eWKkQzyC611oLPYGQ4EbifRsTsCxUZqyUuStGyp8oa [2024-01-19T09:07:41.171Z] aoSKfF6X0+KzGgwwnrjRTUpIl19A92KR0Noo6h622OX+4sZiO/JQdkuX5w/HupK0 [2024-01-19T09:07:41.171Z] A0M0WSMCvU6GOhjGotmh2VTEJwHHY4+TUk0iQYRtv1crONklyZoAQPD76hCrC8Cr [2024-01-19T09:07:41.171Z] IbgsZLfTMC8TWUoMbyUDgvgYkHKMoPm0VGVVuwpRKJxv7+2wXO+pivrrUl2Q9fPe [2024-01-19T09:07:41.171Z] Kk055nJLMV9yPUdig8othUKrRfSxli946AEV1eEOhxddfEwBE3Lt2xn0hhiIedbb [2024-01-19T09:07:41.171Z] Ftf/5kEWFZkXyUmMJK8Ra76Kus2ABueUVEcZ48hrRr1Hf1N9n59VbTUaXgeiZA50 [2024-01-19T09:07:41.171Z] qXf2bymE6F8CAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMB [2024-01-19T09:07:41.171Z] Af8wHQYDVR0OBBYEFMEmdKSKRKDm+iAo2FwjmkWIGHngMA0GCSqGSIb3DQEBCwUA [2024-01-19T09:07:41.171Z] A4ICAQBCPw74M9X/Xx04K1VAES3ypgQYH5bf9FXVDrwhRFSVckria/7dMzoF5wln [2024-01-19T09:07:41.171Z] uq9NGsjkkkDg17AohcQdr8alH4LvPdxpKr3BjpvEcmbqF8xH+MbbeUEnmbSfLI8H [2024-01-19T09:07:41.171Z] sefuhXF9AF/9iYvpVNC8FmJ0OhiVv13VgMQw0CRKkbtjZBf8xaEhq/YqxWVsgOjm [2024-01-19T09:07:41.171Z] dm5CAQ2X0aX7502x8wYRgMnZhA5goC1zVWBVAi8yhhmlhhoDUfg17cXkmaJC5pDd [2024-01-19T09:07:41.171Z] oenZ9NVhW8eDb03MFCrWNvIh89DDeCGWuWfDltDq0n3owyL0IeSn7RfpSclpxVmV [2024-01-19T09:07:41.171Z] /53jkYjwIgxIG7Gsv0LKMbsf6QdBcTjhvfZyMIpBRkTe3zuHd2feKzY9lEkbRvRQ [2024-01-19T09:07:41.171Z] zbh4Ps5YBnG6CKJPTbe2hfi3nhnw/MyEmF3zb0hzvLWNrR9XW3ibb2oL3424XOwc [2024-01-19T09:07:41.171Z] VjrTSCLzO9Rv6s5wi03qoWvKAQQAElqTYRHhynJ3w6wuvKYF5zcZF3MDnrVGLbh1 [2024-01-19T09:07:41.171Z] Q9ePRFBCiXOQ6wPLoUhrrbZ8LpFUFYDXHMtYM7P9sc9IAWoONXREJaO08zgFtMp4 [2024-01-19T09:07:41.171Z] 8iyIYUyQAbsvx8oD2M8kRvrIRSrRJSl6L957b4AFiLIQ/GgV2curs0jje7Edx34c [2024-01-19T09:07:41.171Z] idWw1VrejtwclobqNMVtG3EiPUIpJGpbMcJgbiLSmKkrvQtGng== [2024-01-19T09:07:41.171Z] -----END CERTIFICATE----- [2024-01-19T09:07:41.171Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} from namespace mysql-ibm-pxb-0-84852-01-19-09h05m53s [2024-01-19T09:07:41.172Z] INFO[2024-01-19 09:07:41] Actual:true, Expected: true [2024-01-19T09:07:41.172Z] 2024-01-19 09:07:41 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.AddLabelToResource:#5038] - Adding label [059eb132-a858-4c46-8f77-d637339d05d0=f1086e89-2c30-44b2-906d-336a8bfa3a9a] to ConfigMap kube-****-ca.crt [2024-01-19T09:07:41.430Z] INFO[2024-01-19 09:07:41] Verifying : Description : Adding labels 059eb132-a858-4c46-8f77-d637339d05d0=f1086e89-2c30-44b2-906d-336a8bfa3a9a to resource &ConfigMap{ObjectMeta:{kube-****-ca.crt mysql-ibm-pxb-0-84852-01-19-09h05m53s 46f6918b-b534-424e-95cd-7e32f397a1e3 1831269 0 2024-01-19 09:05:53 +0000 UTC map[059eb132-a858-4c46-8f77-d637339d05d0:f1086e89-2c30-44b2-906d-336a8bfa3a9a] map[kubernetes.io/description:Contains a CA bundle that can be used to verify the kube-apiserver when using internal endpoints such as the internal service IP or kubernetes.default.svc. No other usage is guaranteed across distributions of Kubernetes clusters.] [] [] [{kube-controller-manager Update v1 2024-01-19 09:05:53 +0000 UTC FieldsV1 {"f:data":{".":{},"f:ca.crt":{}},"f:metadata":{"f:annotations":{".":{},"f:kubernetes.io/description":{}}}} }]},Data:map[string]string{ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T09:07:41.430Z] MIIDCjCCAfKgAwIBAgIUWkn6rgBaC+VWV1XdJjGA3DwoRRYwDQYJKoZIhvcNAQEL [2024-01-19T09:07:41.430Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:41.430Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:41.430Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:41.430Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:41.430Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:41.430Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:41.430Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:41.430Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:41.430Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:41.430Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAHbD [2024-01-19T09:07:41.430Z] 4nWTgt/8sat7SbbHxZ+JcMGRAIaeLAjMJxkxyqD8xUq6YQcgbjDVHo1Qbh3PXB3T [2024-01-19T09:07:41.430Z] 5BWUsRbDoKi0eRJx/5cyI3846SYEAOpuUfJrg9X2CH7ZoYhxeJ42dqt3MdoMJEM1 [2024-01-19T09:07:41.430Z] k3e2rRqAKuAxwNXugHryxQM+1ehOJjrEuT5DVK6qu8+JYRcT1hBZb6hrebKPp5eG [2024-01-19T09:07:41.430Z] JvBzHP4YOATIAXhfH0Uj5Y+JhwoOdpXmOFpz+6gITPD4aNzzJZe/0ADSFfOyUJaR [2024-01-19T09:07:41.430Z] g5NnJnkeGqMM3SLsur25r+m3y0i3t+RdocZ/4mHTEnF4KWVWvZy4GjENiEBcXEol [2024-01-19T09:07:41.430Z] o4SZVaP4AF7XQJ5vdA0= [2024-01-19T09:07:41.430Z] -----END CERTIFICATE----- [2024-01-19T09:07:41.430Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:41.430Z] MIIDCjCCAfKgAwIBAgIUXhIIFCc/NxZJn6hNEFoZuPQ8N5UwDQYJKoZIhvcNAQEL [2024-01-19T09:07:41.430Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:41.430Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:41.430Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:41.430Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:41.430Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:41.430Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:41.430Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:41.430Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:41.430Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:41.430Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAKG3 [2024-01-19T09:07:41.430Z] nVfv4IkgHVOxgbrNIoYhjvBvVJcLZub0C9yidexLA3A61p1a7b/OpidG5zShPhH2 [2024-01-19T09:07:41.430Z] HRdQcqgp0HIxaS+/oLD3eElOnnC/EKD359K6n3kN66sMgvNbHxNFM84Au0TXqmbT [2024-01-19T09:07:41.430Z] w8afrKjrfe/VglpHC2GfmgHMF4YH+ZF48C1RZA6z2oXj6Aj+uww0PjiClE9qUYqK [2024-01-19T09:07:41.430Z] R9uRa32prExihsi+Bto7n2Axo92qUiK2KrkEdQitnQfH5z/PojS78vbyIKp1O86N [2024-01-19T09:07:41.430Z] qRImvCYY17CPxP+JUQ7dxWDMF4NTgSnabkp+cd5i/pyoAb0guQFlQQUnHAZa4QNC [2024-01-19T09:07:41.430Z] 7fgTo1IUwWyDmwCnN5U= [2024-01-19T09:07:41.430Z] -----END CERTIFICATE----- [2024-01-19T09:07:41.430Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:41.430Z] MIIDCjCCAfKgAwIBAgIUJPzO3O+wmUt4gHCkl3ojsvwZL6owDQYJKoZIhvcNAQEL [2024-01-19T09:07:41.430Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:41.430Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:41.430Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:41.430Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:41.431Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:41.431Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:41.431Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:41.431Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:41.431Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:41.431Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAGVt [2024-01-19T09:07:41.431Z] g9qX+r8kyGYsvFavmnDxgMK1Zkz80irQJ2BJlDHdK1hp5aqpRbT2z2YgLog5ZMIZ [2024-01-19T09:07:41.431Z] uI36uPCOXo8+Gol1jT7D6vsgMWm0Tk9jCNbO4jPkv+wm2q1B9UZo94UGiS0K+G0S [2024-01-19T09:07:41.431Z] nccl4jmSq1gWpBZIVetdDFNjpipKjhEOVQM7x33APhACxSAWRp9iEr6V/CzMOgkf [2024-01-19T09:07:41.431Z] RHyz5D7ljh4a2XVH7++eKsj2QPfKyiKtIG72tCccQ9zGnzveiXXb4Ai8r+DyKH4Z [2024-01-19T09:07:41.431Z] zDt5Hb4ZFl72E3Rl4bIh/VQBVsr9u/SFR1p/Emyq+kGXsn+9YMiM9tK4mLUtjdox [2024-01-19T09:07:41.431Z] LMWUWTelNRGWAiaTVz8= [2024-01-19T09:07:41.431Z] -----END CERTIFICATE----- [2024-01-19T09:07:41.431Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:41.431Z] MIIDCjCCAfKgAwIBAgIUBxbEoan/HnRZ+2deRj+2RvGMrWQwDQYJKoZIhvcNAQEL [2024-01-19T09:07:41.431Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:41.431Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:41.431Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:41.431Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:41.431Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:41.431Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:41.431Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:41.431Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:41.431Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:41.431Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAB65 [2024-01-19T09:07:41.431Z] nHU+98oTP0U2J0/TsL5GFeBTH7YyUXUNRJvPWxkA8riB3swoScFKbavO6XJ/hrI2 [2024-01-19T09:07:41.431Z] ywApOz77Iyq58OLa5aIC5aaPRf3YE8LbJQYIHmtza21LXhtlKVSrLu4W58PauI8Z [2024-01-19T09:07:41.431Z] 7iaL9QPUihHFK7wrZ2ZpabN2rRJvTGWWKlhmgFTjI+71gPrXWmYBHIKmTv55V5m1 [2024-01-19T09:07:41.431Z] LxORTR/gBXeH9gmM7wz1cbaYNazqs8R1LS5SW76p/y9RLjTzU6N39VYefMMGwbsQ [2024-01-19T09:07:41.431Z] DlD11wvMBxTBp3hQYc3pdVLkVEQt8trVIwEUts7zfwUTCu/z8gH7H5XePpEVNana [2024-01-19T09:07:41.431Z] JhlAcXHCO7nN/ddJmoc= [2024-01-19T09:07:41.431Z] -----END CERTIFICATE----- [2024-01-19T09:07:41.431Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:41.431Z] MIIDGDCCAgCgAwIBAgIUEvLaarpXlv1cVvG6nMuxh2DNCKQwDQYJKoZIhvcNAQEL [2024-01-19T09:07:41.431Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:41.431Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:41.431Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:41.431Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:41.431Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:41.431Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:41.431Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:41.431Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:41.431Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:41.431Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:41.431Z] 9w0BAQsFAAOCAQEAOPakzVUmAqHhmQViZZHSQeRh/xYQBrtn6dnEo6X55SXNWcPU [2024-01-19T09:07:41.431Z] 4EEW3DvwBf75uqxKeGoXHTBpqNHG2x1+IncvIKJmi5OWM3aR/vEyJoBOchPBl+A4 [2024-01-19T09:07:41.431Z] 0xZO1vKiS2djxz8XHMkR70cxD8hv/zDJeIQbgBroF4Wa7rim5+y7jgO0O7Rs1zhT [2024-01-19T09:07:41.431Z] LDDStJ8aUKwcVRmOv7NFQxbQHXNx07kaUE2SQJfhryzZmotoBiymmQT2v0H4/4q0 [2024-01-19T09:07:41.431Z] 5dylRhSCWfwN5+It40/C3b1pbC2d9Iv/eO+B0wqg13dLF3xuA1JblEgnqA9mdNnp [2024-01-19T09:07:41.431Z] P6EgKTQBMiO6nPFyYIPqg1eWYkb1eXu9aEkmbw== [2024-01-19T09:07:41.431Z] -----END CERTIFICATE----- [2024-01-19T09:07:41.431Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:41.431Z] MIIDGDCCAgCgAwIBAgIUWf/qoBANFWF/a/yZr7wfLBMfItAwDQYJKoZIhvcNAQEL [2024-01-19T09:07:41.432Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:41.432Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:41.432Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:41.432Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:41.432Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:41.432Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:41.432Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:41.432Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:41.432Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:41.432Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:41.432Z] 9w0BAQsFAAOCAQEAdCguadZU6WJAuuAtWzvYgftPKkDKNBeGVgbqRwWRCB+QqVFl [2024-01-19T09:07:41.432Z] +3ZOQTStpnXJsY3Ig31hmL88BkBPQ47ENEzs7Czhuz59m+dcBhh/buUoIERSnfUY [2024-01-19T09:07:41.432Z] lWIfqxQevwHYZ3Kjx0l3qliYjKBQJo5xDPnukGb4yUnmrCdZ3fxMzlnr0Ais9rUH [2024-01-19T09:07:41.432Z] MjiDsjP/oUa8naQUTZO4DTRZqeEldzNXnxF6YH+3moTkwRLt9xzirNwECmJ9qxrC [2024-01-19T09:07:41.432Z] eCA7zORxbFSftYLnQiGWUS1dnxBuLOvJHjYPC6EkPPrvqnx2fXcTE1V8GizVRrA6 [2024-01-19T09:07:41.432Z] NPxt+e4qZgU/Nk/Jkpsb7h1FVcKtS8CvmtbdqQ== [2024-01-19T09:07:41.432Z] -----END CERTIFICATE----- [2024-01-19T09:07:41.432Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:41.432Z] MIIDGDCCAgCgAwIBAgIUI6D0WS4tl52N+w2lwAS+6V0DPyUwDQYJKoZIhvcNAQEL [2024-01-19T09:07:41.432Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:41.432Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:41.432Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:41.432Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:41.432Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:41.432Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:41.432Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:41.432Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:41.432Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:41.432Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:41.432Z] 9w0BAQsFAAOCAQEAAS4yElh6TMyP18ipzRA0Wenukevjs/vtkAFOsv1u5GNeDIrv [2024-01-19T09:07:41.432Z] lNi6/VjvtS+DoFNzQnTf3QoO6GTuoWHLQWt6Z/V5AwIsnOXduSXLugMwEv3j9l50 [2024-01-19T09:07:41.432Z] aDkA7vJ3OH5cGqIvIzxQN7IUOEvZNhY67GFP7+zAzVNRpJ1FHYbhQGPuFD//b9G3 [2024-01-19T09:07:41.432Z] hr4E3+iUEc+F/iBA+caA8wRx1zR383Jz6wuoFt7n7HpWNcgMvoh91JlUAK7yUp0B [2024-01-19T09:07:41.432Z] ekuG6q7HtVsk6X+aB1RyWfrSJjih75W7b+p+/bV4vNmQS1wSGSOaSIuVRvMhmpwf [2024-01-19T09:07:41.432Z] FpT+4rUpiCxklD69/MET/1cWxqvao/ppf6phzg== [2024-01-19T09:07:41.432Z] -----END CERTIFICATE----- [2024-01-19T09:07:41.432Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:41.432Z] MIIDGDCCAgCgAwIBAgIUIVCCgYV9enDck18n89YsA5b2IdgwDQYJKoZIhvcNAQEL [2024-01-19T09:07:41.432Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:41.432Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:41.432Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:41.432Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:41.432Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:41.432Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:41.432Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:41.432Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:41.432Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:41.432Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:41.432Z] 9w0BAQsFAAOCAQEAa3eFm1ivFf5Vom6a/7E1gDdjl7cjUZ5cqg332Ky1NdzEOiwW [2024-01-19T09:07:41.432Z] mx3yEk35qCd8LAoFMvPvPIuMx1NDS8kwy3FmIOjKnZapHA31ZUpn/gT/seH4GFIU [2024-01-19T09:07:41.432Z] oZw+rODV5h6jmtCVugAn48u/LxOXbY417JWwaeDx5uw+NzOSyebe+td2PbtUjf5X [2024-01-19T09:07:41.432Z] 6xcwYUmaUGsIAgfIMM12hzXOI2/q1qz5dAqwSgnR7dKBReOXnUoFpNOMpxlJ70p/ [2024-01-19T09:07:41.432Z] WUU1aWIWZqKrNI7Pg1T5jljq7+KbT3+A6+ux0FLExr2c1tw3NndXS2Y9fO55m5/q [2024-01-19T09:07:41.432Z] 0EcrTdKx4+QjbWcja3WdX2pGDk0XGNgx+EXn6Q== [2024-01-19T09:07:41.432Z] -----END CERTIFICATE----- [2024-01-19T09:07:41.432Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:41.432Z] MIIFATCCAumgAwIBAgIRAKc9ZKBASymy5TLOEp57N98wDQYJKoZIhvcNAQELBQAw [2024-01-19T09:07:41.432Z] GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDMyMzIyNTM0NloXDTM2 [2024-01-19T09:07:41.432Z] MDMyMzIyNTM0NlowGjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMIICIjANBgkq [2024-01-19T09:07:41.432Z] hkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA+pYHvQw5iU3v2b3iNuYNKYgsWD6KU7aJ [2024-01-19T09:07:41.433Z] diddtZQxSWYzUI3U0I1UsRPTxnhTifs/M9NW4ZlV13ZfB7APwC8oqKOIiwo7IwlP [2024-01-19T09:07:41.433Z] xg0VKgyz+kT8RJfYr66PPIYP0fpTeu42LpMJ+CKo9sbpgVNDZN2z/qiXrRNX/VtG [2024-01-19T09:07:41.433Z] TkPV7a44fZ5bHHVruAxvDnylpQxJobtCBWlJSsbIRGFHMc2z88eUz9NmIOWUKGGj [2024-01-19T09:07:41.433Z] EmP76x8OfRHpIpuxRSCjn0+i9+hR2siIOpcMOGd+40uVJxbRRP5ZXnUFa2fF5FWd [2024-01-19T09:07:41.433Z] O0u0RPI8HON0ovhrwPJY+4eWKkQzyC611oLPYGQ4EbifRsTsCxUZqyUuStGyp8oa [2024-01-19T09:07:41.433Z] aoSKfF6X0+KzGgwwnrjRTUpIl19A92KR0Noo6h622OX+4sZiO/JQdkuX5w/HupK0 [2024-01-19T09:07:41.433Z] A0M0WSMCvU6GOhjGotmh2VTEJwHHY4+TUk0iQYRtv1crONklyZoAQPD76hCrC8Cr [2024-01-19T09:07:41.433Z] IbgsZLfTMC8TWUoMbyUDgvgYkHKMoPm0VGVVuwpRKJxv7+2wXO+pivrrUl2Q9fPe [2024-01-19T09:07:41.433Z] Kk055nJLMV9yPUdig8othUKrRfSxli946AEV1eEOhxddfEwBE3Lt2xn0hhiIedbb [2024-01-19T09:07:41.433Z] Ftf/5kEWFZkXyUmMJK8Ra76Kus2ABueUVEcZ48hrRr1Hf1N9n59VbTUaXgeiZA50 [2024-01-19T09:07:41.433Z] qXf2bymE6F8CAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMB [2024-01-19T09:07:41.433Z] Af8wHQYDVR0OBBYEFMEmdKSKRKDm+iAo2FwjmkWIGHngMA0GCSqGSIb3DQEBCwUA [2024-01-19T09:07:41.433Z] A4ICAQBCPw74M9X/Xx04K1VAES3ypgQYH5bf9FXVDrwhRFSVckria/7dMzoF5wln [2024-01-19T09:07:41.433Z] uq9NGsjkkkDg17AohcQdr8alH4LvPdxpKr3BjpvEcmbqF8xH+MbbeUEnmbSfLI8H [2024-01-19T09:07:41.433Z] sefuhXF9AF/9iYvpVNC8FmJ0OhiVv13VgMQw0CRKkbtjZBf8xaEhq/YqxWVsgOjm [2024-01-19T09:07:41.433Z] dm5CAQ2X0aX7502x8wYRgMnZhA5goC1zVWBVAi8yhhmlhhoDUfg17cXkmaJC5pDd [2024-01-19T09:07:41.433Z] oenZ9NVhW8eDb03MFCrWNvIh89DDeCGWuWfDltDq0n3owyL0IeSn7RfpSclpxVmV [2024-01-19T09:07:41.433Z] /53jkYjwIgxIG7Gsv0LKMbsf6QdBcTjhvfZyMIpBRkTe3zuHd2feKzY9lEkbRvRQ [2024-01-19T09:07:41.433Z] zbh4Ps5YBnG6CKJPTbe2hfi3nhnw/MyEmF3zb0hzvLWNrR9XW3ibb2oL3424XOwc [2024-01-19T09:07:41.433Z] VjrTSCLzO9Rv6s5wi03qoWvKAQQAElqTYRHhynJ3w6wuvKYF5zcZF3MDnrVGLbh1 [2024-01-19T09:07:41.433Z] Q9ePRFBCiXOQ6wPLoUhrrbZ8LpFUFYDXHMtYM7P9sc9IAWoONXREJaO08zgFtMp4 [2024-01-19T09:07:41.433Z] 8iyIYUyQAbsvx8oD2M8kRvrIRSrRJSl6L957b4AFiLIQ/GgV2curs0jje7Edx34c [2024-01-19T09:07:41.433Z] idWw1VrejtwclobqNMVtG3EiPUIpJGpbMcJgbiLSmKkrvQtGng== [2024-01-19T09:07:41.433Z] -----END CERTIFICATE----- [2024-01-19T09:07:41.433Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} [2024-01-19T09:07:41.433Z] INFO[2024-01-19 09:07:41] Actual:true, Expected: true [2024-01-19T09:07:41.689Z] INFO[2024-01-19 09:07:41] Verifying : Description : Fetching configmap pointer &ConfigMap{ObjectMeta:{openshift-service-ca.crt mysql-ibm-pxb-0-84852-01-19-09h05m53s 5932922b-5646-47c0-8608-42b990f51a9a 1831273 0 2024-01-19 09:05:53 +0000 UTC map[] map[service.beta.openshift.io/inject-cabundle:true] [] [] [{kube-controller-manager Update v1 2024-01-19 09:05:53 +0000 UTC FieldsV1 {"f:data":{},"f:metadata":{"f:annotations":{".":{},"f:service.beta.openshift.io/inject-cabundle":{}}}} } {service-ca-operator Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:data":{"f:service-ca.crt":{}}} }]},Data:map[string]string{service-ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T09:07:41.689Z] MIIDUTCCAjmgAwIBAgIILvRjno8BjvIwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE [2024-01-19T09:07:41.689Z] Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwNTM5NjA4ODAe [2024-01-19T09:07:41.689Z] Fw0yNDAxMTYwOTA4MDhaFw0yNjAzMTYwOTA4MDlaMDYxNDAyBgNVBAMMK29wZW5z [2024-01-19T09:07:41.689Z] aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDUzOTYwODgwggEiMA0GCSqG [2024-01-19T09:07:41.689Z] SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVr8xmLKsMoemVtfnlzzil6B8CGocEhNRo [2024-01-19T09:07:41.689Z] fkzkD8aSOYgIj5zcZo/M0O7nXDXeg9dqx9/h2qrGk+SS2ikU2Xs4Q5gaQDh9U1lB [2024-01-19T09:07:41.689Z] y8paoOEvw06CmJ/1Qqur5A539/OnimRSTdejTu4lwaDwsJiyoGfhvrBJAdjagtYA [2024-01-19T09:07:41.689Z] 0sNMrCgEAT4NWHS5EDx5dF7nMDHmQ0h0gHYei6vUEtiPyLwcTH1T7uVSj22fCEXX [2024-01-19T09:07:41.689Z] LjgN3OSiquJ+Hf4+csPpwgxAEkfjdBldhnImYkunJbg/x4c90uFL1pygZusrbFAf [2024-01-19T09:07:41.689Z] cERW8GngZuL5JY8MQoIstMu+1qS390ouQDDbEcvWjxC8YFNFHywrAgMBAAGjYzBh [2024-01-19T09:07:41.689Z] MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQS5vRk [2024-01-19T09:07:41.689Z] jkBPISX2qYmk1VEQZIXSLzAfBgNVHSMEGDAWgBQS5vRkjkBPISX2qYmk1VEQZIXS [2024-01-19T09:07:41.689Z] LzANBgkqhkiG9w0BAQsFAAOCAQEAZ/qc2vFGJZgDJCEUaPYuufuLRPSF68ixV9fk [2024-01-19T09:07:41.689Z] e5tBkRN8atCsEBygC/RUcwULlhsdgUMGMcREgNC/FYlwkOsAj3gP8KmcXAZ4gHCa [2024-01-19T09:07:41.689Z] T6PIbIcj1hCX0mdX0tvtaj7PWvRUz9CHh6iFzmPsXK5xjaBA9vZbkUAa7odguUit [2024-01-19T09:07:41.689Z] vz1QnKoV1OaZyccDLeQL50khM1XEBn8L+zeY3k85uo2X514HZW4LDPTQ94cpyNnb [2024-01-19T09:07:41.689Z] zqI6utp7JoPJexsHMWNFKGb//HgjluNg7TsywXIeebk+/GZUEDfNKid0oiTpQvzR [2024-01-19T09:07:41.689Z] TFFF+5KO/sa4+qIh1ERm140JxeaurUMaJzn7cFQbekEgt/N+iA== [2024-01-19T09:07:41.689Z] -----END CERTIFICATE----- [2024-01-19T09:07:41.689Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} from namespace mysql-ibm-pxb-0-84852-01-19-09h05m53s [2024-01-19T09:07:41.689Z] INFO[2024-01-19 09:07:41] Actual:true, Expected: true [2024-01-19T09:07:41.689Z] 2024-01-19 09:07:41 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.AddLabelToResource:#5038] - Adding label [059eb132-a858-4c46-8f77-d637339d05d0=f1086e89-2c30-44b2-906d-336a8bfa3a9a] to ConfigMap openshift-service-ca.crt [2024-01-19T09:07:41.946Z] INFO[2024-01-19 09:07:41] Verifying : Description : Adding labels 059eb132-a858-4c46-8f77-d637339d05d0=f1086e89-2c30-44b2-906d-336a8bfa3a9a to resource &ConfigMap{ObjectMeta:{openshift-service-ca.crt mysql-ibm-pxb-0-84852-01-19-09h05m53s 5932922b-5646-47c0-8608-42b990f51a9a 1831273 0 2024-01-19 09:05:53 +0000 UTC map[059eb132-a858-4c46-8f77-d637339d05d0:f1086e89-2c30-44b2-906d-336a8bfa3a9a] map[service.beta.openshift.io/inject-cabundle:true] [] [] [{kube-controller-manager Update v1 2024-01-19 09:05:53 +0000 UTC FieldsV1 {"f:data":{},"f:metadata":{"f:annotations":{".":{},"f:service.beta.openshift.io/inject-cabundle":{}}}} } {service-ca-operator Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:data":{"f:service-ca.crt":{}}} }]},Data:map[string]string{service-ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T09:07:41.946Z] MIIDUTCCAjmgAwIBAgIILvRjno8BjvIwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE [2024-01-19T09:07:41.946Z] Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwNTM5NjA4ODAe [2024-01-19T09:07:41.946Z] Fw0yNDAxMTYwOTA4MDhaFw0yNjAzMTYwOTA4MDlaMDYxNDAyBgNVBAMMK29wZW5z [2024-01-19T09:07:41.946Z] aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDUzOTYwODgwggEiMA0GCSqG [2024-01-19T09:07:41.946Z] SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVr8xmLKsMoemVtfnlzzil6B8CGocEhNRo [2024-01-19T09:07:41.946Z] fkzkD8aSOYgIj5zcZo/M0O7nXDXeg9dqx9/h2qrGk+SS2ikU2Xs4Q5gaQDh9U1lB [2024-01-19T09:07:41.946Z] y8paoOEvw06CmJ/1Qqur5A539/OnimRSTdejTu4lwaDwsJiyoGfhvrBJAdjagtYA [2024-01-19T09:07:41.946Z] 0sNMrCgEAT4NWHS5EDx5dF7nMDHmQ0h0gHYei6vUEtiPyLwcTH1T7uVSj22fCEXX [2024-01-19T09:07:41.946Z] LjgN3OSiquJ+Hf4+csPpwgxAEkfjdBldhnImYkunJbg/x4c90uFL1pygZusrbFAf [2024-01-19T09:07:41.946Z] cERW8GngZuL5JY8MQoIstMu+1qS390ouQDDbEcvWjxC8YFNFHywrAgMBAAGjYzBh [2024-01-19T09:07:41.946Z] MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQS5vRk [2024-01-19T09:07:41.946Z] jkBPISX2qYmk1VEQZIXSLzAfBgNVHSMEGDAWgBQS5vRkjkBPISX2qYmk1VEQZIXS [2024-01-19T09:07:41.946Z] LzANBgkqhkiG9w0BAQsFAAOCAQEAZ/qc2vFGJZgDJCEUaPYuufuLRPSF68ixV9fk [2024-01-19T09:07:41.946Z] e5tBkRN8atCsEBygC/RUcwULlhsdgUMGMcREgNC/FYlwkOsAj3gP8KmcXAZ4gHCa [2024-01-19T09:07:41.946Z] T6PIbIcj1hCX0mdX0tvtaj7PWvRUz9CHh6iFzmPsXK5xjaBA9vZbkUAa7odguUit [2024-01-19T09:07:41.946Z] vz1QnKoV1OaZyccDLeQL50khM1XEBn8L+zeY3k85uo2X514HZW4LDPTQ94cpyNnb [2024-01-19T09:07:41.946Z] zqI6utp7JoPJexsHMWNFKGb//HgjluNg7TsywXIeebk+/GZUEDfNKid0oiTpQvzR [2024-01-19T09:07:41.946Z] TFFF+5KO/sa4+qIh1ERm140JxeaurUMaJzn7cFQbekEgt/N+iA== [2024-01-19T09:07:41.947Z] -----END CERTIFICATE----- [2024-01-19T09:07:41.947Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} [2024-01-19T09:07:41.947Z] INFO[2024-01-19 09:07:41] Actual:true, Expected: true [2024-01-19T09:07:41.948Z] INFO[2024-01-19 09:07:41] Verifying : Description : Fetching pvc list [&PersistentVolumeClaimList{ListMeta:{ 1832811 },Items:[]PersistentVolumeClaim{PersistentVolumeClaim{ObjectMeta:{mysql-data mysql-ibm-pxb-1-84852-01-19-09h05m53s 74694bda-5111-48bf-a87f-0f3d9ddbb2d9 1831649 0 2024-01-19 09:05:54 +0000 UTC map[] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-19 09:06:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-19 09:06:05 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{2147483648 0} {} 2Gi BinarySI},},},VolumeName:pvc-74694bda-5111-48bf-a87f-0f3d9ddbb2d9,Selector:nil,StorageClassName:*mysql-sc,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{10737418240 0} {} 10Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},},},}] from namespace [mysql-ibm-pxb-1-84852-01-19-09h05m53s] [2024-01-19T09:07:41.948Z] INFO[2024-01-19 09:07:41] Actual:true, Expected: true [2024-01-19T09:07:42.207Z] INFO[2024-01-19 09:07:42] Verifying : Description : Fetching pvc pointer &PersistentVolumeClaim{ObjectMeta:{mysql-data mysql-ibm-pxb-1-84852-01-19-09h05m53s 74694bda-5111-48bf-a87f-0f3d9ddbb2d9 1831649 0 2024-01-19 09:05:54 +0000 UTC map[] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-19 09:06:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-19 09:06:05 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{2147483648 0} {} 2Gi BinarySI},},},VolumeName:pvc-74694bda-5111-48bf-a87f-0f3d9ddbb2d9,Selector:nil,StorageClassName:*mysql-sc,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{10737418240 0} {} 10Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},} for pvc {{ } {mysql-data mysql-ibm-pxb-1-84852-01-19-09h05m53s 74694bda-5111-48bf-a87f-0f3d9ddbb2d9 1831649 0 2024-01-19 09:05:54 +0000 UTC map[] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-19 09:06:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-19 09:06:05 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{2147483648 0} {} 2Gi BinarySI}]} pvc-74694bda-5111-48bf-a87f-0f3d9ddbb2d9 0xc00148f350 0xc00148f360 nil nil} {Bound [ReadWriteOnce] map[storage:{{10737418240 0} {} 10Gi BinarySI}] [] map[] }} [2024-01-19T09:07:42.207Z] INFO[2024-01-19 09:07:42] Actual:true, Expected: true [2024-01-19T09:07:42.207Z] 2024-01-19 09:07:42 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.AddLabelToResource:#5030] - Adding label [059eb132-a858-4c46-8f77-d637339d05d0=f1086e89-2c30-44b2-906d-336a8bfa3a9a] to PVC mysql-data [2024-01-19T09:07:42.466Z] INFO[2024-01-19 09:07:42] Verifying : Description : Adding labels 059eb132-a858-4c46-8f77-d637339d05d0=f1086e89-2c30-44b2-906d-336a8bfa3a9a to resource &PersistentVolumeClaim{ObjectMeta:{mysql-data mysql-ibm-pxb-1-84852-01-19-09h05m53s 74694bda-5111-48bf-a87f-0f3d9ddbb2d9 1831649 0 2024-01-19 09:05:54 +0000 UTC map[059eb132-a858-4c46-8f77-d637339d05d0:f1086e89-2c30-44b2-906d-336a8bfa3a9a] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-19 09:06:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-19 09:06:05 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{2147483648 0} {} 2Gi BinarySI},},},VolumeName:pvc-74694bda-5111-48bf-a87f-0f3d9ddbb2d9,Selector:nil,StorageClassName:*mysql-sc,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{10737418240 0} {} 10Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},} [2024-01-19T09:07:42.466Z] INFO[2024-01-19 09:07:42] Actual:true, Expected: true [2024-01-19T09:07:42.727Z] INFO[2024-01-19 09:07:42] Verifying : Description : Fetching configmap list [&ConfigMapList{ListMeta:{ 1832821 },Items:[]ConfigMap{ConfigMap{ObjectMeta:{kube-****-ca.crt mysql-ibm-pxb-1-84852-01-19-09h05m53s 2990e602-b7cf-41d9-a5f0-2bfccb4d6584 1831314 0 2024-01-19 09:05:54 +0000 UTC map[] map[kubernetes.io/description:Contains a CA bundle that can be used to verify the kube-apiserver when using internal endpoints such as the internal service IP or kubernetes.default.svc. No other usage is guaranteed across distributions of Kubernetes clusters.] [] [] [{kube-controller-manager Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:data":{".":{},"f:ca.crt":{}},"f:metadata":{"f:annotations":{".":{},"f:kubernetes.io/description":{}}}} }]},Data:map[string]string{ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.727Z] MIIDCjCCAfKgAwIBAgIUWkn6rgBaC+VWV1XdJjGA3DwoRRYwDQYJKoZIhvcNAQEL [2024-01-19T09:07:42.727Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:42.727Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:42.727Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:42.727Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:42.727Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:42.727Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:42.727Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:42.727Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:42.727Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:42.727Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAHbD [2024-01-19T09:07:42.727Z] 4nWTgt/8sat7SbbHxZ+JcMGRAIaeLAjMJxkxyqD8xUq6YQcgbjDVHo1Qbh3PXB3T [2024-01-19T09:07:42.727Z] 5BWUsRbDoKi0eRJx/5cyI3846SYEAOpuUfJrg9X2CH7ZoYhxeJ42dqt3MdoMJEM1 [2024-01-19T09:07:42.727Z] k3e2rRqAKuAxwNXugHryxQM+1ehOJjrEuT5DVK6qu8+JYRcT1hBZb6hrebKPp5eG [2024-01-19T09:07:42.727Z] JvBzHP4YOATIAXhfH0Uj5Y+JhwoOdpXmOFpz+6gITPD4aNzzJZe/0ADSFfOyUJaR [2024-01-19T09:07:42.727Z] g5NnJnkeGqMM3SLsur25r+m3y0i3t+RdocZ/4mHTEnF4KWVWvZy4GjENiEBcXEol [2024-01-19T09:07:42.727Z] o4SZVaP4AF7XQJ5vdA0= [2024-01-19T09:07:42.727Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.727Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.727Z] MIIDCjCCAfKgAwIBAgIUXhIIFCc/NxZJn6hNEFoZuPQ8N5UwDQYJKoZIhvcNAQEL [2024-01-19T09:07:42.727Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:42.727Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:42.727Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:42.727Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:42.727Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:42.727Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:42.727Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:42.727Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:42.727Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:42.727Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAKG3 [2024-01-19T09:07:42.727Z] nVfv4IkgHVOxgbrNIoYhjvBvVJcLZub0C9yidexLA3A61p1a7b/OpidG5zShPhH2 [2024-01-19T09:07:42.727Z] HRdQcqgp0HIxaS+/oLD3eElOnnC/EKD359K6n3kN66sMgvNbHxNFM84Au0TXqmbT [2024-01-19T09:07:42.727Z] w8afrKjrfe/VglpHC2GfmgHMF4YH+ZF48C1RZA6z2oXj6Aj+uww0PjiClE9qUYqK [2024-01-19T09:07:42.727Z] R9uRa32prExihsi+Bto7n2Axo92qUiK2KrkEdQitnQfH5z/PojS78vbyIKp1O86N [2024-01-19T09:07:42.727Z] qRImvCYY17CPxP+JUQ7dxWDMF4NTgSnabkp+cd5i/pyoAb0guQFlQQUnHAZa4QNC [2024-01-19T09:07:42.727Z] 7fgTo1IUwWyDmwCnN5U= [2024-01-19T09:07:42.727Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.727Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.727Z] MIIDCjCCAfKgAwIBAgIUJPzO3O+wmUt4gHCkl3ojsvwZL6owDQYJKoZIhvcNAQEL [2024-01-19T09:07:42.728Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:42.728Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:42.728Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:42.728Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:42.728Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:42.728Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:42.728Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:42.728Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:42.728Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:42.728Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAGVt [2024-01-19T09:07:42.728Z] g9qX+r8kyGYsvFavmnDxgMK1Zkz80irQJ2BJlDHdK1hp5aqpRbT2z2YgLog5ZMIZ [2024-01-19T09:07:42.728Z] uI36uPCOXo8+Gol1jT7D6vsgMWm0Tk9jCNbO4jPkv+wm2q1B9UZo94UGiS0K+G0S [2024-01-19T09:07:42.728Z] nccl4jmSq1gWpBZIVetdDFNjpipKjhEOVQM7x33APhACxSAWRp9iEr6V/CzMOgkf [2024-01-19T09:07:42.728Z] RHyz5D7ljh4a2XVH7++eKsj2QPfKyiKtIG72tCccQ9zGnzveiXXb4Ai8r+DyKH4Z [2024-01-19T09:07:42.728Z] zDt5Hb4ZFl72E3Rl4bIh/VQBVsr9u/SFR1p/Emyq+kGXsn+9YMiM9tK4mLUtjdox [2024-01-19T09:07:42.728Z] LMWUWTelNRGWAiaTVz8= [2024-01-19T09:07:42.728Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.728Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.728Z] MIIDCjCCAfKgAwIBAgIUBxbEoan/HnRZ+2deRj+2RvGMrWQwDQYJKoZIhvcNAQEL [2024-01-19T09:07:42.728Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:42.728Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:42.728Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:42.728Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:42.728Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:42.728Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:42.728Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:42.728Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:42.728Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:42.728Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAB65 [2024-01-19T09:07:42.728Z] nHU+98oTP0U2J0/TsL5GFeBTH7YyUXUNRJvPWxkA8riB3swoScFKbavO6XJ/hrI2 [2024-01-19T09:07:42.728Z] ywApOz77Iyq58OLa5aIC5aaPRf3YE8LbJQYIHmtza21LXhtlKVSrLu4W58PauI8Z [2024-01-19T09:07:42.728Z] 7iaL9QPUihHFK7wrZ2ZpabN2rRJvTGWWKlhmgFTjI+71gPrXWmYBHIKmTv55V5m1 [2024-01-19T09:07:42.728Z] LxORTR/gBXeH9gmM7wz1cbaYNazqs8R1LS5SW76p/y9RLjTzU6N39VYefMMGwbsQ [2024-01-19T09:07:42.728Z] DlD11wvMBxTBp3hQYc3pdVLkVEQt8trVIwEUts7zfwUTCu/z8gH7H5XePpEVNana [2024-01-19T09:07:42.728Z] JhlAcXHCO7nN/ddJmoc= [2024-01-19T09:07:42.728Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.728Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.728Z] MIIDGDCCAgCgAwIBAgIUEvLaarpXlv1cVvG6nMuxh2DNCKQwDQYJKoZIhvcNAQEL [2024-01-19T09:07:42.728Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:42.728Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:42.728Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:42.728Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:42.728Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:42.728Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:42.728Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:42.729Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:42.729Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:42.729Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:42.729Z] 9w0BAQsFAAOCAQEAOPakzVUmAqHhmQViZZHSQeRh/xYQBrtn6dnEo6X55SXNWcPU [2024-01-19T09:07:42.729Z] 4EEW3DvwBf75uqxKeGoXHTBpqNHG2x1+IncvIKJmi5OWM3aR/vEyJoBOchPBl+A4 [2024-01-19T09:07:42.729Z] 0xZO1vKiS2djxz8XHMkR70cxD8hv/zDJeIQbgBroF4Wa7rim5+y7jgO0O7Rs1zhT [2024-01-19T09:07:42.729Z] LDDStJ8aUKwcVRmOv7NFQxbQHXNx07kaUE2SQJfhryzZmotoBiymmQT2v0H4/4q0 [2024-01-19T09:07:42.729Z] 5dylRhSCWfwN5+It40/C3b1pbC2d9Iv/eO+B0wqg13dLF3xuA1JblEgnqA9mdNnp [2024-01-19T09:07:42.729Z] P6EgKTQBMiO6nPFyYIPqg1eWYkb1eXu9aEkmbw== [2024-01-19T09:07:42.729Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.729Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.729Z] MIIDGDCCAgCgAwIBAgIUWf/qoBANFWF/a/yZr7wfLBMfItAwDQYJKoZIhvcNAQEL [2024-01-19T09:07:42.729Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:42.729Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:42.729Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:42.729Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:42.729Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:42.729Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:42.729Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:42.729Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:42.729Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:42.729Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:42.729Z] 9w0BAQsFAAOCAQEAdCguadZU6WJAuuAtWzvYgftPKkDKNBeGVgbqRwWRCB+QqVFl [2024-01-19T09:07:42.729Z] +3ZOQTStpnXJsY3Ig31hmL88BkBPQ47ENEzs7Czhuz59m+dcBhh/buUoIERSnfUY [2024-01-19T09:07:42.729Z] lWIfqxQevwHYZ3Kjx0l3qliYjKBQJo5xDPnukGb4yUnmrCdZ3fxMzlnr0Ais9rUH [2024-01-19T09:07:42.729Z] MjiDsjP/oUa8naQUTZO4DTRZqeEldzNXnxF6YH+3moTkwRLt9xzirNwECmJ9qxrC [2024-01-19T09:07:42.729Z] eCA7zORxbFSftYLnQiGWUS1dnxBuLOvJHjYPC6EkPPrvqnx2fXcTE1V8GizVRrA6 [2024-01-19T09:07:42.729Z] NPxt+e4qZgU/Nk/Jkpsb7h1FVcKtS8CvmtbdqQ== [2024-01-19T09:07:42.729Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.729Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.729Z] MIIDGDCCAgCgAwIBAgIUI6D0WS4tl52N+w2lwAS+6V0DPyUwDQYJKoZIhvcNAQEL [2024-01-19T09:07:42.729Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:42.729Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:42.729Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:42.729Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:42.729Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:42.729Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:42.729Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:42.729Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:42.729Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:42.729Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:42.729Z] 9w0BAQsFAAOCAQEAAS4yElh6TMyP18ipzRA0Wenukevjs/vtkAFOsv1u5GNeDIrv [2024-01-19T09:07:42.729Z] lNi6/VjvtS+DoFNzQnTf3QoO6GTuoWHLQWt6Z/V5AwIsnOXduSXLugMwEv3j9l50 [2024-01-19T09:07:42.729Z] aDkA7vJ3OH5cGqIvIzxQN7IUOEvZNhY67GFP7+zAzVNRpJ1FHYbhQGPuFD//b9G3 [2024-01-19T09:07:42.729Z] hr4E3+iUEc+F/iBA+caA8wRx1zR383Jz6wuoFt7n7HpWNcgMvoh91JlUAK7yUp0B [2024-01-19T09:07:42.729Z] ekuG6q7HtVsk6X+aB1RyWfrSJjih75W7b+p+/bV4vNmQS1wSGSOaSIuVRvMhmpwf [2024-01-19T09:07:42.729Z] FpT+4rUpiCxklD69/MET/1cWxqvao/ppf6phzg== [2024-01-19T09:07:42.729Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.729Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.729Z] MIIDGDCCAgCgAwIBAgIUIVCCgYV9enDck18n89YsA5b2IdgwDQYJKoZIhvcNAQEL [2024-01-19T09:07:42.729Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:42.729Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:42.729Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:42.729Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:42.729Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:42.730Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:42.730Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:42.730Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:42.730Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:42.730Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:42.730Z] 9w0BAQsFAAOCAQEAa3eFm1ivFf5Vom6a/7E1gDdjl7cjUZ5cqg332Ky1NdzEOiwW [2024-01-19T09:07:42.730Z] mx3yEk35qCd8LAoFMvPvPIuMx1NDS8kwy3FmIOjKnZapHA31ZUpn/gT/seH4GFIU [2024-01-19T09:07:42.730Z] oZw+rODV5h6jmtCVugAn48u/LxOXbY417JWwaeDx5uw+NzOSyebe+td2PbtUjf5X [2024-01-19T09:07:42.730Z] 6xcwYUmaUGsIAgfIMM12hzXOI2/q1qz5dAqwSgnR7dKBReOXnUoFpNOMpxlJ70p/ [2024-01-19T09:07:42.730Z] WUU1aWIWZqKrNI7Pg1T5jljq7+KbT3+A6+ux0FLExr2c1tw3NndXS2Y9fO55m5/q [2024-01-19T09:07:42.730Z] 0EcrTdKx4+QjbWcja3WdX2pGDk0XGNgx+EXn6Q== [2024-01-19T09:07:42.730Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.730Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.730Z] MIIFATCCAumgAwIBAgIRAKc9ZKBASymy5TLOEp57N98wDQYJKoZIhvcNAQELBQAw [2024-01-19T09:07:42.730Z] GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDMyMzIyNTM0NloXDTM2 [2024-01-19T09:07:42.730Z] MDMyMzIyNTM0NlowGjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMIICIjANBgkq [2024-01-19T09:07:42.730Z] hkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA+pYHvQw5iU3v2b3iNuYNKYgsWD6KU7aJ [2024-01-19T09:07:42.730Z] diddtZQxSWYzUI3U0I1UsRPTxnhTifs/M9NW4ZlV13ZfB7APwC8oqKOIiwo7IwlP [2024-01-19T09:07:42.730Z] xg0VKgyz+kT8RJfYr66PPIYP0fpTeu42LpMJ+CKo9sbpgVNDZN2z/qiXrRNX/VtG [2024-01-19T09:07:42.730Z] TkPV7a44fZ5bHHVruAxvDnylpQxJobtCBWlJSsbIRGFHMc2z88eUz9NmIOWUKGGj [2024-01-19T09:07:42.730Z] EmP76x8OfRHpIpuxRSCjn0+i9+hR2siIOpcMOGd+40uVJxbRRP5ZXnUFa2fF5FWd [2024-01-19T09:07:42.730Z] O0u0RPI8HON0ovhrwPJY+4eWKkQzyC611oLPYGQ4EbifRsTsCxUZqyUuStGyp8oa [2024-01-19T09:07:42.730Z] aoSKfF6X0+KzGgwwnrjRTUpIl19A92KR0Noo6h622OX+4sZiO/JQdkuX5w/HupK0 [2024-01-19T09:07:42.730Z] A0M0WSMCvU6GOhjGotmh2VTEJwHHY4+TUk0iQYRtv1crONklyZoAQPD76hCrC8Cr [2024-01-19T09:07:42.730Z] IbgsZLfTMC8TWUoMbyUDgvgYkHKMoPm0VGVVuwpRKJxv7+2wXO+pivrrUl2Q9fPe [2024-01-19T09:07:42.730Z] Kk055nJLMV9yPUdig8othUKrRfSxli946AEV1eEOhxddfEwBE3Lt2xn0hhiIedbb [2024-01-19T09:07:42.730Z] Ftf/5kEWFZkXyUmMJK8Ra76Kus2ABueUVEcZ48hrRr1Hf1N9n59VbTUaXgeiZA50 [2024-01-19T09:07:42.730Z] qXf2bymE6F8CAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMB [2024-01-19T09:07:42.730Z] Af8wHQYDVR0OBBYEFMEmdKSKRKDm+iAo2FwjmkWIGHngMA0GCSqGSIb3DQEBCwUA [2024-01-19T09:07:42.730Z] A4ICAQBCPw74M9X/Xx04K1VAES3ypgQYH5bf9FXVDrwhRFSVckria/7dMzoF5wln [2024-01-19T09:07:42.730Z] uq9NGsjkkkDg17AohcQdr8alH4LvPdxpKr3BjpvEcmbqF8xH+MbbeUEnmbSfLI8H [2024-01-19T09:07:42.730Z] sefuhXF9AF/9iYvpVNC8FmJ0OhiVv13VgMQw0CRKkbtjZBf8xaEhq/YqxWVsgOjm [2024-01-19T09:07:42.730Z] dm5CAQ2X0aX7502x8wYRgMnZhA5goC1zVWBVAi8yhhmlhhoDUfg17cXkmaJC5pDd [2024-01-19T09:07:42.730Z] oenZ9NVhW8eDb03MFCrWNvIh89DDeCGWuWfDltDq0n3owyL0IeSn7RfpSclpxVmV [2024-01-19T09:07:42.730Z] /53jkYjwIgxIG7Gsv0LKMbsf6QdBcTjhvfZyMIpBRkTe3zuHd2feKzY9lEkbRvRQ [2024-01-19T09:07:42.730Z] zbh4Ps5YBnG6CKJPTbe2hfi3nhnw/MyEmF3zb0hzvLWNrR9XW3ibb2oL3424XOwc [2024-01-19T09:07:42.730Z] VjrTSCLzO9Rv6s5wi03qoWvKAQQAElqTYRHhynJ3w6wuvKYF5zcZF3MDnrVGLbh1 [2024-01-19T09:07:42.730Z] Q9ePRFBCiXOQ6wPLoUhrrbZ8LpFUFYDXHMtYM7P9sc9IAWoONXREJaO08zgFtMp4 [2024-01-19T09:07:42.730Z] 8iyIYUyQAbsvx8oD2M8kRvrIRSrRJSl6L957b4AFiLIQ/GgV2curs0jje7Edx34c [2024-01-19T09:07:42.730Z] idWw1VrejtwclobqNMVtG3EiPUIpJGpbMcJgbiLSmKkrvQtGng== [2024-01-19T09:07:42.730Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.731Z] ,},BinaryData:map[string][]byte{},Immutable:nil,},ConfigMap{ObjectMeta:{openshift-service-ca.crt mysql-ibm-pxb-1-84852-01-19-09h05m53s a90cfeb2-87ca-48c4-a156-25230d74432d 1831328 0 2024-01-19 09:05:54 +0000 UTC map[] map[service.beta.openshift.io/inject-cabundle:true] [] [] [{kube-controller-manager Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:data":{},"f:metadata":{"f:annotations":{".":{},"f:service.beta.openshift.io/inject-cabundle":{}}}} } {service-ca-operator Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:data":{"f:service-ca.crt":{}}} }]},Data:map[string]string{service-ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.731Z] MIIDUTCCAjmgAwIBAgIILvRjno8BjvIwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE [2024-01-19T09:07:42.731Z] Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwNTM5NjA4ODAe [2024-01-19T09:07:42.731Z] Fw0yNDAxMTYwOTA4MDhaFw0yNjAzMTYwOTA4MDlaMDYxNDAyBgNVBAMMK29wZW5z [2024-01-19T09:07:42.731Z] aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDUzOTYwODgwggEiMA0GCSqG [2024-01-19T09:07:42.731Z] SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVr8xmLKsMoemVtfnlzzil6B8CGocEhNRo [2024-01-19T09:07:42.731Z] fkzkD8aSOYgIj5zcZo/M0O7nXDXeg9dqx9/h2qrGk+SS2ikU2Xs4Q5gaQDh9U1lB [2024-01-19T09:07:42.731Z] y8paoOEvw06CmJ/1Qqur5A539/OnimRSTdejTu4lwaDwsJiyoGfhvrBJAdjagtYA [2024-01-19T09:07:42.731Z] 0sNMrCgEAT4NWHS5EDx5dF7nMDHmQ0h0gHYei6vUEtiPyLwcTH1T7uVSj22fCEXX [2024-01-19T09:07:42.731Z] LjgN3OSiquJ+Hf4+csPpwgxAEkfjdBldhnImYkunJbg/x4c90uFL1pygZusrbFAf [2024-01-19T09:07:42.731Z] cERW8GngZuL5JY8MQoIstMu+1qS390ouQDDbEcvWjxC8YFNFHywrAgMBAAGjYzBh [2024-01-19T09:07:42.731Z] MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQS5vRk [2024-01-19T09:07:42.731Z] jkBPISX2qYmk1VEQZIXSLzAfBgNVHSMEGDAWgBQS5vRkjkBPISX2qYmk1VEQZIXS [2024-01-19T09:07:42.731Z] LzANBgkqhkiG9w0BAQsFAAOCAQEAZ/qc2vFGJZgDJCEUaPYuufuLRPSF68ixV9fk [2024-01-19T09:07:42.731Z] e5tBkRN8atCsEBygC/RUcwULlhsdgUMGMcREgNC/FYlwkOsAj3gP8KmcXAZ4gHCa [2024-01-19T09:07:42.731Z] T6PIbIcj1hCX0mdX0tvtaj7PWvRUz9CHh6iFzmPsXK5xjaBA9vZbkUAa7odguUit [2024-01-19T09:07:42.731Z] vz1QnKoV1OaZyccDLeQL50khM1XEBn8L+zeY3k85uo2X514HZW4LDPTQ94cpyNnb [2024-01-19T09:07:42.731Z] zqI6utp7JoPJexsHMWNFKGb//HgjluNg7TsywXIeebk+/GZUEDfNKid0oiTpQvzR [2024-01-19T09:07:42.731Z] TFFF+5KO/sa4+qIh1ERm140JxeaurUMaJzn7cFQbekEgt/N+iA== [2024-01-19T09:07:42.731Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.731Z] ,},BinaryData:map[string][]byte{},Immutable:nil,},},}] from namespace [mysql-ibm-pxb-1-84852-01-19-09h05m53s] [2024-01-19T09:07:42.731Z] INFO[2024-01-19 09:07:42] Actual:true, Expected: true [2024-01-19T09:07:42.731Z] INFO[2024-01-19 09:07:42] Verifying : Description : Fetching configmap pointer &ConfigMap{ObjectMeta:{kube-****-ca.crt mysql-ibm-pxb-1-84852-01-19-09h05m53s 2990e602-b7cf-41d9-a5f0-2bfccb4d6584 1831314 0 2024-01-19 09:05:54 +0000 UTC map[] map[kubernetes.io/description:Contains a CA bundle that can be used to verify the kube-apiserver when using internal endpoints such as the internal service IP or kubernetes.default.svc. No other usage is guaranteed across distributions of Kubernetes clusters.] [] [] [{kube-controller-manager Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:data":{".":{},"f:ca.crt":{}},"f:metadata":{"f:annotations":{".":{},"f:kubernetes.io/description":{}}}} }]},Data:map[string]string{ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.731Z] MIIDCjCCAfKgAwIBAgIUWkn6rgBaC+VWV1XdJjGA3DwoRRYwDQYJKoZIhvcNAQEL [2024-01-19T09:07:42.731Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:42.731Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:42.731Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:42.731Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:42.731Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:42.731Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:42.731Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:42.731Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:42.731Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:42.731Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAHbD [2024-01-19T09:07:42.731Z] 4nWTgt/8sat7SbbHxZ+JcMGRAIaeLAjMJxkxyqD8xUq6YQcgbjDVHo1Qbh3PXB3T [2024-01-19T09:07:42.731Z] 5BWUsRbDoKi0eRJx/5cyI3846SYEAOpuUfJrg9X2CH7ZoYhxeJ42dqt3MdoMJEM1 [2024-01-19T09:07:42.731Z] k3e2rRqAKuAxwNXugHryxQM+1ehOJjrEuT5DVK6qu8+JYRcT1hBZb6hrebKPp5eG [2024-01-19T09:07:42.731Z] JvBzHP4YOATIAXhfH0Uj5Y+JhwoOdpXmOFpz+6gITPD4aNzzJZe/0ADSFfOyUJaR [2024-01-19T09:07:42.731Z] g5NnJnkeGqMM3SLsur25r+m3y0i3t+RdocZ/4mHTEnF4KWVWvZy4GjENiEBcXEol [2024-01-19T09:07:42.731Z] o4SZVaP4AF7XQJ5vdA0= [2024-01-19T09:07:42.732Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.732Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.732Z] MIIDCjCCAfKgAwIBAgIUXhIIFCc/NxZJn6hNEFoZuPQ8N5UwDQYJKoZIhvcNAQEL [2024-01-19T09:07:42.732Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:42.732Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:42.732Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:42.732Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:42.732Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:42.732Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:42.732Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:42.732Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:42.732Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:42.732Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAKG3 [2024-01-19T09:07:42.732Z] nVfv4IkgHVOxgbrNIoYhjvBvVJcLZub0C9yidexLA3A61p1a7b/OpidG5zShPhH2 [2024-01-19T09:07:42.732Z] HRdQcqgp0HIxaS+/oLD3eElOnnC/EKD359K6n3kN66sMgvNbHxNFM84Au0TXqmbT [2024-01-19T09:07:42.732Z] w8afrKjrfe/VglpHC2GfmgHMF4YH+ZF48C1RZA6z2oXj6Aj+uww0PjiClE9qUYqK [2024-01-19T09:07:42.732Z] R9uRa32prExihsi+Bto7n2Axo92qUiK2KrkEdQitnQfH5z/PojS78vbyIKp1O86N [2024-01-19T09:07:42.732Z] qRImvCYY17CPxP+JUQ7dxWDMF4NTgSnabkp+cd5i/pyoAb0guQFlQQUnHAZa4QNC [2024-01-19T09:07:42.732Z] 7fgTo1IUwWyDmwCnN5U= [2024-01-19T09:07:42.732Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.732Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.732Z] MIIDCjCCAfKgAwIBAgIUJPzO3O+wmUt4gHCkl3ojsvwZL6owDQYJKoZIhvcNAQEL [2024-01-19T09:07:42.732Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:42.732Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:42.732Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:42.732Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:42.732Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:42.732Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:42.732Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:42.732Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:42.732Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:42.732Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAGVt [2024-01-19T09:07:42.732Z] g9qX+r8kyGYsvFavmnDxgMK1Zkz80irQJ2BJlDHdK1hp5aqpRbT2z2YgLog5ZMIZ [2024-01-19T09:07:42.732Z] uI36uPCOXo8+Gol1jT7D6vsgMWm0Tk9jCNbO4jPkv+wm2q1B9UZo94UGiS0K+G0S [2024-01-19T09:07:42.732Z] nccl4jmSq1gWpBZIVetdDFNjpipKjhEOVQM7x33APhACxSAWRp9iEr6V/CzMOgkf [2024-01-19T09:07:42.732Z] RHyz5D7ljh4a2XVH7++eKsj2QPfKyiKtIG72tCccQ9zGnzveiXXb4Ai8r+DyKH4Z [2024-01-19T09:07:42.732Z] zDt5Hb4ZFl72E3Rl4bIh/VQBVsr9u/SFR1p/Emyq+kGXsn+9YMiM9tK4mLUtjdox [2024-01-19T09:07:42.732Z] LMWUWTelNRGWAiaTVz8= [2024-01-19T09:07:42.732Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.732Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.732Z] MIIDCjCCAfKgAwIBAgIUBxbEoan/HnRZ+2deRj+2RvGMrWQwDQYJKoZIhvcNAQEL [2024-01-19T09:07:42.732Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:42.732Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:42.732Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:42.732Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:42.732Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:42.732Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:42.732Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:42.732Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:42.732Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:42.732Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAB65 [2024-01-19T09:07:42.732Z] nHU+98oTP0U2J0/TsL5GFeBTH7YyUXUNRJvPWxkA8riB3swoScFKbavO6XJ/hrI2 [2024-01-19T09:07:42.732Z] ywApOz77Iyq58OLa5aIC5aaPRf3YE8LbJQYIHmtza21LXhtlKVSrLu4W58PauI8Z [2024-01-19T09:07:42.732Z] 7iaL9QPUihHFK7wrZ2ZpabN2rRJvTGWWKlhmgFTjI+71gPrXWmYBHIKmTv55V5m1 [2024-01-19T09:07:42.732Z] LxORTR/gBXeH9gmM7wz1cbaYNazqs8R1LS5SW76p/y9RLjTzU6N39VYefMMGwbsQ [2024-01-19T09:07:42.732Z] DlD11wvMBxTBp3hQYc3pdVLkVEQt8trVIwEUts7zfwUTCu/z8gH7H5XePpEVNana [2024-01-19T09:07:42.732Z] JhlAcXHCO7nN/ddJmoc= [2024-01-19T09:07:42.732Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.732Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.732Z] MIIDGDCCAgCgAwIBAgIUEvLaarpXlv1cVvG6nMuxh2DNCKQwDQYJKoZIhvcNAQEL [2024-01-19T09:07:42.733Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:42.733Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:42.733Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:42.733Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:42.733Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:42.733Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:42.733Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:42.733Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:42.733Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:42.733Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:42.733Z] 9w0BAQsFAAOCAQEAOPakzVUmAqHhmQViZZHSQeRh/xYQBrtn6dnEo6X55SXNWcPU [2024-01-19T09:07:42.733Z] 4EEW3DvwBf75uqxKeGoXHTBpqNHG2x1+IncvIKJmi5OWM3aR/vEyJoBOchPBl+A4 [2024-01-19T09:07:42.733Z] 0xZO1vKiS2djxz8XHMkR70cxD8hv/zDJeIQbgBroF4Wa7rim5+y7jgO0O7Rs1zhT [2024-01-19T09:07:42.733Z] LDDStJ8aUKwcVRmOv7NFQxbQHXNx07kaUE2SQJfhryzZmotoBiymmQT2v0H4/4q0 [2024-01-19T09:07:42.733Z] 5dylRhSCWfwN5+It40/C3b1pbC2d9Iv/eO+B0wqg13dLF3xuA1JblEgnqA9mdNnp [2024-01-19T09:07:42.733Z] P6EgKTQBMiO6nPFyYIPqg1eWYkb1eXu9aEkmbw== [2024-01-19T09:07:42.733Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.733Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.733Z] MIIDGDCCAgCgAwIBAgIUWf/qoBANFWF/a/yZr7wfLBMfItAwDQYJKoZIhvcNAQEL [2024-01-19T09:07:42.733Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:42.733Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:42.733Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:42.733Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:42.733Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:42.733Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:42.733Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:42.733Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:42.733Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:42.733Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:42.733Z] 9w0BAQsFAAOCAQEAdCguadZU6WJAuuAtWzvYgftPKkDKNBeGVgbqRwWRCB+QqVFl [2024-01-19T09:07:42.733Z] +3ZOQTStpnXJsY3Ig31hmL88BkBPQ47ENEzs7Czhuz59m+dcBhh/buUoIERSnfUY [2024-01-19T09:07:42.733Z] lWIfqxQevwHYZ3Kjx0l3qliYjKBQJo5xDPnukGb4yUnmrCdZ3fxMzlnr0Ais9rUH [2024-01-19T09:07:42.733Z] MjiDsjP/oUa8naQUTZO4DTRZqeEldzNXnxF6YH+3moTkwRLt9xzirNwECmJ9qxrC [2024-01-19T09:07:42.733Z] eCA7zORxbFSftYLnQiGWUS1dnxBuLOvJHjYPC6EkPPrvqnx2fXcTE1V8GizVRrA6 [2024-01-19T09:07:42.733Z] NPxt+e4qZgU/Nk/Jkpsb7h1FVcKtS8CvmtbdqQ== [2024-01-19T09:07:42.733Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.733Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.733Z] MIIDGDCCAgCgAwIBAgIUI6D0WS4tl52N+w2lwAS+6V0DPyUwDQYJKoZIhvcNAQEL [2024-01-19T09:07:42.733Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:42.733Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:42.733Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:42.733Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:42.733Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:42.733Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:42.733Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:42.733Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:42.733Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:42.733Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:42.733Z] 9w0BAQsFAAOCAQEAAS4yElh6TMyP18ipzRA0Wenukevjs/vtkAFOsv1u5GNeDIrv [2024-01-19T09:07:42.733Z] lNi6/VjvtS+DoFNzQnTf3QoO6GTuoWHLQWt6Z/V5AwIsnOXduSXLugMwEv3j9l50 [2024-01-19T09:07:42.733Z] aDkA7vJ3OH5cGqIvIzxQN7IUOEvZNhY67GFP7+zAzVNRpJ1FHYbhQGPuFD//b9G3 [2024-01-19T09:07:42.733Z] hr4E3+iUEc+F/iBA+caA8wRx1zR383Jz6wuoFt7n7HpWNcgMvoh91JlUAK7yUp0B [2024-01-19T09:07:42.733Z] ekuG6q7HtVsk6X+aB1RyWfrSJjih75W7b+p+/bV4vNmQS1wSGSOaSIuVRvMhmpwf [2024-01-19T09:07:42.733Z] FpT+4rUpiCxklD69/MET/1cWxqvao/ppf6phzg== [2024-01-19T09:07:42.733Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.733Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.733Z] MIIDGDCCAgCgAwIBAgIUIVCCgYV9enDck18n89YsA5b2IdgwDQYJKoZIhvcNAQEL [2024-01-19T09:07:42.733Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:42.733Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:42.734Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:42.734Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:42.734Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:42.734Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:42.734Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:42.734Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:42.734Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:42.734Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:42.734Z] 9w0BAQsFAAOCAQEAa3eFm1ivFf5Vom6a/7E1gDdjl7cjUZ5cqg332Ky1NdzEOiwW [2024-01-19T09:07:42.734Z] mx3yEk35qCd8LAoFMvPvPIuMx1NDS8kwy3FmIOjKnZapHA31ZUpn/gT/seH4GFIU [2024-01-19T09:07:42.734Z] oZw+rODV5h6jmtCVugAn48u/LxOXbY417JWwaeDx5uw+NzOSyebe+td2PbtUjf5X [2024-01-19T09:07:42.734Z] 6xcwYUmaUGsIAgfIMM12hzXOI2/q1qz5dAqwSgnR7dKBReOXnUoFpNOMpxlJ70p/ [2024-01-19T09:07:42.734Z] WUU1aWIWZqKrNI7Pg1T5jljq7+KbT3+A6+ux0FLExr2c1tw3NndXS2Y9fO55m5/q [2024-01-19T09:07:42.734Z] 0EcrTdKx4+QjbWcja3WdX2pGDk0XGNgx+EXn6Q== [2024-01-19T09:07:42.734Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.734Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.734Z] MIIFATCCAumgAwIBAgIRAKc9ZKBASymy5TLOEp57N98wDQYJKoZIhvcNAQELBQAw [2024-01-19T09:07:42.734Z] GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDMyMzIyNTM0NloXDTM2 [2024-01-19T09:07:42.734Z] MDMyMzIyNTM0NlowGjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMIICIjANBgkq [2024-01-19T09:07:42.734Z] hkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA+pYHvQw5iU3v2b3iNuYNKYgsWD6KU7aJ [2024-01-19T09:07:42.734Z] diddtZQxSWYzUI3U0I1UsRPTxnhTifs/M9NW4ZlV13ZfB7APwC8oqKOIiwo7IwlP [2024-01-19T09:07:42.734Z] xg0VKgyz+kT8RJfYr66PPIYP0fpTeu42LpMJ+CKo9sbpgVNDZN2z/qiXrRNX/VtG [2024-01-19T09:07:42.734Z] TkPV7a44fZ5bHHVruAxvDnylpQxJobtCBWlJSsbIRGFHMc2z88eUz9NmIOWUKGGj [2024-01-19T09:07:42.734Z] EmP76x8OfRHpIpuxRSCjn0+i9+hR2siIOpcMOGd+40uVJxbRRP5ZXnUFa2fF5FWd [2024-01-19T09:07:42.734Z] O0u0RPI8HON0ovhrwPJY+4eWKkQzyC611oLPYGQ4EbifRsTsCxUZqyUuStGyp8oa [2024-01-19T09:07:42.734Z] aoSKfF6X0+KzGgwwnrjRTUpIl19A92KR0Noo6h622OX+4sZiO/JQdkuX5w/HupK0 [2024-01-19T09:07:42.734Z] A0M0WSMCvU6GOhjGotmh2VTEJwHHY4+TUk0iQYRtv1crONklyZoAQPD76hCrC8Cr [2024-01-19T09:07:42.734Z] IbgsZLfTMC8TWUoMbyUDgvgYkHKMoPm0VGVVuwpRKJxv7+2wXO+pivrrUl2Q9fPe [2024-01-19T09:07:42.734Z] Kk055nJLMV9yPUdig8othUKrRfSxli946AEV1eEOhxddfEwBE3Lt2xn0hhiIedbb [2024-01-19T09:07:42.734Z] Ftf/5kEWFZkXyUmMJK8Ra76Kus2ABueUVEcZ48hrRr1Hf1N9n59VbTUaXgeiZA50 [2024-01-19T09:07:42.734Z] qXf2bymE6F8CAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMB [2024-01-19T09:07:42.734Z] Af8wHQYDVR0OBBYEFMEmdKSKRKDm+iAo2FwjmkWIGHngMA0GCSqGSIb3DQEBCwUA [2024-01-19T09:07:42.734Z] A4ICAQBCPw74M9X/Xx04K1VAES3ypgQYH5bf9FXVDrwhRFSVckria/7dMzoF5wln [2024-01-19T09:07:42.734Z] uq9NGsjkkkDg17AohcQdr8alH4LvPdxpKr3BjpvEcmbqF8xH+MbbeUEnmbSfLI8H [2024-01-19T09:07:42.734Z] sefuhXF9AF/9iYvpVNC8FmJ0OhiVv13VgMQw0CRKkbtjZBf8xaEhq/YqxWVsgOjm [2024-01-19T09:07:42.734Z] dm5CAQ2X0aX7502x8wYRgMnZhA5goC1zVWBVAi8yhhmlhhoDUfg17cXkmaJC5pDd [2024-01-19T09:07:42.734Z] oenZ9NVhW8eDb03MFCrWNvIh89DDeCGWuWfDltDq0n3owyL0IeSn7RfpSclpxVmV [2024-01-19T09:07:42.734Z] /53jkYjwIgxIG7Gsv0LKMbsf6QdBcTjhvfZyMIpBRkTe3zuHd2feKzY9lEkbRvRQ [2024-01-19T09:07:42.734Z] zbh4Ps5YBnG6CKJPTbe2hfi3nhnw/MyEmF3zb0hzvLWNrR9XW3ibb2oL3424XOwc [2024-01-19T09:07:42.734Z] VjrTSCLzO9Rv6s5wi03qoWvKAQQAElqTYRHhynJ3w6wuvKYF5zcZF3MDnrVGLbh1 [2024-01-19T09:07:42.734Z] Q9ePRFBCiXOQ6wPLoUhrrbZ8LpFUFYDXHMtYM7P9sc9IAWoONXREJaO08zgFtMp4 [2024-01-19T09:07:42.734Z] 8iyIYUyQAbsvx8oD2M8kRvrIRSrRJSl6L957b4AFiLIQ/GgV2curs0jje7Edx34c [2024-01-19T09:07:42.734Z] idWw1VrejtwclobqNMVtG3EiPUIpJGpbMcJgbiLSmKkrvQtGng== [2024-01-19T09:07:42.734Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.734Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} from namespace mysql-ibm-pxb-1-84852-01-19-09h05m53s [2024-01-19T09:07:42.734Z] INFO[2024-01-19 09:07:42] Actual:true, Expected: true [2024-01-19T09:07:42.734Z] 2024-01-19 09:07:42 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.AddLabelToResource:#5038] - Adding label [059eb132-a858-4c46-8f77-d637339d05d0=f1086e89-2c30-44b2-906d-336a8bfa3a9a] to ConfigMap kube-****-ca.crt [2024-01-19T09:07:42.991Z] INFO[2024-01-19 09:07:42] Verifying : Description : Adding labels 059eb132-a858-4c46-8f77-d637339d05d0=f1086e89-2c30-44b2-906d-336a8bfa3a9a to resource &ConfigMap{ObjectMeta:{kube-****-ca.crt mysql-ibm-pxb-1-84852-01-19-09h05m53s 2990e602-b7cf-41d9-a5f0-2bfccb4d6584 1831314 0 2024-01-19 09:05:54 +0000 UTC map[059eb132-a858-4c46-8f77-d637339d05d0:f1086e89-2c30-44b2-906d-336a8bfa3a9a] map[kubernetes.io/description:Contains a CA bundle that can be used to verify the kube-apiserver when using internal endpoints such as the internal service IP or kubernetes.default.svc. No other usage is guaranteed across distributions of Kubernetes clusters.] [] [] [{kube-controller-manager Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:data":{".":{},"f:ca.crt":{}},"f:metadata":{"f:annotations":{".":{},"f:kubernetes.io/description":{}}}} }]},Data:map[string]string{ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.991Z] MIIDCjCCAfKgAwIBAgIUWkn6rgBaC+VWV1XdJjGA3DwoRRYwDQYJKoZIhvcNAQEL [2024-01-19T09:07:42.991Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:42.991Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:42.991Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:42.991Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:42.991Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:42.991Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:42.991Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:42.991Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:42.991Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:42.991Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAHbD [2024-01-19T09:07:42.991Z] 4nWTgt/8sat7SbbHxZ+JcMGRAIaeLAjMJxkxyqD8xUq6YQcgbjDVHo1Qbh3PXB3T [2024-01-19T09:07:42.991Z] 5BWUsRbDoKi0eRJx/5cyI3846SYEAOpuUfJrg9X2CH7ZoYhxeJ42dqt3MdoMJEM1 [2024-01-19T09:07:42.991Z] k3e2rRqAKuAxwNXugHryxQM+1ehOJjrEuT5DVK6qu8+JYRcT1hBZb6hrebKPp5eG [2024-01-19T09:07:42.991Z] JvBzHP4YOATIAXhfH0Uj5Y+JhwoOdpXmOFpz+6gITPD4aNzzJZe/0ADSFfOyUJaR [2024-01-19T09:07:42.991Z] g5NnJnkeGqMM3SLsur25r+m3y0i3t+RdocZ/4mHTEnF4KWVWvZy4GjENiEBcXEol [2024-01-19T09:07:42.991Z] o4SZVaP4AF7XQJ5vdA0= [2024-01-19T09:07:42.991Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.991Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.991Z] MIIDCjCCAfKgAwIBAgIUXhIIFCc/NxZJn6hNEFoZuPQ8N5UwDQYJKoZIhvcNAQEL [2024-01-19T09:07:42.991Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:42.991Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:42.991Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:42.991Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:42.991Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:42.991Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:42.991Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:42.991Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:42.991Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:42.991Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAKG3 [2024-01-19T09:07:42.991Z] nVfv4IkgHVOxgbrNIoYhjvBvVJcLZub0C9yidexLA3A61p1a7b/OpidG5zShPhH2 [2024-01-19T09:07:42.991Z] HRdQcqgp0HIxaS+/oLD3eElOnnC/EKD359K6n3kN66sMgvNbHxNFM84Au0TXqmbT [2024-01-19T09:07:42.991Z] w8afrKjrfe/VglpHC2GfmgHMF4YH+ZF48C1RZA6z2oXj6Aj+uww0PjiClE9qUYqK [2024-01-19T09:07:42.991Z] R9uRa32prExihsi+Bto7n2Axo92qUiK2KrkEdQitnQfH5z/PojS78vbyIKp1O86N [2024-01-19T09:07:42.991Z] qRImvCYY17CPxP+JUQ7dxWDMF4NTgSnabkp+cd5i/pyoAb0guQFlQQUnHAZa4QNC [2024-01-19T09:07:42.991Z] 7fgTo1IUwWyDmwCnN5U= [2024-01-19T09:07:42.991Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.991Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.991Z] MIIDCjCCAfKgAwIBAgIUJPzO3O+wmUt4gHCkl3ojsvwZL6owDQYJKoZIhvcNAQEL [2024-01-19T09:07:42.992Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:42.992Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:42.992Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:42.992Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:42.992Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:42.992Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:42.992Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:42.992Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:42.992Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:42.992Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAGVt [2024-01-19T09:07:42.992Z] g9qX+r8kyGYsvFavmnDxgMK1Zkz80irQJ2BJlDHdK1hp5aqpRbT2z2YgLog5ZMIZ [2024-01-19T09:07:42.992Z] uI36uPCOXo8+Gol1jT7D6vsgMWm0Tk9jCNbO4jPkv+wm2q1B9UZo94UGiS0K+G0S [2024-01-19T09:07:42.992Z] nccl4jmSq1gWpBZIVetdDFNjpipKjhEOVQM7x33APhACxSAWRp9iEr6V/CzMOgkf [2024-01-19T09:07:42.992Z] RHyz5D7ljh4a2XVH7++eKsj2QPfKyiKtIG72tCccQ9zGnzveiXXb4Ai8r+DyKH4Z [2024-01-19T09:07:42.992Z] zDt5Hb4ZFl72E3Rl4bIh/VQBVsr9u/SFR1p/Emyq+kGXsn+9YMiM9tK4mLUtjdox [2024-01-19T09:07:42.992Z] LMWUWTelNRGWAiaTVz8= [2024-01-19T09:07:42.992Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.992Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.992Z] MIIDCjCCAfKgAwIBAgIUBxbEoan/HnRZ+2deRj+2RvGMrWQwDQYJKoZIhvcNAQEL [2024-01-19T09:07:42.992Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:42.992Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:42.992Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:42.992Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:42.992Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:42.992Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:42.992Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:42.992Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:42.992Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:42.992Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAB65 [2024-01-19T09:07:42.992Z] nHU+98oTP0U2J0/TsL5GFeBTH7YyUXUNRJvPWxkA8riB3swoScFKbavO6XJ/hrI2 [2024-01-19T09:07:42.992Z] ywApOz77Iyq58OLa5aIC5aaPRf3YE8LbJQYIHmtza21LXhtlKVSrLu4W58PauI8Z [2024-01-19T09:07:42.992Z] 7iaL9QPUihHFK7wrZ2ZpabN2rRJvTGWWKlhmgFTjI+71gPrXWmYBHIKmTv55V5m1 [2024-01-19T09:07:42.992Z] LxORTR/gBXeH9gmM7wz1cbaYNazqs8R1LS5SW76p/y9RLjTzU6N39VYefMMGwbsQ [2024-01-19T09:07:42.992Z] DlD11wvMBxTBp3hQYc3pdVLkVEQt8trVIwEUts7zfwUTCu/z8gH7H5XePpEVNana [2024-01-19T09:07:42.992Z] JhlAcXHCO7nN/ddJmoc= [2024-01-19T09:07:42.992Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.992Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.992Z] MIIDGDCCAgCgAwIBAgIUEvLaarpXlv1cVvG6nMuxh2DNCKQwDQYJKoZIhvcNAQEL [2024-01-19T09:07:42.992Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:42.992Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:42.992Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:42.992Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:42.992Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:42.992Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:42.992Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:42.992Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:42.992Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:42.992Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:42.992Z] 9w0BAQsFAAOCAQEAOPakzVUmAqHhmQViZZHSQeRh/xYQBrtn6dnEo6X55SXNWcPU [2024-01-19T09:07:42.992Z] 4EEW3DvwBf75uqxKeGoXHTBpqNHG2x1+IncvIKJmi5OWM3aR/vEyJoBOchPBl+A4 [2024-01-19T09:07:42.992Z] 0xZO1vKiS2djxz8XHMkR70cxD8hv/zDJeIQbgBroF4Wa7rim5+y7jgO0O7Rs1zhT [2024-01-19T09:07:42.992Z] LDDStJ8aUKwcVRmOv7NFQxbQHXNx07kaUE2SQJfhryzZmotoBiymmQT2v0H4/4q0 [2024-01-19T09:07:42.992Z] 5dylRhSCWfwN5+It40/C3b1pbC2d9Iv/eO+B0wqg13dLF3xuA1JblEgnqA9mdNnp [2024-01-19T09:07:42.992Z] P6EgKTQBMiO6nPFyYIPqg1eWYkb1eXu9aEkmbw== [2024-01-19T09:07:42.992Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.992Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.992Z] MIIDGDCCAgCgAwIBAgIUWf/qoBANFWF/a/yZr7wfLBMfItAwDQYJKoZIhvcNAQEL [2024-01-19T09:07:42.992Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:42.992Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:42.992Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:42.992Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:42.992Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:42.993Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:42.993Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:42.993Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:42.993Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:42.993Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:42.993Z] 9w0BAQsFAAOCAQEAdCguadZU6WJAuuAtWzvYgftPKkDKNBeGVgbqRwWRCB+QqVFl [2024-01-19T09:07:42.993Z] +3ZOQTStpnXJsY3Ig31hmL88BkBPQ47ENEzs7Czhuz59m+dcBhh/buUoIERSnfUY [2024-01-19T09:07:42.993Z] lWIfqxQevwHYZ3Kjx0l3qliYjKBQJo5xDPnukGb4yUnmrCdZ3fxMzlnr0Ais9rUH [2024-01-19T09:07:42.993Z] MjiDsjP/oUa8naQUTZO4DTRZqeEldzNXnxF6YH+3moTkwRLt9xzirNwECmJ9qxrC [2024-01-19T09:07:42.993Z] eCA7zORxbFSftYLnQiGWUS1dnxBuLOvJHjYPC6EkPPrvqnx2fXcTE1V8GizVRrA6 [2024-01-19T09:07:42.993Z] NPxt+e4qZgU/Nk/Jkpsb7h1FVcKtS8CvmtbdqQ== [2024-01-19T09:07:42.993Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.993Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.993Z] MIIDGDCCAgCgAwIBAgIUI6D0WS4tl52N+w2lwAS+6V0DPyUwDQYJKoZIhvcNAQEL [2024-01-19T09:07:42.993Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:42.993Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:42.993Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:42.993Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:42.993Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:42.993Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:42.993Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:42.993Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:42.993Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:42.993Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:42.993Z] 9w0BAQsFAAOCAQEAAS4yElh6TMyP18ipzRA0Wenukevjs/vtkAFOsv1u5GNeDIrv [2024-01-19T09:07:42.993Z] lNi6/VjvtS+DoFNzQnTf3QoO6GTuoWHLQWt6Z/V5AwIsnOXduSXLugMwEv3j9l50 [2024-01-19T09:07:42.993Z] aDkA7vJ3OH5cGqIvIzxQN7IUOEvZNhY67GFP7+zAzVNRpJ1FHYbhQGPuFD//b9G3 [2024-01-19T09:07:42.993Z] hr4E3+iUEc+F/iBA+caA8wRx1zR383Jz6wuoFt7n7HpWNcgMvoh91JlUAK7yUp0B [2024-01-19T09:07:42.993Z] ekuG6q7HtVsk6X+aB1RyWfrSJjih75W7b+p+/bV4vNmQS1wSGSOaSIuVRvMhmpwf [2024-01-19T09:07:42.993Z] FpT+4rUpiCxklD69/MET/1cWxqvao/ppf6phzg== [2024-01-19T09:07:42.993Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.993Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.993Z] MIIDGDCCAgCgAwIBAgIUIVCCgYV9enDck18n89YsA5b2IdgwDQYJKoZIhvcNAQEL [2024-01-19T09:07:42.993Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:42.993Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:42.993Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:42.993Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:42.993Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:42.993Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:42.993Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:42.993Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:42.993Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:42.993Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:42.993Z] 9w0BAQsFAAOCAQEAa3eFm1ivFf5Vom6a/7E1gDdjl7cjUZ5cqg332Ky1NdzEOiwW [2024-01-19T09:07:42.993Z] mx3yEk35qCd8LAoFMvPvPIuMx1NDS8kwy3FmIOjKnZapHA31ZUpn/gT/seH4GFIU [2024-01-19T09:07:42.993Z] oZw+rODV5h6jmtCVugAn48u/LxOXbY417JWwaeDx5uw+NzOSyebe+td2PbtUjf5X [2024-01-19T09:07:42.993Z] 6xcwYUmaUGsIAgfIMM12hzXOI2/q1qz5dAqwSgnR7dKBReOXnUoFpNOMpxlJ70p/ [2024-01-19T09:07:42.993Z] WUU1aWIWZqKrNI7Pg1T5jljq7+KbT3+A6+ux0FLExr2c1tw3NndXS2Y9fO55m5/q [2024-01-19T09:07:42.993Z] 0EcrTdKx4+QjbWcja3WdX2pGDk0XGNgx+EXn6Q== [2024-01-19T09:07:42.993Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.993Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:42.993Z] MIIFATCCAumgAwIBAgIRAKc9ZKBASymy5TLOEp57N98wDQYJKoZIhvcNAQELBQAw [2024-01-19T09:07:42.993Z] GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDMyMzIyNTM0NloXDTM2 [2024-01-19T09:07:42.993Z] MDMyMzIyNTM0NlowGjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMIICIjANBgkq [2024-01-19T09:07:42.993Z] hkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA+pYHvQw5iU3v2b3iNuYNKYgsWD6KU7aJ [2024-01-19T09:07:42.993Z] diddtZQxSWYzUI3U0I1UsRPTxnhTifs/M9NW4ZlV13ZfB7APwC8oqKOIiwo7IwlP [2024-01-19T09:07:42.993Z] xg0VKgyz+kT8RJfYr66PPIYP0fpTeu42LpMJ+CKo9sbpgVNDZN2z/qiXrRNX/VtG [2024-01-19T09:07:42.993Z] TkPV7a44fZ5bHHVruAxvDnylpQxJobtCBWlJSsbIRGFHMc2z88eUz9NmIOWUKGGj [2024-01-19T09:07:42.993Z] EmP76x8OfRHpIpuxRSCjn0+i9+hR2siIOpcMOGd+40uVJxbRRP5ZXnUFa2fF5FWd [2024-01-19T09:07:42.993Z] O0u0RPI8HON0ovhrwPJY+4eWKkQzyC611oLPYGQ4EbifRsTsCxUZqyUuStGyp8oa [2024-01-19T09:07:42.993Z] aoSKfF6X0+KzGgwwnrjRTUpIl19A92KR0Noo6h622OX+4sZiO/JQdkuX5w/HupK0 [2024-01-19T09:07:42.993Z] A0M0WSMCvU6GOhjGotmh2VTEJwHHY4+TUk0iQYRtv1crONklyZoAQPD76hCrC8Cr [2024-01-19T09:07:42.993Z] IbgsZLfTMC8TWUoMbyUDgvgYkHKMoPm0VGVVuwpRKJxv7+2wXO+pivrrUl2Q9fPe [2024-01-19T09:07:42.994Z] Kk055nJLMV9yPUdig8othUKrRfSxli946AEV1eEOhxddfEwBE3Lt2xn0hhiIedbb [2024-01-19T09:07:42.994Z] Ftf/5kEWFZkXyUmMJK8Ra76Kus2ABueUVEcZ48hrRr1Hf1N9n59VbTUaXgeiZA50 [2024-01-19T09:07:42.994Z] qXf2bymE6F8CAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMB [2024-01-19T09:07:42.994Z] Af8wHQYDVR0OBBYEFMEmdKSKRKDm+iAo2FwjmkWIGHngMA0GCSqGSIb3DQEBCwUA [2024-01-19T09:07:42.994Z] A4ICAQBCPw74M9X/Xx04K1VAES3ypgQYH5bf9FXVDrwhRFSVckria/7dMzoF5wln [2024-01-19T09:07:42.994Z] uq9NGsjkkkDg17AohcQdr8alH4LvPdxpKr3BjpvEcmbqF8xH+MbbeUEnmbSfLI8H [2024-01-19T09:07:42.994Z] sefuhXF9AF/9iYvpVNC8FmJ0OhiVv13VgMQw0CRKkbtjZBf8xaEhq/YqxWVsgOjm [2024-01-19T09:07:42.994Z] dm5CAQ2X0aX7502x8wYRgMnZhA5goC1zVWBVAi8yhhmlhhoDUfg17cXkmaJC5pDd [2024-01-19T09:07:42.994Z] oenZ9NVhW8eDb03MFCrWNvIh89DDeCGWuWfDltDq0n3owyL0IeSn7RfpSclpxVmV [2024-01-19T09:07:42.994Z] /53jkYjwIgxIG7Gsv0LKMbsf6QdBcTjhvfZyMIpBRkTe3zuHd2feKzY9lEkbRvRQ [2024-01-19T09:07:42.994Z] zbh4Ps5YBnG6CKJPTbe2hfi3nhnw/MyEmF3zb0hzvLWNrR9XW3ibb2oL3424XOwc [2024-01-19T09:07:42.994Z] VjrTSCLzO9Rv6s5wi03qoWvKAQQAElqTYRHhynJ3w6wuvKYF5zcZF3MDnrVGLbh1 [2024-01-19T09:07:42.994Z] Q9ePRFBCiXOQ6wPLoUhrrbZ8LpFUFYDXHMtYM7P9sc9IAWoONXREJaO08zgFtMp4 [2024-01-19T09:07:42.994Z] 8iyIYUyQAbsvx8oD2M8kRvrIRSrRJSl6L957b4AFiLIQ/GgV2curs0jje7Edx34c [2024-01-19T09:07:42.994Z] idWw1VrejtwclobqNMVtG3EiPUIpJGpbMcJgbiLSmKkrvQtGng== [2024-01-19T09:07:42.994Z] -----END CERTIFICATE----- [2024-01-19T09:07:42.994Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} [2024-01-19T09:07:42.994Z] INFO[2024-01-19 09:07:42] Actual:true, Expected: true [2024-01-19T09:07:43.251Z] INFO[2024-01-19 09:07:43] Verifying : Description : Fetching configmap pointer &ConfigMap{ObjectMeta:{openshift-service-ca.crt mysql-ibm-pxb-1-84852-01-19-09h05m53s a90cfeb2-87ca-48c4-a156-25230d74432d 1831328 0 2024-01-19 09:05:54 +0000 UTC map[] map[service.beta.openshift.io/inject-cabundle:true] [] [] [{kube-controller-manager Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:data":{},"f:metadata":{"f:annotations":{".":{},"f:service.beta.openshift.io/inject-cabundle":{}}}} } {service-ca-operator Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:data":{"f:service-ca.crt":{}}} }]},Data:map[string]string{service-ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T09:07:43.251Z] MIIDUTCCAjmgAwIBAgIILvRjno8BjvIwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE [2024-01-19T09:07:43.251Z] Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwNTM5NjA4ODAe [2024-01-19T09:07:43.251Z] Fw0yNDAxMTYwOTA4MDhaFw0yNjAzMTYwOTA4MDlaMDYxNDAyBgNVBAMMK29wZW5z [2024-01-19T09:07:43.251Z] aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDUzOTYwODgwggEiMA0GCSqG [2024-01-19T09:07:43.251Z] SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVr8xmLKsMoemVtfnlzzil6B8CGocEhNRo [2024-01-19T09:07:43.251Z] fkzkD8aSOYgIj5zcZo/M0O7nXDXeg9dqx9/h2qrGk+SS2ikU2Xs4Q5gaQDh9U1lB [2024-01-19T09:07:43.251Z] y8paoOEvw06CmJ/1Qqur5A539/OnimRSTdejTu4lwaDwsJiyoGfhvrBJAdjagtYA [2024-01-19T09:07:43.251Z] 0sNMrCgEAT4NWHS5EDx5dF7nMDHmQ0h0gHYei6vUEtiPyLwcTH1T7uVSj22fCEXX [2024-01-19T09:07:43.251Z] LjgN3OSiquJ+Hf4+csPpwgxAEkfjdBldhnImYkunJbg/x4c90uFL1pygZusrbFAf [2024-01-19T09:07:43.251Z] cERW8GngZuL5JY8MQoIstMu+1qS390ouQDDbEcvWjxC8YFNFHywrAgMBAAGjYzBh [2024-01-19T09:07:43.251Z] MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQS5vRk [2024-01-19T09:07:43.251Z] jkBPISX2qYmk1VEQZIXSLzAfBgNVHSMEGDAWgBQS5vRkjkBPISX2qYmk1VEQZIXS [2024-01-19T09:07:43.251Z] LzANBgkqhkiG9w0BAQsFAAOCAQEAZ/qc2vFGJZgDJCEUaPYuufuLRPSF68ixV9fk [2024-01-19T09:07:43.251Z] e5tBkRN8atCsEBygC/RUcwULlhsdgUMGMcREgNC/FYlwkOsAj3gP8KmcXAZ4gHCa [2024-01-19T09:07:43.251Z] T6PIbIcj1hCX0mdX0tvtaj7PWvRUz9CHh6iFzmPsXK5xjaBA9vZbkUAa7odguUit [2024-01-19T09:07:43.251Z] vz1QnKoV1OaZyccDLeQL50khM1XEBn8L+zeY3k85uo2X514HZW4LDPTQ94cpyNnb [2024-01-19T09:07:43.252Z] zqI6utp7JoPJexsHMWNFKGb//HgjluNg7TsywXIeebk+/GZUEDfNKid0oiTpQvzR [2024-01-19T09:07:43.252Z] TFFF+5KO/sa4+qIh1ERm140JxeaurUMaJzn7cFQbekEgt/N+iA== [2024-01-19T09:07:43.252Z] -----END CERTIFICATE----- [2024-01-19T09:07:43.252Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} from namespace mysql-ibm-pxb-1-84852-01-19-09h05m53s [2024-01-19T09:07:43.252Z] INFO[2024-01-19 09:07:43] Actual:true, Expected: true [2024-01-19T09:07:43.252Z] 2024-01-19 09:07:43 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.AddLabelToResource:#5038] - Adding label [059eb132-a858-4c46-8f77-d637339d05d0=f1086e89-2c30-44b2-906d-336a8bfa3a9a] to ConfigMap openshift-service-ca.crt [2024-01-19T09:07:43.509Z] INFO[2024-01-19 09:07:43] Verifying : Description : Adding labels 059eb132-a858-4c46-8f77-d637339d05d0=f1086e89-2c30-44b2-906d-336a8bfa3a9a to resource &ConfigMap{ObjectMeta:{openshift-service-ca.crt mysql-ibm-pxb-1-84852-01-19-09h05m53s a90cfeb2-87ca-48c4-a156-25230d74432d 1831328 0 2024-01-19 09:05:54 +0000 UTC map[059eb132-a858-4c46-8f77-d637339d05d0:f1086e89-2c30-44b2-906d-336a8bfa3a9a] map[service.beta.openshift.io/inject-cabundle:true] [] [] [{kube-controller-manager Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:data":{},"f:metadata":{"f:annotations":{".":{},"f:service.beta.openshift.io/inject-cabundle":{}}}} } {service-ca-operator Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:data":{"f:service-ca.crt":{}}} }]},Data:map[string]string{service-ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T09:07:43.509Z] MIIDUTCCAjmgAwIBAgIILvRjno8BjvIwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE [2024-01-19T09:07:43.509Z] Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwNTM5NjA4ODAe [2024-01-19T09:07:43.509Z] Fw0yNDAxMTYwOTA4MDhaFw0yNjAzMTYwOTA4MDlaMDYxNDAyBgNVBAMMK29wZW5z [2024-01-19T09:07:43.509Z] aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDUzOTYwODgwggEiMA0GCSqG [2024-01-19T09:07:43.509Z] SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVr8xmLKsMoemVtfnlzzil6B8CGocEhNRo [2024-01-19T09:07:43.509Z] fkzkD8aSOYgIj5zcZo/M0O7nXDXeg9dqx9/h2qrGk+SS2ikU2Xs4Q5gaQDh9U1lB [2024-01-19T09:07:43.509Z] y8paoOEvw06CmJ/1Qqur5A539/OnimRSTdejTu4lwaDwsJiyoGfhvrBJAdjagtYA [2024-01-19T09:07:43.509Z] 0sNMrCgEAT4NWHS5EDx5dF7nMDHmQ0h0gHYei6vUEtiPyLwcTH1T7uVSj22fCEXX [2024-01-19T09:07:43.509Z] LjgN3OSiquJ+Hf4+csPpwgxAEkfjdBldhnImYkunJbg/x4c90uFL1pygZusrbFAf [2024-01-19T09:07:43.509Z] cERW8GngZuL5JY8MQoIstMu+1qS390ouQDDbEcvWjxC8YFNFHywrAgMBAAGjYzBh [2024-01-19T09:07:43.509Z] MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQS5vRk [2024-01-19T09:07:43.509Z] jkBPISX2qYmk1VEQZIXSLzAfBgNVHSMEGDAWgBQS5vRkjkBPISX2qYmk1VEQZIXS [2024-01-19T09:07:43.509Z] LzANBgkqhkiG9w0BAQsFAAOCAQEAZ/qc2vFGJZgDJCEUaPYuufuLRPSF68ixV9fk [2024-01-19T09:07:43.509Z] e5tBkRN8atCsEBygC/RUcwULlhsdgUMGMcREgNC/FYlwkOsAj3gP8KmcXAZ4gHCa [2024-01-19T09:07:43.509Z] T6PIbIcj1hCX0mdX0tvtaj7PWvRUz9CHh6iFzmPsXK5xjaBA9vZbkUAa7odguUit [2024-01-19T09:07:43.509Z] vz1QnKoV1OaZyccDLeQL50khM1XEBn8L+zeY3k85uo2X514HZW4LDPTQ94cpyNnb [2024-01-19T09:07:43.509Z] zqI6utp7JoPJexsHMWNFKGb//HgjluNg7TsywXIeebk+/GZUEDfNKid0oiTpQvzR [2024-01-19T09:07:43.509Z] TFFF+5KO/sa4+qIh1ERm140JxeaurUMaJzn7cFQbekEgt/N+iA== [2024-01-19T09:07:43.509Z] -----END CERTIFICATE----- [2024-01-19T09:07:43.509Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} [2024-01-19T09:07:43.509Z] INFO[2024-01-19 09:07:43] Actual:true, Expected: true [2024-01-19T09:07:43.510Z] INFO[2024-01-19 09:07:43] Verifying : Description : Fetching pvc list [&PersistentVolumeClaimList{ListMeta:{ 1832833 },Items:[]PersistentVolumeClaim{PersistentVolumeClaim{ObjectMeta:{mysql-data mysql-ibm-pxb-2-84852-01-19-09h05m53s e91bb5cb-b8d5-450f-a2bb-d312c22b46c0 1831656 0 2024-01-19 09:05:54 +0000 UTC map[] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-19 09:06:06 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-19 09:06:06 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{2147483648 0} {} 2Gi BinarySI},},},VolumeName:pvc-e91bb5cb-b8d5-450f-a2bb-d312c22b46c0,Selector:nil,StorageClassName:*mysql-sc,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{10737418240 0} {} 10Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},},},}] from namespace [mysql-ibm-pxb-2-84852-01-19-09h05m53s] [2024-01-19T09:07:43.510Z] INFO[2024-01-19 09:07:43] Actual:true, Expected: true [2024-01-19T09:07:43.766Z] INFO[2024-01-19 09:07:43] Verifying : Description : Fetching pvc pointer &PersistentVolumeClaim{ObjectMeta:{mysql-data mysql-ibm-pxb-2-84852-01-19-09h05m53s e91bb5cb-b8d5-450f-a2bb-d312c22b46c0 1831656 0 2024-01-19 09:05:54 +0000 UTC map[] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-19 09:06:06 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-19 09:06:06 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{2147483648 0} {} 2Gi BinarySI},},},VolumeName:pvc-e91bb5cb-b8d5-450f-a2bb-d312c22b46c0,Selector:nil,StorageClassName:*mysql-sc,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{10737418240 0} {} 10Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},} for pvc {{ } {mysql-data mysql-ibm-pxb-2-84852-01-19-09h05m53s e91bb5cb-b8d5-450f-a2bb-d312c22b46c0 1831656 0 2024-01-19 09:05:54 +0000 UTC map[] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-19 09:06:06 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-19 09:06:06 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]} {[ReadWriteOnce] nil {map[] map[storage:{{2147483648 0} {} 2Gi BinarySI}]} pvc-e91bb5cb-b8d5-450f-a2bb-d312c22b46c0 0xc0015f7250 0xc0015f7270 nil nil} {Bound [ReadWriteOnce] map[storage:{{10737418240 0} {} 10Gi BinarySI}] [] map[] }} [2024-01-19T09:07:43.766Z] INFO[2024-01-19 09:07:43] Actual:true, Expected: true [2024-01-19T09:07:43.766Z] 2024-01-19 09:07:43 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.AddLabelToResource:#5030] - Adding label [059eb132-a858-4c46-8f77-d637339d05d0=f1086e89-2c30-44b2-906d-336a8bfa3a9a] to PVC mysql-data [2024-01-19T09:07:44.022Z] INFO[2024-01-19 09:07:43] Verifying : Description : Adding labels 059eb132-a858-4c46-8f77-d637339d05d0=f1086e89-2c30-44b2-906d-336a8bfa3a9a to resource &PersistentVolumeClaim{ObjectMeta:{mysql-data mysql-ibm-pxb-2-84852-01-19-09h05m53s e91bb5cb-b8d5-450f-a2bb-d312c22b46c0 1831656 0 2024-01-19 09:05:54 +0000 UTC map[059eb132-a858-4c46-8f77-d637339d05d0:f1086e89-2c30-44b2-906d-336a8bfa3a9a] map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io volume.kubernetes.io/storage-provisioner:vpc.block.csi.ibm.io] [] [kubernetes.io/pvc-protection] [{backup.test Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} } {kube-controller-manager Update v1 2024-01-19 09:06:06 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}} } {kube-controller-manager Update v1 2024-01-19 09:06:06 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}} status}]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{2147483648 0} {} 2Gi BinarySI},},},VolumeName:pvc-e91bb5cb-b8d5-450f-a2bb-d312c22b46c0,Selector:nil,StorageClassName:*mysql-sc,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{10737418240 0} {} 10Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},ResizeStatus:nil,},} [2024-01-19T09:07:44.022Z] INFO[2024-01-19 09:07:43] Actual:true, Expected: true [2024-01-19T09:07:44.282Z] INFO[2024-01-19 09:07:44] Verifying : Description : Fetching configmap list [&ConfigMapList{ListMeta:{ 1832847 },Items:[]ConfigMap{ConfigMap{ObjectMeta:{kube-****-ca.crt mysql-ibm-pxb-2-84852-01-19-09h05m53s 9a770ad9-b498-4aaf-af19-78d43d7df872 1831385 0 2024-01-19 09:05:54 +0000 UTC map[] map[kubernetes.io/description:Contains a CA bundle that can be used to verify the kube-apiserver when using internal endpoints such as the internal service IP or kubernetes.default.svc. No other usage is guaranteed across distributions of Kubernetes clusters.] [] [] [{kube-controller-manager Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:data":{".":{},"f:ca.crt":{}},"f:metadata":{"f:annotations":{".":{},"f:kubernetes.io/description":{}}}} }]},Data:map[string]string{ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.282Z] MIIDCjCCAfKgAwIBAgIUWkn6rgBaC+VWV1XdJjGA3DwoRRYwDQYJKoZIhvcNAQEL [2024-01-19T09:07:44.282Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:44.282Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:44.282Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:44.282Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:44.282Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:44.282Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:44.282Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:44.282Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:44.282Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:44.282Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAHbD [2024-01-19T09:07:44.282Z] 4nWTgt/8sat7SbbHxZ+JcMGRAIaeLAjMJxkxyqD8xUq6YQcgbjDVHo1Qbh3PXB3T [2024-01-19T09:07:44.282Z] 5BWUsRbDoKi0eRJx/5cyI3846SYEAOpuUfJrg9X2CH7ZoYhxeJ42dqt3MdoMJEM1 [2024-01-19T09:07:44.282Z] k3e2rRqAKuAxwNXugHryxQM+1ehOJjrEuT5DVK6qu8+JYRcT1hBZb6hrebKPp5eG [2024-01-19T09:07:44.282Z] JvBzHP4YOATIAXhfH0Uj5Y+JhwoOdpXmOFpz+6gITPD4aNzzJZe/0ADSFfOyUJaR [2024-01-19T09:07:44.282Z] g5NnJnkeGqMM3SLsur25r+m3y0i3t+RdocZ/4mHTEnF4KWVWvZy4GjENiEBcXEol [2024-01-19T09:07:44.282Z] o4SZVaP4AF7XQJ5vdA0= [2024-01-19T09:07:44.282Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.282Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.282Z] MIIDCjCCAfKgAwIBAgIUXhIIFCc/NxZJn6hNEFoZuPQ8N5UwDQYJKoZIhvcNAQEL [2024-01-19T09:07:44.282Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:44.282Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:44.282Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:44.282Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:44.282Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:44.282Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:44.282Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:44.282Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:44.282Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:44.282Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAKG3 [2024-01-19T09:07:44.282Z] nVfv4IkgHVOxgbrNIoYhjvBvVJcLZub0C9yidexLA3A61p1a7b/OpidG5zShPhH2 [2024-01-19T09:07:44.282Z] HRdQcqgp0HIxaS+/oLD3eElOnnC/EKD359K6n3kN66sMgvNbHxNFM84Au0TXqmbT [2024-01-19T09:07:44.282Z] w8afrKjrfe/VglpHC2GfmgHMF4YH+ZF48C1RZA6z2oXj6Aj+uww0PjiClE9qUYqK [2024-01-19T09:07:44.282Z] R9uRa32prExihsi+Bto7n2Axo92qUiK2KrkEdQitnQfH5z/PojS78vbyIKp1O86N [2024-01-19T09:07:44.282Z] qRImvCYY17CPxP+JUQ7dxWDMF4NTgSnabkp+cd5i/pyoAb0guQFlQQUnHAZa4QNC [2024-01-19T09:07:44.282Z] 7fgTo1IUwWyDmwCnN5U= [2024-01-19T09:07:44.282Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.282Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.282Z] MIIDCjCCAfKgAwIBAgIUJPzO3O+wmUt4gHCkl3ojsvwZL6owDQYJKoZIhvcNAQEL [2024-01-19T09:07:44.283Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:44.283Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:44.283Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:44.283Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:44.283Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:44.283Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:44.283Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:44.283Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:44.283Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:44.283Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAGVt [2024-01-19T09:07:44.283Z] g9qX+r8kyGYsvFavmnDxgMK1Zkz80irQJ2BJlDHdK1hp5aqpRbT2z2YgLog5ZMIZ [2024-01-19T09:07:44.283Z] uI36uPCOXo8+Gol1jT7D6vsgMWm0Tk9jCNbO4jPkv+wm2q1B9UZo94UGiS0K+G0S [2024-01-19T09:07:44.283Z] nccl4jmSq1gWpBZIVetdDFNjpipKjhEOVQM7x33APhACxSAWRp9iEr6V/CzMOgkf [2024-01-19T09:07:44.283Z] RHyz5D7ljh4a2XVH7++eKsj2QPfKyiKtIG72tCccQ9zGnzveiXXb4Ai8r+DyKH4Z [2024-01-19T09:07:44.283Z] zDt5Hb4ZFl72E3Rl4bIh/VQBVsr9u/SFR1p/Emyq+kGXsn+9YMiM9tK4mLUtjdox [2024-01-19T09:07:44.283Z] LMWUWTelNRGWAiaTVz8= [2024-01-19T09:07:44.283Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.283Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.283Z] MIIDCjCCAfKgAwIBAgIUBxbEoan/HnRZ+2deRj+2RvGMrWQwDQYJKoZIhvcNAQEL [2024-01-19T09:07:44.283Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:44.283Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:44.283Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:44.283Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:44.283Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:44.283Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:44.283Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:44.283Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:44.283Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:44.283Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAB65 [2024-01-19T09:07:44.283Z] nHU+98oTP0U2J0/TsL5GFeBTH7YyUXUNRJvPWxkA8riB3swoScFKbavO6XJ/hrI2 [2024-01-19T09:07:44.283Z] ywApOz77Iyq58OLa5aIC5aaPRf3YE8LbJQYIHmtza21LXhtlKVSrLu4W58PauI8Z [2024-01-19T09:07:44.283Z] 7iaL9QPUihHFK7wrZ2ZpabN2rRJvTGWWKlhmgFTjI+71gPrXWmYBHIKmTv55V5m1 [2024-01-19T09:07:44.283Z] LxORTR/gBXeH9gmM7wz1cbaYNazqs8R1LS5SW76p/y9RLjTzU6N39VYefMMGwbsQ [2024-01-19T09:07:44.283Z] DlD11wvMBxTBp3hQYc3pdVLkVEQt8trVIwEUts7zfwUTCu/z8gH7H5XePpEVNana [2024-01-19T09:07:44.283Z] JhlAcXHCO7nN/ddJmoc= [2024-01-19T09:07:44.283Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.283Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.283Z] MIIDGDCCAgCgAwIBAgIUEvLaarpXlv1cVvG6nMuxh2DNCKQwDQYJKoZIhvcNAQEL [2024-01-19T09:07:44.283Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:44.283Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:44.283Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:44.283Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:44.283Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:44.283Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:44.283Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:44.283Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:44.283Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:44.283Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:44.283Z] 9w0BAQsFAAOCAQEAOPakzVUmAqHhmQViZZHSQeRh/xYQBrtn6dnEo6X55SXNWcPU [2024-01-19T09:07:44.283Z] 4EEW3DvwBf75uqxKeGoXHTBpqNHG2x1+IncvIKJmi5OWM3aR/vEyJoBOchPBl+A4 [2024-01-19T09:07:44.283Z] 0xZO1vKiS2djxz8XHMkR70cxD8hv/zDJeIQbgBroF4Wa7rim5+y7jgO0O7Rs1zhT [2024-01-19T09:07:44.283Z] LDDStJ8aUKwcVRmOv7NFQxbQHXNx07kaUE2SQJfhryzZmotoBiymmQT2v0H4/4q0 [2024-01-19T09:07:44.283Z] 5dylRhSCWfwN5+It40/C3b1pbC2d9Iv/eO+B0wqg13dLF3xuA1JblEgnqA9mdNnp [2024-01-19T09:07:44.283Z] P6EgKTQBMiO6nPFyYIPqg1eWYkb1eXu9aEkmbw== [2024-01-19T09:07:44.283Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.283Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.283Z] MIIDGDCCAgCgAwIBAgIUWf/qoBANFWF/a/yZr7wfLBMfItAwDQYJKoZIhvcNAQEL [2024-01-19T09:07:44.283Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:44.283Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:44.283Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:44.283Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:44.283Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:44.283Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:44.284Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:44.284Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:44.284Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:44.284Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:44.284Z] 9w0BAQsFAAOCAQEAdCguadZU6WJAuuAtWzvYgftPKkDKNBeGVgbqRwWRCB+QqVFl [2024-01-19T09:07:44.284Z] +3ZOQTStpnXJsY3Ig31hmL88BkBPQ47ENEzs7Czhuz59m+dcBhh/buUoIERSnfUY [2024-01-19T09:07:44.284Z] lWIfqxQevwHYZ3Kjx0l3qliYjKBQJo5xDPnukGb4yUnmrCdZ3fxMzlnr0Ais9rUH [2024-01-19T09:07:44.284Z] MjiDsjP/oUa8naQUTZO4DTRZqeEldzNXnxF6YH+3moTkwRLt9xzirNwECmJ9qxrC [2024-01-19T09:07:44.284Z] eCA7zORxbFSftYLnQiGWUS1dnxBuLOvJHjYPC6EkPPrvqnx2fXcTE1V8GizVRrA6 [2024-01-19T09:07:44.284Z] NPxt+e4qZgU/Nk/Jkpsb7h1FVcKtS8CvmtbdqQ== [2024-01-19T09:07:44.284Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.284Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.284Z] MIIDGDCCAgCgAwIBAgIUI6D0WS4tl52N+w2lwAS+6V0DPyUwDQYJKoZIhvcNAQEL [2024-01-19T09:07:44.284Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:44.284Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:44.284Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:44.284Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:44.284Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:44.284Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:44.284Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:44.284Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:44.284Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:44.284Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:44.284Z] 9w0BAQsFAAOCAQEAAS4yElh6TMyP18ipzRA0Wenukevjs/vtkAFOsv1u5GNeDIrv [2024-01-19T09:07:44.284Z] lNi6/VjvtS+DoFNzQnTf3QoO6GTuoWHLQWt6Z/V5AwIsnOXduSXLugMwEv3j9l50 [2024-01-19T09:07:44.284Z] aDkA7vJ3OH5cGqIvIzxQN7IUOEvZNhY67GFP7+zAzVNRpJ1FHYbhQGPuFD//b9G3 [2024-01-19T09:07:44.284Z] hr4E3+iUEc+F/iBA+caA8wRx1zR383Jz6wuoFt7n7HpWNcgMvoh91JlUAK7yUp0B [2024-01-19T09:07:44.284Z] ekuG6q7HtVsk6X+aB1RyWfrSJjih75W7b+p+/bV4vNmQS1wSGSOaSIuVRvMhmpwf [2024-01-19T09:07:44.284Z] FpT+4rUpiCxklD69/MET/1cWxqvao/ppf6phzg== [2024-01-19T09:07:44.284Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.284Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.284Z] MIIDGDCCAgCgAwIBAgIUIVCCgYV9enDck18n89YsA5b2IdgwDQYJKoZIhvcNAQEL [2024-01-19T09:07:44.284Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:44.284Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:44.284Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:44.284Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:44.284Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:44.284Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:44.284Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:44.284Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:44.284Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:44.284Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:44.284Z] 9w0BAQsFAAOCAQEAa3eFm1ivFf5Vom6a/7E1gDdjl7cjUZ5cqg332Ky1NdzEOiwW [2024-01-19T09:07:44.284Z] mx3yEk35qCd8LAoFMvPvPIuMx1NDS8kwy3FmIOjKnZapHA31ZUpn/gT/seH4GFIU [2024-01-19T09:07:44.284Z] oZw+rODV5h6jmtCVugAn48u/LxOXbY417JWwaeDx5uw+NzOSyebe+td2PbtUjf5X [2024-01-19T09:07:44.284Z] 6xcwYUmaUGsIAgfIMM12hzXOI2/q1qz5dAqwSgnR7dKBReOXnUoFpNOMpxlJ70p/ [2024-01-19T09:07:44.284Z] WUU1aWIWZqKrNI7Pg1T5jljq7+KbT3+A6+ux0FLExr2c1tw3NndXS2Y9fO55m5/q [2024-01-19T09:07:44.284Z] 0EcrTdKx4+QjbWcja3WdX2pGDk0XGNgx+EXn6Q== [2024-01-19T09:07:44.284Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.284Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.284Z] MIIFATCCAumgAwIBAgIRAKc9ZKBASymy5TLOEp57N98wDQYJKoZIhvcNAQELBQAw [2024-01-19T09:07:44.284Z] GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDMyMzIyNTM0NloXDTM2 [2024-01-19T09:07:44.284Z] MDMyMzIyNTM0NlowGjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMIICIjANBgkq [2024-01-19T09:07:44.284Z] hkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA+pYHvQw5iU3v2b3iNuYNKYgsWD6KU7aJ [2024-01-19T09:07:44.284Z] diddtZQxSWYzUI3U0I1UsRPTxnhTifs/M9NW4ZlV13ZfB7APwC8oqKOIiwo7IwlP [2024-01-19T09:07:44.284Z] xg0VKgyz+kT8RJfYr66PPIYP0fpTeu42LpMJ+CKo9sbpgVNDZN2z/qiXrRNX/VtG [2024-01-19T09:07:44.284Z] TkPV7a44fZ5bHHVruAxvDnylpQxJobtCBWlJSsbIRGFHMc2z88eUz9NmIOWUKGGj [2024-01-19T09:07:44.284Z] EmP76x8OfRHpIpuxRSCjn0+i9+hR2siIOpcMOGd+40uVJxbRRP5ZXnUFa2fF5FWd [2024-01-19T09:07:44.284Z] O0u0RPI8HON0ovhrwPJY+4eWKkQzyC611oLPYGQ4EbifRsTsCxUZqyUuStGyp8oa [2024-01-19T09:07:44.284Z] aoSKfF6X0+KzGgwwnrjRTUpIl19A92KR0Noo6h622OX+4sZiO/JQdkuX5w/HupK0 [2024-01-19T09:07:44.284Z] A0M0WSMCvU6GOhjGotmh2VTEJwHHY4+TUk0iQYRtv1crONklyZoAQPD76hCrC8Cr [2024-01-19T09:07:44.284Z] IbgsZLfTMC8TWUoMbyUDgvgYkHKMoPm0VGVVuwpRKJxv7+2wXO+pivrrUl2Q9fPe [2024-01-19T09:07:44.285Z] Kk055nJLMV9yPUdig8othUKrRfSxli946AEV1eEOhxddfEwBE3Lt2xn0hhiIedbb [2024-01-19T09:07:44.285Z] Ftf/5kEWFZkXyUmMJK8Ra76Kus2ABueUVEcZ48hrRr1Hf1N9n59VbTUaXgeiZA50 [2024-01-19T09:07:44.285Z] qXf2bymE6F8CAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMB [2024-01-19T09:07:44.285Z] Af8wHQYDVR0OBBYEFMEmdKSKRKDm+iAo2FwjmkWIGHngMA0GCSqGSIb3DQEBCwUA [2024-01-19T09:07:44.285Z] A4ICAQBCPw74M9X/Xx04K1VAES3ypgQYH5bf9FXVDrwhRFSVckria/7dMzoF5wln [2024-01-19T09:07:44.285Z] uq9NGsjkkkDg17AohcQdr8alH4LvPdxpKr3BjpvEcmbqF8xH+MbbeUEnmbSfLI8H [2024-01-19T09:07:44.285Z] sefuhXF9AF/9iYvpVNC8FmJ0OhiVv13VgMQw0CRKkbtjZBf8xaEhq/YqxWVsgOjm [2024-01-19T09:07:44.285Z] dm5CAQ2X0aX7502x8wYRgMnZhA5goC1zVWBVAi8yhhmlhhoDUfg17cXkmaJC5pDd [2024-01-19T09:07:44.285Z] oenZ9NVhW8eDb03MFCrWNvIh89DDeCGWuWfDltDq0n3owyL0IeSn7RfpSclpxVmV [2024-01-19T09:07:44.285Z] /53jkYjwIgxIG7Gsv0LKMbsf6QdBcTjhvfZyMIpBRkTe3zuHd2feKzY9lEkbRvRQ [2024-01-19T09:07:44.285Z] zbh4Ps5YBnG6CKJPTbe2hfi3nhnw/MyEmF3zb0hzvLWNrR9XW3ibb2oL3424XOwc [2024-01-19T09:07:44.285Z] VjrTSCLzO9Rv6s5wi03qoWvKAQQAElqTYRHhynJ3w6wuvKYF5zcZF3MDnrVGLbh1 [2024-01-19T09:07:44.285Z] Q9ePRFBCiXOQ6wPLoUhrrbZ8LpFUFYDXHMtYM7P9sc9IAWoONXREJaO08zgFtMp4 [2024-01-19T09:07:44.285Z] 8iyIYUyQAbsvx8oD2M8kRvrIRSrRJSl6L957b4AFiLIQ/GgV2curs0jje7Edx34c [2024-01-19T09:07:44.285Z] idWw1VrejtwclobqNMVtG3EiPUIpJGpbMcJgbiLSmKkrvQtGng== [2024-01-19T09:07:44.285Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.285Z] ,},BinaryData:map[string][]byte{},Immutable:nil,},ConfigMap{ObjectMeta:{openshift-service-ca.crt mysql-ibm-pxb-2-84852-01-19-09h05m53s 7ed3f1d7-f781-409d-89e8-6ad5ea8e3f5a 1831392 0 2024-01-19 09:05:54 +0000 UTC map[] map[service.beta.openshift.io/inject-cabundle:true] [] [] [{kube-controller-manager Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:data":{},"f:metadata":{"f:annotations":{".":{},"f:service.beta.openshift.io/inject-cabundle":{}}}} } {service-ca-operator Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:data":{"f:service-ca.crt":{}}} }]},Data:map[string]string{service-ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.285Z] MIIDUTCCAjmgAwIBAgIILvRjno8BjvIwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE [2024-01-19T09:07:44.285Z] Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwNTM5NjA4ODAe [2024-01-19T09:07:44.285Z] Fw0yNDAxMTYwOTA4MDhaFw0yNjAzMTYwOTA4MDlaMDYxNDAyBgNVBAMMK29wZW5z [2024-01-19T09:07:44.285Z] aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDUzOTYwODgwggEiMA0GCSqG [2024-01-19T09:07:44.285Z] SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVr8xmLKsMoemVtfnlzzil6B8CGocEhNRo [2024-01-19T09:07:44.285Z] fkzkD8aSOYgIj5zcZo/M0O7nXDXeg9dqx9/h2qrGk+SS2ikU2Xs4Q5gaQDh9U1lB [2024-01-19T09:07:44.285Z] y8paoOEvw06CmJ/1Qqur5A539/OnimRSTdejTu4lwaDwsJiyoGfhvrBJAdjagtYA [2024-01-19T09:07:44.285Z] 0sNMrCgEAT4NWHS5EDx5dF7nMDHmQ0h0gHYei6vUEtiPyLwcTH1T7uVSj22fCEXX [2024-01-19T09:07:44.285Z] LjgN3OSiquJ+Hf4+csPpwgxAEkfjdBldhnImYkunJbg/x4c90uFL1pygZusrbFAf [2024-01-19T09:07:44.285Z] cERW8GngZuL5JY8MQoIstMu+1qS390ouQDDbEcvWjxC8YFNFHywrAgMBAAGjYzBh [2024-01-19T09:07:44.285Z] MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQS5vRk [2024-01-19T09:07:44.285Z] jkBPISX2qYmk1VEQZIXSLzAfBgNVHSMEGDAWgBQS5vRkjkBPISX2qYmk1VEQZIXS [2024-01-19T09:07:44.285Z] LzANBgkqhkiG9w0BAQsFAAOCAQEAZ/qc2vFGJZgDJCEUaPYuufuLRPSF68ixV9fk [2024-01-19T09:07:44.285Z] e5tBkRN8atCsEBygC/RUcwULlhsdgUMGMcREgNC/FYlwkOsAj3gP8KmcXAZ4gHCa [2024-01-19T09:07:44.285Z] T6PIbIcj1hCX0mdX0tvtaj7PWvRUz9CHh6iFzmPsXK5xjaBA9vZbkUAa7odguUit [2024-01-19T09:07:44.285Z] vz1QnKoV1OaZyccDLeQL50khM1XEBn8L+zeY3k85uo2X514HZW4LDPTQ94cpyNnb [2024-01-19T09:07:44.285Z] zqI6utp7JoPJexsHMWNFKGb//HgjluNg7TsywXIeebk+/GZUEDfNKid0oiTpQvzR [2024-01-19T09:07:44.285Z] TFFF+5KO/sa4+qIh1ERm140JxeaurUMaJzn7cFQbekEgt/N+iA== [2024-01-19T09:07:44.285Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.285Z] ,},BinaryData:map[string][]byte{},Immutable:nil,},},}] from namespace [mysql-ibm-pxb-2-84852-01-19-09h05m53s] [2024-01-19T09:07:44.285Z] INFO[2024-01-19 09:07:44] Actual:true, Expected: true [2024-01-19T09:07:44.544Z] INFO[2024-01-19 09:07:44] Verifying : Description : Fetching configmap pointer &ConfigMap{ObjectMeta:{kube-****-ca.crt mysql-ibm-pxb-2-84852-01-19-09h05m53s 9a770ad9-b498-4aaf-af19-78d43d7df872 1831385 0 2024-01-19 09:05:54 +0000 UTC map[] map[kubernetes.io/description:Contains a CA bundle that can be used to verify the kube-apiserver when using internal endpoints such as the internal service IP or kubernetes.default.svc. No other usage is guaranteed across distributions of Kubernetes clusters.] [] [] [{kube-controller-manager Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:data":{".":{},"f:ca.crt":{}},"f:metadata":{"f:annotations":{".":{},"f:kubernetes.io/description":{}}}} }]},Data:map[string]string{ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.544Z] MIIDCjCCAfKgAwIBAgIUWkn6rgBaC+VWV1XdJjGA3DwoRRYwDQYJKoZIhvcNAQEL [2024-01-19T09:07:44.544Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:44.545Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:44.545Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:44.545Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:44.545Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:44.545Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:44.545Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:44.546Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:44.546Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:44.546Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAHbD [2024-01-19T09:07:44.546Z] 4nWTgt/8sat7SbbHxZ+JcMGRAIaeLAjMJxkxyqD8xUq6YQcgbjDVHo1Qbh3PXB3T [2024-01-19T09:07:44.546Z] 5BWUsRbDoKi0eRJx/5cyI3846SYEAOpuUfJrg9X2CH7ZoYhxeJ42dqt3MdoMJEM1 [2024-01-19T09:07:44.546Z] k3e2rRqAKuAxwNXugHryxQM+1ehOJjrEuT5DVK6qu8+JYRcT1hBZb6hrebKPp5eG [2024-01-19T09:07:44.546Z] JvBzHP4YOATIAXhfH0Uj5Y+JhwoOdpXmOFpz+6gITPD4aNzzJZe/0ADSFfOyUJaR [2024-01-19T09:07:44.546Z] g5NnJnkeGqMM3SLsur25r+m3y0i3t+RdocZ/4mHTEnF4KWVWvZy4GjENiEBcXEol [2024-01-19T09:07:44.546Z] o4SZVaP4AF7XQJ5vdA0= [2024-01-19T09:07:44.546Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.546Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.546Z] MIIDCjCCAfKgAwIBAgIUXhIIFCc/NxZJn6hNEFoZuPQ8N5UwDQYJKoZIhvcNAQEL [2024-01-19T09:07:44.546Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:44.546Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:44.547Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:44.547Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:44.547Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:44.547Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:44.547Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:44.547Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:44.547Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:44.547Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAKG3 [2024-01-19T09:07:44.547Z] nVfv4IkgHVOxgbrNIoYhjvBvVJcLZub0C9yidexLA3A61p1a7b/OpidG5zShPhH2 [2024-01-19T09:07:44.547Z] HRdQcqgp0HIxaS+/oLD3eElOnnC/EKD359K6n3kN66sMgvNbHxNFM84Au0TXqmbT [2024-01-19T09:07:44.547Z] w8afrKjrfe/VglpHC2GfmgHMF4YH+ZF48C1RZA6z2oXj6Aj+uww0PjiClE9qUYqK [2024-01-19T09:07:44.547Z] R9uRa32prExihsi+Bto7n2Axo92qUiK2KrkEdQitnQfH5z/PojS78vbyIKp1O86N [2024-01-19T09:07:44.547Z] qRImvCYY17CPxP+JUQ7dxWDMF4NTgSnabkp+cd5i/pyoAb0guQFlQQUnHAZa4QNC [2024-01-19T09:07:44.547Z] 7fgTo1IUwWyDmwCnN5U= [2024-01-19T09:07:44.547Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.547Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.547Z] MIIDCjCCAfKgAwIBAgIUJPzO3O+wmUt4gHCkl3ojsvwZL6owDQYJKoZIhvcNAQEL [2024-01-19T09:07:44.547Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:44.547Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:44.547Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:44.547Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:44.547Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:44.548Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:44.548Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:44.548Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:44.548Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:44.548Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAGVt [2024-01-19T09:07:44.548Z] g9qX+r8kyGYsvFavmnDxgMK1Zkz80irQJ2BJlDHdK1hp5aqpRbT2z2YgLog5ZMIZ [2024-01-19T09:07:44.548Z] uI36uPCOXo8+Gol1jT7D6vsgMWm0Tk9jCNbO4jPkv+wm2q1B9UZo94UGiS0K+G0S [2024-01-19T09:07:44.548Z] nccl4jmSq1gWpBZIVetdDFNjpipKjhEOVQM7x33APhACxSAWRp9iEr6V/CzMOgkf [2024-01-19T09:07:44.548Z] RHyz5D7ljh4a2XVH7++eKsj2QPfKyiKtIG72tCccQ9zGnzveiXXb4Ai8r+DyKH4Z [2024-01-19T09:07:44.548Z] zDt5Hb4ZFl72E3Rl4bIh/VQBVsr9u/SFR1p/Emyq+kGXsn+9YMiM9tK4mLUtjdox [2024-01-19T09:07:44.548Z] LMWUWTelNRGWAiaTVz8= [2024-01-19T09:07:44.548Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.548Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.548Z] MIIDCjCCAfKgAwIBAgIUBxbEoan/HnRZ+2deRj+2RvGMrWQwDQYJKoZIhvcNAQEL [2024-01-19T09:07:44.548Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:44.548Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:44.548Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:44.548Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:44.548Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:44.548Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:44.548Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:44.548Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:44.548Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:44.548Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAB65 [2024-01-19T09:07:44.548Z] nHU+98oTP0U2J0/TsL5GFeBTH7YyUXUNRJvPWxkA8riB3swoScFKbavO6XJ/hrI2 [2024-01-19T09:07:44.548Z] ywApOz77Iyq58OLa5aIC5aaPRf3YE8LbJQYIHmtza21LXhtlKVSrLu4W58PauI8Z [2024-01-19T09:07:44.548Z] 7iaL9QPUihHFK7wrZ2ZpabN2rRJvTGWWKlhmgFTjI+71gPrXWmYBHIKmTv55V5m1 [2024-01-19T09:07:44.548Z] LxORTR/gBXeH9gmM7wz1cbaYNazqs8R1LS5SW76p/y9RLjTzU6N39VYefMMGwbsQ [2024-01-19T09:07:44.548Z] DlD11wvMBxTBp3hQYc3pdVLkVEQt8trVIwEUts7zfwUTCu/z8gH7H5XePpEVNana [2024-01-19T09:07:44.548Z] JhlAcXHCO7nN/ddJmoc= [2024-01-19T09:07:44.548Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.548Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.548Z] MIIDGDCCAgCgAwIBAgIUEvLaarpXlv1cVvG6nMuxh2DNCKQwDQYJKoZIhvcNAQEL [2024-01-19T09:07:44.548Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:44.548Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:44.548Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:44.548Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:44.548Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:44.548Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:44.548Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:44.548Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:44.548Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:44.548Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:44.548Z] 9w0BAQsFAAOCAQEAOPakzVUmAqHhmQViZZHSQeRh/xYQBrtn6dnEo6X55SXNWcPU [2024-01-19T09:07:44.548Z] 4EEW3DvwBf75uqxKeGoXHTBpqNHG2x1+IncvIKJmi5OWM3aR/vEyJoBOchPBl+A4 [2024-01-19T09:07:44.548Z] 0xZO1vKiS2djxz8XHMkR70cxD8hv/zDJeIQbgBroF4Wa7rim5+y7jgO0O7Rs1zhT [2024-01-19T09:07:44.548Z] LDDStJ8aUKwcVRmOv7NFQxbQHXNx07kaUE2SQJfhryzZmotoBiymmQT2v0H4/4q0 [2024-01-19T09:07:44.548Z] 5dylRhSCWfwN5+It40/C3b1pbC2d9Iv/eO+B0wqg13dLF3xuA1JblEgnqA9mdNnp [2024-01-19T09:07:44.548Z] P6EgKTQBMiO6nPFyYIPqg1eWYkb1eXu9aEkmbw== [2024-01-19T09:07:44.548Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.548Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.548Z] MIIDGDCCAgCgAwIBAgIUWf/qoBANFWF/a/yZr7wfLBMfItAwDQYJKoZIhvcNAQEL [2024-01-19T09:07:44.548Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:44.548Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:44.548Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:44.548Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:44.548Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:44.548Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:44.548Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:44.548Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:44.548Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:44.549Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:44.549Z] 9w0BAQsFAAOCAQEAdCguadZU6WJAuuAtWzvYgftPKkDKNBeGVgbqRwWRCB+QqVFl [2024-01-19T09:07:44.549Z] +3ZOQTStpnXJsY3Ig31hmL88BkBPQ47ENEzs7Czhuz59m+dcBhh/buUoIERSnfUY [2024-01-19T09:07:44.549Z] lWIfqxQevwHYZ3Kjx0l3qliYjKBQJo5xDPnukGb4yUnmrCdZ3fxMzlnr0Ais9rUH [2024-01-19T09:07:44.549Z] MjiDsjP/oUa8naQUTZO4DTRZqeEldzNXnxF6YH+3moTkwRLt9xzirNwECmJ9qxrC [2024-01-19T09:07:44.549Z] eCA7zORxbFSftYLnQiGWUS1dnxBuLOvJHjYPC6EkPPrvqnx2fXcTE1V8GizVRrA6 [2024-01-19T09:07:44.549Z] NPxt+e4qZgU/Nk/Jkpsb7h1FVcKtS8CvmtbdqQ== [2024-01-19T09:07:44.549Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.549Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.549Z] MIIDGDCCAgCgAwIBAgIUI6D0WS4tl52N+w2lwAS+6V0DPyUwDQYJKoZIhvcNAQEL [2024-01-19T09:07:44.549Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:44.549Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:44.549Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:44.549Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:44.549Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:44.549Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:44.549Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:44.549Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:44.549Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:44.549Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:44.549Z] 9w0BAQsFAAOCAQEAAS4yElh6TMyP18ipzRA0Wenukevjs/vtkAFOsv1u5GNeDIrv [2024-01-19T09:07:44.549Z] lNi6/VjvtS+DoFNzQnTf3QoO6GTuoWHLQWt6Z/V5AwIsnOXduSXLugMwEv3j9l50 [2024-01-19T09:07:44.549Z] aDkA7vJ3OH5cGqIvIzxQN7IUOEvZNhY67GFP7+zAzVNRpJ1FHYbhQGPuFD//b9G3 [2024-01-19T09:07:44.549Z] hr4E3+iUEc+F/iBA+caA8wRx1zR383Jz6wuoFt7n7HpWNcgMvoh91JlUAK7yUp0B [2024-01-19T09:07:44.549Z] ekuG6q7HtVsk6X+aB1RyWfrSJjih75W7b+p+/bV4vNmQS1wSGSOaSIuVRvMhmpwf [2024-01-19T09:07:44.549Z] FpT+4rUpiCxklD69/MET/1cWxqvao/ppf6phzg== [2024-01-19T09:07:44.549Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.549Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.549Z] MIIDGDCCAgCgAwIBAgIUIVCCgYV9enDck18n89YsA5b2IdgwDQYJKoZIhvcNAQEL [2024-01-19T09:07:44.549Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:44.549Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:44.549Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:44.549Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:44.549Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:44.549Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:44.549Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:44.549Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:44.549Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:44.549Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:44.549Z] 9w0BAQsFAAOCAQEAa3eFm1ivFf5Vom6a/7E1gDdjl7cjUZ5cqg332Ky1NdzEOiwW [2024-01-19T09:07:44.549Z] mx3yEk35qCd8LAoFMvPvPIuMx1NDS8kwy3FmIOjKnZapHA31ZUpn/gT/seH4GFIU [2024-01-19T09:07:44.549Z] oZw+rODV5h6jmtCVugAn48u/LxOXbY417JWwaeDx5uw+NzOSyebe+td2PbtUjf5X [2024-01-19T09:07:44.549Z] 6xcwYUmaUGsIAgfIMM12hzXOI2/q1qz5dAqwSgnR7dKBReOXnUoFpNOMpxlJ70p/ [2024-01-19T09:07:44.549Z] WUU1aWIWZqKrNI7Pg1T5jljq7+KbT3+A6+ux0FLExr2c1tw3NndXS2Y9fO55m5/q [2024-01-19T09:07:44.549Z] 0EcrTdKx4+QjbWcja3WdX2pGDk0XGNgx+EXn6Q== [2024-01-19T09:07:44.549Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.549Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.549Z] MIIFATCCAumgAwIBAgIRAKc9ZKBASymy5TLOEp57N98wDQYJKoZIhvcNAQELBQAw [2024-01-19T09:07:44.549Z] GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDMyMzIyNTM0NloXDTM2 [2024-01-19T09:07:44.549Z] MDMyMzIyNTM0NlowGjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMIICIjANBgkq [2024-01-19T09:07:44.549Z] hkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA+pYHvQw5iU3v2b3iNuYNKYgsWD6KU7aJ [2024-01-19T09:07:44.549Z] diddtZQxSWYzUI3U0I1UsRPTxnhTifs/M9NW4ZlV13ZfB7APwC8oqKOIiwo7IwlP [2024-01-19T09:07:44.549Z] xg0VKgyz+kT8RJfYr66PPIYP0fpTeu42LpMJ+CKo9sbpgVNDZN2z/qiXrRNX/VtG [2024-01-19T09:07:44.549Z] TkPV7a44fZ5bHHVruAxvDnylpQxJobtCBWlJSsbIRGFHMc2z88eUz9NmIOWUKGGj [2024-01-19T09:07:44.549Z] EmP76x8OfRHpIpuxRSCjn0+i9+hR2siIOpcMOGd+40uVJxbRRP5ZXnUFa2fF5FWd [2024-01-19T09:07:44.549Z] O0u0RPI8HON0ovhrwPJY+4eWKkQzyC611oLPYGQ4EbifRsTsCxUZqyUuStGyp8oa [2024-01-19T09:07:44.549Z] aoSKfF6X0+KzGgwwnrjRTUpIl19A92KR0Noo6h622OX+4sZiO/JQdkuX5w/HupK0 [2024-01-19T09:07:44.549Z] A0M0WSMCvU6GOhjGotmh2VTEJwHHY4+TUk0iQYRtv1crONklyZoAQPD76hCrC8Cr [2024-01-19T09:07:44.549Z] IbgsZLfTMC8TWUoMbyUDgvgYkHKMoPm0VGVVuwpRKJxv7+2wXO+pivrrUl2Q9fPe [2024-01-19T09:07:44.549Z] Kk055nJLMV9yPUdig8othUKrRfSxli946AEV1eEOhxddfEwBE3Lt2xn0hhiIedbb [2024-01-19T09:07:44.549Z] Ftf/5kEWFZkXyUmMJK8Ra76Kus2ABueUVEcZ48hrRr1Hf1N9n59VbTUaXgeiZA50 [2024-01-19T09:07:44.549Z] qXf2bymE6F8CAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMB [2024-01-19T09:07:44.549Z] Af8wHQYDVR0OBBYEFMEmdKSKRKDm+iAo2FwjmkWIGHngMA0GCSqGSIb3DQEBCwUA [2024-01-19T09:07:44.549Z] A4ICAQBCPw74M9X/Xx04K1VAES3ypgQYH5bf9FXVDrwhRFSVckria/7dMzoF5wln [2024-01-19T09:07:44.550Z] uq9NGsjkkkDg17AohcQdr8alH4LvPdxpKr3BjpvEcmbqF8xH+MbbeUEnmbSfLI8H [2024-01-19T09:07:44.550Z] sefuhXF9AF/9iYvpVNC8FmJ0OhiVv13VgMQw0CRKkbtjZBf8xaEhq/YqxWVsgOjm [2024-01-19T09:07:44.550Z] dm5CAQ2X0aX7502x8wYRgMnZhA5goC1zVWBVAi8yhhmlhhoDUfg17cXkmaJC5pDd [2024-01-19T09:07:44.550Z] oenZ9NVhW8eDb03MFCrWNvIh89DDeCGWuWfDltDq0n3owyL0IeSn7RfpSclpxVmV [2024-01-19T09:07:44.550Z] /53jkYjwIgxIG7Gsv0LKMbsf6QdBcTjhvfZyMIpBRkTe3zuHd2feKzY9lEkbRvRQ [2024-01-19T09:07:44.550Z] zbh4Ps5YBnG6CKJPTbe2hfi3nhnw/MyEmF3zb0hzvLWNrR9XW3ibb2oL3424XOwc [2024-01-19T09:07:44.550Z] VjrTSCLzO9Rv6s5wi03qoWvKAQQAElqTYRHhynJ3w6wuvKYF5zcZF3MDnrVGLbh1 [2024-01-19T09:07:44.550Z] Q9ePRFBCiXOQ6wPLoUhrrbZ8LpFUFYDXHMtYM7P9sc9IAWoONXREJaO08zgFtMp4 [2024-01-19T09:07:44.550Z] 8iyIYUyQAbsvx8oD2M8kRvrIRSrRJSl6L957b4AFiLIQ/GgV2curs0jje7Edx34c [2024-01-19T09:07:44.550Z] idWw1VrejtwclobqNMVtG3EiPUIpJGpbMcJgbiLSmKkrvQtGng== [2024-01-19T09:07:44.550Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.550Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} from namespace mysql-ibm-pxb-2-84852-01-19-09h05m53s [2024-01-19T09:07:44.550Z] INFO[2024-01-19 09:07:44] Actual:true, Expected: true [2024-01-19T09:07:44.550Z] 2024-01-19 09:07:44 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.AddLabelToResource:#5038] - Adding label [059eb132-a858-4c46-8f77-d637339d05d0=f1086e89-2c30-44b2-906d-336a8bfa3a9a] to ConfigMap kube-****-ca.crt [2024-01-19T09:07:44.550Z] INFO[2024-01-19 09:07:44] Verifying : Description : Adding labels 059eb132-a858-4c46-8f77-d637339d05d0=f1086e89-2c30-44b2-906d-336a8bfa3a9a to resource &ConfigMap{ObjectMeta:{kube-****-ca.crt mysql-ibm-pxb-2-84852-01-19-09h05m53s 9a770ad9-b498-4aaf-af19-78d43d7df872 1831385 0 2024-01-19 09:05:54 +0000 UTC map[059eb132-a858-4c46-8f77-d637339d05d0:f1086e89-2c30-44b2-906d-336a8bfa3a9a] map[kubernetes.io/description:Contains a CA bundle that can be used to verify the kube-apiserver when using internal endpoints such as the internal service IP or kubernetes.default.svc. No other usage is guaranteed across distributions of Kubernetes clusters.] [] [] [{kube-controller-manager Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:data":{".":{},"f:ca.crt":{}},"f:metadata":{"f:annotations":{".":{},"f:kubernetes.io/description":{}}}} }]},Data:map[string]string{ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.550Z] MIIDCjCCAfKgAwIBAgIUWkn6rgBaC+VWV1XdJjGA3DwoRRYwDQYJKoZIhvcNAQEL [2024-01-19T09:07:44.550Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:44.550Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:44.550Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:44.550Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:44.550Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:44.550Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:44.550Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:44.550Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:44.550Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:44.550Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAHbD [2024-01-19T09:07:44.550Z] 4nWTgt/8sat7SbbHxZ+JcMGRAIaeLAjMJxkxyqD8xUq6YQcgbjDVHo1Qbh3PXB3T [2024-01-19T09:07:44.550Z] 5BWUsRbDoKi0eRJx/5cyI3846SYEAOpuUfJrg9X2CH7ZoYhxeJ42dqt3MdoMJEM1 [2024-01-19T09:07:44.550Z] k3e2rRqAKuAxwNXugHryxQM+1ehOJjrEuT5DVK6qu8+JYRcT1hBZb6hrebKPp5eG [2024-01-19T09:07:44.550Z] JvBzHP4YOATIAXhfH0Uj5Y+JhwoOdpXmOFpz+6gITPD4aNzzJZe/0ADSFfOyUJaR [2024-01-19T09:07:44.550Z] g5NnJnkeGqMM3SLsur25r+m3y0i3t+RdocZ/4mHTEnF4KWVWvZy4GjENiEBcXEol [2024-01-19T09:07:44.550Z] o4SZVaP4AF7XQJ5vdA0= [2024-01-19T09:07:44.550Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.550Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.550Z] MIIDCjCCAfKgAwIBAgIUXhIIFCc/NxZJn6hNEFoZuPQ8N5UwDQYJKoZIhvcNAQEL [2024-01-19T09:07:44.550Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:44.550Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:44.551Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:44.551Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:44.551Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:44.551Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:44.551Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:44.551Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:44.551Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:44.551Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAKG3 [2024-01-19T09:07:44.551Z] nVfv4IkgHVOxgbrNIoYhjvBvVJcLZub0C9yidexLA3A61p1a7b/OpidG5zShPhH2 [2024-01-19T09:07:44.551Z] HRdQcqgp0HIxaS+/oLD3eElOnnC/EKD359K6n3kN66sMgvNbHxNFM84Au0TXqmbT [2024-01-19T09:07:44.551Z] w8afrKjrfe/VglpHC2GfmgHMF4YH+ZF48C1RZA6z2oXj6Aj+uww0PjiClE9qUYqK [2024-01-19T09:07:44.551Z] R9uRa32prExihsi+Bto7n2Axo92qUiK2KrkEdQitnQfH5z/PojS78vbyIKp1O86N [2024-01-19T09:07:44.551Z] qRImvCYY17CPxP+JUQ7dxWDMF4NTgSnabkp+cd5i/pyoAb0guQFlQQUnHAZa4QNC [2024-01-19T09:07:44.551Z] 7fgTo1IUwWyDmwCnN5U= [2024-01-19T09:07:44.551Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.551Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.551Z] MIIDCjCCAfKgAwIBAgIUJPzO3O+wmUt4gHCkl3ojsvwZL6owDQYJKoZIhvcNAQEL [2024-01-19T09:07:44.551Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:44.551Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:44.551Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:44.551Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:44.551Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:44.551Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:44.551Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:44.551Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:44.551Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:44.551Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAGVt [2024-01-19T09:07:44.551Z] g9qX+r8kyGYsvFavmnDxgMK1Zkz80irQJ2BJlDHdK1hp5aqpRbT2z2YgLog5ZMIZ [2024-01-19T09:07:44.551Z] uI36uPCOXo8+Gol1jT7D6vsgMWm0Tk9jCNbO4jPkv+wm2q1B9UZo94UGiS0K+G0S [2024-01-19T09:07:44.551Z] nccl4jmSq1gWpBZIVetdDFNjpipKjhEOVQM7x33APhACxSAWRp9iEr6V/CzMOgkf [2024-01-19T09:07:44.551Z] RHyz5D7ljh4a2XVH7++eKsj2QPfKyiKtIG72tCccQ9zGnzveiXXb4Ai8r+DyKH4Z [2024-01-19T09:07:44.551Z] zDt5Hb4ZFl72E3Rl4bIh/VQBVsr9u/SFR1p/Emyq+kGXsn+9YMiM9tK4mLUtjdox [2024-01-19T09:07:44.551Z] LMWUWTelNRGWAiaTVz8= [2024-01-19T09:07:44.551Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.551Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.551Z] MIIDCjCCAfKgAwIBAgIUBxbEoan/HnRZ+2deRj+2RvGMrWQwDQYJKoZIhvcNAQEL [2024-01-19T09:07:44.551Z] BQAwHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2MB4XDTI0MDExNjA4MzAw [2024-01-19T09:07:44.551Z] MFoXDTM0MDExMzA4MzAwMFowHTEbMBkGA1UEAxMScm9vdC1jYS0xNzA1Mzk0MDI2 [2024-01-19T09:07:44.551Z] MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4OWc5cCqIz1u7UAd16Mj [2024-01-19T09:07:44.551Z] UOXUET9+7t0Wkju31j5Lh5hK+fBKMflSXqjV96gEsL814sD3hHPxcPxhfI1zryqc [2024-01-19T09:07:44.551Z] IT0ttTXCoXVxzBJTf2pIwovgnCYT4N2Q9LQxuDf3H43y0cyqtYSkqwcKXVIUYJCE [2024-01-19T09:07:44.551Z] gLLYKGnP8f+2ndguxKUpEnDkwNSWeyHKhq7Ke8KWe6BHsTHqAadbsP7nXjRMDD08 [2024-01-19T09:07:44.551Z] YCogAcYfYkgD5+AMLrXzL1KFytQCpSwrUFd5DiVwIlpDydfCsmG0zIJWfpcsjkyy [2024-01-19T09:07:44.551Z] XnmFFHw8a7tnQ71K+OXJ8nGroKRkz9up+0EXHUY8ccdhR/hgD6Iza0Y7IGnj55PU [2024-01-19T09:07:44.551Z] dQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV [2024-01-19T09:07:44.551Z] HQ4EFgQUqOsEN3C63ffcbVRtn1IRZvWcUTcwDQYJKoZIhvcNAQELBQADggEBAB65 [2024-01-19T09:07:44.551Z] nHU+98oTP0U2J0/TsL5GFeBTH7YyUXUNRJvPWxkA8riB3swoScFKbavO6XJ/hrI2 [2024-01-19T09:07:44.551Z] ywApOz77Iyq58OLa5aIC5aaPRf3YE8LbJQYIHmtza21LXhtlKVSrLu4W58PauI8Z [2024-01-19T09:07:44.551Z] 7iaL9QPUihHFK7wrZ2ZpabN2rRJvTGWWKlhmgFTjI+71gPrXWmYBHIKmTv55V5m1 [2024-01-19T09:07:44.551Z] LxORTR/gBXeH9gmM7wz1cbaYNazqs8R1LS5SW76p/y9RLjTzU6N39VYefMMGwbsQ [2024-01-19T09:07:44.551Z] DlD11wvMBxTBp3hQYc3pdVLkVEQt8trVIwEUts7zfwUTCu/z8gH7H5XePpEVNana [2024-01-19T09:07:44.551Z] JhlAcXHCO7nN/ddJmoc= [2024-01-19T09:07:44.551Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.551Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.551Z] MIIDGDCCAgCgAwIBAgIUEvLaarpXlv1cVvG6nMuxh2DNCKQwDQYJKoZIhvcNAQEL [2024-01-19T09:07:44.551Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:44.551Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:44.551Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:44.551Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:44.551Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:44.551Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:44.551Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:44.551Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:44.552Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:44.552Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:44.552Z] 9w0BAQsFAAOCAQEAOPakzVUmAqHhmQViZZHSQeRh/xYQBrtn6dnEo6X55SXNWcPU [2024-01-19T09:07:44.552Z] 4EEW3DvwBf75uqxKeGoXHTBpqNHG2x1+IncvIKJmi5OWM3aR/vEyJoBOchPBl+A4 [2024-01-19T09:07:44.552Z] 0xZO1vKiS2djxz8XHMkR70cxD8hv/zDJeIQbgBroF4Wa7rim5+y7jgO0O7Rs1zhT [2024-01-19T09:07:44.552Z] LDDStJ8aUKwcVRmOv7NFQxbQHXNx07kaUE2SQJfhryzZmotoBiymmQT2v0H4/4q0 [2024-01-19T09:07:44.552Z] 5dylRhSCWfwN5+It40/C3b1pbC2d9Iv/eO+B0wqg13dLF3xuA1JblEgnqA9mdNnp [2024-01-19T09:07:44.552Z] P6EgKTQBMiO6nPFyYIPqg1eWYkb1eXu9aEkmbw== [2024-01-19T09:07:44.552Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.552Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.552Z] MIIDGDCCAgCgAwIBAgIUWf/qoBANFWF/a/yZr7wfLBMfItAwDQYJKoZIhvcNAQEL [2024-01-19T09:07:44.552Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:44.552Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:44.552Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:44.552Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:44.552Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:44.552Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:44.552Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:44.552Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:44.552Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:44.552Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:44.552Z] 9w0BAQsFAAOCAQEAdCguadZU6WJAuuAtWzvYgftPKkDKNBeGVgbqRwWRCB+QqVFl [2024-01-19T09:07:44.552Z] +3ZOQTStpnXJsY3Ig31hmL88BkBPQ47ENEzs7Czhuz59m+dcBhh/buUoIERSnfUY [2024-01-19T09:07:44.552Z] lWIfqxQevwHYZ3Kjx0l3qliYjKBQJo5xDPnukGb4yUnmrCdZ3fxMzlnr0Ais9rUH [2024-01-19T09:07:44.552Z] MjiDsjP/oUa8naQUTZO4DTRZqeEldzNXnxF6YH+3moTkwRLt9xzirNwECmJ9qxrC [2024-01-19T09:07:44.552Z] eCA7zORxbFSftYLnQiGWUS1dnxBuLOvJHjYPC6EkPPrvqnx2fXcTE1V8GizVRrA6 [2024-01-19T09:07:44.552Z] NPxt+e4qZgU/Nk/Jkpsb7h1FVcKtS8CvmtbdqQ== [2024-01-19T09:07:44.552Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.552Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.552Z] MIIDGDCCAgCgAwIBAgIUI6D0WS4tl52N+w2lwAS+6V0DPyUwDQYJKoZIhvcNAQEL [2024-01-19T09:07:44.552Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:44.552Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:44.552Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:44.552Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:44.552Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:44.552Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:44.552Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:44.552Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:44.552Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:44.552Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:44.552Z] 9w0BAQsFAAOCAQEAAS4yElh6TMyP18ipzRA0Wenukevjs/vtkAFOsv1u5GNeDIrv [2024-01-19T09:07:44.552Z] lNi6/VjvtS+DoFNzQnTf3QoO6GTuoWHLQWt6Z/V5AwIsnOXduSXLugMwEv3j9l50 [2024-01-19T09:07:44.552Z] aDkA7vJ3OH5cGqIvIzxQN7IUOEvZNhY67GFP7+zAzVNRpJ1FHYbhQGPuFD//b9G3 [2024-01-19T09:07:44.552Z] hr4E3+iUEc+F/iBA+caA8wRx1zR383Jz6wuoFt7n7HpWNcgMvoh91JlUAK7yUp0B [2024-01-19T09:07:44.552Z] ekuG6q7HtVsk6X+aB1RyWfrSJjih75W7b+p+/bV4vNmQS1wSGSOaSIuVRvMhmpwf [2024-01-19T09:07:44.552Z] FpT+4rUpiCxklD69/MET/1cWxqvao/ppf6phzg== [2024-01-19T09:07:44.552Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.552Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.552Z] MIIDGDCCAgCgAwIBAgIUIVCCgYV9enDck18n89YsA5b2IdgwDQYJKoZIhvcNAQEL [2024-01-19T09:07:44.552Z] BQAwJDEiMCAGA1UEAxMZY2x1c3Rlci1zaWduZXItMTcwNTM5NDAyNjAeFw0yNDAx [2024-01-19T09:07:44.552Z] MTYwODMwMDBaFw0zNDAxMTMwODMwMDBaMCQxIjAgBgNVBAMTGWNsdXN0ZXItc2ln [2024-01-19T09:07:44.552Z] bmVyLTE3MDUzOTQwMjYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt [2024-01-19T09:07:44.552Z] bRY9u6adE9wbSIzZ3NcVd3dJg8Pm8qIPHJoa1Ce4FvPT+PrvHFphfBUapbB6TBAH [2024-01-19T09:07:44.552Z] wAZaPuApIefUiNFPrKcCS2oD/CXNQ/9Ej91N/Q1DdEKxhIaCDGWugkEm51XzW0zX [2024-01-19T09:07:44.552Z] OZ72ADz4GYHlU5qZkSIRzQlnRKZdhVq6kBgm0Mfm/pJp0z+sS6hSBYmCNVit/rkk [2024-01-19T09:07:44.552Z] ZZVTVDTjRyhIj9iO3r/JZ/GeRJpXmnBJFdJpM5QywGEoU9Ax3z1dsVWj/TBd8rg2 [2024-01-19T09:07:44.552Z] xJcT+rqrqwvby/9H6Xr2rGwc+cWGhsVMLDQlHahB3+8s113WRi3jbHu0ZPfu9jF+ [2024-01-19T09:07:44.552Z] pO3fFRbspMtmzP34p8T/AgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMB [2024-01-19T09:07:44.552Z] Af8EBTADAQH/MB0GA1UdDgQWBBTgihiMxfCH3xU5U51LZ+K2pt5KZTANBgkqhkiG [2024-01-19T09:07:44.552Z] 9w0BAQsFAAOCAQEAa3eFm1ivFf5Vom6a/7E1gDdjl7cjUZ5cqg332Ky1NdzEOiwW [2024-01-19T09:07:44.552Z] mx3yEk35qCd8LAoFMvPvPIuMx1NDS8kwy3FmIOjKnZapHA31ZUpn/gT/seH4GFIU [2024-01-19T09:07:44.553Z] oZw+rODV5h6jmtCVugAn48u/LxOXbY417JWwaeDx5uw+NzOSyebe+td2PbtUjf5X [2024-01-19T09:07:44.553Z] 6xcwYUmaUGsIAgfIMM12hzXOI2/q1qz5dAqwSgnR7dKBReOXnUoFpNOMpxlJ70p/ [2024-01-19T09:07:44.553Z] WUU1aWIWZqKrNI7Pg1T5jljq7+KbT3+A6+ux0FLExr2c1tw3NndXS2Y9fO55m5/q [2024-01-19T09:07:44.553Z] 0EcrTdKx4+QjbWcja3WdX2pGDk0XGNgx+EXn6Q== [2024-01-19T09:07:44.553Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.553Z] -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.553Z] MIIFATCCAumgAwIBAgIRAKc9ZKBASymy5TLOEp57N98wDQYJKoZIhvcNAQELBQAw [2024-01-19T09:07:44.553Z] GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDMyMzIyNTM0NloXDTM2 [2024-01-19T09:07:44.553Z] MDMyMzIyNTM0NlowGjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMIICIjANBgkq [2024-01-19T09:07:44.553Z] hkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA+pYHvQw5iU3v2b3iNuYNKYgsWD6KU7aJ [2024-01-19T09:07:44.553Z] diddtZQxSWYzUI3U0I1UsRPTxnhTifs/M9NW4ZlV13ZfB7APwC8oqKOIiwo7IwlP [2024-01-19T09:07:44.553Z] xg0VKgyz+kT8RJfYr66PPIYP0fpTeu42LpMJ+CKo9sbpgVNDZN2z/qiXrRNX/VtG [2024-01-19T09:07:44.553Z] TkPV7a44fZ5bHHVruAxvDnylpQxJobtCBWlJSsbIRGFHMc2z88eUz9NmIOWUKGGj [2024-01-19T09:07:44.553Z] EmP76x8OfRHpIpuxRSCjn0+i9+hR2siIOpcMOGd+40uVJxbRRP5ZXnUFa2fF5FWd [2024-01-19T09:07:44.553Z] O0u0RPI8HON0ovhrwPJY+4eWKkQzyC611oLPYGQ4EbifRsTsCxUZqyUuStGyp8oa [2024-01-19T09:07:44.553Z] aoSKfF6X0+KzGgwwnrjRTUpIl19A92KR0Noo6h622OX+4sZiO/JQdkuX5w/HupK0 [2024-01-19T09:07:44.553Z] A0M0WSMCvU6GOhjGotmh2VTEJwHHY4+TUk0iQYRtv1crONklyZoAQPD76hCrC8Cr [2024-01-19T09:07:44.553Z] IbgsZLfTMC8TWUoMbyUDgvgYkHKMoPm0VGVVuwpRKJxv7+2wXO+pivrrUl2Q9fPe [2024-01-19T09:07:44.553Z] Kk055nJLMV9yPUdig8othUKrRfSxli946AEV1eEOhxddfEwBE3Lt2xn0hhiIedbb [2024-01-19T09:07:44.553Z] Ftf/5kEWFZkXyUmMJK8Ra76Kus2ABueUVEcZ48hrRr1Hf1N9n59VbTUaXgeiZA50 [2024-01-19T09:07:44.553Z] qXf2bymE6F8CAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMB [2024-01-19T09:07:44.553Z] Af8wHQYDVR0OBBYEFMEmdKSKRKDm+iAo2FwjmkWIGHngMA0GCSqGSIb3DQEBCwUA [2024-01-19T09:07:44.553Z] A4ICAQBCPw74M9X/Xx04K1VAES3ypgQYH5bf9FXVDrwhRFSVckria/7dMzoF5wln [2024-01-19T09:07:44.553Z] uq9NGsjkkkDg17AohcQdr8alH4LvPdxpKr3BjpvEcmbqF8xH+MbbeUEnmbSfLI8H [2024-01-19T09:07:44.553Z] sefuhXF9AF/9iYvpVNC8FmJ0OhiVv13VgMQw0CRKkbtjZBf8xaEhq/YqxWVsgOjm [2024-01-19T09:07:44.553Z] dm5CAQ2X0aX7502x8wYRgMnZhA5goC1zVWBVAi8yhhmlhhoDUfg17cXkmaJC5pDd [2024-01-19T09:07:44.553Z] oenZ9NVhW8eDb03MFCrWNvIh89DDeCGWuWfDltDq0n3owyL0IeSn7RfpSclpxVmV [2024-01-19T09:07:44.553Z] /53jkYjwIgxIG7Gsv0LKMbsf6QdBcTjhvfZyMIpBRkTe3zuHd2feKzY9lEkbRvRQ [2024-01-19T09:07:44.553Z] zbh4Ps5YBnG6CKJPTbe2hfi3nhnw/MyEmF3zb0hzvLWNrR9XW3ibb2oL3424XOwc [2024-01-19T09:07:44.553Z] VjrTSCLzO9Rv6s5wi03qoWvKAQQAElqTYRHhynJ3w6wuvKYF5zcZF3MDnrVGLbh1 [2024-01-19T09:07:44.553Z] Q9ePRFBCiXOQ6wPLoUhrrbZ8LpFUFYDXHMtYM7P9sc9IAWoONXREJaO08zgFtMp4 [2024-01-19T09:07:44.553Z] 8iyIYUyQAbsvx8oD2M8kRvrIRSrRJSl6L957b4AFiLIQ/GgV2curs0jje7Edx34c [2024-01-19T09:07:44.553Z] idWw1VrejtwclobqNMVtG3EiPUIpJGpbMcJgbiLSmKkrvQtGng== [2024-01-19T09:07:44.553Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.553Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} [2024-01-19T09:07:44.553Z] INFO[2024-01-19 09:07:44] Actual:true, Expected: true [2024-01-19T09:07:44.811Z] INFO[2024-01-19 09:07:44] Verifying : Description : Fetching configmap pointer &ConfigMap{ObjectMeta:{openshift-service-ca.crt mysql-ibm-pxb-2-84852-01-19-09h05m53s 7ed3f1d7-f781-409d-89e8-6ad5ea8e3f5a 1831392 0 2024-01-19 09:05:54 +0000 UTC map[] map[service.beta.openshift.io/inject-cabundle:true] [] [] [{kube-controller-manager Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:data":{},"f:metadata":{"f:annotations":{".":{},"f:service.beta.openshift.io/inject-cabundle":{}}}} } {service-ca-operator Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:data":{"f:service-ca.crt":{}}} }]},Data:map[string]string{service-ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T09:07:44.811Z] MIIDUTCCAjmgAwIBAgIILvRjno8BjvIwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE [2024-01-19T09:07:44.811Z] Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwNTM5NjA4ODAe [2024-01-19T09:07:44.811Z] Fw0yNDAxMTYwOTA4MDhaFw0yNjAzMTYwOTA4MDlaMDYxNDAyBgNVBAMMK29wZW5z [2024-01-19T09:07:44.811Z] aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDUzOTYwODgwggEiMA0GCSqG [2024-01-19T09:07:44.811Z] SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVr8xmLKsMoemVtfnlzzil6B8CGocEhNRo [2024-01-19T09:07:44.811Z] fkzkD8aSOYgIj5zcZo/M0O7nXDXeg9dqx9/h2qrGk+SS2ikU2Xs4Q5gaQDh9U1lB [2024-01-19T09:07:44.811Z] y8paoOEvw06CmJ/1Qqur5A539/OnimRSTdejTu4lwaDwsJiyoGfhvrBJAdjagtYA [2024-01-19T09:07:44.811Z] 0sNMrCgEAT4NWHS5EDx5dF7nMDHmQ0h0gHYei6vUEtiPyLwcTH1T7uVSj22fCEXX [2024-01-19T09:07:44.811Z] LjgN3OSiquJ+Hf4+csPpwgxAEkfjdBldhnImYkunJbg/x4c90uFL1pygZusrbFAf [2024-01-19T09:07:44.811Z] cERW8GngZuL5JY8MQoIstMu+1qS390ouQDDbEcvWjxC8YFNFHywrAgMBAAGjYzBh [2024-01-19T09:07:44.811Z] MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQS5vRk [2024-01-19T09:07:44.811Z] jkBPISX2qYmk1VEQZIXSLzAfBgNVHSMEGDAWgBQS5vRkjkBPISX2qYmk1VEQZIXS [2024-01-19T09:07:44.811Z] LzANBgkqhkiG9w0BAQsFAAOCAQEAZ/qc2vFGJZgDJCEUaPYuufuLRPSF68ixV9fk [2024-01-19T09:07:44.811Z] e5tBkRN8atCsEBygC/RUcwULlhsdgUMGMcREgNC/FYlwkOsAj3gP8KmcXAZ4gHCa [2024-01-19T09:07:44.811Z] T6PIbIcj1hCX0mdX0tvtaj7PWvRUz9CHh6iFzmPsXK5xjaBA9vZbkUAa7odguUit [2024-01-19T09:07:44.811Z] vz1QnKoV1OaZyccDLeQL50khM1XEBn8L+zeY3k85uo2X514HZW4LDPTQ94cpyNnb [2024-01-19T09:07:44.811Z] zqI6utp7JoPJexsHMWNFKGb//HgjluNg7TsywXIeebk+/GZUEDfNKid0oiTpQvzR [2024-01-19T09:07:44.811Z] TFFF+5KO/sa4+qIh1ERm140JxeaurUMaJzn7cFQbekEgt/N+iA== [2024-01-19T09:07:44.811Z] -----END CERTIFICATE----- [2024-01-19T09:07:44.811Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} from namespace mysql-ibm-pxb-2-84852-01-19-09h05m53s [2024-01-19T09:07:44.811Z] INFO[2024-01-19 09:07:44] Actual:true, Expected: true [2024-01-19T09:07:44.811Z] 2024-01-19 09:07:44 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.AddLabelToResource:#5038] - Adding label [059eb132-a858-4c46-8f77-d637339d05d0=f1086e89-2c30-44b2-906d-336a8bfa3a9a] to ConfigMap openshift-service-ca.crt [2024-01-19T09:07:45.068Z] INFO[2024-01-19 09:07:44] Verifying : Description : Adding labels 059eb132-a858-4c46-8f77-d637339d05d0=f1086e89-2c30-44b2-906d-336a8bfa3a9a to resource &ConfigMap{ObjectMeta:{openshift-service-ca.crt mysql-ibm-pxb-2-84852-01-19-09h05m53s 7ed3f1d7-f781-409d-89e8-6ad5ea8e3f5a 1831392 0 2024-01-19 09:05:54 +0000 UTC map[059eb132-a858-4c46-8f77-d637339d05d0:f1086e89-2c30-44b2-906d-336a8bfa3a9a] map[service.beta.openshift.io/inject-cabundle:true] [] [] [{kube-controller-manager Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:data":{},"f:metadata":{"f:annotations":{".":{},"f:service.beta.openshift.io/inject-cabundle":{}}}} } {service-ca-operator Update v1 2024-01-19 09:05:54 +0000 UTC FieldsV1 {"f:data":{"f:service-ca.crt":{}}} }]},Data:map[string]string{service-ca.crt: -----BEGIN CERTIFICATE----- [2024-01-19T09:07:45.068Z] MIIDUTCCAjmgAwIBAgIILvRjno8BjvIwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE [2024-01-19T09:07:45.068Z] Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwNTM5NjA4ODAe [2024-01-19T09:07:45.068Z] Fw0yNDAxMTYwOTA4MDhaFw0yNjAzMTYwOTA4MDlaMDYxNDAyBgNVBAMMK29wZW5z [2024-01-19T09:07:45.068Z] aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDUzOTYwODgwggEiMA0GCSqG [2024-01-19T09:07:45.068Z] SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVr8xmLKsMoemVtfnlzzil6B8CGocEhNRo [2024-01-19T09:07:45.068Z] fkzkD8aSOYgIj5zcZo/M0O7nXDXeg9dqx9/h2qrGk+SS2ikU2Xs4Q5gaQDh9U1lB [2024-01-19T09:07:45.068Z] y8paoOEvw06CmJ/1Qqur5A539/OnimRSTdejTu4lwaDwsJiyoGfhvrBJAdjagtYA [2024-01-19T09:07:45.068Z] 0sNMrCgEAT4NWHS5EDx5dF7nMDHmQ0h0gHYei6vUEtiPyLwcTH1T7uVSj22fCEXX [2024-01-19T09:07:45.068Z] LjgN3OSiquJ+Hf4+csPpwgxAEkfjdBldhnImYkunJbg/x4c90uFL1pygZusrbFAf [2024-01-19T09:07:45.068Z] cERW8GngZuL5JY8MQoIstMu+1qS390ouQDDbEcvWjxC8YFNFHywrAgMBAAGjYzBh [2024-01-19T09:07:45.068Z] MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQS5vRk [2024-01-19T09:07:45.068Z] jkBPISX2qYmk1VEQZIXSLzAfBgNVHSMEGDAWgBQS5vRkjkBPISX2qYmk1VEQZIXS [2024-01-19T09:07:45.068Z] LzANBgkqhkiG9w0BAQsFAAOCAQEAZ/qc2vFGJZgDJCEUaPYuufuLRPSF68ixV9fk [2024-01-19T09:07:45.068Z] e5tBkRN8atCsEBygC/RUcwULlhsdgUMGMcREgNC/FYlwkOsAj3gP8KmcXAZ4gHCa [2024-01-19T09:07:45.068Z] T6PIbIcj1hCX0mdX0tvtaj7PWvRUz9CHh6iFzmPsXK5xjaBA9vZbkUAa7odguUit [2024-01-19T09:07:45.068Z] vz1QnKoV1OaZyccDLeQL50khM1XEBn8L+zeY3k85uo2X514HZW4LDPTQ94cpyNnb [2024-01-19T09:07:45.068Z] zqI6utp7JoPJexsHMWNFKGb//HgjluNg7TsywXIeebk+/GZUEDfNKid0oiTpQvzR [2024-01-19T09:07:45.068Z] TFFF+5KO/sa4+qIh1ERm140JxeaurUMaJzn7cFQbekEgt/N+iA== [2024-01-19T09:07:45.068Z] -----END CERTIFICATE----- [2024-01-19T09:07:45.068Z] ,},BinaryData:map[string][]byte{},Immutable:nil,} [2024-01-19T09:07:45.068Z] INFO[2024-01-19 09:07:44] Actual:true, Expected: true [2024-01-19T09:07:45.068Z] 2024-01-19 09:07:44 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Creating cloud credentials and registering backup location [2024-01-19T09:07:46.436Z] 2024-01-19 09:07:46 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [autogenerated-cred-1705655266] for org [default] with [aws] as provider [2024-01-19T09:07:46.436Z] 2024-01-19 09:07:46 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.CreateCloudCredential:#4389] - Create creds for Aws [2024-01-19T09:07:46.436Z] 2024-01-19 09:07:46 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-1705655266] is present [2024-01-19T09:07:46.436Z] INFO[2024-01-19 09:07:46] Verifying : Description : Creating cloud credentials autogenerated-cred-1705655266 with provider aws [2024-01-19T09:07:46.436Z] INFO[2024-01-19 09:07:46] Actual:true, Expected: true [2024-01-19T09:07:46.436Z] 2024-01-19 09:07:46 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Created cloud credentials with name - autogenerated-cred-1705655266 [2024-01-19T09:07:49.707Z] INFO[2024-01-19 09:07:49] Verifying : Description : Creating backup location autogenerated-backup-location-1705655266 [2024-01-19T09:07:49.707Z] INFO[2024-01-19 09:07:49] Actual:true, Expected: true [2024-01-19T09:07:49.707Z] 2024-01-19 09:07:49 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Adding source and destination clusters with px-central-admin ctx [2024-01-19T09:07:49.707Z] INFO[2024-01-19 09:07:49] Verifying : Description : Getting KUBECONFIGS Environment variable [2024-01-19T09:07:49.707Z] INFO[2024-01-19 09:07:49] Actual:true, Expected: true [2024-01-19T09:07:49.707Z] 2024-01-19 09:07:49 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.dumpKubeConfigs:#5950] - dump kubeconfigs to file system [2024-01-19T09:07:49.707Z] 2024-01-19 09:07:49 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.dumpKubeConfigs:#5956] - Get over kubeconfig list [source-config destination-config] [2024-01-19T09:07:49.707Z] 2024-01-19 09:07:49 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/source-config [2024-01-19T09:07:49.707Z] 2024-01-19 09:07:49 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.dumpKubeConfigs:#5965] - Save kubeconfig to /tmp/destination-config [2024-01-19T09:07:49.707Z] 2024-01-19 09:07:49 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Create cluster [source-cluster] in org [default] [2024-01-19T09:07:49.707Z] 2024-01-19 09:07:49 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T09:07:49.707Z] 2024-01-19 09:07:49 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.CreateApplicationClusters:#4061] - Save cluster source-cluster kubeconfig to /tmp/source-config [2024-01-19T09:07:49.708Z] 2024-01-19 09:07:49 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Create cluster [destination-cluster] in org [default] [2024-01-19T09:07:49.708Z] 2024-01-19 09:07:49 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T09:07:49.708Z] 2024-01-19 09:07:49 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.CreateApplicationClusters:#4068] - Save cluster destination-cluster kubeconfig to /tmp/destination-config [2024-01-19T09:07:49.708Z] 2024-01-19 09:07:49 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-source-config-cloud-cred-qaeks] for IBM [2024-01-19T09:07:49.708Z] 2024-01-19 09:07:49 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-source-config-cloud-cred-qaeks] for org [default] with [ibm] as provider [2024-01-19T09:07:49.708Z] 2024-01-19 09:07:49 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T09:07:49.708Z] 2024-01-19 09:07:49 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T09:07:49.963Z] 2024-01-19 09:07:49 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-source-config-cloud-cred-qaeks] is present [2024-01-19T09:07:49.963Z] 2024-01-19 09:07:49 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Create cluster [source-cluster] in org [default] [2024-01-19T09:07:54.148Z] 2024-01-19 09:07:53 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.CreateApplicationClusters:#4231] - Cluster credential with name [ibm-destination-config-cloud-cred-robsb] for IBM [2024-01-19T09:07:54.149Z] 2024-01-19 09:07:53 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.CreateCloudCredential:#4385] - Create cloud credential with name [ibm-destination-config-cloud-cred-robsb] for org [default] with [ibm] as provider [2024-01-19T09:07:54.149Z] 2024-01-19 09:07:53 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.CreateCloudCredential:#4462] - Create creds for IBM [2024-01-19T09:07:54.149Z] 2024-01-19 09:07:53 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.GetIBMApiKey:#5136] - Snigdha: %!(EXTRA backup.IBMCredential={ **** }) [2024-01-19T09:07:54.149Z] 2024-01-19 09:07:53 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [ibm-destination-config-cloud-cred-robsb] is present [2024-01-19T09:07:54.149Z] 2024-01-19 09:07:53 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Create cluster [destination-cluster] in org [default] [2024-01-19T09:07:54.149Z] INFO[2024-01-19 09:07:54] Verifying : Description : Verifying creation of source [source-cluster] and destination [destination-cluster] clusters [2024-01-19T09:07:54.149Z] INFO[2024-01-19 09:07:54] Actual:true, Expected: true [2024-01-19T09:07:54.149Z] INFO[2024-01-19 09:07:54] Verifying : Description : Verifying if [destination-cluster] cluster is online [2024-01-19T09:07:54.149Z] INFO[2024-01-19 09:07:54] Actual:Online, Expected: Online [2024-01-19T09:07:54.149Z] INFO[2024-01-19 09:07:54] Verifying : Description : Fetching [destination-cluster] cluster uid [2024-01-19T09:07:54.149Z] INFO[2024-01-19 09:07:54] Actual:true, Expected: true [2024-01-19T09:07:54.149Z] 2024-01-19 09:07:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Uid of [destination-cluster] cluster is 83de4b4b-9f8e-4fb2-946b-2f5b8022d68c [2024-01-19T09:07:54.149Z] 2024-01-19 09:07:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Creating a schedule policy [2024-01-19T09:07:54.404Z] INFO[2024-01-19 09:07:54] Verifying : Description : Fetching px-central-admin ctx [2024-01-19T09:07:54.404Z] INFO[2024-01-19 09:07:54] Actual:true, Expected: true [2024-01-19T09:07:54.404Z] 2024-01-19 09:07:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Create Backup Schedule Policy [2024-01-19T09:07:55.332Z] INFO[2024-01-19 09:07:54] Verifying : Description : Verifying creation of periodic schedule policy of interval 15 minutes named [periodic-1705655274] [2024-01-19T09:07:55.332Z] INFO[2024-01-19 09:07:54] Actual:true, Expected: true [2024-01-19T09:07:55.332Z] INFO[2024-01-19 09:07:55] Verifying : Description : Fetching uid of periodic schedule policy named [periodic-1705655274] [2024-01-19T09:07:55.332Z] INFO[2024-01-19 09:07:55] Actual:true, Expected: true [2024-01-19T09:07:55.332Z] 2024-01-19 09:07:55 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Creating a schedule backup using namespace label and resource label [2024-01-19T09:07:56.698Z] INFO[2024-01-19 09:07:56] Verifying : Description : Fetching px-central-admin ctx [2024-01-19T09:07:56.698Z] INFO[2024-01-19 09:07:56] Actual:true, Expected: true [2024-01-19T09:07:56.698Z] 2024-01-19 09:07:56 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [backup.AdditionalScheduledBackupRequestParams:#4012] - Detected backup type - native_csi [2024-01-19T09:07:56.698Z] 2024-01-19 09:07:56 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Volume snapshot classes found in the cluster - [2024-01-19T09:07:56.698Z] 2024-01-19 09:07:56 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] ibmc-vpcblock-snapshot [2024-01-19T09:07:56.698Z] 2024-01-19 09:07:56 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Picking the volume snapshot class [ibmc-vpcblock-snapshot] from env variable [VOLUME_SNAPSHOT_CLASS] [2024-01-19T09:09:04.321Z] 2024-01-19 09:08:56 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] first schedule backup for schedule name [tp-backup-schedule-1705655276] is [tp-backup-schedule-1705655276-ff78f6a-interval-2024-01-19-090756] [2024-01-19T09:09:04.321Z] 2024/01/19 09:08:56 DoRetryWithTimeout - Error: {backup status for [tp-backup-schedule-1705655276-ff78f6a-interval-2024-01-19-090756] expected was [[Success]] but got [InProgress] because of [Application resources backup is in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T09:09:30.841Z] 2024-01-19 09:09:26 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [backup.CreateScheduleBackupWithNamespaceLabel:#3059] - Successfully created schedule backup [tp-backup-schedule-1705655276-ff78f6a-interval-2024-01-19-090756] with namespace label [1-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,5-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,11-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,12-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,13-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,3-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,4-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,6-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,18-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,2-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,7-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,8-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,9-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,10-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,14-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,16-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,19-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,0-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,15-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,17-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a] [2024-01-19T09:09:30.841Z] INFO[2024-01-19 09:09:26] Verifying : Description : Verifying creation of schedule backup with schedule name [tp-backup-schedule-1705655276] [2024-01-19T09:09:30.841Z] INFO[2024-01-19 09:09:26] Actual:true, Expected: true [2024-01-19T09:09:30.841Z] INFO[2024-01-19 09:09:26] Verifying : Description : Fetching the name of the first schedule backup [tp-backup-schedule-1705655276-ff78f6a-interval-2024-01-19-090756] [2024-01-19T09:09:30.841Z] INFO[2024-01-19 09:09:26] Actual:true, Expected: true [2024-01-19T09:09:30.841Z] 2024-01-19 09:09:26 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup tp-backup-schedule-1705655276-ff78f6a-interval-2024-01-19-090756 [2024-01-19T09:09:30.841Z] 2024-01-19 09:09:26 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [mysql-ibm-pxb-0-84852-01-19-09h05m53s mysql-ibm-pxb-1-84852-01-19-09h05m53s mysql-ibm-pxb-2-84852-01-19-09h05m53s] [2024-01-19T09:09:30.842Z] 2024-01-19 09:09:26 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are 1-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,5-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,11-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,12-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,13-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,3-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,4-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,6-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,18-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,2-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,7-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,8-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,9-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,10-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,14-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,16-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,19-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,0-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,15-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,17-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a [2024-01-19T09:09:30.842Z] INFO[2024-01-19 09:09:26] Verifying : Description : Verifying the labeled namespaces [[mysql-ibm-pxb-0-84852-01-19-09h05m53s mysql-ibm-pxb-1-84852-01-19-09h05m53s mysql-ibm-pxb-2-84852-01-19-09h05m53s]] are backed up and checks for labels [1-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,5-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,11-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,12-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,13-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,3-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,4-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,6-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,18-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,2-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,7-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,8-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,9-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,10-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,14-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,16-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,19-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,0-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,15-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,17-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a] applied to the backup [tp-backup-schedule-1705655276-ff78f6a-interval-2024-01-19-090756] [2024-01-19T09:09:30.842Z] INFO[2024-01-19 09:09:26] Actual:true, Expected: true [2024-01-19T09:09:30.842Z] 2024-01-19 09:09:26 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Waiting for 15 minutes for the next schedule backup to be triggered [2024-01-19T09:15:22.365Z] INFO[2024-01-19 09:15:14] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T09:15:22.365Z] INFO[2024-01-19 09:15:14] sched stats: tick interval: 63674 < 1.2s (100%) [2024-01-19T09:15:22.365Z] INFO[2024-01-19 09:15:14] sched stats: tick process durations: 63673 < 500ms (100%) [2024-01-19T09:24:28.941Z] INFO[2024-01-19 09:24:27] Verifying : Description : Fetching the name of the second schedule backup [tp-backup-schedule-1705655276-ff78f6a-interval-2024-01-19-092259] [2024-01-19T09:24:28.941Z] INFO[2024-01-19 09:24:27] Actual:true, Expected: true [2024-01-19T09:24:28.941Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup tp-backup-schedule-1705655276-ff78f6a-interval-2024-01-19-092259 [2024-01-19T09:24:28.941Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [mysql-ibm-pxb-0-84852-01-19-09h05m53s mysql-ibm-pxb-1-84852-01-19-09h05m53s mysql-ibm-pxb-2-84852-01-19-09h05m53s] [2024-01-19T09:24:28.941Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are 1-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,5-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,11-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,12-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,13-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,3-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,4-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,6-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,18-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,2-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,7-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,8-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,9-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,10-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,14-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,16-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,19-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,0-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,15-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,17-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a [2024-01-19T09:24:28.942Z] INFO[2024-01-19 09:24:27] Verifying : Description : Verifying the labeled namespace [[mysql-ibm-pxb-0-84852-01-19-09h05m53s mysql-ibm-pxb-1-84852-01-19-09h05m53s mysql-ibm-pxb-2-84852-01-19-09h05m53s]] is backed up and checks for labels [1-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,5-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,11-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,12-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,13-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,3-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,4-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,6-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,18-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,2-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,7-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,8-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,9-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,10-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,14-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,16-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,19-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,0-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,15-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,17-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a] applied to the backup [tp-backup-schedule-1705655276-ff78f6a-interval-2024-01-19-092259] [2024-01-19T09:24:28.942Z] INFO[2024-01-19 09:24:27] Actual:true, Expected: true [2024-01-19T09:24:28.942Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Scheduling applications to create new namespaces [2024-01-19T09:24:28.942Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T09:24:28.942Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T09:24:28.942Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T09:24:28.942Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T09:24:28.942Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T09:24:28.942Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T09:24:28.943Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T09:24:28.943Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T09:24:28.944Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T09:24:28.944Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T09:24:28.944Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Scheduled application with namespace [mysql-ibm-new-namespace-0-84852-01-19-09h05m53s] [2024-01-19T09:24:28.944Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.CreateScheduleOptions:#1744] - Creating ScheduleOptions [2024-01-19T09:24:28.944Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.CreateScheduleOptions:#1783] - ScheduleOptions: Scheduling Apps with hyper-converged [2024-01-19T09:24:28.944Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createStorageObject:#1886] - Setting provisioner of mysql-sc to vpc.block.csi.ibm.io [2024-01-19T09:24:28.944Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createStorageObject:#1904] - [mysql-ibm] Found existing storage class: mysql-sc [2024-01-19T09:24:28.944Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createStorageObject:#1990] - [mysql-ibm] Created PVC: mysql-data [2024-01-19T09:24:28.944Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createCoreObject:#2468] - [mysql-ibm] Created Service: mysql [2024-01-19T09:24:28.945Z] {Volumes:[{Name:mysql-persistent-storage VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:mysql-data,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:mysql Image:mysql:5.6 Command:[] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:3306 Protocol: HostIP:}] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:mysql-persistent-storage ReadOnly:false MountPath:/var/lib/mysql SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysqladmin -u **** -p$MYSQL_ROOT_PASSWORD ping],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:70,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[sh -c mysql -u **** -p$MYSQL_ROOT_PASSWORD -e "select 1"],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:0,FailureThreshold:0,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T09:24:28.945Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysql [2024-01-19T09:24:28.946Z] {Volumes:[] InitContainers:[] Containers:[{Name:mysqlslap Image:adityadani/mysqlslap Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:MYSQL_ROOT_PASSWORD Value:password ValueFrom:nil} {Name:MYSQL_SERVICE_PORT Value:3306 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds: ActiveDeadlineSeconds: DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken: NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace: SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority: DNSConfig:nil ReadinessGates:[] RuntimeClassName: EnableServiceLinks: PreemptionPolicy: Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN: OS:nil HostUsers:} [2024-01-19T09:24:28.946Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).createCoreObject:#2359] - [mysql-ibm] Created deployment: mysqlslap [2024-01-19T09:24:28.946Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Scheduled application with namespace [mysql-ibm-new-namespace-1-84852-01-19-09h05m53s] [2024-01-19T09:24:28.946Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Validating new namespaces [2024-01-19T09:24:28.946Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Validate applications [2024-01-19T09:24:28.946Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Validating mysql-ibm app [2024-01-19T09:24:28.946Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Validating mysql-ibm app's volumes [2024-01-19T09:24:28.946Z] 2024-01-19 09:24:27 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T09:24:28.946Z] 2024/01/19 09:24:27 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T09:24:38.914Z] 2024/01/19 09:24:37 DoRetryWithTimeout - Error: {PVC mysql-data is not ready yet. Cause: PVC expected status: Bound PVC actual status: Pending}, Next try in [10s], timeout [8m0s] [2024-01-19T09:24:48.885Z] 2024-01-19 09:24:47 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-new-namespace-0-84852-01-19-09h05m53s [2024-01-19T09:24:48.885Z] 2024-01-19 09:24:47 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T09:24:48.885Z] 2024-01-19 09:24:47 +0000:[WARNING] [{BackupScheduleForOldAndNewNS}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T09:24:48.885Z] 2024-01-19 09:24:47 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] wait for mysql-ibm app to start running [2024-01-19T09:24:48.885Z] 2024-01-19 09:24:47 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T09:24:48.886Z] 2024/01/19 09:24:48 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T09:24:48.886Z] pod name:mysql-7d766cd65c-4plhs namespace:mysql-ibm-new-namespace-0-84852-01-19-09h05m53s running:false ready:false node:10.240.64.8 [2024-01-19T09:24:48.886Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T09:24:58.822Z] 2024/01/19 09:24:58 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T09:24:58.822Z] pod name:mysql-7d766cd65c-4plhs namespace:mysql-ibm-new-namespace-0-84852-01-19-09h05m53s running:true ready:false node:10.240.64.8 [2024-01-19T09:24:58.822Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T09:25:08.786Z] 2024/01/19 09:25:08 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T09:25:08.786Z] pod name:mysql-7d766cd65c-4plhs namespace:mysql-ibm-new-namespace-0-84852-01-19-09h05m53s running:true ready:false node:10.240.64.8 [2024-01-19T09:25:08.786Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T09:25:16.893Z] INFO[2024-01-19 09:25:15] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T09:25:16.893Z] INFO[2024-01-19 09:25:15] sched stats: tick interval: 64275 < 1.2s (100%) [2024-01-19T09:25:16.893Z] INFO[2024-01-19 09:25:15] sched stats: tick process durations: 64274 < 500ms (100%) [2024-01-19T09:25:18.784Z] 2024/01/19 09:25:18 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T09:25:18.784Z] pod name:mysql-7d766cd65c-4plhs namespace:mysql-ibm-new-namespace-0-84852-01-19-09h05m53s running:true ready:false node:10.240.64.8 [2024-01-19T09:25:18.784Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T09:25:28.722Z] 2024/01/19 09:25:28 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T09:25:28.722Z] pod name:mysql-7d766cd65c-4plhs namespace:mysql-ibm-new-namespace-0-84852-01-19-09h05m53s running:true ready:false node:10.240.64.8 [2024-01-19T09:25:28.722Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T09:25:38.663Z] 2024/01/19 09:25:38 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T09:25:38.663Z] pod name:mysql-7d766cd65c-4plhs namespace:mysql-ibm-new-namespace-0-84852-01-19-09h05m53s running:true ready:false node:10.240.64.8 [2024-01-19T09:25:38.663Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T09:25:50.829Z] 2024/01/19 09:25:48 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T09:25:50.829Z] pod name:mysql-7d766cd65c-4plhs namespace:mysql-ibm-new-namespace-0-84852-01-19-09h05m53s running:true ready:false node:10.240.64.8 [2024-01-19T09:25:50.829Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T09:25:58.925Z] 2024-01-19 09:25:58 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T09:25:58.925Z] 2024-01-19 09:25:58 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T09:25:59.189Z] 2024-01-19 09:25:59 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] validate if mysql-ibm app's volumes are setup [2024-01-19T09:25:59.189Z] 2024-01-19 09:25:59 +0000:[WARNING] [{BackupScheduleForOldAndNewNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T09:25:59.189Z] 2024-01-19 09:25:59 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T09:25:59.189Z] 2024-01-19 09:25:59 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T09:25:59.189Z] 2024-01-19 09:25:59 +0000:[WARNING] [{BackupScheduleForOldAndNewNS}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T09:25:59.189Z] 2024-01-19 09:25:59 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Validating mysql-ibm app [2024-01-19T09:25:59.189Z] 2024-01-19 09:25:59 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Validating mysql-ibm app's volumes [2024-01-19T09:25:59.189Z] 2024-01-19 09:25:59 +0000:[WARNING] [{BackupScheduleForOldAndNewNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T09:25:59.189Z] 2024-01-19 09:25:59 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T09:25:59.189Z] 2024-01-19 09:25:59 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).ValidateVolumes:#3800] - [mysql-ibm] Validated PVC: mysql-data, Namespace: mysql-ibm-new-namespace-1-84852-01-19-09h05m53s [2024-01-19T09:25:59.446Z] 2024-01-19 09:25:59 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).ValidateVolumes:#3839] - [mysql-ibm] Autopilot is not enabled, skipping PVC: mysql-data size validation [2024-01-19T09:25:59.446Z] 2024-01-19 09:25:59 +0000:[WARNING] [{BackupScheduleForOldAndNewNS}] [ibm.(*ibm).ValidateCreateVolume:#51] - ValidateCreateVolume function has not been implemented for volume driver - ibm [2024-01-19T09:25:59.446Z] 2024-01-19 09:25:59 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] wait for mysql-ibm app to start running [2024-01-19T09:25:59.446Z] 2024-01-19 09:25:59 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).WaitForRunning:#2957] - [mysql-ibm] Validated Service: mysql [2024-01-19T09:25:59.446Z] 2024/01/19 09:25:59 DoRetryWithTimeout - Error: {app mysql is not ready yet. Cause: Expected replicas: 1 Available replicas: 0 Current pods overview: [2024-01-19T09:25:59.447Z] pod name:mysql-7d766cd65c-ktr4q namespace:mysql-ibm-new-namespace-1-84852-01-19-09h05m53s running:true ready:false node:10.240.64.27 [2024-01-19T09:25:59.447Z] }, Next try in [10s], timeout [10m0s] [2024-01-19T09:26:11.614Z] 2024-01-19 09:26:09 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysql [2024-01-19T09:26:11.614Z] 2024-01-19 09:26:09 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).WaitForRunning:#2920] - [mysql-ibm] Validated deployment: mysqlslap [2024-01-19T09:26:11.614Z] 2024-01-19 09:26:09 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] validate if mysql-ibm app's volumes are setup [2024-01-19T09:26:11.614Z] 2024-01-19 09:26:09 +0000:[WARNING] [{BackupScheduleForOldAndNewNS}] [ibm.(*ibm).InspectVolume:#91] - InspectVolume function has not been implemented for volume driver - ibm [2024-01-19T09:26:11.614Z] 2024-01-19 09:26:09 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).GetVolumes:#4298] - K8s.GetVolumes() found volume mysql-data for app mysql-ibm [2024-01-19T09:26:11.614Z] 2024-01-19 09:26:09 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.ValidateContext.func2.4.1:#702] - validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T09:26:11.614Z] 2024-01-19 09:26:09 +0000:[WARNING] [{BackupScheduleForOldAndNewNS}] [ibm.(*ibm).ValidateVolumeSetup:#57] - ValidateVolumeSetup function has not been implemented for volume driver - ibm [2024-01-19T09:26:11.614Z] 2024-01-19 09:26:09 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Apply same namespace labels to new namespaces [2024-01-19T09:26:11.615Z] INFO[2024-01-19 09:26:09] Verifying : Description : Applying same namespace labels map[0-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 1-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 10-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 11-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 12-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 13-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 14-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 15-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 16-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 17-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 18-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 19-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 2-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 3-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 4-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 5-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 6-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 7-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 8-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 9-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a] to new namespace [mysql-ibm-new-namespace-0-84852-01-19-09h05m53s mysql-ibm-new-namespace-1-84852-01-19-09h05m53s] [2024-01-19T09:26:11.615Z] INFO[2024-01-19 09:26:09] Actual:true, Expected: true [2024-01-19T09:26:11.615Z] 2024-01-19 09:26:09 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Verifying new applications namespace new application namespaces with same namespace label are included in next schedule backup [2024-01-19T09:26:11.615Z] INFO[2024-01-19 09:26:10] Verifying : Description : Fetching px-central-admin ctx [2024-01-19T09:26:11.615Z] INFO[2024-01-19 09:26:10] Actual:true, Expected: true [2024-01-19T09:26:11.615Z] 2024-01-19 09:26:10 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Waiting for 15ns minutes for the next schedule backup to be triggered [2024-01-19T09:35:18.225Z] INFO[2024-01-19 09:35:16] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T09:35:18.225Z] INFO[2024-01-19 09:35:16] sched stats: tick interval: 64876 < 1.2s (100%) [2024-01-19T09:35:18.225Z] INFO[2024-01-19 09:35:16] sched stats: tick process durations: 64875 < 500ms (100%) [2024-01-19T09:41:24.766Z] 2024-01-19 09:41:10 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Next schedule backup name [tp-backup-schedule-1705655276-ff78f6a-interval-2024-01-19-093802] [2024-01-19T09:41:24.766Z] INFO[2024-01-19 09:41:10] Verifying : Description : Verifying backup success for tp-backup-schedule-1705655276-ff78f6a-interval-2024-01-19-093802 schedule backup [2024-01-19T09:41:24.766Z] INFO[2024-01-19 09:41:10] Actual:true, Expected: true [2024-01-19T09:41:24.766Z] 2024-01-19 09:41:10 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Total namespaces are [[mysql-ibm-pxb-0-84852-01-19-09h05m53s mysql-ibm-pxb-1-84852-01-19-09h05m53s mysql-ibm-pxb-2-84852-01-19-09h05m53s mysql-ibm-new-namespace-0-84852-01-19-09h05m53s mysql-ibm-new-namespace-1-84852-01-19-09h05m53s]] [2024-01-19T09:41:24.766Z] 2024-01-19 09:41:10 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [backup.NamespaceLabelBackupSuccessCheck:#3234] - Getting the Uid of backup tp-backup-schedule-1705655276-ff78f6a-interval-2024-01-19-093802 [2024-01-19T09:41:24.766Z] 2024-01-19 09:41:10 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [backup.NamespaceLabelBackupSuccessCheck:#3249] - The list of namespaces backed up are [mysql-ibm-new-namespace-0-84852-01-19-09h05m53s mysql-ibm-new-namespace-1-84852-01-19-09h05m53s mysql-ibm-pxb-0-84852-01-19-09h05m53s mysql-ibm-pxb-1-84852-01-19-09h05m53s mysql-ibm-pxb-2-84852-01-19-09h05m53s] [2024-01-19T09:41:24.767Z] 2024-01-19 09:41:10 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [backup.NamespaceLabelBackupSuccessCheck:#3254] - The list of labels applied to backup are 1-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,5-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,11-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,12-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,13-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,3-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,4-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,6-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,18-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,2-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,7-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,8-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,9-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,10-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,14-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,16-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,19-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,0-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,15-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,17-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a [2024-01-19T09:41:24.767Z] INFO[2024-01-19 09:41:10] Verifying : Description : Verifying the labeled namespace [[mysql-ibm-new-namespace-0-84852-01-19-09h05m53s mysql-ibm-new-namespace-1-84852-01-19-09h05m53s mysql-ibm-pxb-0-84852-01-19-09h05m53s mysql-ibm-pxb-1-84852-01-19-09h05m53s mysql-ibm-pxb-2-84852-01-19-09h05m53s]] is backed up and checks for labels [1-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,5-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,11-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,12-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,13-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,3-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,4-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,6-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,18-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,2-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,7-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,8-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,9-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,10-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,14-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,16-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,19-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,0-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,15-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a,17-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a=a1cd28bb-fbbf-4adc-bb9c-15f855daba3a] applied to the backup [tp-backup-schedule-1705655276-ff78f6a-interval-2024-01-19-093802] [2024-01-19T09:41:24.767Z] INFO[2024-01-19 09:41:10] Actual:true, Expected: true [2024-01-19T09:41:24.767Z] 2024-01-19 09:41:10 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Restoring scheduled backups [2024-01-19T09:41:24.767Z] INFO[2024-01-19 09:41:11] Verifying : Description : Fetching px-central-admin ctx [2024-01-19T09:41:24.767Z] INFO[2024-01-19 09:41:11] Actual:true, Expected: true [2024-01-19T09:41:24.767Z] INFO[2024-01-19 09:41:11] Verifying : Description : Fetching names of all schedule backups of schedule named [tp-backup-schedule-1705655276] [2024-01-19T09:41:24.767Z] INFO[2024-01-19 09:41:11] Actual:true, Expected: true [2024-01-19T09:41:24.767Z] 2024/01/19 09:41:11 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-schedule-1705655276-1705657271] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T09:41:42.833Z] 2024/01/19 09:41:41 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-schedule-1705655276-1705657271] expected was [[PartialSuccess Success]] but got [InProgress] because of [Volume restores are in progress]}, Next try in [30s], timeout [40m0s] [2024-01-19T09:42:14.868Z] 2024-01-19 09:42:11 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [backup.CreateRestore:#936] - Restore [tp-restore-tp-backup-schedule-1705655276-1705657271] created successfully [2024-01-19T09:42:14.868Z] INFO[2024-01-19 09:42:11] Verifying : Description : Verification of restoring scheduled backups - tp-restore-tp-backup-schedule-1705655276-1705657271 [2024-01-19T09:42:14.868Z] INFO[2024-01-19 09:42:11] Actual:true, Expected: true [2024-01-19T09:42:14.869Z] 2024/01/19 09:42:11 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-schedule-1705655276-1705657331] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T09:42:46.894Z] 2024-01-19 09:42:42 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [backup.CreateRestore:#936] - Restore [tp-restore-tp-backup-schedule-1705655276-1705657331] created successfully [2024-01-19T09:42:46.894Z] INFO[2024-01-19 09:42:42] Verifying : Description : Verification of restoring scheduled backups - tp-restore-tp-backup-schedule-1705655276-1705657331 [2024-01-19T09:42:46.894Z] INFO[2024-01-19 09:42:42] Actual:true, Expected: true [2024-01-19T09:42:46.894Z] 2024/01/19 09:42:42 DoRetryWithTimeout - Error: {restore status for [tp-restore-tp-backup-schedule-1705655276-1705657362] expected was [[PartialSuccess Success]] but got [Pending] because of []}, Next try in [30s], timeout [40m0s] [2024-01-19T09:43:13.386Z] 2024-01-19 09:43:12 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [backup.CreateRestore:#936] - Restore [tp-restore-tp-backup-schedule-1705655276-1705657362] created successfully [2024-01-19T09:43:13.386Z] INFO[2024-01-19 09:43:12] Verifying : Description : Verification of restoring scheduled backups - tp-restore-tp-backup-schedule-1705655276-1705657362 [2024-01-19T09:43:13.386Z] INFO[2024-01-19 09:43:12] Actual:true, Expected: true [2024-01-19T09:43:13.386Z] INFO[2024-01-19 09:43:12] Verifying : Description : Fetching px-central-admin ctx [2024-01-19T09:43:13.386Z] INFO[2024-01-19 09:43:12] Actual:true, Expected: true [2024-01-19T09:43:13.386Z] 2024/01/19 09:43:12 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705655276] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T09:43:23.338Z] 2024/01/19 09:43:22 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705655276] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T09:43:33.328Z] 2024/01/19 09:43:32 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705655276] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T09:43:43.265Z] 2024/01/19 09:43:42 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705655276] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T09:43:53.233Z] 2024/01/19 09:43:52 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705655276] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T09:44:03.211Z] 2024/01/19 09:44:03 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705655276] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T09:44:13.171Z] 2024/01/19 09:44:13 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705655276] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T09:44:23.116Z] 2024/01/19 09:44:23 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705655276] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T09:44:35.312Z] 2024/01/19 09:44:33 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705655276] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T09:44:43.403Z] 2024/01/19 09:44:43 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705655276] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T09:44:53.354Z] 2024/01/19 09:44:53 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705655276] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T09:45:04.076Z] 2024/01/19 09:45:03 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705655276] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T09:45:14.018Z] 2024/01/19 09:45:13 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705655276] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T09:45:18.182Z] INFO[2024-01-19 09:45:17] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T09:45:18.182Z] INFO[2024-01-19 09:45:17] sched stats: tick interval: 65477 < 1.2s (100%) [2024-01-19T09:45:18.182Z] INFO[2024-01-19 09:45:17] sched stats: tick process durations: 65476 < 500ms (100%) [2024-01-19T09:45:23.508Z] 2024/01/19 09:45:23 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705655276] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T09:45:33.444Z] 2024/01/19 09:45:33 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705655276] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T09:45:43.388Z] 2024/01/19 09:45:43 DoRetryWithTimeout - Error: {backupSchedule [tp-backup-schedule-1705655276] is in [DeletePending] state}, Next try in [10s], timeout [1h0m0s] [2024-01-19T09:45:53.326Z] INFO[2024-01-19 09:45:53] Verifying : Description : Verification of deleting backup schedule - tp-backup-schedule-1705655276 [2024-01-19T09:45:53.326Z] INFO[2024-01-19 09:45:53] Actual:true, Expected: true [2024-01-19T09:45:53.888Z] INFO[2024-01-19 09:45:53] Verifying : Description : Deleting backup schedule policies [periodic-1705655274] [2024-01-19T09:45:53.888Z] INFO[2024-01-19 09:45:53] Actual:true, Expected: true [2024-01-19T09:45:53.888Z] INFO[2024-01-19 09:45:53] Verifying : Description : Getting the backup driver [2024-01-19T09:45:53.888Z] INFO[2024-01-19 09:45:53] Actual:true, Expected: true [2024-01-19T09:45:53.888Z] INFO[2024-01-19 09:45:53] Verifying : Description : Deleting restore [tp-restore-tp-backup-schedule-1705655276-1705657271] [2024-01-19T09:45:53.888Z] INFO[2024-01-19 09:45:53] Actual:true, Expected: true [2024-01-19T09:45:53.888Z] INFO[2024-01-19 09:45:53] Verifying : Description : Getting the backup driver [2024-01-19T09:45:53.888Z] INFO[2024-01-19 09:45:53] Actual:true, Expected: true [2024-01-19T09:45:53.888Z] INFO[2024-01-19 09:45:53] Verifying : Description : Deleting restore [tp-restore-tp-backup-schedule-1705655276-1705657331] [2024-01-19T09:45:53.888Z] INFO[2024-01-19 09:45:53] Actual:true, Expected: true [2024-01-19T09:45:53.888Z] INFO[2024-01-19 09:45:53] Verifying : Description : Getting the backup driver [2024-01-19T09:45:53.888Z] INFO[2024-01-19 09:45:53] Actual:true, Expected: true [2024-01-19T09:45:53.888Z] INFO[2024-01-19 09:45:53] Verifying : Description : Deleting restore [tp-restore-tp-backup-schedule-1705655276-1705657362] [2024-01-19T09:45:53.888Z] INFO[2024-01-19 09:45:53] Actual:true, Expected: true [2024-01-19T09:45:53.888Z] 2024-01-19 09:45:53 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Deleting labels from namespaces- [mysql-ibm-new-namespace-0-84852-01-19-09h05m53s mysql-ibm-new-namespace-1-84852-01-19-09h05m53s mysql-ibm-pxb-0-84852-01-19-09h05m53s mysql-ibm-pxb-1-84852-01-19-09h05m53s mysql-ibm-pxb-2-84852-01-19-09h05m53s] [2024-01-19T09:45:54.820Z] INFO[2024-01-19 09:45:54] Verifying : Description : Deleting labels [map[0-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 1-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 10-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 11-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 12-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 13-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 14-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 15-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 16-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 17-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 18-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 19-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 2-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 3-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 4-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 5-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 6-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 7-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 8-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a 9-a1cd28bb-fbbf-4adc-bb9c-15f855daba3a:a1cd28bb-fbbf-4adc-bb9c-15f855daba3a]] from namespaces [[mysql-ibm-new-namespace-0-84852-01-19-09h05m53s mysql-ibm-new-namespace-1-84852-01-19-09h05m53s mysql-ibm-pxb-0-84852-01-19-09h05m53s mysql-ibm-pxb-1-84852-01-19-09h05m53s mysql-ibm-pxb-2-84852-01-19-09h05m53s]] [2024-01-19T09:45:54.820Z] INFO[2024-01-19 09:45:54] Actual:true, Expected: true [2024-01-19T09:45:54.820Z] 2024-01-19 09:45:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Deleting deployed namespaces - [mysql-ibm-new-namespace-0-84852-01-19-09h05m53s mysql-ibm-new-namespace-1-84852-01-19-09h05m53s mysql-ibm-pxb-0-84852-01-19-09h05m53s mysql-ibm-pxb-1-84852-01-19-09h05m53s mysql-ibm-pxb-2-84852-01-19-09h05m53s] [2024-01-19T09:45:54.820Z] 2024-01-19 09:45:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Destroying apps [2024-01-19T09:45:54.820Z] 2024-01-19 09:45:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T09:45:55.076Z] 2024-01-19 09:45:55 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T09:45:55.076Z] 2024-01-19 09:45:55 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] start destroying mysql-ibm app [2024-01-19T09:45:55.331Z] 2024-01-19 09:45:55 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T09:45:55.331Z] 2024-01-19 09:45:55 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T09:45:55.331Z] 2024-01-19 09:45:55 +0000:[WARNING] [{BackupScheduleForOldAndNewNS}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T09:45:55.331Z] INFO[2024-01-19 09:45:55] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T09:45:55.331Z] INFO[2024-01-19 09:45:55] Actual:true, Expected: true [2024-01-19T09:45:55.331Z] 2024-01-19 09:45:55 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.DeleteVolumes.func1:#1684] - destroy the mysql-ibm app's volumes [2024-01-19T09:45:55.892Z] 2024-01-19 09:45:55 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).DeleteVolumes:#4051] - [mysql-ibm] Destroyed PVC: mysql-data [2024-01-19T09:45:55.892Z] 2024-01-19 09:45:55 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] start destroying mysql-ibm app [2024-01-19T09:45:55.892Z] 2024-01-19 09:45:55 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).destroyCoreObject:#2685] - [mysql-ibm] Destroyed Service: mysql [2024-01-19T09:45:56.148Z] 2024-01-19 09:45:55 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T09:45:56.148Z] 2024-01-19 09:45:55 +0000:[WARNING] [{BackupScheduleForOldAndNewNS}] [ibm.(*ibm).ValidateDeleteVolume:#63] - ValidateDeleteVolume function has not been implemented for volume driver - ibm [2024-01-19T09:45:56.148Z] INFO[2024-01-19 09:45:55] Verifying : Description : mysql-ibm's volume mysql-data deleted successfully? [2024-01-19T09:45:56.148Z] INFO[2024-01-19 09:45:55] Actual:true, Expected: true [2024-01-19T09:45:56.148Z] 2024-01-19 09:45:55 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Cleaning backup locations in map [map[73ab26a5-844b-4ef6-98a2-9d4b41a31a4e:autogenerated-backup-location-1705655266]], cloud credential [autogenerated-cred-1705655266], source [source-cluster] and destination [destination-cluster] cluster [2024-01-19T09:45:56.149Z] INFO[2024-01-19 09:45:56] Verifying : Description : Verifying deletion of backup location [autogenerated-backup-location-1705655266] [2024-01-19T09:45:56.149Z] INFO[2024-01-19 09:45:56] Actual:true, Expected: true [2024-01-19T09:45:56.149Z] 2024-01-19 09:45:56 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705655266] is present [2024-01-19T09:45:56.405Z] 2024/01/19 09:45:56 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705655266 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T09:46:28.451Z] 2024-01-19 09:46:26 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [backup.IsBackupLocationPresent:#2442] - Backup location [autogenerated-backup-location-1705655266] is present [2024-01-19T09:46:28.451Z] 2024/01/19 09:46:26 DoRetryWithTimeout - Error: {backup location autogenerated-backup-location-1705655266 is not deleted yet. Status - [status:DeletePending reason:"Deletion is in progress" ]}, Next try in [30s], timeout [1h0m0s] [2024-01-19T09:47:00.478Z] 2024-01-19 09:46:56 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [backup.IsBackupLocationPresent:#2446] - Backup locations fetched - [aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 autogenerated-backup-location-omphu aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 aws-global-aws-px-backup-system-test-4cac76dc-bl autogenerated-backup-location-1705628002 autogenerated-backup-location-1705634221 autogenerated-backup-location-1705635982 autogenerated-backup-location-wkyfv] [2024-01-19T09:47:00.478Z] INFO[2024-01-19 09:46:56] Verifying : Description : Verifying backup location deletion status autogenerated-backup-location-1705655266 [2024-01-19T09:47:00.478Z] INFO[2024-01-19 09:46:56] Actual:true, Expected: true [2024-01-19T09:47:00.478Z] 2024-01-19 09:46:56 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.IsCloudCredPresent:#9926] - Cloud Credential [autogenerated-cred-1705655266] is present [2024-01-19T09:47:00.478Z] INFO[2024-01-19 09:46:56] Verifying : Description : Verifying if cloud cred [autogenerated-cred-1705655266] is present [2024-01-19T09:47:00.478Z] INFO[2024-01-19 09:46:56] Actual:true, Expected: true [2024-01-19T09:47:00.478Z] INFO[2024-01-19 09:46:56] Verifying : Description : Verifying deletion of cloud cred [autogenerated-cred-1705655266] [2024-01-19T09:47:00.478Z] INFO[2024-01-19 09:46:56] Actual:true, Expected: true [2024-01-19T09:47:00.478Z] INFO[2024-01-19 09:46:56] Verifying : Description : Deleting cloud cred autogenerated-cred-1705655266 [2024-01-19T09:47:00.478Z] INFO[2024-01-19 09:46:56] Actual:true, Expected: true [2024-01-19T09:47:00.478Z] INFO[2024-01-19 09:46:56] Verifying : Description : Verifying enumerate cluster in organization default [2024-01-19T09:47:00.478Z] INFO[2024-01-19 09:46:56] Actual:true, Expected: true [2024-01-19T09:47:00.478Z] INFO[2024-01-19 09:46:56] Verifying : Description : Deleting cluster source-cluster [2024-01-19T09:47:00.479Z] ERRO[2024-01-19 09:46:56] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl], cannot be deleted, Expected:  [2024-01-19T09:47:00.479Z] INFO[2024-01-19 09:46:57] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-hrfiy] [2024-01-19T09:47:00.479Z] ERRO[2024-01-19 09:46:57] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-hrfiy:72b1b409-7bb1-4f22-92df-0ba5e77c11d3], Expected:  [2024-01-19T09:47:00.479Z] INFO[2024-01-19 09:46:57] Verifying : Description : Deleting cluster source-cluster [2024-01-19T09:47:00.479Z] ERRO[2024-01-19 09:46:57] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh], cannot be deleted, Expected:  [2024-01-19T09:47:00.479Z] INFO[2024-01-19 09:46:57] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-fbvvo] [2024-01-19T09:47:00.479Z] ERRO[2024-01-19 09:46:57] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-fbvvo:7cddb5aa-c7ff-4044-9d27-458be2bd5cde], Expected:  [2024-01-19T09:47:00.479Z] INFO[2024-01-19 09:46:57] Verifying : Description : Deleting cluster source-cluster [2024-01-19T09:47:00.480Z] ERRO[2024-01-19 09:46:57] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo], cannot be deleted, Expected:  [2024-01-19T09:47:00.480Z] INFO[2024-01-19 09:46:58] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-enzij] [2024-01-19T09:47:00.480Z] ERRO[2024-01-19 09:46:58] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-enzij:4e0ddfc7-8811-4ed5-8e87-ed0841582111], Expected:  [2024-01-19T09:47:00.480Z] INFO[2024-01-19 09:46:59] Verifying : Description : Deleting cluster source-cluster [2024-01-19T09:47:00.480Z] ERRO[2024-01-19 09:46:59] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo], cannot be deleted, Expected:  [2024-01-19T09:47:00.480Z] INFO[2024-01-19 09:47:00] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-wcpfm] [2024-01-19T09:47:00.480Z] ERRO[2024-01-19 09:47:00] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-wcpfm:c5ff084a-a913-4412-99fb-e2b0a003a014], Expected:  [2024-01-19T09:47:00.738Z] INFO[2024-01-19 09:47:00] Verifying : Description : Deleting cluster source-cluster [2024-01-19T09:47:00.738Z] ERRO[2024-01-19 09:47:00] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx], cannot be deleted, Expected:  [2024-01-19T09:47:02.103Z] INFO[2024-01-19 09:47:01] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-vlbwu] [2024-01-19T09:47:02.103Z] ERRO[2024-01-19 09:47:01] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-vlbwu:f0465bb5-2565-4948-bb27-7e45a4b883ff], Expected:  [2024-01-19T09:47:02.103Z] INFO[2024-01-19 09:47:01] Verifying : Description : Deleting cluster source-cluster [2024-01-19T09:47:02.104Z] ERRO[2024-01-19 09:47:01] Actual:rpc error: code = FailedPrecondition desc = cluster [source-cluster] in use by backupschedule [tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa], cannot be deleted, Expected:  [2024-01-19T09:47:03.467Z] INFO[2024-01-19 09:47:03] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-mhxzi] [2024-01-19T09:47:03.467Z] ERRO[2024-01-19 09:47:03] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-mhxzi:f551fb81-1a78-4d7f-b804-759c6e6b562e], Expected:  [2024-01-19T09:47:03.723Z] INFO[2024-01-19 09:47:03] Verifying : Description : Deleting cluster source-cluster [2024-01-19T09:47:03.723Z] INFO[2024-01-19 09:47:03] Actual:true, Expected: true [2024-01-19T09:47:05.088Z] INFO[2024-01-19 09:47:04] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-source-config-cloud-cred-qaeks] [2024-01-19T09:47:05.088Z] INFO[2024-01-19 09:47:04] Actual:true, Expected: true [2024-01-19T09:47:05.088Z] INFO[2024-01-19 09:47:04] Verifying : Description : Deleting cluster destination-cluster [2024-01-19T09:47:05.088Z] INFO[2024-01-19 09:47:04] Actual:true, Expected: true [2024-01-19T09:47:06.455Z] INFO[2024-01-19 09:47:06] Verifying : Description : Verifying deletion of cluster cloud cred [ibm-destination-config-cloud-cred-robsb] [2024-01-19T09:47:06.455Z] INFO[2024-01-19 09:47:06] Actual:true, Expected: true [2024-01-19T09:47:06.455Z] INFO[2024-01-19 09:47:06] --------Test End------ [2024-01-19T09:47:06.455Z] INFO[2024-01-19 09:47:06] #Test: [2024-01-19T09:47:06.455Z] INFO[2024-01-19 09:47:06] #Description: [2024-01-19T09:47:06.455Z] INFO[2024-01-19 09:47:06] ------------------------ [2024-01-19T09:47:06.455Z] 2024-01-19 09:47:06 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-new-namespace-0-84852-01-19-09h05m53s] [2024-01-19T09:47:06.455Z] 2024-01-19 09:47:06 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-new-namespace-1-84852-01-19-09h05m53s] [2024-01-19T09:47:06.710Z] 2024-01-19 09:47:06 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-84852-01-19-09h05m53s] [2024-01-19T09:47:06.966Z] 2024-01-19 09:47:06 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-1-84852-01-19-09h05m53s] [2024-01-19T09:47:06.966Z] 2024-01-19 09:47:06 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-2-84852-01-19-09h05m53s] [2024-01-19T09:47:07.221Z] 2024-01-19 09:47:07 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.GetDestinationClusterConfigPath:#5100] - Destination config path: /tmp/destination-config [2024-01-19T09:47:07.221Z] 2024-01-19 09:47:07 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Switching context: The context is already [/tmp/destination-config] [2024-01-19T09:47:07.476Z] 2024-01-19 09:47:07 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-new-namespace-0-84852-01-19-09h05m53s] [2024-01-19T09:47:07.732Z] 2024-01-19 09:47:07 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-new-namespace-1-84852-01-19-09h05m53s] [2024-01-19T09:47:07.987Z] 2024-01-19 09:47:07 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-0-84852-01-19-09h05m53s] [2024-01-19T09:47:07.987Z] 2024-01-19 09:47:07 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-1-84852-01-19-09h05m53s] [2024-01-19T09:47:08.242Z] 2024-01-19 09:47:08 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.DeleteAllNamespacesCreatedByTestCase:#10193] - Deleting namespace [mysql-ibm-pxb-2-84852-01-19-09h05m53s] [2024-01-19T09:47:08.500Z] 2024-01-19 09:47:08 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.EndPxBackupTorpedoTest:#7288] - >>>> Collecting logs for testcase : {BackupScheduleForOldAndNewNS} Schedule backup using namespace label for old and new namespaces [2024-01-19T09:47:08.500Z] 2024-01-19 09:47:08 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.EndPxBackupTorpedoTest:#7295] - Creating a directory [/tmp/px-backup-test-logs] to store logs [2024-01-19T09:47:08.500Z] 2024/01/19 09:47:08 DoRetryWithTimeout - Error: {dial tcp 10.240.128.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T09:47:13.743Z] 2024/01/19 09:47:13 DoRetryWithTimeout - Error: {dial tcp 10.240.128.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T09:47:18.984Z] 2024/01/19 09:47:18 DoRetryWithTimeout - Error: {dial tcp 10.240.128.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T09:47:24.223Z] 2024/01/19 09:47:23 DoRetryWithTimeout - Error: {dial tcp 10.240.128.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T09:47:28.386Z] 2024/01/19 09:47:28 DoRetryWithTimeout - Error: {dial tcp 10.240.128.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T09:47:33.626Z] 2024/01/19 09:47:33 DoRetryWithTimeout - Error: {dial tcp 10.240.128.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T09:47:38.866Z] 2024/01/19 09:47:38 DoRetryWithTimeout - Error: {dial tcp 10.240.128.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T09:47:44.109Z] 2024/01/19 09:47:43 DoRetryWithTimeout - Error: {dial tcp 10.240.128.6:22: connect: connection refused}, Next try in [5s], timeout [20s] [2024-01-19T09:47:49.353Z] 2024/01/19 09:47:48 DoRetryWithTimeout - Error: {Failed to run command on: 10.240.128.6. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.6 10.240.128.6]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T09:47:49.353Z] {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T09:47:49.353Z] {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T09:47:49.353Z] {dial tcp 10.240.128.6:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README}, Next try in [5s], timeout [20s] [2024-01-19T09:47:53.525Z] 2024-01-19 09:47:53 +0000:[WARNING] [{BackupScheduleForOldAndNewNS}] [tests.runCmd:#2437] - failed to run cmd: mkdir -p /tmp/px-backup-test-logs. err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.6. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.6 10.240.128.6]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T09:47:53.525Z] {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T09:47:53.525Z] {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T09:47:53.525Z] {dial tcp 10.240.128.6:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T09:47:53.525Z] 2024-01-19 09:47:53 +0000:[ERROR] [{BackupScheduleForOldAndNewNS}] [tests.EndPxBackupTorpedoTest:#7298] - Error in creating a directory [/tmp/px-backup-test-logs] to store logs. Err: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Failed to run command on: 10.240.128.6. Cause: unable to run cmd (mkdir -p /tmp/px-backup-test-logs): Failed to run command on: . Cause: failed to dial: no usable address found. Tried: [10.240.128.6 10.240.128.6]. Error: timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T09:47:53.525Z] {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T09:47:53.525Z] {dial tcp 10.240.128.6:22: connect: connection refused} [2024-01-19T09:47:53.525Z] {dial tcp 10.240.128.6:22: connect: connection refused}Ensure you have setup the nodes for ssh access as per the README} [2024-01-19T09:47:53.525Z] 2024-01-19 09:47:53 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [tests.GetSourceClusterConfigPath:#5083] - Source config path: /tmp/source-config [2024-01-19T09:47:53.525Z] 2024-01-19 09:47:53 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] Switching context to [/tmp/source-config] [2024-01-19T09:47:53.525Z] 2024-01-19 09:47:53 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.41] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T09:47:53.525Z] 2024-01-19 09:47:53 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.7] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T09:47:53.781Z] 2024-01-19 09:47:53 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.0.8] as Type: Master, Zone: us-south-1, Region us-south [2024-01-19T09:47:53.781Z] 2024-01-19 09:47:53 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.20] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T09:47:53.781Z] 2024-01-19 09:47:53 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.6] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T09:47:53.781Z] 2024-01-19 09:47:53 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.128.7] as Type: Master, Zone: us-south-3, Region us-south [2024-01-19T09:47:54.341Z] 2024-01-19 09:47:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.27] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T09:47:54.596Z] 2024-01-19 09:47:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.8] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T09:47:55.156Z] 2024-01-19 09:47:54 +0000:[INFO] [{BackupScheduleForOldAndNewNS}] [k8s.(*K8s).parseK8SNode:#879] - Parsed node [10.240.64.9] as Type: Master, Zone: us-south-2, Region us-south [2024-01-19T09:47:55.411Z] 2024/01/19 09:47:55 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:48:05.346Z] 2024/01/19 09:48:05 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:48:15.290Z] 2024/01/19 09:48:15 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:48:27.454Z] 2024/01/19 09:48:25 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:48:35.530Z] 2024/01/19 09:48:35 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:48:45.473Z] 2024/01/19 09:48:45 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:48:55.412Z] 2024/01/19 09:48:55 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:49:07.573Z] 2024/01/19 09:49:05 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:49:15.660Z] 2024/01/19 09:49:15 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:49:25.609Z] 2024/01/19 09:49:25 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:49:35.554Z] 2024/01/19 09:49:35 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:49:47.732Z] 2024/01/19 09:49:45 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:49:55.814Z] 2024/01/19 09:49:55 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:50:05.755Z] 2024/01/19 09:50:05 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:50:17.928Z] 2024/01/19 09:50:15 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:50:26.019Z] 2024/01/19 09:50:25 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:50:35.960Z] 2024/01/19 09:50:35 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:50:45.901Z] 2024/01/19 09:50:45 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:50:55.864Z] 2024/01/19 09:50:55 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:51:08.039Z] 2024/01/19 09:51:05 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:51:16.131Z] 2024/01/19 09:51:15 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:51:26.078Z] 2024/01/19 09:51:25 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:51:36.017Z] 2024/01/19 09:51:35 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:51:48.180Z] 2024/01/19 09:51:45 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:51:56.274Z] 2024/01/19 09:51:55 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:52:06.218Z] 2024/01/19 09:52:05 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:52:17.338Z] 2024/01/19 09:52:15 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:52:27.274Z] 2024/01/19 09:52:26 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:52:37.217Z] 2024/01/19 09:52:36 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:52:47.154Z] 2024/01/19 09:52:46 DoRetryWithTimeout - Error: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}, Next try in [10s], timeout [5m0s] [2024-01-19T09:52:57.122Z] 2024-01-19 09:52:56 +0000:[ERROR] [{BackupScheduleForOldAndNewNS}] [tests.EndPxBackupTorpedoTest.func1:#7283] - failed to switch context to source cluster. Err: failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.122Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.122Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.122Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.122Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.122Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.122Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.122Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.122Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.122Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.122Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.123Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.123Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.123Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.123Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.123Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.123Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.123Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.123Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.123Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.123Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.123Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.123Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.124Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.124Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.124Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.124Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.124Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.124Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.124Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T09:52:57.124Z] [2024-01-19T09:52:57.124Z] ------------------------------ [2024-01-19T09:52:57.124Z] • Failure [2822.123 seconds] [2024-01-19T09:52:57.124Z] {BackupScheduleForOldAndNewNS} [2024-01-19T09:52:57.124Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_namespace_labelled_test.go:210 [2024-01-19T09:52:57.124Z] Schedule backup using namespace label for old and new namespaces [] [2024-01-19T09:52:57.124Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_namespace_labelled_test.go:257 [2024-01-19T09:52:57.124Z] [2024-01-19T09:52:57.124Z] Unexpected error: [2024-01-19T09:52:57.124Z] <*errors.errorString | 0xc001635eb0>: { [2024-01-19T09:52:57.126Z] s: "failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}\n{No... [2024-01-19T09:52:57.126Z] [2024-01-19T09:52:57.126Z] Gomega truncated this representation as it exceeds 'format.MaxLength'. [2024-01-19T09:52:57.126Z] Consider having the object provide a custom 'GomegaStringer' representation [2024-01-19T09:52:57.126Z] or adjust the parameters in Gomega's 'format' package. [2024-01-19T09:52:57.126Z] [2024-01-19T09:52:57.126Z] Learn more here: https://onsi.github.io/gomega/#adjusting-output [2024-01-19T09:52:57.126Z] [2024-01-19T09:52:57.126Z] failed to switch to context. RefreshNodeRegistry Error: [timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.126Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.126Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.126Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.126Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.126Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.126Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.126Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.126Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.126Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.126Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.126Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.126Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.126Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.126Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.126Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.126Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.127Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.127Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.127Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.127Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.127Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.127Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.127Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.127Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.127Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.127Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.127Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.127Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown} [2024-01-19T09:52:57.127Z] {Node: 10.240.64.9 is not ready due to err: node: 10.240.64.9 is not ready as condition: Ready (Kubelet stopped posting node status.) is Unknown. Reason: NodeStatusUnknown}] [2024-01-19T09:52:57.127Z] occurred [2024-01-19T09:52:57.127Z] [2024-01-19T09:52:57.127Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T09:52:57.127Z] [2024-01-19T09:52:57.127Z] Full Stack Trace [2024-01-19T09:52:57.127Z] github.com/portworx/torpedo/pkg/log.FailOnError({0x88a5060, 0xc001635eb0}, {0x7f74bfd?, 0x6cc0320?}, {0x0?, 0x6cc0320?, 0xc0013a0430?}) [2024-01-19T09:52:57.127Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 +0x290 [2024-01-19T09:52:57.127Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest.func1() [2024-01-19T09:52:57.127Z] /go/src/github.com/portworx/torpedo/tests/common.go:7283 +0x32 [2024-01-19T09:52:57.127Z] github.com/portworx/torpedo/tests.EndPxBackupTorpedoTest({0xc00117a040, 0x2, 0x2}) [2024-01-19T09:52:57.127Z] /go/src/github.com/portworx/torpedo/tests/common.go:7299 +0x778 [2024-01-19T09:52:57.127Z] github.com/portworx/torpedo/tests/backup.glob..func19.3() [2024-01-19T09:52:57.127Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_namespace_labelled_test.go:432 +0x5bd [2024-01-19T09:52:57.127Z] github.com/portworx/torpedo/tests/backup.TestBasic(0x0?) [2024-01-19T09:52:57.127Z] /go/src/github.com/portworx/torpedo/tests/backup/backup_basic_test.go:93 +0xb9 [2024-01-19T09:52:57.127Z] testing.tRunner(0xc000ca6b60, 0x8103de0) [2024-01-19T09:52:57.127Z] /usr/local/go/src/testing/testing.go:1446 +0x10b [2024-01-19T09:52:57.127Z] created by testing.(*T).Run [2024-01-19T09:52:57.127Z] /usr/local/go/src/testing/testing.go:1493 +0x35f [2024-01-19T09:52:57.127Z] ------------------------------ [2024-01-19T09:52:57.127Z] STEP: schedule applications [2024-01-19T09:52:57.127Z] STEP: schedule applications [2024-01-19T09:52:57.127Z] STEP: schedule applications [2024-01-19T09:52:57.127Z] STEP: Validate applications [2024-01-19T09:52:57.127Z] STEP: validate applications [2024-01-19T09:52:57.127Z] STEP: validate mysql-ibm app's volumes [2024-01-19T09:52:57.127Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T09:52:57.127Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T09:52:57.128Z] STEP: get mysql-ibm app's volume: pvc-ea35ece2-66a9-4466-8e54-1fdf00cca63a inspected by the volume driver [2024-01-19T09:52:57.128Z] STEP: wait for mysql-ibm app to start running [2024-01-19T09:52:57.128Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T09:52:57.128Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T09:52:57.128Z] STEP: Validate Px pod restart count [2024-01-19T09:52:57.128Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T09:52:57.128Z] STEP: validate mysql-ibm app's volumes [2024-01-19T09:52:57.128Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T09:52:57.128Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T09:52:57.128Z] STEP: get mysql-ibm app's volume: pvc-74694bda-5111-48bf-a87f-0f3d9ddbb2d9 inspected by the volume driver [2024-01-19T09:52:57.128Z] STEP: wait for mysql-ibm app to start running [2024-01-19T09:52:57.128Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T09:52:57.128Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T09:52:57.128Z] STEP: Validate Px pod restart count [2024-01-19T09:52:57.128Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T09:52:57.128Z] STEP: validate mysql-ibm app's volumes [2024-01-19T09:52:57.128Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T09:52:57.128Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T09:52:57.128Z] STEP: get mysql-ibm app's volume: pvc-e91bb5cb-b8d5-450f-a2bb-d312c22b46c0 inspected by the volume driver [2024-01-19T09:52:57.128Z] STEP: wait for mysql-ibm app to start running [2024-01-19T09:52:57.128Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T09:52:57.128Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T09:52:57.128Z] STEP: Validate Px pod restart count [2024-01-19T09:52:57.128Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T09:52:57.128Z] STEP: Adding labels to namespaces [2024-01-19T09:52:57.128Z] STEP: Generating namespace label string from label map for namespaces [2024-01-19T09:52:57.128Z] STEP: Adding labels to resources [2024-01-19T09:52:57.128Z] STEP: Creating cloud credentials and backup location [2024-01-19T09:52:57.128Z] STEP: Add source and destination clusters with px-central-admin ctx [2024-01-19T09:52:57.128Z] STEP: Create schedule policy [2024-01-19T09:52:57.128Z] STEP: Creating a schedule backup using namespace label and resource label [2024-01-19T09:52:57.128Z] STEP: Schedule applications to create new namespaces [2024-01-19T09:52:57.128Z] STEP: schedule applications [2024-01-19T09:52:57.128Z] STEP: schedule applications [2024-01-19T09:52:57.128Z] STEP: Validate new namespaces [2024-01-19T09:52:57.128Z] STEP: validate applications [2024-01-19T09:52:57.128Z] STEP: validate mysql-ibm app's volumes [2024-01-19T09:52:57.128Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T09:52:57.128Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T09:52:57.128Z] STEP: get mysql-ibm app's volume: pvc-82b0a4c9-9724-476d-b58c-8c791a55c3d3 inspected by the volume driver [2024-01-19T09:52:57.128Z] STEP: wait for mysql-ibm app to start running [2024-01-19T09:52:57.128Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T09:52:57.128Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T09:52:57.128Z] STEP: Validate Px pod restart count [2024-01-19T09:52:57.128Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T09:52:57.128Z] STEP: validate mysql-ibm app's volumes [2024-01-19T09:52:57.128Z] STEP: inspect mysql-ibm app's volumes [2024-01-19T09:52:57.128Z] STEP: get mysql-ibm app's volume's custom parameters [2024-01-19T09:52:57.128Z] STEP: get mysql-ibm app's volume: pvc-1ad9344c-e6ea-44a6-918d-3d717f09cb4b inspected by the volume driver [2024-01-19T09:52:57.128Z] STEP: wait for mysql-ibm app to start running [2024-01-19T09:52:57.128Z] STEP: validate if mysql-ibm app's volumes are setup [2024-01-19T09:52:57.128Z] STEP: validate if mysql-ibm app's volume: mysql-data is setup [2024-01-19T09:52:57.128Z] STEP: Validate Px pod restart count [2024-01-19T09:52:57.128Z] STEP: Getting current restart counts for portworx pods and matching [2024-01-19T09:52:57.128Z] STEP: Apply same namespace labels to new namespaces [2024-01-19T09:52:57.128Z] STEP: Verify new application namespaces with same namespace label are included in next schedule backup [2024-01-19T09:52:57.128Z] STEP: Restoring scheduled backups [2024-01-19T09:52:57.128Z] STEP: destroy apps [2024-01-19T09:52:57.128Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T09:52:57.128Z] STEP: start destroying mysql-ibm app [2024-01-19T09:52:57.128Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T09:52:57.129Z] STEP: destroy the mysql-ibm app's volumes [2024-01-19T09:52:57.129Z] STEP: start destroying mysql-ibm app [2024-01-19T09:52:57.129Z] STEP: validate mysql-ibm app's volume mysql-data has been deleted in the volume driver [2024-01-19T09:52:57.129Z] SS2024-01-19 09:52:56 +0000:[INFO] Log Dir: /testresults//Environment cleanup.log [2024-01-19T09:52:57.129Z] INFO[2024-01-19 09:52:56] --------Test Start------ [2024-01-19T09:52:57.129Z] INFO[2024-01-19 09:52:56] #Test: Environment cleanup [2024-01-19T09:52:57.129Z] INFO[2024-01-19 09:52:56] #Description: Removing Px-Backup entities created during the test execution [2024-01-19T09:52:57.129Z] INFO[2024-01-19 09:52:56] ------------------------ [2024-01-19T09:52:57.129Z] INFO[2024-01-19 09:52:57] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-094232 [2024-01-19T09:52:57.129Z] INFO[2024-01-19 09:52:57] Actual:true, Expected: true [2024-01-19T09:52:57.385Z] INFO[2024-01-19 09:52:57] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-094232 [2024-01-19T09:52:57.385Z] INFO[2024-01-19 09:52:57] Actual:true, Expected: true [2024-01-19T09:52:57.385Z] INFO[2024-01-19 09:52:57] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-094223 [2024-01-19T09:52:57.385Z] INFO[2024-01-19 09:52:57] Actual:true, Expected: true [2024-01-19T09:52:57.385Z] INFO[2024-01-19 09:52:57] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-094223 [2024-01-19T09:52:57.385Z] INFO[2024-01-19 09:52:57] Actual:true, Expected: true [2024-01-19T09:52:57.385Z] INFO[2024-01-19 09:52:57] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-094203 [2024-01-19T09:52:57.385Z] INFO[2024-01-19 09:52:57] Actual:true, Expected: true [2024-01-19T09:52:57.641Z] INFO[2024-01-19 09:52:57] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-094203 [2024-01-19T09:52:57.641Z] INFO[2024-01-19 09:52:57] Actual:true, Expected: true [2024-01-19T09:52:57.641Z] INFO[2024-01-19 09:52:57] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-092729 [2024-01-19T09:52:57.641Z] INFO[2024-01-19 09:52:57] Actual:true, Expected: true [2024-01-19T09:52:57.897Z] INFO[2024-01-19 09:52:57] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-092729 [2024-01-19T09:52:57.897Z] INFO[2024-01-19 09:52:57] Actual:true, Expected: true [2024-01-19T09:52:57.897Z] INFO[2024-01-19 09:52:57] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-092719 [2024-01-19T09:52:57.897Z] INFO[2024-01-19 09:52:57] Actual:true, Expected: true [2024-01-19T09:52:57.897Z] INFO[2024-01-19 09:52:57] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-092719 [2024-01-19T09:52:57.897Z] INFO[2024-01-19 09:52:57] Actual:true, Expected: true [2024-01-19T09:52:57.897Z] INFO[2024-01-19 09:52:57] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-092659 [2024-01-19T09:52:57.897Z] INFO[2024-01-19 09:52:57] Actual:true, Expected: true [2024-01-19T09:52:58.154Z] INFO[2024-01-19 09:52:57] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-092659 [2024-01-19T09:52:58.154Z] INFO[2024-01-19 09:52:57] Actual:true, Expected: true [2024-01-19T09:52:58.155Z] INFO[2024-01-19 09:52:58] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-091226 [2024-01-19T09:52:58.155Z] INFO[2024-01-19 09:52:58] Actual:true, Expected: true [2024-01-19T09:52:58.411Z] INFO[2024-01-19 09:52:58] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-091226 [2024-01-19T09:52:58.411Z] INFO[2024-01-19 09:52:58] Actual:true, Expected: true [2024-01-19T09:52:58.411Z] INFO[2024-01-19 09:52:58] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-091216 [2024-01-19T09:52:58.411Z] INFO[2024-01-19 09:52:58] Actual:true, Expected: true [2024-01-19T09:52:58.411Z] INFO[2024-01-19 09:52:58] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-091216 [2024-01-19T09:52:58.411Z] INFO[2024-01-19 09:52:58] Actual:true, Expected: true [2024-01-19T09:52:58.667Z] INFO[2024-01-19 09:52:58] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-091156 [2024-01-19T09:52:58.667Z] INFO[2024-01-19 09:52:58] Actual:true, Expected: true [2024-01-19T09:52:58.667Z] INFO[2024-01-19 09:52:58] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-091156 [2024-01-19T09:52:58.667Z] INFO[2024-01-19 09:52:58] Actual:true, Expected: true [2024-01-19T09:52:58.667Z] INFO[2024-01-19 09:52:58] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-085723 [2024-01-19T09:52:58.667Z] INFO[2024-01-19 09:52:58] Actual:true, Expected: true [2024-01-19T09:52:58.667Z] INFO[2024-01-19 09:52:58] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-085723 [2024-01-19T09:52:58.667Z] INFO[2024-01-19 09:52:58] Actual:true, Expected: true [2024-01-19T09:52:58.923Z] INFO[2024-01-19 09:52:58] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-085713 [2024-01-19T09:52:58.924Z] INFO[2024-01-19 09:52:58] Actual:true, Expected: true [2024-01-19T09:52:59.181Z] INFO[2024-01-19 09:52:58] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-085713 [2024-01-19T09:52:59.181Z] INFO[2024-01-19 09:52:58] Actual:true, Expected: true [2024-01-19T09:52:59.181Z] INFO[2024-01-19 09:52:59] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-085653 [2024-01-19T09:52:59.181Z] INFO[2024-01-19 09:52:59] Actual:true, Expected: true [2024-01-19T09:52:59.181Z] INFO[2024-01-19 09:52:59] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-085653 [2024-01-19T09:52:59.181Z] INFO[2024-01-19 09:52:59] Actual:true, Expected: true [2024-01-19T09:52:59.437Z] INFO[2024-01-19 09:52:59] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-084220 [2024-01-19T09:52:59.437Z] INFO[2024-01-19 09:52:59] Actual:true, Expected: true [2024-01-19T09:52:59.437Z] INFO[2024-01-19 09:52:59] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-084220 [2024-01-19T09:52:59.437Z] INFO[2024-01-19 09:52:59] Actual:true, Expected: true [2024-01-19T09:52:59.437Z] INFO[2024-01-19 09:52:59] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-084210 [2024-01-19T09:52:59.437Z] INFO[2024-01-19 09:52:59] Actual:true, Expected: true [2024-01-19T09:52:59.694Z] INFO[2024-01-19 09:52:59] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-084210 [2024-01-19T09:52:59.694Z] INFO[2024-01-19 09:52:59] Actual:true, Expected: true [2024-01-19T09:52:59.694Z] INFO[2024-01-19 09:52:59] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-084150 [2024-01-19T09:52:59.694Z] INFO[2024-01-19 09:52:59] Actual:true, Expected: true [2024-01-19T09:52:59.694Z] INFO[2024-01-19 09:52:59] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-084150 [2024-01-19T09:52:59.694Z] INFO[2024-01-19 09:52:59] Actual:true, Expected: true [2024-01-19T09:52:59.694Z] INFO[2024-01-19 09:52:59] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-061147-autogenerated-backup-location-wkyfv-9e4361f [2024-01-19T09:52:59.694Z] INFO[2024-01-19 09:52:59] Actual:true, Expected: true [2024-01-19T09:52:59.950Z] INFO[2024-01-19 09:52:59] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-061147-autogenerated-backup-location-wkyfv-9e4361f [2024-01-19T09:52:59.950Z] INFO[2024-01-19 09:52:59] Actual:true, Expected: true [2024-01-19T09:52:59.950Z] INFO[2024-01-19 09:52:59] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-061147-autogenerated-backup-location-1705635982-9e4361f [2024-01-19T09:52:59.950Z] INFO[2024-01-19 09:52:59] Actual:true, Expected: true [2024-01-19T09:53:00.209Z] INFO[2024-01-19 09:53:00] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-061147-autogenerated-backup-location-1705635982-9e4361f [2024-01-19T09:53:00.209Z] INFO[2024-01-19 09:53:00] Actual:true, Expected: true [2024-01-19T09:53:00.209Z] INFO[2024-01-19 09:53:00] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-061147-autogenerated-backup-location-1705634221-9e4361f [2024-01-19T09:53:00.209Z] INFO[2024-01-19 09:53:00] Actual:true, Expected: true [2024-01-19T09:53:00.209Z] INFO[2024-01-19 09:53:00] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-061147-autogenerated-backup-location-1705634221-9e4361f [2024-01-19T09:53:00.209Z] INFO[2024-01-19 09:53:00] Actual:true, Expected: true [2024-01-19T09:53:00.465Z] INFO[2024-01-19 09:53:00] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-061147-autogenerated-backup-location-1705628002-9e4361f [2024-01-19T09:53:00.465Z] INFO[2024-01-19 09:53:00] Actual:true, Expected: true [2024-01-19T09:53:00.465Z] INFO[2024-01-19 09:53:00] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-061147-autogenerated-backup-location-1705628002-9e4361f [2024-01-19T09:53:00.465Z] INFO[2024-01-19 09:53:00] Actual:true, Expected: true [2024-01-19T09:53:00.465Z] INFO[2024-01-19 09:53:00] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-061147-aws-global-aws-px-backup-system-test-4cac76dc-bl-9e4361f [2024-01-19T09:53:00.465Z] INFO[2024-01-19 09:53:00] Actual:true, Expected: true [2024-01-19T09:53:00.720Z] INFO[2024-01-19 09:53:00] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-061147-aws-global-aws-px-backup-system-test-4cac76dc-bl-9e4361f [2024-01-19T09:53:00.720Z] INFO[2024-01-19 09:53:00] Actual:true, Expected: true [2024-01-19T09:53:00.720Z] INFO[2024-01-19 09:53:00] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-061147-aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598-9e4361f [2024-01-19T09:53:00.720Z] INFO[2024-01-19 09:53:00] Actual:true, Expected: true [2024-01-19T09:53:00.720Z] INFO[2024-01-19 09:53:00] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-061147-aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598-9e4361f [2024-01-19T09:53:00.720Z] INFO[2024-01-19 09:53:00] Actual:true, Expected: true [2024-01-19T09:53:00.998Z] INFO[2024-01-19 09:53:00] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-061138-autogenerated-backup-location-wkyfv-d0207b8 [2024-01-19T09:53:00.998Z] INFO[2024-01-19 09:53:00] Actual:true, Expected: true [2024-01-19T09:53:00.998Z] INFO[2024-01-19 09:53:00] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-061138-autogenerated-backup-location-wkyfv-d0207b8 [2024-01-19T09:53:00.998Z] INFO[2024-01-19 09:53:00] Actual:true, Expected: true [2024-01-19T09:53:00.998Z] INFO[2024-01-19 09:53:00] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-061138-autogenerated-backup-location-1705635982-d0207b8 [2024-01-19T09:53:00.998Z] INFO[2024-01-19 09:53:00] Actual:true, Expected: true [2024-01-19T09:53:01.260Z] INFO[2024-01-19 09:53:00] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-061138-autogenerated-backup-location-1705635982-d0207b8 [2024-01-19T09:53:01.260Z] INFO[2024-01-19 09:53:00] Actual:true, Expected: true [2024-01-19T09:53:01.260Z] INFO[2024-01-19 09:53:01] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-061138-autogenerated-backup-location-1705634221-d0207b8 [2024-01-19T09:53:01.260Z] INFO[2024-01-19 09:53:01] Actual:true, Expected: true [2024-01-19T09:53:01.517Z] INFO[2024-01-19 09:53:01] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-061138-autogenerated-backup-location-1705634221-d0207b8 [2024-01-19T09:53:01.517Z] INFO[2024-01-19 09:53:01] Actual:true, Expected: true [2024-01-19T09:53:01.517Z] INFO[2024-01-19 09:53:01] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-061138-autogenerated-backup-location-1705628002-d0207b8 [2024-01-19T09:53:01.517Z] INFO[2024-01-19 09:53:01] Actual:true, Expected: true [2024-01-19T09:53:01.773Z] INFO[2024-01-19 09:53:01] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-061138-autogenerated-backup-location-1705628002-d0207b8 [2024-01-19T09:53:01.773Z] INFO[2024-01-19 09:53:01] Actual:true, Expected: true [2024-01-19T09:53:01.774Z] INFO[2024-01-19 09:53:01] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-061138-aws-global-aws-px-backup-system-test-4cac76dc-bl-d0207b8 [2024-01-19T09:53:01.774Z] INFO[2024-01-19 09:53:01] Actual:true, Expected: true [2024-01-19T09:53:01.774Z] INFO[2024-01-19 09:53:01] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-061138-aws-global-aws-px-backup-system-test-4cac76dc-bl-d0207b8 [2024-01-19T09:53:01.774Z] INFO[2024-01-19 09:53:01] Actual:true, Expected: true [2024-01-19T09:53:01.774Z] INFO[2024-01-19 09:53:01] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-061138-aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598-d0207b8 [2024-01-19T09:53:01.774Z] INFO[2024-01-19 09:53:01] Actual:true, Expected: true [2024-01-19T09:53:02.029Z] INFO[2024-01-19 09:53:01] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-061138-aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598-d0207b8 [2024-01-19T09:53:02.029Z] INFO[2024-01-19 09:53:01] Actual:true, Expected: true [2024-01-19T09:53:02.030Z] INFO[2024-01-19 09:53:01] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-061117-autogenerated-backup-location-wkyfv-91ca43c [2024-01-19T09:53:02.030Z] INFO[2024-01-19 09:53:01] Actual:true, Expected: true [2024-01-19T09:53:02.030Z] INFO[2024-01-19 09:53:01] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-061117-autogenerated-backup-location-wkyfv-91ca43c [2024-01-19T09:53:02.030Z] INFO[2024-01-19 09:53:01] Actual:true, Expected: true [2024-01-19T09:53:02.286Z] INFO[2024-01-19 09:53:02] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-061117-autogenerated-backup-location-1705635982-91ca43c [2024-01-19T09:53:02.286Z] INFO[2024-01-19 09:53:02] Actual:true, Expected: true [2024-01-19T09:53:02.286Z] INFO[2024-01-19 09:53:02] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-061117-autogenerated-backup-location-1705635982-91ca43c [2024-01-19T09:53:02.286Z] INFO[2024-01-19 09:53:02] Actual:true, Expected: true [2024-01-19T09:53:02.286Z] INFO[2024-01-19 09:53:02] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-061117-autogenerated-backup-location-1705634221-91ca43c [2024-01-19T09:53:02.286Z] INFO[2024-01-19 09:53:02] Actual:true, Expected: true [2024-01-19T09:53:02.542Z] INFO[2024-01-19 09:53:02] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-061117-autogenerated-backup-location-1705634221-91ca43c [2024-01-19T09:53:02.542Z] INFO[2024-01-19 09:53:02] Actual:true, Expected: true [2024-01-19T09:53:02.542Z] INFO[2024-01-19 09:53:02] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-061117-autogenerated-backup-location-1705628002-91ca43c [2024-01-19T09:53:02.542Z] INFO[2024-01-19 09:53:02] Actual:true, Expected: true [2024-01-19T09:53:02.798Z] INFO[2024-01-19 09:53:02] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-061117-autogenerated-backup-location-1705628002-91ca43c [2024-01-19T09:53:02.798Z] INFO[2024-01-19 09:53:02] Actual:true, Expected: true [2024-01-19T09:53:02.798Z] INFO[2024-01-19 09:53:02] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-061117-aws-global-aws-px-backup-system-test-4cac76dc-bl-91ca43c [2024-01-19T09:53:02.798Z] INFO[2024-01-19 09:53:02] Actual:true, Expected: true [2024-01-19T09:53:02.798Z] INFO[2024-01-19 09:53:02] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-061117-aws-global-aws-px-backup-system-test-4cac76dc-bl-91ca43c [2024-01-19T09:53:02.798Z] INFO[2024-01-19 09:53:02] Actual:true, Expected: true [2024-01-19T09:53:02.798Z] INFO[2024-01-19 09:53:02] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-061117-aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598-91ca43c [2024-01-19T09:53:02.798Z] INFO[2024-01-19 09:53:02] Actual:true, Expected: true [2024-01-19T09:53:03.054Z] INFO[2024-01-19 09:53:02] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-061117-aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598-91ca43c [2024-01-19T09:53:03.054Z] INFO[2024-01-19 09:53:02] Actual:true, Expected: true [2024-01-19T09:53:03.054Z] INFO[2024-01-19 09:53:02] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643-autogenerated-backup-location-wkyfv-91bbc09 [2024-01-19T09:53:03.054Z] INFO[2024-01-19 09:53:02] Actual:true, Expected: true [2024-01-19T09:53:03.313Z] INFO[2024-01-19 09:53:03] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643-autogenerated-backup-location-wkyfv-91bbc09 [2024-01-19T09:53:03.313Z] INFO[2024-01-19 09:53:03] Actual:true, Expected: true [2024-01-19T09:53:03.313Z] INFO[2024-01-19 09:53:03] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643-autogenerated-backup-location-1705635982-91bbc09 [2024-01-19T09:53:03.313Z] INFO[2024-01-19 09:53:03] Actual:true, Expected: true [2024-01-19T09:53:03.313Z] INFO[2024-01-19 09:53:03] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643-autogenerated-backup-location-1705635982-91bbc09 [2024-01-19T09:53:03.313Z] INFO[2024-01-19 09:53:03] Actual:true, Expected: true [2024-01-19T09:53:03.568Z] INFO[2024-01-19 09:53:03] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643-autogenerated-backup-location-1705634221-91bbc09 [2024-01-19T09:53:03.568Z] INFO[2024-01-19 09:53:03] Actual:true, Expected: true [2024-01-19T09:53:03.568Z] INFO[2024-01-19 09:53:03] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643-autogenerated-backup-location-1705634221-91bbc09 [2024-01-19T09:53:03.568Z] INFO[2024-01-19 09:53:03] Actual:true, Expected: true [2024-01-19T09:53:03.568Z] INFO[2024-01-19 09:53:03] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643-autogenerated-backup-location-1705628002-91bbc09 [2024-01-19T09:53:03.568Z] INFO[2024-01-19 09:53:03] Actual:true, Expected: true [2024-01-19T09:53:03.824Z] INFO[2024-01-19 09:53:03] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643-autogenerated-backup-location-1705628002-91bbc09 [2024-01-19T09:53:03.824Z] INFO[2024-01-19 09:53:03] Actual:true, Expected: true [2024-01-19T09:53:03.824Z] INFO[2024-01-19 09:53:03] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643-aws-global-aws-px-backup-system-test-4cac76dc-bl-91bbc09 [2024-01-19T09:53:03.825Z] INFO[2024-01-19 09:53:03] Actual:true, Expected: true [2024-01-19T09:53:03.825Z] INFO[2024-01-19 09:53:03] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643-aws-global-aws-px-backup-system-test-4cac76dc-bl-91bbc09 [2024-01-19T09:53:03.825Z] INFO[2024-01-19 09:53:03] Actual:true, Expected: true [2024-01-19T09:53:04.083Z] INFO[2024-01-19 09:53:03] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643-aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598-91bbc09 [2024-01-19T09:53:04.083Z] INFO[2024-01-19 09:53:03] Actual:true, Expected: true [2024-01-19T09:53:04.083Z] INFO[2024-01-19 09:53:03] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637505-with-rules-zvfc-3533a38-interval-2024-01-19-055643-aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598-91bbc09 [2024-01-19T09:53:04.083Z] INFO[2024-01-19 09:53:03] Actual:true, Expected: true [2024-01-19T09:53:04.083Z] INFO[2024-01-19 09:53:03] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633-autogenerated-backup-location-wkyfv-6b37bd8 [2024-01-19T09:53:04.083Z] INFO[2024-01-19 09:53:03] Actual:true, Expected: true [2024-01-19T09:53:04.339Z] INFO[2024-01-19 09:53:04] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633-autogenerated-backup-location-wkyfv-6b37bd8 [2024-01-19T09:53:04.339Z] INFO[2024-01-19 09:53:04] Actual:true, Expected: true [2024-01-19T09:53:04.339Z] INFO[2024-01-19 09:53:04] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633-autogenerated-backup-location-1705635982-6b37bd8 [2024-01-19T09:53:04.339Z] INFO[2024-01-19 09:53:04] Actual:true, Expected: true [2024-01-19T09:53:04.339Z] INFO[2024-01-19 09:53:04] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633-autogenerated-backup-location-1705635982-6b37bd8 [2024-01-19T09:53:04.339Z] INFO[2024-01-19 09:53:04] Actual:true, Expected: true [2024-01-19T09:53:04.597Z] INFO[2024-01-19 09:53:04] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633-autogenerated-backup-location-1705634221-6b37bd8 [2024-01-19T09:53:04.598Z] INFO[2024-01-19 09:53:04] Actual:true, Expected: true [2024-01-19T09:53:04.606Z] INFO[2024-01-19 09:53:04] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633-autogenerated-backup-location-1705634221-6b37bd8 [2024-01-19T09:53:04.606Z] INFO[2024-01-19 09:53:04] Actual:true, Expected: true [2024-01-19T09:53:04.606Z] INFO[2024-01-19 09:53:04] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633-autogenerated-backup-location-1705628002-6b37bd8 [2024-01-19T09:53:04.607Z] INFO[2024-01-19 09:53:04] Actual:true, Expected: true [2024-01-19T09:53:04.607Z] INFO[2024-01-19 09:53:04] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633-autogenerated-backup-location-1705628002-6b37bd8 [2024-01-19T09:53:04.607Z] INFO[2024-01-19 09:53:04] Actual:true, Expected: true [2024-01-19T09:53:04.862Z] INFO[2024-01-19 09:53:04] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633-aws-global-aws-px-backup-system-test-4cac76dc-bl-6b37bd8 [2024-01-19T09:53:04.862Z] INFO[2024-01-19 09:53:04] Actual:true, Expected: true [2024-01-19T09:53:04.863Z] INFO[2024-01-19 09:53:04] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633-aws-global-aws-px-backup-system-test-4cac76dc-bl-6b37bd8 [2024-01-19T09:53:04.863Z] INFO[2024-01-19 09:53:04] Actual:true, Expected: true [2024-01-19T09:53:04.863Z] INFO[2024-01-19 09:53:04] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633-aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598-6b37bd8 [2024-01-19T09:53:04.863Z] INFO[2024-01-19 09:53:04] Actual:true, Expected: true [2024-01-19T09:53:05.120Z] INFO[2024-01-19 09:53:04] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637501-with-rules-pulv-9aec100-interval-2024-01-19-055633-aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598-6b37bd8 [2024-01-19T09:53:05.120Z] INFO[2024-01-19 09:53:04] Actual:true, Expected: true [2024-01-19T09:53:05.120Z] INFO[2024-01-19 09:53:04] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612-autogenerated-backup-location-wkyfv-3a02d01 [2024-01-19T09:53:05.120Z] INFO[2024-01-19 09:53:04] Actual:true, Expected: true [2024-01-19T09:53:05.120Z] INFO[2024-01-19 09:53:05] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612-autogenerated-backup-location-wkyfv-3a02d01 [2024-01-19T09:53:05.120Z] INFO[2024-01-19 09:53:05] Actual:true, Expected: true [2024-01-19T09:53:05.378Z] INFO[2024-01-19 09:53:05] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612-autogenerated-backup-location-1705635982-3a02d01 [2024-01-19T09:53:05.378Z] INFO[2024-01-19 09:53:05] Actual:true, Expected: true [2024-01-19T09:53:05.378Z] INFO[2024-01-19 09:53:05] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612-autogenerated-backup-location-1705635982-3a02d01 [2024-01-19T09:53:05.378Z] INFO[2024-01-19 09:53:05] Actual:true, Expected: true [2024-01-19T09:53:05.378Z] INFO[2024-01-19 09:53:05] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612-autogenerated-backup-location-1705634221-3a02d01 [2024-01-19T09:53:05.378Z] INFO[2024-01-19 09:53:05] Actual:true, Expected: true [2024-01-19T09:53:05.634Z] INFO[2024-01-19 09:53:05] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612-autogenerated-backup-location-1705634221-3a02d01 [2024-01-19T09:53:05.634Z] INFO[2024-01-19 09:53:05] Actual:true, Expected: true [2024-01-19T09:53:05.634Z] INFO[2024-01-19 09:53:05] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612-autogenerated-backup-location-1705628002-3a02d01 [2024-01-19T09:53:05.634Z] INFO[2024-01-19 09:53:05] Actual:true, Expected: true [2024-01-19T09:53:05.634Z] INFO[2024-01-19 09:53:05] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612-autogenerated-backup-location-1705628002-3a02d01 [2024-01-19T09:53:05.634Z] INFO[2024-01-19 09:53:05] Actual:true, Expected: true [2024-01-19T09:53:05.890Z] INFO[2024-01-19 09:53:05] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612-aws-global-aws-px-backup-system-test-4cac76dc-bl-3a02d01 [2024-01-19T09:53:05.890Z] INFO[2024-01-19 09:53:05] Actual:true, Expected: true [2024-01-19T09:53:05.890Z] INFO[2024-01-19 09:53:05] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612-aws-global-aws-px-backup-system-test-4cac76dc-bl-3a02d01 [2024-01-19T09:53:05.890Z] INFO[2024-01-19 09:53:05] Actual:true, Expected: true [2024-01-19T09:53:05.890Z] INFO[2024-01-19 09:53:05] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612-aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598-3a02d01 [2024-01-19T09:53:05.890Z] INFO[2024-01-19 09:53:05] Actual:true, Expected: true [2024-01-19T09:53:06.145Z] INFO[2024-01-19 09:53:05] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637494-with-rules-ipvs-2290b23-interval-2024-01-19-055612-aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598-3a02d01 [2024-01-19T09:53:06.145Z] INFO[2024-01-19 09:53:05] Actual:true, Expected: true [2024-01-19T09:53:06.145Z] INFO[2024-01-19 09:53:05] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641 [2024-01-19T09:53:06.146Z] INFO[2024-01-19 09:53:05] Actual:true, Expected: true [2024-01-19T09:53:06.146Z] INFO[2024-01-19 09:53:06] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637513-with-rules-wpck-35ab8d9-interval-2024-01-19-042641 [2024-01-19T09:53:06.146Z] INFO[2024-01-19 09:53:06] Actual:true, Expected: true [2024-01-19T09:53:06.402Z] INFO[2024-01-19 09:53:06] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 [2024-01-19T09:53:06.402Z] INFO[2024-01-19 09:53:06] Actual:true, Expected: true [2024-01-19T09:53:06.403Z] INFO[2024-01-19 09:53:06] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637509-with-rules-bhgj-5555180-interval-2024-01-19-042631 [2024-01-19T09:53:06.403Z] INFO[2024-01-19 09:53:06] Actual:true, Expected: true [2024-01-19T09:53:06.403Z] INFO[2024-01-19 09:53:06] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602 [2024-01-19T09:53:06.403Z] INFO[2024-01-19 09:53:06] Actual:true, Expected: true [2024-01-19T09:53:06.658Z] INFO[2024-01-19 09:53:06] Verifying : Description : Verifying backup deletion - tp-backup-schedule-nslabel-testuser1-1705637497-with-rules-aosm-c356103-interval-2024-01-19-042602 [2024-01-19T09:53:06.658Z] INFO[2024-01-19 09:53:06] Actual:true, Expected: true [2024-01-19T09:53:06.658Z] INFO[2024-01-19 09:53:06] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923 [2024-01-19T09:53:06.659Z] INFO[2024-01-19 09:53:06] Actual:true, Expected: true [2024-01-19T09:53:06.659Z] INFO[2024-01-19 09:53:06] Verifying : Description : Verifying backup deletion - tp-backup-schedule-multiple-ns-testuser1-1705637513-with-rules-nzjf-253fd1d-interval-2024-01-19-041923 [2024-01-19T09:53:06.659Z] INFO[2024-01-19 09:53:06] Actual:true, Expected: true [2024-01-19T09:53:06.915Z] INFO[2024-01-19 09:53:06] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 [2024-01-19T09:53:06.916Z] INFO[2024-01-19 09:53:06] Actual:true, Expected: true [2024-01-19T09:53:06.916Z] INFO[2024-01-19 09:53:06] Verifying : Description : Verifying backup deletion - tp-backup-schedule-multiple-ns-testuser1-1705637509-with-rules-nwwo-15cb0d5-interval-2024-01-19-041913 [2024-01-19T09:53:06.916Z] INFO[2024-01-19 09:53:06] Actual:true, Expected: true [2024-01-19T09:53:06.916Z] INFO[2024-01-19 09:53:06] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 [2024-01-19T09:53:06.916Z] INFO[2024-01-19 09:53:06] Actual:true, Expected: true [2024-01-19T09:53:07.171Z] INFO[2024-01-19 09:53:07] Verifying : Description : Verifying backup deletion - tp-backup-schedule-multiple-ns-testuser1-1705637505-with-rules-ledn-21f9e4e-interval-2024-01-19-041903 [2024-01-19T09:53:07.171Z] INFO[2024-01-19 09:53:07] Actual:true, Expected: true [2024-01-19T09:53:07.172Z] INFO[2024-01-19 09:53:07] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 [2024-01-19T09:53:07.172Z] INFO[2024-01-19 09:53:07] Actual:true, Expected: true [2024-01-19T09:53:07.428Z] INFO[2024-01-19 09:53:07] Verifying : Description : Verifying backup deletion - tp-backup-schedule-multiple-ns-testuser1-1705637501-with-rules-ckpb-893465a-interval-2024-01-19-041853 [2024-01-19T09:53:07.428Z] INFO[2024-01-19 09:53:07] Actual:true, Expected: true [2024-01-19T09:53:07.428Z] INFO[2024-01-19 09:53:07] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 [2024-01-19T09:53:07.428Z] INFO[2024-01-19 09:53:07] Actual:true, Expected: true [2024-01-19T09:53:07.684Z] INFO[2024-01-19 09:53:07] Verifying : Description : Verifying backup deletion - tp-backup-schedule-multiple-ns-testuser1-1705637497-with-rules-qofg-36c480f-interval-2024-01-19-041843 [2024-01-19T09:53:07.684Z] INFO[2024-01-19 09:53:07] Actual:true, Expected: true [2024-01-19T09:53:07.684Z] INFO[2024-01-19 09:53:07] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 [2024-01-19T09:53:07.684Z] INFO[2024-01-19 09:53:07] Actual:true, Expected: true [2024-01-19T09:53:07.684Z] INFO[2024-01-19 09:53:07] Verifying : Description : Verifying backup deletion - tp-backup-schedule-multiple-ns-testuser1-1705637494-with-rules-hrmn-bfdedd9-interval-2024-01-19-041833 [2024-01-19T09:53:07.684Z] INFO[2024-01-19 09:53:07] Actual:true, Expected: true [2024-01-19T09:53:07.684Z] INFO[2024-01-19 09:53:07] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342 [2024-01-19T09:53:07.684Z] INFO[2024-01-19 09:53:07] Actual:true, Expected: true [2024-01-19T09:53:07.940Z] INFO[2024-01-19 09:53:07] Verifying : Description : Verifying backup deletion - tp-backup-schedule-single-ns-testuser1-1705637513-without-rules-tyxa-fbead3a-interval-2024-01-19-041342 [2024-01-19T09:53:07.940Z] INFO[2024-01-19 09:53:07] Actual:true, Expected: true [2024-01-19T09:53:07.941Z] INFO[2024-01-19 09:53:07] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332 [2024-01-19T09:53:07.941Z] INFO[2024-01-19 09:53:07] Actual:true, Expected: true [2024-01-19T09:53:07.941Z] INFO[2024-01-19 09:53:07] Verifying : Description : Verifying backup deletion - tp-backup-schedule-single-ns-testuser1-1705637509-without-rules-capx-455ee79-interval-2024-01-19-041332 [2024-01-19T09:53:07.941Z] INFO[2024-01-19 09:53:07] Actual:true, Expected: true [2024-01-19T09:53:08.196Z] INFO[2024-01-19 09:53:07] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 [2024-01-19T09:53:08.196Z] INFO[2024-01-19 09:53:07] Actual:true, Expected: true [2024-01-19T09:53:08.196Z] INFO[2024-01-19 09:53:08] Verifying : Description : Verifying backup deletion - tp-backup-schedule-single-ns-testuser1-1705637505-without-rules-izzo-9f474fe-interval-2024-01-19-041322 [2024-01-19T09:53:08.197Z] INFO[2024-01-19 09:53:08] Actual:true, Expected: true [2024-01-19T09:53:08.197Z] INFO[2024-01-19 09:53:08] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312 [2024-01-19T09:53:08.197Z] INFO[2024-01-19 09:53:08] Actual:true, Expected: true [2024-01-19T09:53:08.452Z] INFO[2024-01-19 09:53:08] Verifying : Description : Verifying backup deletion - tp-backup-schedule-single-ns-testuser1-1705637501-without-rules-lcuo-6bde591-interval-2024-01-19-041312 [2024-01-19T09:53:08.452Z] INFO[2024-01-19 09:53:08] Actual:true, Expected: true [2024-01-19T09:53:08.452Z] INFO[2024-01-19 09:53:08] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302 [2024-01-19T09:53:08.452Z] INFO[2024-01-19 09:53:08] Actual:true, Expected: true [2024-01-19T09:53:08.452Z] INFO[2024-01-19 09:53:08] Verifying : Description : Verifying backup deletion - tp-backup-schedule-single-ns-testuser1-1705637497-without-rules-cixh-c455deb-interval-2024-01-19-041302 [2024-01-19T09:53:08.452Z] INFO[2024-01-19 09:53:08] Actual:true, Expected: true [2024-01-19T09:53:08.453Z] INFO[2024-01-19 09:53:08] Verifying : Description : Getting backuip UID for backup tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl-7c3a711-interval-2024-01-19-041252 [2024-01-19T09:53:08.453Z] INFO[2024-01-19 09:53:08] Actual:true, Expected: true [2024-01-19T09:53:08.708Z] INFO[2024-01-19 09:53:08] Verifying : Description : Verifying backup deletion - tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl-7c3a711-interval-2024-01-19-041252 [2024-01-19T09:53:08.708Z] INFO[2024-01-19 09:53:08] Actual:true, Expected: true [2024-01-19T09:53:08.708Z] INFO[2024-01-19 09:53:08] Verifying : Description : Getting backuip UID for backup tp-backup-1705636048 [2024-01-19T09:53:08.708Z] INFO[2024-01-19 09:53:08] Actual:true, Expected: true [2024-01-19T09:53:08.964Z] INFO[2024-01-19 09:53:08] Verifying : Description : Verifying backup deletion - tp-backup-1705636048 [2024-01-19T09:53:08.964Z] INFO[2024-01-19 09:53:08] Actual:true, Expected: true [2024-01-19T09:53:08.964Z] INFO[2024-01-19 09:53:08] Verifying : Description : Getting backuip UID for backup tp-backup-1705636038 [2024-01-19T09:53:08.964Z] INFO[2024-01-19 09:53:08] Actual:true, Expected: true [2024-01-19T09:53:09.220Z] INFO[2024-01-19 09:53:08] Verifying : Description : Verifying backup deletion - tp-backup-1705636038 [2024-01-19T09:53:09.220Z] INFO[2024-01-19 09:53:08] Actual:true, Expected: true [2024-01-19T09:53:09.220Z] INFO[2024-01-19 09:53:08] Verifying : Description : Getting backuip UID for backup tp-backup-1705634226 [2024-01-19T09:53:09.220Z] INFO[2024-01-19 09:53:08] Actual:true, Expected: true [2024-01-19T09:53:09.220Z] INFO[2024-01-19 09:53:09] Verifying : Description : Verifying backup deletion - tp-backup-1705634226 [2024-01-19T09:53:09.220Z] INFO[2024-01-19 09:53:09] Actual:true, Expected: true [2024-01-19T09:53:09.783Z] INFO[2024-01-19 09:53:09] Verifying : Description : Getting the backup driver [2024-01-19T09:53:09.783Z] INFO[2024-01-19 09:53:09] Actual:true, Expected: true [2024-01-19T09:53:09.783Z] INFO[2024-01-19 09:53:09] Verifying : Description : Verifying restore deletion - tp-restore-1705654428 [2024-01-19T09:53:09.783Z] INFO[2024-01-19 09:53:09] Actual:true, Expected: true [2024-01-19T09:53:09.783Z] INFO[2024-01-19 09:53:09] Verifying : Description : Getting the backup driver [2024-01-19T09:53:09.783Z] INFO[2024-01-19 09:53:09] Actual:true, Expected: true [2024-01-19T09:53:09.783Z] INFO[2024-01-19 09:53:09] Verifying : Description : Verifying restore deletion - testuser1-1705637494-test-restore-single-ns-pbdt [2024-01-19T09:53:09.783Z] INFO[2024-01-19 09:53:09] Actual:true, Expected: true [2024-01-19T09:53:09.783Z] INFO[2024-01-19 09:53:09] Verifying : Description : Getting the backup driver [2024-01-19T09:53:09.783Z] INFO[2024-01-19 09:53:09] Actual:true, Expected: true [2024-01-19T09:53:10.039Z] INFO[2024-01-19 09:53:09] Verifying : Description : Verifying restore deletion - tp-restore-tp-backup-manual-single-ns-with-rules-bfdr [2024-01-19T09:53:10.039Z] INFO[2024-01-19 09:53:09] Actual:true, Expected: true [2024-01-19T09:53:10.039Z] INFO[2024-01-19 09:53:09] Verifying : Description : Getting the backup driver [2024-01-19T09:53:10.039Z] INFO[2024-01-19 09:53:09] Actual:true, Expected: true [2024-01-19T09:53:10.295Z] INFO[2024-01-19 09:53:10] Verifying : Description : Verifying restore deletion - tp-restore-1705636305 [2024-01-19T09:53:10.295Z] INFO[2024-01-19 09:53:10] Actual:true, Expected: true [2024-01-19T09:53:10.295Z] INFO[2024-01-19 09:53:10] Verifying : Description : Getting the backup driver [2024-01-19T09:53:10.295Z] INFO[2024-01-19 09:53:10] Actual:true, Expected: true [2024-01-19T09:53:10.295Z] INFO[2024-01-19 09:53:10] Verifying : Description : Verifying restore deletion - tp-restore-1705634322 [2024-01-19T09:53:10.295Z] INFO[2024-01-19 09:53:10] Actual:true, Expected: true [2024-01-19T09:53:10.295Z] INFO[2024-01-19 09:53:10] Verifying : Description : Getting the backup driver [2024-01-19T09:53:10.296Z] INFO[2024-01-19 09:53:10] Actual:true, Expected: true [2024-01-19T09:53:10.296Z] INFO[2024-01-19 09:53:10] Verifying : Description : Verifying restore deletion - tp-restore-backup-1705633224 [2024-01-19T09:53:10.296Z] INFO[2024-01-19 09:53:10] Actual:true, Expected: true [2024-01-19T09:53:10.296Z] INFO[2024-01-19 09:53:10] Verifying : Description : Getting the backup driver [2024-01-19T09:53:10.296Z] INFO[2024-01-19 09:53:10] Actual:true, Expected: true [2024-01-19T09:53:10.297Z] INFO[2024-01-19 09:53:10] Verifying : Description : Verifying restore deletion - tp-restore-1705628372 [2024-01-19T09:53:10.297Z] INFO[2024-01-19 09:53:10] Actual:true, Expected: true [2024-01-19T09:53:10.297Z] INFO[2024-01-19 09:53:10] Verifying : Description : Getting the backup driver [2024-01-19T09:53:10.297Z] INFO[2024-01-19 09:53:10] Actual:true, Expected: true [2024-01-19T09:53:10.554Z] INFO[2024-01-19 09:53:10] Verifying : Description : Verifying restore deletion - tp-restore-backup-schedule-nxxtr-c26d2e0-interval-2024-01-19-010855 [2024-01-19T09:53:10.554Z] INFO[2024-01-19 09:53:10] Actual:true, Expected: true [2024-01-19T09:53:10.555Z] 2024-01-19 09:53:10 +0000:[INFO] [backup.getAllBackupLocations:#1603] - The backup locations and their UID are map[4418c3bb-e07a-4cdc-83e7-b25be38d1e4c:aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 4b199f31-3f2e-4a73-a7ab-aeb20f3ab33a:autogenerated-backup-location-1705628002 4c14da25-9552-4acb-a4e8-a02da0fa1688:aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 654a1515-d800-4a43-bf14-8b83a0c5ec1e:autogenerated-backup-location-1705634221 b4816cdb-0621-48dc-a333-0ee37ec2c86c:autogenerated-backup-location-1705635982 cd190df6-e916-41da-b969-ae70c2b830cb:autogenerated-backup-location-omphu e32820d4-7435-4cf7-a3c7-d857b223c325:aws-global-aws-px-backup-system-test-4cac76dc-bl fb5a14e2-2342-4596-b12d-cfadc2b7ea70:autogenerated-backup-location-wkyfv] [2024-01-19T09:53:10.555Z] INFO[2024-01-19 09:53:10] Verifying : Description : Verifying fetching of all backup locations [2024-01-19T09:53:10.555Z] INFO[2024-01-19 09:53:10] Actual:true, Expected: true [2024-01-19T09:53:11.115Z] INFO[2024-01-19 09:53:10] Verifying : Description : Verifying backup location deletion - aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 [2024-01-19T09:53:11.115Z] INFO[2024-01-19 09:53:10] Actual:true, Expected: true [2024-01-19T09:53:11.678Z] INFO[2024-01-19 09:53:11] Verifying : Description : Verifying backup location deletion - autogenerated-backup-location-omphu [2024-01-19T09:53:11.678Z] INFO[2024-01-19 09:53:11] Actual:true, Expected: true [2024-01-19T09:53:13.041Z] INFO[2024-01-19 09:53:13] Verifying : Description : Verifying backup location deletion - aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 [2024-01-19T09:53:13.041Z] INFO[2024-01-19 09:53:13] Actual:true, Expected: true [2024-01-19T09:53:14.413Z] INFO[2024-01-19 09:53:14] Verifying : Description : Verifying backup location deletion - aws-global-aws-px-backup-system-test-4cac76dc-bl [2024-01-19T09:53:14.413Z] INFO[2024-01-19 09:53:14] Actual:true, Expected: true [2024-01-19T09:53:16.302Z] INFO[2024-01-19 09:53:15] Verifying : Description : Verifying backup location deletion - autogenerated-backup-location-1705628002 [2024-01-19T09:53:16.302Z] INFO[2024-01-19 09:53:15] Actual:true, Expected: true [2024-01-19T09:53:17.228Z] INFO[2024-01-19 09:53:17] Verifying : Description : Verifying backup location deletion - autogenerated-backup-location-1705634221 [2024-01-19T09:53:17.228Z] INFO[2024-01-19 09:53:17] Actual:true, Expected: true [2024-01-19T09:53:18.594Z] INFO[2024-01-19 09:53:18] Verifying : Description : Verifying backup location deletion - autogenerated-backup-location-1705635982 [2024-01-19T09:53:18.594Z] INFO[2024-01-19 09:53:18] Actual:true, Expected: true [2024-01-19T09:53:19.963Z] INFO[2024-01-19 09:53:19] Verifying : Description : Verifying backup location deletion - autogenerated-backup-location-wkyfv [2024-01-19T09:53:19.963Z] ERRO[2024-01-19 09:53:19] Actual:rpc error: code = Unknown desc = backup schedule object [tp-backup-schedule-single-ns-testuser1-1705637494-without-rules-aocl] has reference to backuplocation [autogenerated-backup-location-wkyfv:fb5a14e2-2342-4596-b12d-cfadc2b7ea70], Expected:  [2024-01-19T09:53:20.219Z] 2024-01-19 09:53:19 +0000:[INFO] [backup.getAllBackupLocations:#1603] - The backup locations and their UID are map[4418c3bb-e07a-4cdc-83e7-b25be38d1e4c:aws-global-aws-px-backup-system-test-2983c664-bl-1705577909 4b199f31-3f2e-4a73-a7ab-aeb20f3ab33a:autogenerated-backup-location-1705628002 4c14da25-9552-4acb-a4e8-a02da0fa1688:aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 654a1515-d800-4a43-bf14-8b83a0c5ec1e:autogenerated-backup-location-1705634221 b4816cdb-0621-48dc-a333-0ee37ec2c86c:autogenerated-backup-location-1705635982 cd190df6-e916-41da-b969-ae70c2b830cb:autogenerated-backup-location-omphu e32820d4-7435-4cf7-a3c7-d857b223c325:aws-global-aws-px-backup-system-test-4cac76dc-bl fb5a14e2-2342-4596-b12d-cfadc2b7ea70:autogenerated-backup-location-wkyfv] [2024-01-19T09:53:20.219Z] INFO[2024-01-19 09:53:19] Verifying : Description : Verifying fetching of all backup locations [2024-01-19T09:53:20.219Z] INFO[2024-01-19 09:53:19] Actual:true, Expected: true [2024-01-19T09:53:20.219Z] 2024/01/19 09:53:19 DoRetryWithTimeout - Error: {found 8 backup locations}, Next try in [30s], timeout [5m0s] [2024-01-19T09:53:52.247Z] 2024-01-19 09:53:49 +0000:[INFO] [backup.getAllBackupLocations:#1603] - The backup locations and their UID are map[4b199f31-3f2e-4a73-a7ab-aeb20f3ab33a:autogenerated-backup-location-1705628002 4c14da25-9552-4acb-a4e8-a02da0fa1688:aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 654a1515-d800-4a43-bf14-8b83a0c5ec1e:autogenerated-backup-location-1705634221 b4816cdb-0621-48dc-a333-0ee37ec2c86c:autogenerated-backup-location-1705635982 e32820d4-7435-4cf7-a3c7-d857b223c325:aws-global-aws-px-backup-system-test-4cac76dc-bl fb5a14e2-2342-4596-b12d-cfadc2b7ea70:autogenerated-backup-location-wkyfv] [2024-01-19T09:53:52.247Z] INFO[2024-01-19 09:53:49] Verifying : Description : Verifying fetching of all backup locations [2024-01-19T09:53:52.247Z] INFO[2024-01-19 09:53:49] Actual:true, Expected: true [2024-01-19T09:53:52.247Z] 2024/01/19 09:53:49 DoRetryWithTimeout - Error: {found 6 backup locations}, Next try in [30s], timeout [5m0s] [2024-01-19T09:54:24.266Z] 2024-01-19 09:54:20 +0000:[INFO] [backup.getAllBackupLocations:#1603] - The backup locations and their UID are map[4b199f31-3f2e-4a73-a7ab-aeb20f3ab33a:autogenerated-backup-location-1705628002 4c14da25-9552-4acb-a4e8-a02da0fa1688:aws-global-aws-px-backup-system-test-4cac76dc-bl-1705605598 654a1515-d800-4a43-bf14-8b83a0c5ec1e:autogenerated-backup-location-1705634221 b4816cdb-0621-48dc-a333-0ee37ec2c86c:autogenerated-backup-location-1705635982 e32820d4-7435-4cf7-a3c7-d857b223c325:aws-global-aws-px-backup-system-test-4cac76dc-bl fb5a14e2-2342-4596-b12d-cfadc2b7ea70:autogenerated-backup-location-wkyfv] [2024-01-19T09:54:24.266Z] INFO[2024-01-19 09:54:20] Verifying : Description : Verifying fetching of all backup locations [2024-01-19T09:54:24.266Z] INFO[2024-01-19 09:54:20] Actual:true, Expected: true [2024-01-19T09:54:24.266Z] 2024/01/19 09:54:20 DoRetryWithTimeout - Error: {found 6 backup locations}, Next try in [30s], timeout [5m0s] [2024-01-19T09:54:50.761Z] 2024-01-19 09:54:50 +0000:[INFO] [backup.getAllBackupLocations:#1603] - The backup locations and their UID are map[654a1515-d800-4a43-bf14-8b83a0c5ec1e:autogenerated-backup-location-1705634221 b4816cdb-0621-48dc-a333-0ee37ec2c86c:autogenerated-backup-location-1705635982 fb5a14e2-2342-4596-b12d-cfadc2b7ea70:autogenerated-backup-location-wkyfv] [2024-01-19T09:54:50.761Z] INFO[2024-01-19 09:54:50] Verifying : Description : Verifying fetching of all backup locations [2024-01-19T09:54:50.761Z] INFO[2024-01-19 09:54:50] Actual:true, Expected: true [2024-01-19T09:54:50.761Z] 2024/01/19 09:54:50 DoRetryWithTimeout - Error: {found 3 backup locations}, Next try in [30s], timeout [5m0s] [2024-01-19T09:55:22.817Z] INFO[2024-01-19 09:55:18] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T09:55:22.817Z] INFO[2024-01-19 09:55:18] sched stats: tick interval: 66078 < 1.2s (100%) [2024-01-19T09:55:22.817Z] INFO[2024-01-19 09:55:18] sched stats: tick process durations: 66077 < 500ms (100%) [2024-01-19T09:55:22.817Z] 2024-01-19 09:55:20 +0000:[INFO] [backup.getAllBackupLocations:#1603] - The backup locations and their UID are map[654a1515-d800-4a43-bf14-8b83a0c5ec1e:autogenerated-backup-location-1705634221 b4816cdb-0621-48dc-a333-0ee37ec2c86c:autogenerated-backup-location-1705635982 fb5a14e2-2342-4596-b12d-cfadc2b7ea70:autogenerated-backup-location-wkyfv] [2024-01-19T09:55:22.817Z] INFO[2024-01-19 09:55:20] Verifying : Description : Verifying fetching of all backup locations [2024-01-19T09:55:22.817Z] INFO[2024-01-19 09:55:20] Actual:true, Expected: true [2024-01-19T09:55:22.817Z] 2024/01/19 09:55:20 DoRetryWithTimeout - Error: {found 3 backup locations}, Next try in [30s], timeout [5m0s] [2024-01-19T09:55:54.856Z] 2024-01-19 09:55:50 +0000:[INFO] [backup.getAllBackupLocations:#1603] - The backup locations and their UID are map[654a1515-d800-4a43-bf14-8b83a0c5ec1e:autogenerated-backup-location-1705634221 b4816cdb-0621-48dc-a333-0ee37ec2c86c:autogenerated-backup-location-1705635982 fb5a14e2-2342-4596-b12d-cfadc2b7ea70:autogenerated-backup-location-wkyfv] [2024-01-19T09:55:54.856Z] INFO[2024-01-19 09:55:50] Verifying : Description : Verifying fetching of all backup locations [2024-01-19T09:55:54.856Z] INFO[2024-01-19 09:55:50] Actual:true, Expected: true [2024-01-19T09:55:54.856Z] 2024/01/19 09:55:50 DoRetryWithTimeout - Error: {found 3 backup locations}, Next try in [30s], timeout [5m0s] [2024-01-19T09:56:21.358Z] 2024-01-19 09:56:20 +0000:[INFO] [backup.getAllBackupLocations:#1603] - The backup locations and their UID are map[654a1515-d800-4a43-bf14-8b83a0c5ec1e:autogenerated-backup-location-1705634221 b4816cdb-0621-48dc-a333-0ee37ec2c86c:autogenerated-backup-location-1705635982 fb5a14e2-2342-4596-b12d-cfadc2b7ea70:autogenerated-backup-location-wkyfv] [2024-01-19T09:56:21.358Z] INFO[2024-01-19 09:56:20] Verifying : Description : Verifying fetching of all backup locations [2024-01-19T09:56:21.358Z] INFO[2024-01-19 09:56:20] Actual:true, Expected: true [2024-01-19T09:56:21.359Z] 2024/01/19 09:56:20 DoRetryWithTimeout - Error: {found 3 backup locations}, Next try in [30s], timeout [5m0s] [2024-01-19T09:56:53.396Z] 2024-01-19 09:56:50 +0000:[INFO] [backup.getAllBackupLocations:#1603] - The backup locations and their UID are map[654a1515-d800-4a43-bf14-8b83a0c5ec1e:autogenerated-backup-location-1705634221 b4816cdb-0621-48dc-a333-0ee37ec2c86c:autogenerated-backup-location-1705635982 fb5a14e2-2342-4596-b12d-cfadc2b7ea70:autogenerated-backup-location-wkyfv] [2024-01-19T09:56:53.396Z] INFO[2024-01-19 09:56:50] Verifying : Description : Verifying fetching of all backup locations [2024-01-19T09:56:53.396Z] INFO[2024-01-19 09:56:50] Actual:true, Expected: true [2024-01-19T09:56:53.396Z] 2024/01/19 09:56:50 DoRetryWithTimeout - Error: {found 3 backup locations}, Next try in [30s], timeout [5m0s] [2024-01-19T09:57:25.432Z] 2024-01-19 09:57:20 +0000:[INFO] [backup.getAllBackupLocations:#1603] - The backup locations and their UID are map[654a1515-d800-4a43-bf14-8b83a0c5ec1e:autogenerated-backup-location-1705634221 b4816cdb-0621-48dc-a333-0ee37ec2c86c:autogenerated-backup-location-1705635982 fb5a14e2-2342-4596-b12d-cfadc2b7ea70:autogenerated-backup-location-wkyfv] [2024-01-19T09:57:25.432Z] INFO[2024-01-19 09:57:20] Verifying : Description : Verifying fetching of all backup locations [2024-01-19T09:57:25.432Z] INFO[2024-01-19 09:57:20] Actual:true, Expected: true [2024-01-19T09:57:25.432Z] 2024/01/19 09:57:20 DoRetryWithTimeout - Error: {found 3 backup locations}, Next try in [30s], timeout [5m0s] [2024-01-19T09:57:52.361Z] 2024-01-19 09:57:50 +0000:[INFO] [backup.getAllBackupLocations:#1603] - The backup locations and their UID are map[654a1515-d800-4a43-bf14-8b83a0c5ec1e:autogenerated-backup-location-1705634221 b4816cdb-0621-48dc-a333-0ee37ec2c86c:autogenerated-backup-location-1705635982 fb5a14e2-2342-4596-b12d-cfadc2b7ea70:autogenerated-backup-location-wkyfv] [2024-01-19T09:57:52.361Z] INFO[2024-01-19 09:57:50] Verifying : Description : Verifying fetching of all backup locations [2024-01-19T09:57:52.361Z] INFO[2024-01-19 09:57:50] Actual:true, Expected: true [2024-01-19T09:57:52.361Z] 2024/01/19 09:57:50 DoRetryWithTimeout - Error: {found 3 backup locations}, Next try in [30s], timeout [5m0s] [2024-01-19T09:58:24.396Z] INFO[2024-01-19 09:58:20] Verifying : Description : Verifying backup location deletion success [2024-01-19T09:58:24.396Z] ERRO[2024-01-19 09:58:20] Actual:timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {found 8 backup locations} [2024-01-19T09:58:24.396Z] {found 6 backup locations} [2024-01-19T09:58:24.396Z] {found 6 backup locations} [2024-01-19T09:58:24.396Z] {found 3 backup locations} [2024-01-19T09:58:24.396Z] {found 3 backup locations} [2024-01-19T09:58:24.396Z] {found 3 backup locations} [2024-01-19T09:58:24.396Z] {found 3 backup locations} [2024-01-19T09:58:24.396Z] {found 3 backup locations} [2024-01-19T09:58:24.396Z] {found 3 backup locations} [2024-01-19T09:58:24.396Z] {found 3 backup locations}, Expected:  [2024-01-19T09:58:24.397Z] 2024-01-19 09:58:20 +0000:[INFO] [backup.getAllCloudCredentials:#1624] - The cloud credentials and their UID are map[003d42b9-10c5-415b-860a-ef2401c4efe0:ibm-source-config-cloud-cred-xjkmp 007e50ce-bf9a-4337-bfc0-d6bbeddea8c8:ibm-destination-config-cloud-cred-wvqmp 03603a66-be2a-44dd-be0b-8e71410ff3e8:ibm-destination-config-cloud-cred-pkgor 05c1ce23-8059-4e66-b024-3fbbbc031f4b:ibm-destination-config-cloud-cred-ughzr 14e588f0-5305-4564-a50d-12eaadb0b1f0:ibm-destination-config-cloud-cred-zrovm 1d3dcaf0-e6c1-4039-90a2-ddb19cb8d3cd:ibm-source-config-cloud-cred-bimhn 1e9bffde-8075-4419-b2af-dcb2e204b565:ibm-destination-config-cloud-cred-gyeyl 2384d278-f42b-4310-ac99-23f40f56faeb:ibm-destination-config-cloud-cred-jfzok 242230f4-6757-4901-a866-c9945bee2eaf:autogenerated-cred-1705635982 2c5d8e0d-ff33-4b61-94f7-c8712a726319:ibm-destination-config-cloud-cred-emwoc 34166b66-22d2-4a9c-a419-19ed40fafa49:ibm-destination-config-cloud-cred-rpaqk 384f63f1-0fe9-46e0-bbd2-7fe781ca8ee3:autogenerated-cred-1705634221 3dd61eb8-d980-41bb-9b95-e0f5b990f9cf:ibm-source-config-cloud-cred-nvnow 3e89294f-cd10-4357-9e62-9807b2e7abca:cred-aws-1705608690 3f9f3525-a2f7-4e0b-8a0e-cc6d4936fd1f:ibm-destination-config-cloud-cred-dapwp 4526f4e4-5f42-4eb4-836b-4e562b4515b8:ibm-destination-config-cloud-cred-ggszx 48f219bc-3c87-4b44-bb04-a629bc2026fc:cred-aws-1705605598 4cadddda-0ca8-49f2-bde2-7f4384b9855b:ibm-destination-config-cloud-cred-ddvlx 4d960f42-b816-4842-b66f-2ebc91eb93ff:ibm-destination-config-cloud-cred-fitnv 4e0ddfc7-8811-4ed5-8e87-ed0841582111:ibm-source-config-cloud-cred-enzij 4e20d0f0-1904-466a-ad3a-1e19d7169cae:ibm-destination-config-cloud-cred-buvoh 4f3c25d0-c35c-4838-8e74-c0eb5f9c2cf6:ibm-source-config-cloud-cred-jiipm 52ceb6bb-9aa9-4972-9d7c-f143c76280cd:ibm-destination-config-cloud-cred-jeepg 5432c506-e79c-4494-871e-471635aa123a:ibm-destination-config-cloud-cred-sztik 57f7fd68-30b9-41c6-847d-b815e358a6a5:ibm-destination-config-cloud-cred-ansrj 59b6b644-c8ed-43cb-b62d-60c0f233bdd7:ibm-source-config-cloud-cred-vrzih 5c7121db-0835-4cca-8680-a3fcb710d1d6:ibm-destination-config-cloud-cred-ziypr 611175c8-09e6-4f7c-a7ba-953634c139b0:cred-aws-kudis 6d424701-6848-4cdb-b24c-e67fceec9dcd:ibm-source-config-cloud-cred-xnwmi 72b1b409-7bb1-4f22-92df-0ba5e77c11d3:ibm-source-config-cloud-cred-hrfiy 7b349e59-a61d-4c07-bccb-6ba223f821b5:ibm-source-config-cloud-cred-vmlxp 7cddb5aa-c7ff-4044-9d27-458be2bd5cde:ibm-source-config-cloud-cred-fbvvo 7e6ad931-e339-4f31-9674-a563626aad91:ibm-source-config-cloud-cred-rcmdm 8039f83c-2bad-4565-a4a3-932d11ca3c35:ibm-source-config-cloud-cred-ckjxv 832e19de-64ac-4aa2-83bc-96f99917be8a:autogenerated-cred-meoyc 84f974bc-b70c-4f60-8f28-a9b620cd9cdc:autogenerated-cred-1705634224 8545f392-9619-4801-9b9d-9913b47ba56e:invalid-autogenerated-cred-testuser1-1705618999-glndt 87a15256-4a8a-4244-8df9-2f79276576b6:cred-aws-1705577909 87aedb2a-2f2b-40ad-9e10-b9fc68f0fbc6:ibm-source-config-cloud-cred-cufev 8c9a2571-b3b0-48bf-b1cf-eeef4421a95e:ibm-source-config-cloud-cred-jpmbk 8f15f902-c815-42a1-a5ea-1ae8a24d6fea:ibm-destination-config-cloud-cred-myzyc 9312af98-b05c-4c55-8f00-164bdb7b60f9:ibm-destination-config-cloud-cred-iuupt 9841bce2-74d5-4fc2-831e-caf981277058:ibm-destination-config-cloud-cred-zzzrt 9a8bf73f-fe61-4c03-9503-7027e331fbae:ibm-source-config-cloud-cred-eockv 9f8af29c-79bb-495e-87af-b11583b908aa:autogenerated-cred-testuser1-1705643043-dvtop a12b5b6d-216c-433c-a2a3-62d797b4b228:ibm-source-config-cloud-cred-jyrnu a271ea5b-dc9d-40f1-a82a-651539c7e486:ibm-destination-config-cloud-cred-zbbvt a69408ff-7799-4032-bff5-cf0f9daad9b7:ibm-source-config-cloud-cred-sejkd ab4a9ad9-5834-4329-a070-07e162922643:ibm-destination-config-cloud-cred-oxkoo b36f64b8-40fe-4f3e-b68c-9eddccab1dba:ibm-destination-config-cloud-cred-gcbql b5dc5235-d58e-4094-96e1-db8248a9f4e2:autogenerated-cred-testuser1-1705570507-vjsmz bba39e51-7244-4145-89c9-c91dfe3df075:ibm-destination-config-cloud-cred-pnouo c51bf4bb-493a-4784-af65-fdbc0167456c:ibm-source-config-cloud-cred-bfxwa c5ff084a-a913-4412-99fb-e2b0a003a014:ibm-source-config-cloud-cred-wcpfm c6ba4a7b-d732-4dcd-b84f-8aaba0d31812:ibm-source-config-cloud-cred-xnntd cd1e34e3-2679-43d3-a3f8-0e08a6291df6:ibm-destination-config-cloud-cred-qprlh d2557807-ac3a-4d0f-8747-9603d343da4a:ibm-destination-config-cloud-cred-ffjdm d3b1b808-0bab-4c91-a304-f60c8c64f1e0:ibm-destination-config-cloud-cred-eqjzn d64d258b-5f8c-4690-ad2a-05909c63cfa1:ibm-source-config-cloud-cred-fkfzj e2e6c523-fa9f-444e-848a-22c5300d3806:cred-aws-1705592040 e334b23e-f120-4865-bb3b-0a25a9e951a3:ibm-destination-config-cloud-cred-jeilk e677fd08-4059-4889-b566-3ad1933fd8a6:ibm-source-config-cloud-cred-giori e754dfb8-04d0-4031-8571-aa8c1812669d:ibm-source-config-cloud-cred-jdydf e86ca3c8-a8f5-4aa5-96b8-8105f2db2a72:autogenerated-cred-testuser1-1705641008-jsxlu ead03c67-6f14-4ebb-a588-c621f8cb3f2b:ibm-source-config-cloud-cred-gzmjh ec6d2d18-1421-41d4-bad3-ceb41fd17fe0:ibm-destination-config-cloud-cred-jinlc edb6d233-6918-4eb4-b9dc-f051a90533a3:ibm-destination-config-cloud-cred-lrlgh f0465bb5-2565-4948-bb27-7e45a4b883ff:ibm-source-config-cloud-cred-vlbwu f14fb5fc-2a0f-4c58-8481-600f05d0a19f:ibm-destination-config-cloud-cred-bgfur f551fb81-1a78-4d7f-b804-759c6e6b562e:ibm-source-config-cloud-cred-mhxzi f7f10871-8f0b-4f94-8f70-032b91b20ce3:autogenerated-cred-oacqz f823e120-b319-41dc-8876-ec5ae0147d64:ibm-destination-config-cloud-cred-gzpiz fe53b7a2-3045-4494-90af-c5a010a82974:ibm-destination-config-cloud-cred-flqaz] [2024-01-19T09:58:24.397Z] INFO[2024-01-19 09:58:20] Verifying : Description : Verifying fetching of all cloud credentials [2024-01-19T09:58:24.397Z] INFO[2024-01-19 09:58:20] Actual:true, Expected: true [2024-01-19T09:58:24.397Z] INFO[2024-01-19 09:58:20] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-ffjdm [2024-01-19T09:58:24.397Z] INFO[2024-01-19 09:58:20] Actual:true, Expected: true [2024-01-19T09:58:24.398Z] INFO[2024-01-19 09:58:21] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-ziypr [2024-01-19T09:58:24.398Z] INFO[2024-01-19 09:58:21] Actual:true, Expected: true [2024-01-19T09:58:24.398Z] INFO[2024-01-19 09:58:22] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-jeilk [2024-01-19T09:58:24.398Z] INFO[2024-01-19 09:58:22] Actual:true, Expected: true [2024-01-19T09:58:24.398Z] INFO[2024-01-19 09:58:24] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-jiipm [2024-01-19T09:58:24.398Z] INFO[2024-01-19 09:58:24] Actual:true, Expected: true [2024-01-19T09:58:25.817Z] INFO[2024-01-19 09:58:25] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-vrzih [2024-01-19T09:58:25.817Z] INFO[2024-01-19 09:58:25] Actual:true, Expected: true [2024-01-19T09:58:27.181Z] INFO[2024-01-19 09:58:27] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-eockv [2024-01-19T09:58:27.181Z] INFO[2024-01-19 09:58:27] Actual:true, Expected: true [2024-01-19T09:58:28.550Z] INFO[2024-01-19 09:58:28] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-xnwmi [2024-01-19T09:58:28.550Z] INFO[2024-01-19 09:58:28] Actual:true, Expected: true [2024-01-19T09:58:29.955Z] INFO[2024-01-19 09:58:29] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-ddvlx [2024-01-19T09:58:29.955Z] INFO[2024-01-19 09:58:29] Actual:true, Expected: true [2024-01-19T09:58:31.320Z] INFO[2024-01-19 09:58:31] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-gcbql [2024-01-19T09:58:31.320Z] INFO[2024-01-19 09:58:31] Actual:true, Expected: true [2024-01-19T09:58:33.207Z] INFO[2024-01-19 09:58:32] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-iuupt [2024-01-19T09:58:33.208Z] INFO[2024-01-19 09:58:32] Actual:true, Expected: true [2024-01-19T09:58:34.155Z] INFO[2024-01-19 09:58:34] Verifying : Description : Deleting cloud cred cred-aws-kudis [2024-01-19T09:58:34.155Z] INFO[2024-01-19 09:58:34] Actual:true, Expected: true [2024-01-19T09:58:35.536Z] INFO[2024-01-19 09:58:35] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-fkfzj [2024-01-19T09:58:35.536Z] INFO[2024-01-19 09:58:35] Actual:true, Expected: true [2024-01-19T09:58:36.902Z] INFO[2024-01-19 09:58:36] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-xnntd [2024-01-19T09:58:36.903Z] INFO[2024-01-19 09:58:36] Actual:true, Expected: true [2024-01-19T09:58:38.327Z] INFO[2024-01-19 09:58:38] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-vlbwu [2024-01-19T09:58:38.327Z] ERRO[2024-01-19 09:58:38] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-vlbwu:f0465bb5-2565-4948-bb27-7e45a4b883ff], Expected:  [2024-01-19T09:58:39.695Z] INFO[2024-01-19 09:58:39] Verifying : Description : Deleting cloud cred autogenerated-cred-testuser1-1705641008-jsxlu [2024-01-19T09:58:39.695Z] INFO[2024-01-19 09:58:39] Actual:true, Expected: true [2024-01-19T09:58:41.584Z] INFO[2024-01-19 09:58:41] Verifying : Description : Deleting cloud cred cred-aws-1705577909 [2024-01-19T09:58:41.584Z] INFO[2024-01-19 09:58:41] Actual:true, Expected: true [2024-01-19T09:58:42.513Z] INFO[2024-01-19 09:58:42] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-jinlc [2024-01-19T09:58:42.513Z] INFO[2024-01-19 09:58:42] Actual:true, Expected: true [2024-01-19T09:58:43.877Z] INFO[2024-01-19 09:58:43] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-jyrnu [2024-01-19T09:58:43.877Z] INFO[2024-01-19 09:58:43] Actual:true, Expected: true [2024-01-19T09:58:45.766Z] INFO[2024-01-19 09:58:45] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-bfxwa [2024-01-19T09:58:45.766Z] INFO[2024-01-19 09:58:45] Actual:true, Expected: true [2024-01-19T09:58:46.693Z] INFO[2024-01-19 09:58:46] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-sztik [2024-01-19T09:58:46.693Z] INFO[2024-01-19 09:58:46] Actual:true, Expected: true [2024-01-19T09:58:48.580Z] INFO[2024-01-19 09:58:48] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-flqaz [2024-01-19T09:58:48.581Z] INFO[2024-01-19 09:58:48] Actual:true, Expected: true [2024-01-19T09:58:49.519Z] INFO[2024-01-19 09:58:49] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-zbbvt [2024-01-19T09:58:49.519Z] INFO[2024-01-19 09:58:49] Actual:true, Expected: true [2024-01-19T09:58:50.885Z] INFO[2024-01-19 09:58:50] Verifying : Description : Deleting cloud cred cred-aws-1705592040 [2024-01-19T09:58:50.885Z] INFO[2024-01-19 09:58:50] Actual:true, Expected: true [2024-01-19T09:58:52.777Z] INFO[2024-01-19 09:58:52] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-jpmbk [2024-01-19T09:58:52.777Z] INFO[2024-01-19 09:58:52] Actual:true, Expected: true [2024-01-19T09:58:53.703Z] INFO[2024-01-19 09:58:53] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-zzzrt [2024-01-19T09:58:53.703Z] INFO[2024-01-19 09:58:53] Actual:true, Expected: true [2024-01-19T09:58:55.590Z] INFO[2024-01-19 09:58:55] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-cufev [2024-01-19T09:58:55.590Z] INFO[2024-01-19 09:58:55] Actual:true, Expected: true [2024-01-19T09:58:56.518Z] INFO[2024-01-19 09:58:56] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-gzmjh [2024-01-19T09:58:56.518Z] INFO[2024-01-19 09:58:56] Actual:true, Expected: true [2024-01-19T09:58:57.883Z] INFO[2024-01-19 09:58:57] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-nvnow [2024-01-19T09:58:57.883Z] INFO[2024-01-19 09:58:57] Actual:true, Expected: true [2024-01-19T09:58:59.781Z] INFO[2024-01-19 09:58:59] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-ughzr [2024-01-19T09:58:59.781Z] INFO[2024-01-19 09:58:59] Actual:true, Expected: true [2024-01-19T09:59:00.708Z] INFO[2024-01-19 09:59:00] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-buvoh [2024-01-19T09:59:00.708Z] INFO[2024-01-19 09:59:00] Actual:true, Expected: true [2024-01-19T09:59:02.113Z] INFO[2024-01-19 09:59:02] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-ggszx [2024-01-19T09:59:02.113Z] INFO[2024-01-19 09:59:02] Actual:true, Expected: true [2024-01-19T09:59:03.479Z] INFO[2024-01-19 09:59:03] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-lrlgh [2024-01-19T09:59:03.479Z] INFO[2024-01-19 09:59:03] Actual:true, Expected: true [2024-01-19T09:59:05.368Z] INFO[2024-01-19 09:59:04] Verifying : Description : Deleting cloud cred autogenerated-cred-testuser1-1705643043-dvtop [2024-01-19T09:59:05.368Z] INFO[2024-01-19 09:59:04] Actual:true, Expected: true [2024-01-19T09:59:06.733Z] INFO[2024-01-19 09:59:06] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-giori [2024-01-19T09:59:06.733Z] INFO[2024-01-19 09:59:06] Actual:true, Expected: true [2024-01-19T09:59:07.659Z] INFO[2024-01-19 09:59:07] Verifying : Description : Deleting cloud cred autogenerated-cred-1705634221 [2024-01-19T09:59:07.659Z] ERRO[2024-01-19 09:59:07] Actual:rpc error: code = Unknown desc = backup location [autogenerated-backup-location-1705634221] has reference to cloud credential [autogenerated-cred-1705634221:384f63f1-0fe9-46e0-bbd2-7fe781ca8ee3], Expected:  [2024-01-19T09:59:09.553Z] INFO[2024-01-19 09:59:09] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-eqjzn [2024-01-19T09:59:09.553Z] INFO[2024-01-19 09:59:09] Actual:true, Expected: true [2024-01-19T09:59:10.481Z] INFO[2024-01-19 09:59:10] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-qprlh [2024-01-19T09:59:10.481Z] INFO[2024-01-19 09:59:10] Actual:true, Expected: true [2024-01-19T09:59:11.877Z] INFO[2024-01-19 09:59:11] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-zrovm [2024-01-19T09:59:11.877Z] INFO[2024-01-19 09:59:11] Actual:true, Expected: true [2024-01-19T09:59:13.766Z] INFO[2024-01-19 09:59:13] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-gyeyl [2024-01-19T09:59:13.766Z] INFO[2024-01-19 09:59:13] Actual:true, Expected: true [2024-01-19T09:59:14.695Z] INFO[2024-01-19 09:59:14] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-emwoc [2024-01-19T09:59:14.695Z] INFO[2024-01-19 09:59:14] Actual:true, Expected: true [2024-01-19T09:59:16.586Z] INFO[2024-01-19 09:59:16] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-pkgor [2024-01-19T09:59:16.586Z] INFO[2024-01-19 09:59:16] Actual:true, Expected: true [2024-01-19T09:59:17.566Z] INFO[2024-01-19 09:59:17] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-dapwp [2024-01-19T09:59:17.566Z] INFO[2024-01-19 09:59:17] Actual:true, Expected: true [2024-01-19T09:59:18.936Z] INFO[2024-01-19 09:59:18] Verifying : Description : Deleting cloud cred cred-aws-1705605598 [2024-01-19T09:59:18.936Z] INFO[2024-01-19 09:59:18] Actual:true, Expected: true [2024-01-19T09:59:20.299Z] INFO[2024-01-19 09:59:20] Verifying : Description : Deleting cloud cred cred-aws-1705608690 [2024-01-19T09:59:20.299Z] INFO[2024-01-19 09:59:20] Actual:true, Expected: true [2024-01-19T09:59:21.669Z] INFO[2024-01-19 09:59:21] Verifying : Description : Deleting cloud cred invalid-autogenerated-cred-testuser1-1705618999-glndt [2024-01-19T09:59:21.669Z] INFO[2024-01-19 09:59:21] Actual:true, Expected: true [2024-01-19T09:59:23.558Z] INFO[2024-01-19 09:59:23] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-mhxzi [2024-01-19T09:59:23.558Z] ERRO[2024-01-19 09:59:23] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-mhxzi:f551fb81-1a78-4d7f-b804-759c6e6b562e], Expected:  [2024-01-19T09:59:24.485Z] INFO[2024-01-19 09:59:24] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-rcmdm [2024-01-19T09:59:24.485Z] INFO[2024-01-19 09:59:24] Actual:true, Expected: true [2024-01-19T09:59:26.373Z] INFO[2024-01-19 09:59:25] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-xjkmp [2024-01-19T09:59:26.373Z] INFO[2024-01-19 09:59:25] Actual:true, Expected: true [2024-01-19T09:59:27.299Z] INFO[2024-01-19 09:59:27] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-jeepg [2024-01-19T09:59:27.299Z] INFO[2024-01-19 09:59:27] Actual:true, Expected: true [2024-01-19T09:59:29.191Z] INFO[2024-01-19 09:59:28] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-jdydf [2024-01-19T09:59:29.191Z] INFO[2024-01-19 09:59:28] Actual:true, Expected: true [2024-01-19T09:59:30.121Z] INFO[2024-01-19 09:59:30] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-gzpiz [2024-01-19T09:59:30.121Z] INFO[2024-01-19 09:59:30] Actual:true, Expected: true [2024-01-19T09:59:31.497Z] INFO[2024-01-19 09:59:31] Verifying : Description : Deleting cloud cred autogenerated-cred-1705635982 [2024-01-19T09:59:31.497Z] ERRO[2024-01-19 09:59:31] Actual:rpc error: code = Unknown desc = backup location [autogenerated-backup-location-1705635982] has reference to cloud credential [autogenerated-cred-1705635982:242230f4-6757-4901-a866-c9945bee2eaf], Expected:  [2024-01-19T09:59:32.881Z] INFO[2024-01-19 09:59:32] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-fitnv [2024-01-19T09:59:32.881Z] INFO[2024-01-19 09:59:32] Actual:true, Expected: true [2024-01-19T09:59:34.776Z] INFO[2024-01-19 09:59:34] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-jfzok [2024-01-19T09:59:34.776Z] INFO[2024-01-19 09:59:34] Actual:true, Expected: true [2024-01-19T09:59:35.706Z] INFO[2024-01-19 09:59:35] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-myzyc [2024-01-19T09:59:35.706Z] INFO[2024-01-19 09:59:35] Actual:true, Expected: true [2024-01-19T09:59:37.595Z] INFO[2024-01-19 09:59:37] Verifying : Description : Deleting cloud cred autogenerated-cred-testuser1-1705570507-vjsmz [2024-01-19T09:59:37.595Z] INFO[2024-01-19 09:59:37] Actual:true, Expected: true [2024-01-19T09:59:38.522Z] INFO[2024-01-19 09:59:38] Verifying : Description : Deleting cloud cred autogenerated-cred-1705634224 [2024-01-19T09:59:38.522Z] INFO[2024-01-19 09:59:38] Actual:true, Expected: true [2024-01-19T09:59:39.886Z] INFO[2024-01-19 09:59:39] Verifying : Description : Deleting cloud cred autogenerated-cred-meoyc [2024-01-19T09:59:39.887Z] ERRO[2024-01-19 09:59:39] Actual:rpc error: code = Unknown desc = backup location [autogenerated-backup-location-wkyfv] has reference to cloud credential [autogenerated-cred-meoyc:832e19de-64ac-4aa2-83bc-96f99917be8a], Expected:  [2024-01-19T09:59:41.775Z] INFO[2024-01-19 09:59:41] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-pnouo [2024-01-19T09:59:41.775Z] INFO[2024-01-19 09:59:41] Actual:true, Expected: true [2024-01-19T09:59:42.748Z] INFO[2024-01-19 09:59:42] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-hrfiy [2024-01-19T09:59:42.748Z] ERRO[2024-01-19 09:59:42] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-hrfiy:72b1b409-7bb1-4f22-92df-0ba5e77c11d3], Expected:  [2024-01-19T09:59:44.114Z] INFO[2024-01-19 09:59:44] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-bimhn [2024-01-19T09:59:44.114Z] INFO[2024-01-19 09:59:44] Actual:true, Expected: true [2024-01-19T09:59:46.004Z] INFO[2024-01-19 09:59:45] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-vmlxp [2024-01-19T09:59:46.004Z] INFO[2024-01-19 09:59:45] Actual:true, Expected: true [2024-01-19T09:59:46.933Z] INFO[2024-01-19 09:59:46] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-oxkoo [2024-01-19T09:59:46.933Z] INFO[2024-01-19 09:59:46] Actual:true, Expected: true [2024-01-19T09:59:48.303Z] INFO[2024-01-19 09:59:48] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-ckjxv [2024-01-19T09:59:48.303Z] INFO[2024-01-19 09:59:48] Actual:true, Expected: true [2024-01-19T09:59:50.194Z] INFO[2024-01-19 09:59:49] Verifying : Description : Deleting cloud cred autogenerated-cred-oacqz [2024-01-19T09:59:50.194Z] INFO[2024-01-19 09:59:49] Actual:true, Expected: true [2024-01-19T09:59:51.122Z] INFO[2024-01-19 09:59:51] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-sejkd [2024-01-19T09:59:51.122Z] INFO[2024-01-19 09:59:51] Actual:true, Expected: true [2024-01-19T09:59:52.491Z] INFO[2024-01-19 09:59:52] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-ansrj [2024-01-19T09:59:52.491Z] INFO[2024-01-19 09:59:52] Actual:true, Expected: true [2024-01-19T09:59:53.863Z] INFO[2024-01-19 09:59:53] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-wvqmp [2024-01-19T09:59:53.863Z] INFO[2024-01-19 09:59:53] Actual:true, Expected: true [2024-01-19T09:59:55.778Z] INFO[2024-01-19 09:59:55] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-bgfur [2024-01-19T09:59:55.778Z] INFO[2024-01-19 09:59:55] Actual:true, Expected: true [2024-01-19T09:59:56.706Z] INFO[2024-01-19 09:59:56] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-wcpfm [2024-01-19T09:59:56.706Z] ERRO[2024-01-19 09:59:56] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-wcpfm:c5ff084a-a913-4412-99fb-e2b0a003a014], Expected:  [2024-01-19T09:59:58.076Z] INFO[2024-01-19 09:59:58] Verifying : Description : Deleting cloud cred ibm-destination-config-cloud-cred-rpaqk [2024-01-19T09:59:58.076Z] INFO[2024-01-19 09:59:58] Actual:true, Expected: true [2024-01-19T09:59:59.447Z] INFO[2024-01-19 09:59:59] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-fbvvo [2024-01-19T09:59:59.447Z] ERRO[2024-01-19 09:59:59] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-fbvvo:7cddb5aa-c7ff-4044-9d27-458be2bd5cde], Expected:  [2024-01-19T10:00:01.339Z] INFO[2024-01-19 10:00:00] Verifying : Description : Deleting cloud cred ibm-source-config-cloud-cred-enzij [2024-01-19T10:00:01.339Z] ERRO[2024-01-19 10:00:00] Actual:rpc error: code = Unknown desc = cluster [source-cluster] has reference to cloud credential [ibm-source-config-cloud-cred-enzij:4e0ddfc7-8811-4ed5-8e87-ed0841582111], Expected:  [2024-01-19T10:00:01.339Z] 2024-01-19 10:00:00 +0000:[INFO] [backup.getAllCloudCredentials:#1624] - The cloud credentials and their UID are map[242230f4-6757-4901-a866-c9945bee2eaf:autogenerated-cred-1705635982 384f63f1-0fe9-46e0-bbd2-7fe781ca8ee3:autogenerated-cred-1705634221 4e0ddfc7-8811-4ed5-8e87-ed0841582111:ibm-source-config-cloud-cred-enzij 72b1b409-7bb1-4f22-92df-0ba5e77c11d3:ibm-source-config-cloud-cred-hrfiy 7cddb5aa-c7ff-4044-9d27-458be2bd5cde:ibm-source-config-cloud-cred-fbvvo 832e19de-64ac-4aa2-83bc-96f99917be8a:autogenerated-cred-meoyc c5ff084a-a913-4412-99fb-e2b0a003a014:ibm-source-config-cloud-cred-wcpfm f0465bb5-2565-4948-bb27-7e45a4b883ff:ibm-source-config-cloud-cred-vlbwu f551fb81-1a78-4d7f-b804-759c6e6b562e:ibm-source-config-cloud-cred-mhxzi] [2024-01-19T10:00:01.339Z] INFO[2024-01-19 10:00:00] Verifying : Description : Verifying fetching of all cloud credentials [2024-01-19T10:00:01.339Z] INFO[2024-01-19 10:00:00] Actual:true, Expected: true [2024-01-19T10:00:01.339Z] 2024/01/19 10:00:00 DoRetryWithTimeout - Error: {found 8 cloud credentials}, Next try in [30s], timeout [5m0s] [2024-01-19T10:00:33.397Z] 2024-01-19 10:00:30 +0000:[INFO] [backup.getAllCloudCredentials:#1624] - The cloud credentials and their UID are map[242230f4-6757-4901-a866-c9945bee2eaf:autogenerated-cred-1705635982 384f63f1-0fe9-46e0-bbd2-7fe781ca8ee3:autogenerated-cred-1705634221 4e0ddfc7-8811-4ed5-8e87-ed0841582111:ibm-source-config-cloud-cred-enzij 72b1b409-7bb1-4f22-92df-0ba5e77c11d3:ibm-source-config-cloud-cred-hrfiy 7cddb5aa-c7ff-4044-9d27-458be2bd5cde:ibm-source-config-cloud-cred-fbvvo 832e19de-64ac-4aa2-83bc-96f99917be8a:autogenerated-cred-meoyc c5ff084a-a913-4412-99fb-e2b0a003a014:ibm-source-config-cloud-cred-wcpfm f0465bb5-2565-4948-bb27-7e45a4b883ff:ibm-source-config-cloud-cred-vlbwu f551fb81-1a78-4d7f-b804-759c6e6b562e:ibm-source-config-cloud-cred-mhxzi] [2024-01-19T10:00:33.397Z] INFO[2024-01-19 10:00:30] Verifying : Description : Verifying fetching of all cloud credentials [2024-01-19T10:00:33.397Z] INFO[2024-01-19 10:00:30] Actual:true, Expected: true [2024-01-19T10:00:33.397Z] 2024/01/19 10:00:30 DoRetryWithTimeout - Error: {found 8 cloud credentials}, Next try in [30s], timeout [5m0s] [2024-01-19T10:01:05.438Z] 2024-01-19 10:01:00 +0000:[INFO] [backup.getAllCloudCredentials:#1624] - The cloud credentials and their UID are map[242230f4-6757-4901-a866-c9945bee2eaf:autogenerated-cred-1705635982 384f63f1-0fe9-46e0-bbd2-7fe781ca8ee3:autogenerated-cred-1705634221 4e0ddfc7-8811-4ed5-8e87-ed0841582111:ibm-source-config-cloud-cred-enzij 72b1b409-7bb1-4f22-92df-0ba5e77c11d3:ibm-source-config-cloud-cred-hrfiy 7cddb5aa-c7ff-4044-9d27-458be2bd5cde:ibm-source-config-cloud-cred-fbvvo 832e19de-64ac-4aa2-83bc-96f99917be8a:autogenerated-cred-meoyc c5ff084a-a913-4412-99fb-e2b0a003a014:ibm-source-config-cloud-cred-wcpfm f0465bb5-2565-4948-bb27-7e45a4b883ff:ibm-source-config-cloud-cred-vlbwu f551fb81-1a78-4d7f-b804-759c6e6b562e:ibm-source-config-cloud-cred-mhxzi] [2024-01-19T10:01:05.438Z] INFO[2024-01-19 10:01:00] Verifying : Description : Verifying fetching of all cloud credentials [2024-01-19T10:01:05.438Z] INFO[2024-01-19 10:01:00] Actual:true, Expected: true [2024-01-19T10:01:05.438Z] 2024/01/19 10:01:00 DoRetryWithTimeout - Error: {found 8 cloud credentials}, Next try in [30s], timeout [5m0s] [2024-01-19T10:01:31.953Z] 2024-01-19 10:01:31 +0000:[INFO] [backup.getAllCloudCredentials:#1624] - The cloud credentials and their UID are map[242230f4-6757-4901-a866-c9945bee2eaf:autogenerated-cred-1705635982 384f63f1-0fe9-46e0-bbd2-7fe781ca8ee3:autogenerated-cred-1705634221 4e0ddfc7-8811-4ed5-8e87-ed0841582111:ibm-source-config-cloud-cred-enzij 72b1b409-7bb1-4f22-92df-0ba5e77c11d3:ibm-source-config-cloud-cred-hrfiy 7cddb5aa-c7ff-4044-9d27-458be2bd5cde:ibm-source-config-cloud-cred-fbvvo 832e19de-64ac-4aa2-83bc-96f99917be8a:autogenerated-cred-meoyc c5ff084a-a913-4412-99fb-e2b0a003a014:ibm-source-config-cloud-cred-wcpfm f0465bb5-2565-4948-bb27-7e45a4b883ff:ibm-source-config-cloud-cred-vlbwu f551fb81-1a78-4d7f-b804-759c6e6b562e:ibm-source-config-cloud-cred-mhxzi] [2024-01-19T10:01:31.953Z] INFO[2024-01-19 10:01:31] Verifying : Description : Verifying fetching of all cloud credentials [2024-01-19T10:01:31.953Z] INFO[2024-01-19 10:01:31] Actual:true, Expected: true [2024-01-19T10:01:31.953Z] 2024/01/19 10:01:31 DoRetryWithTimeout - Error: {found 8 cloud credentials}, Next try in [30s], timeout [5m0s] [2024-01-19T10:02:03.978Z] 2024-01-19 10:02:01 +0000:[INFO] [backup.getAllCloudCredentials:#1624] - The cloud credentials and their UID are map[242230f4-6757-4901-a866-c9945bee2eaf:autogenerated-cred-1705635982 384f63f1-0fe9-46e0-bbd2-7fe781ca8ee3:autogenerated-cred-1705634221 4e0ddfc7-8811-4ed5-8e87-ed0841582111:ibm-source-config-cloud-cred-enzij 72b1b409-7bb1-4f22-92df-0ba5e77c11d3:ibm-source-config-cloud-cred-hrfiy 7cddb5aa-c7ff-4044-9d27-458be2bd5cde:ibm-source-config-cloud-cred-fbvvo 832e19de-64ac-4aa2-83bc-96f99917be8a:autogenerated-cred-meoyc c5ff084a-a913-4412-99fb-e2b0a003a014:ibm-source-config-cloud-cred-wcpfm f0465bb5-2565-4948-bb27-7e45a4b883ff:ibm-source-config-cloud-cred-vlbwu f551fb81-1a78-4d7f-b804-759c6e6b562e:ibm-source-config-cloud-cred-mhxzi] [2024-01-19T10:02:03.978Z] INFO[2024-01-19 10:02:01] Verifying : Description : Verifying fetching of all cloud credentials [2024-01-19T10:02:03.978Z] INFO[2024-01-19 10:02:01] Actual:true, Expected: true [2024-01-19T10:02:03.978Z] 2024/01/19 10:02:01 DoRetryWithTimeout - Error: {found 8 cloud credentials}, Next try in [30s], timeout [5m0s] [2024-01-19T10:02:36.021Z] 2024-01-19 10:02:31 +0000:[INFO] [backup.getAllCloudCredentials:#1624] - The cloud credentials and their UID are map[242230f4-6757-4901-a866-c9945bee2eaf:autogenerated-cred-1705635982 384f63f1-0fe9-46e0-bbd2-7fe781ca8ee3:autogenerated-cred-1705634221 4e0ddfc7-8811-4ed5-8e87-ed0841582111:ibm-source-config-cloud-cred-enzij 72b1b409-7bb1-4f22-92df-0ba5e77c11d3:ibm-source-config-cloud-cred-hrfiy 7cddb5aa-c7ff-4044-9d27-458be2bd5cde:ibm-source-config-cloud-cred-fbvvo 832e19de-64ac-4aa2-83bc-96f99917be8a:autogenerated-cred-meoyc c5ff084a-a913-4412-99fb-e2b0a003a014:ibm-source-config-cloud-cred-wcpfm f0465bb5-2565-4948-bb27-7e45a4b883ff:ibm-source-config-cloud-cred-vlbwu f551fb81-1a78-4d7f-b804-759c6e6b562e:ibm-source-config-cloud-cred-mhxzi] [2024-01-19T10:02:36.021Z] INFO[2024-01-19 10:02:31] Verifying : Description : Verifying fetching of all cloud credentials [2024-01-19T10:02:36.021Z] INFO[2024-01-19 10:02:31] Actual:true, Expected: true [2024-01-19T10:02:36.021Z] 2024/01/19 10:02:31 DoRetryWithTimeout - Error: {found 8 cloud credentials}, Next try in [30s], timeout [5m0s] [2024-01-19T10:03:03.698Z] 2024-01-19 10:03:01 +0000:[INFO] [backup.getAllCloudCredentials:#1624] - The cloud credentials and their UID are map[242230f4-6757-4901-a866-c9945bee2eaf:autogenerated-cred-1705635982 384f63f1-0fe9-46e0-bbd2-7fe781ca8ee3:autogenerated-cred-1705634221 4e0ddfc7-8811-4ed5-8e87-ed0841582111:ibm-source-config-cloud-cred-enzij 72b1b409-7bb1-4f22-92df-0ba5e77c11d3:ibm-source-config-cloud-cred-hrfiy 7cddb5aa-c7ff-4044-9d27-458be2bd5cde:ibm-source-config-cloud-cred-fbvvo 832e19de-64ac-4aa2-83bc-96f99917be8a:autogenerated-cred-meoyc c5ff084a-a913-4412-99fb-e2b0a003a014:ibm-source-config-cloud-cred-wcpfm f0465bb5-2565-4948-bb27-7e45a4b883ff:ibm-source-config-cloud-cred-vlbwu f551fb81-1a78-4d7f-b804-759c6e6b562e:ibm-source-config-cloud-cred-mhxzi] [2024-01-19T10:03:03.698Z] INFO[2024-01-19 10:03:01] Verifying : Description : Verifying fetching of all cloud credentials [2024-01-19T10:03:03.698Z] INFO[2024-01-19 10:03:01] Actual:true, Expected: true [2024-01-19T10:03:03.698Z] 2024/01/19 10:03:01 DoRetryWithTimeout - Error: {found 8 cloud credentials}, Next try in [30s], timeout [5m0s] [2024-01-19T10:03:35.738Z] 2024-01-19 10:03:31 +0000:[INFO] [backup.getAllCloudCredentials:#1624] - The cloud credentials and their UID are map[242230f4-6757-4901-a866-c9945bee2eaf:autogenerated-cred-1705635982 384f63f1-0fe9-46e0-bbd2-7fe781ca8ee3:autogenerated-cred-1705634221 4e0ddfc7-8811-4ed5-8e87-ed0841582111:ibm-source-config-cloud-cred-enzij 72b1b409-7bb1-4f22-92df-0ba5e77c11d3:ibm-source-config-cloud-cred-hrfiy 7cddb5aa-c7ff-4044-9d27-458be2bd5cde:ibm-source-config-cloud-cred-fbvvo 832e19de-64ac-4aa2-83bc-96f99917be8a:autogenerated-cred-meoyc c5ff084a-a913-4412-99fb-e2b0a003a014:ibm-source-config-cloud-cred-wcpfm f0465bb5-2565-4948-bb27-7e45a4b883ff:ibm-source-config-cloud-cred-vlbwu f551fb81-1a78-4d7f-b804-759c6e6b562e:ibm-source-config-cloud-cred-mhxzi] [2024-01-19T10:03:35.739Z] INFO[2024-01-19 10:03:31] Verifying : Description : Verifying fetching of all cloud credentials [2024-01-19T10:03:35.739Z] INFO[2024-01-19 10:03:31] Actual:true, Expected: true [2024-01-19T10:03:35.739Z] 2024/01/19 10:03:31 DoRetryWithTimeout - Error: {found 8 cloud credentials}, Next try in [30s], timeout [5m0s] [2024-01-19T10:04:02.244Z] 2024-01-19 10:04:01 +0000:[INFO] [backup.getAllCloudCredentials:#1624] - The cloud credentials and their UID are map[242230f4-6757-4901-a866-c9945bee2eaf:autogenerated-cred-1705635982 384f63f1-0fe9-46e0-bbd2-7fe781ca8ee3:autogenerated-cred-1705634221 4e0ddfc7-8811-4ed5-8e87-ed0841582111:ibm-source-config-cloud-cred-enzij 72b1b409-7bb1-4f22-92df-0ba5e77c11d3:ibm-source-config-cloud-cred-hrfiy 7cddb5aa-c7ff-4044-9d27-458be2bd5cde:ibm-source-config-cloud-cred-fbvvo 832e19de-64ac-4aa2-83bc-96f99917be8a:autogenerated-cred-meoyc c5ff084a-a913-4412-99fb-e2b0a003a014:ibm-source-config-cloud-cred-wcpfm f0465bb5-2565-4948-bb27-7e45a4b883ff:ibm-source-config-cloud-cred-vlbwu f551fb81-1a78-4d7f-b804-759c6e6b562e:ibm-source-config-cloud-cred-mhxzi] [2024-01-19T10:04:02.244Z] INFO[2024-01-19 10:04:01] Verifying : Description : Verifying fetching of all cloud credentials [2024-01-19T10:04:02.244Z] INFO[2024-01-19 10:04:01] Actual:true, Expected: true [2024-01-19T10:04:02.244Z] 2024/01/19 10:04:01 DoRetryWithTimeout - Error: {found 8 cloud credentials}, Next try in [30s], timeout [5m0s] [2024-01-19T10:04:34.353Z] 2024-01-19 10:04:31 +0000:[INFO] [backup.getAllCloudCredentials:#1624] - The cloud credentials and their UID are map[242230f4-6757-4901-a866-c9945bee2eaf:autogenerated-cred-1705635982 384f63f1-0fe9-46e0-bbd2-7fe781ca8ee3:autogenerated-cred-1705634221 4e0ddfc7-8811-4ed5-8e87-ed0841582111:ibm-source-config-cloud-cred-enzij 72b1b409-7bb1-4f22-92df-0ba5e77c11d3:ibm-source-config-cloud-cred-hrfiy 7cddb5aa-c7ff-4044-9d27-458be2bd5cde:ibm-source-config-cloud-cred-fbvvo 832e19de-64ac-4aa2-83bc-96f99917be8a:autogenerated-cred-meoyc c5ff084a-a913-4412-99fb-e2b0a003a014:ibm-source-config-cloud-cred-wcpfm f0465bb5-2565-4948-bb27-7e45a4b883ff:ibm-source-config-cloud-cred-vlbwu f551fb81-1a78-4d7f-b804-759c6e6b562e:ibm-source-config-cloud-cred-mhxzi] [2024-01-19T10:04:34.353Z] INFO[2024-01-19 10:04:31] Verifying : Description : Verifying fetching of all cloud credentials [2024-01-19T10:04:34.353Z] INFO[2024-01-19 10:04:31] Actual:true, Expected: true [2024-01-19T10:04:34.353Z] 2024/01/19 10:04:31 DoRetryWithTimeout - Error: {found 8 cloud credentials}, Next try in [30s], timeout [5m0s] [2024-01-19T10:05:06.411Z] INFO[2024-01-19 10:05:01] Verifying : Description : Verifying cloud credential deletion success [2024-01-19T10:05:06.411Z] ERRO[2024-01-19 10:05:01] Actual:timed out performing task., Error was: DoRetryWithTimeout timed out. Errors generated in retries: {found 8 cloud credentials} [2024-01-19T10:05:06.411Z] {found 8 cloud credentials} [2024-01-19T10:05:06.411Z] {found 8 cloud credentials} [2024-01-19T10:05:06.411Z] {found 8 cloud credentials} [2024-01-19T10:05:06.411Z] {found 8 cloud credentials} [2024-01-19T10:05:06.411Z] {found 8 cloud credentials} [2024-01-19T10:05:06.411Z] {found 8 cloud credentials} [2024-01-19T10:05:06.411Z] {found 8 cloud credentials} [2024-01-19T10:05:06.411Z] {found 8 cloud credentials} [2024-01-19T10:05:06.411Z] {found 8 cloud credentials}, Expected:  [2024-01-19T10:05:06.411Z] 2024-01-19 10:05:02 +0000:[INFO] [backup.glob..func2:#366] - Bucket deleted - global-aws-px-backup-system-test-4cac76dc [2024-01-19T10:05:06.411Z] INFO[2024-01-19 10:05:02] Verifying : Description : Verifying cleaning up of all users from keycloak [2024-01-19T10:05:06.411Z] INFO[2024-01-19 10:05:02] Actual:true, Expected: true [2024-01-19T10:05:06.411Z] 2024-01-19 10:05:02 +0000:[INFO] [backup.glob..func2:#387] - User admin was not deleted [2024-01-19T10:05:06.411Z] 2024-01-19 10:05:02 +0000:[INFO] [backup.glob..func2:#387] - User px-central-admin was not deleted [2024-01-19T10:05:06.411Z] 2024-01-19 10:05:03 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser10-1705628096 - b35bb037-122f-498a-b15f-1688fe27cb60 [2024-01-19T10:05:06.411Z] 2024-01-19 10:05:03 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser10-1705628096] deletion - [2024-01-19T10:05:06.411Z] 2024-01-19 10:05:03 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser10-1705628096 [2024-01-19T10:05:06.411Z] INFO[2024-01-19 10:05:03] Verifying : Description : Verifying user [testuser10-1705628096] deletion [2024-01-19T10:05:06.411Z] INFO[2024-01-19 10:05:03] Actual:true, Expected: true [2024-01-19T10:05:06.411Z] 2024-01-19 10:05:04 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705568457 - 4605f33c-86f9-4970-9b86-12e9f605796f [2024-01-19T10:05:06.411Z] 2024-01-19 10:05:04 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705568457] deletion - [2024-01-19T10:05:06.411Z] 2024-01-19 10:05:04 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705568457 [2024-01-19T10:05:06.411Z] INFO[2024-01-19 10:05:04] Verifying : Description : Verifying user [testuser1-1705568457] deletion [2024-01-19T10:05:06.411Z] INFO[2024-01-19 10:05:04] Actual:true, Expected: true [2024-01-19T10:05:06.411Z] 2024-01-19 10:05:05 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705569021 - ed4b72a8-ec8c-40b6-abaa-c93284e25e23 [2024-01-19T10:05:06.411Z] 2024-01-19 10:05:06 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705569021] deletion - [2024-01-19T10:05:06.411Z] 2024-01-19 10:05:06 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705569021 [2024-01-19T10:05:06.411Z] INFO[2024-01-19 10:05:06] Verifying : Description : Verifying user [testuser1-1705569021] deletion [2024-01-19T10:05:06.412Z] INFO[2024-01-19 10:05:06] Actual:true, Expected: true [2024-01-19T10:05:07.341Z] 2024-01-19 10:05:07 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705569023 - 116f8b28-3f3d-4104-ac70-d745f95c8bb8 [2024-01-19T10:05:07.918Z] 2024-01-19 10:05:07 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705569023] deletion - [2024-01-19T10:05:07.918Z] 2024-01-19 10:05:07 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705569023 [2024-01-19T10:05:07.918Z] INFO[2024-01-19 10:05:07] Verifying : Description : Verifying user [testuser1-1705569023] deletion [2024-01-19T10:05:07.918Z] INFO[2024-01-19 10:05:07] Actual:true, Expected: true [2024-01-19T10:05:09.282Z] 2024-01-19 10:05:08 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705569027 - 4eee8e6b-6400-4f53-9da8-bc1b3aac1650 [2024-01-19T10:05:09.538Z] 2024-01-19 10:05:09 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705569027] deletion - [2024-01-19T10:05:09.538Z] 2024-01-19 10:05:09 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705569027 [2024-01-19T10:05:09.538Z] INFO[2024-01-19 10:05:09] Verifying : Description : Verifying user [testuser1-1705569027] deletion [2024-01-19T10:05:09.538Z] INFO[2024-01-19 10:05:09] Actual:true, Expected: true [2024-01-19T10:05:10.904Z] 2024-01-19 10:05:10 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705569626 - 26139790-a1ff-4a92-bfe1-263a51333904 [2024-01-19T10:05:11.194Z] 2024-01-19 10:05:11 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705569626] deletion - [2024-01-19T10:05:11.194Z] 2024-01-19 10:05:11 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705569626 [2024-01-19T10:05:11.194Z] INFO[2024-01-19 10:05:11] Verifying : Description : Verifying user [testuser1-1705569626] deletion [2024-01-19T10:05:11.194Z] INFO[2024-01-19 10:05:11] Actual:true, Expected: true [2024-01-19T10:05:12.560Z] 2024-01-19 10:05:12 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705570505 - 0b812160-e12b-4afd-ad7c-3dd2cba19397 [2024-01-19T10:05:12.816Z] 2024-01-19 10:05:12 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705570505] deletion - [2024-01-19T10:05:12.816Z] 2024-01-19 10:05:12 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705570505 [2024-01-19T10:05:12.816Z] INFO[2024-01-19 10:05:12] Verifying : Description : Verifying user [testuser1-1705570505] deletion [2024-01-19T10:05:12.816Z] INFO[2024-01-19 10:05:12] Actual:true, Expected: true [2024-01-19T10:05:14.184Z] 2024-01-19 10:05:13 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705570507 - 87fb392d-e166-41af-a25d-91f860298727 [2024-01-19T10:05:14.441Z] 2024-01-19 10:05:14 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705570507] deletion - [2024-01-19T10:05:14.441Z] 2024-01-19 10:05:14 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705570507 [2024-01-19T10:05:14.442Z] INFO[2024-01-19 10:05:14] Verifying : Description : Verifying user [testuser1-1705570507] deletion [2024-01-19T10:05:14.442Z] INFO[2024-01-19 10:05:14] Actual:true, Expected: true [2024-01-19T10:05:15.369Z] 2024-01-19 10:05:15 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705570511 - f84b844a-2598-4a87-ada6-632c428f6b60 [2024-01-19T10:05:15.931Z] 2024-01-19 10:05:15 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705570511] deletion - [2024-01-19T10:05:15.931Z] 2024-01-19 10:05:15 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705570511 [2024-01-19T10:05:15.931Z] INFO[2024-01-19 10:05:15] Verifying : Description : Verifying user [testuser1-1705570511] deletion [2024-01-19T10:05:15.931Z] INFO[2024-01-19 10:05:15] Actual:true, Expected: true [2024-01-19T10:05:17.296Z] 2024-01-19 10:05:16 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705576411 - cebf960d-bc25-4c89-ab94-9cfe0742e96a [2024-01-19T10:05:17.554Z] 2024-01-19 10:05:17 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705576411] deletion - [2024-01-19T10:05:17.554Z] 2024-01-19 10:05:17 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705576411 [2024-01-19T10:05:17.554Z] INFO[2024-01-19 10:05:17] Verifying : Description : Verifying user [testuser1-1705576411] deletion [2024-01-19T10:05:17.554Z] INFO[2024-01-19 10:05:17] Actual:true, Expected: true [2024-01-19T10:05:18.918Z] 2024-01-19 10:05:18 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705576804 - f98d1f44-ac1d-4ad8-803e-97dd3b7068e1 [2024-01-19T10:05:19.174Z] 2024-01-19 10:05:19 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705576804] deletion - [2024-01-19T10:05:19.174Z] 2024-01-19 10:05:19 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705576804 [2024-01-19T10:05:19.174Z] INFO[2024-01-19 10:05:19] Verifying : Description : Verifying user [testuser1-1705576804] deletion [2024-01-19T10:05:19.174Z] INFO[2024-01-19 10:05:19] Actual:true, Expected: true [2024-01-19T10:05:19.738Z] INFO[2024-01-19 10:05:19] sched stats: workers: current=10, started=10, exited=0 [2024-01-19T10:05:19.738Z] INFO[2024-01-19 10:05:19] sched stats: tick interval: 66679 < 1.2s (100%) [2024-01-19T10:05:19.738Z] INFO[2024-01-19 10:05:19] sched stats: tick process durations: 66678 < 500ms (100%) [2024-01-19T10:05:20.302Z] 2024-01-19 10:05:20 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705592708 - 157b6684-6119-46de-ac11-165e33ccc7ac [2024-01-19T10:05:20.863Z] 2024-01-19 10:05:20 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705592708] deletion - [2024-01-19T10:05:20.863Z] 2024-01-19 10:05:20 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705592708 [2024-01-19T10:05:20.863Z] INFO[2024-01-19 10:05:20] Verifying : Description : Verifying user [testuser1-1705592708] deletion [2024-01-19T10:05:20.863Z] INFO[2024-01-19 10:05:20] Actual:true, Expected: true [2024-01-19T10:05:21.794Z] 2024-01-19 10:05:21 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705595980 - cf87c088-7fb7-4cc7-a18d-95f00d2a9974 [2024-01-19T10:05:22.355Z] 2024-01-19 10:05:22 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705595980] deletion - [2024-01-19T10:05:22.355Z] 2024-01-19 10:05:22 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705595980 [2024-01-19T10:05:22.355Z] INFO[2024-01-19 10:05:22] Verifying : Description : Verifying user [testuser1-1705595980] deletion [2024-01-19T10:05:22.355Z] INFO[2024-01-19 10:05:22] Actual:true, Expected: true [2024-01-19T10:05:23.721Z] 2024-01-19 10:05:23 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705596515 - ed18de04-e42b-48a4-9469-b00c20c8ddec [2024-01-19T10:05:23.978Z] 2024-01-19 10:05:23 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705596515] deletion - [2024-01-19T10:05:23.978Z] 2024-01-19 10:05:23 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705596515 [2024-01-19T10:05:23.979Z] INFO[2024-01-19 10:05:23] Verifying : Description : Verifying user [testuser1-1705596515] deletion [2024-01-19T10:05:23.979Z] INFO[2024-01-19 10:05:23] Actual:true, Expected: true [2024-01-19T10:05:25.346Z] 2024-01-19 10:05:24 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705596517 - 88de5c6a-97e3-49f5-8bb8-8933dfbb79af [2024-01-19T10:05:25.602Z] 2024-01-19 10:05:25 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705596517] deletion - [2024-01-19T10:05:25.602Z] 2024-01-19 10:05:25 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705596517 [2024-01-19T10:05:25.602Z] INFO[2024-01-19 10:05:25] Verifying : Description : Verifying user [testuser1-1705596517] deletion [2024-01-19T10:05:25.602Z] INFO[2024-01-19 10:05:25] Actual:true, Expected: true [2024-01-19T10:05:26.532Z] 2024-01-19 10:05:26 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705596520 - 0439fb16-36f9-47d1-88fe-1c5d55b096f3 [2024-01-19T10:05:27.093Z] 2024-01-19 10:05:27 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705596520] deletion - [2024-01-19T10:05:27.094Z] 2024-01-19 10:05:27 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705596520 [2024-01-19T10:05:27.094Z] INFO[2024-01-19 10:05:27] Verifying : Description : Verifying user [testuser1-1705596520] deletion [2024-01-19T10:05:27.094Z] INFO[2024-01-19 10:05:27] Actual:true, Expected: true [2024-01-19T10:05:28.457Z] 2024-01-19 10:05:28 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705597117 - 70a91b8d-127c-4fba-bbf6-15c954a0a95c [2024-01-19T10:05:28.713Z] 2024-01-19 10:05:28 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705597117] deletion - [2024-01-19T10:05:28.713Z] 2024-01-19 10:05:28 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705597117 [2024-01-19T10:05:28.713Z] INFO[2024-01-19 10:05:28] Verifying : Description : Verifying user [testuser1-1705597117] deletion [2024-01-19T10:05:28.713Z] INFO[2024-01-19 10:05:28] Actual:true, Expected: true [2024-01-19T10:05:30.080Z] 2024-01-19 10:05:29 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705603039 - 46d3e5a9-a13d-4004-a271-a2c0cfd5258d [2024-01-19T10:05:30.337Z] 2024-01-19 10:05:30 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705603039] deletion - [2024-01-19T10:05:30.337Z] 2024-01-19 10:05:30 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705603039 [2024-01-19T10:05:30.337Z] INFO[2024-01-19 10:05:30] Verifying : Description : Verifying user [testuser1-1705603039] deletion [2024-01-19T10:05:30.337Z] INFO[2024-01-19 10:05:30] Actual:true, Expected: true [2024-01-19T10:05:31.702Z] 2024-01-19 10:05:31 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705618999 - c9ebbb13-e7ea-4d88-9b3b-e3c540acc148 [2024-01-19T10:05:31.959Z] 2024-01-19 10:05:31 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705618999] deletion - [2024-01-19T10:05:31.959Z] 2024-01-19 10:05:31 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705618999 [2024-01-19T10:05:31.959Z] INFO[2024-01-19 10:05:31] Verifying : Description : Verifying user [testuser1-1705618999] deletion [2024-01-19T10:05:31.959Z] INFO[2024-01-19 10:05:31] Actual:true, Expected: true [2024-01-19T10:05:33.324Z] 2024-01-19 10:05:32 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705625950 - bf03dd75-8285-4424-8ca9-3bc5ace8efbf [2024-01-19T10:05:33.887Z] 2024-01-19 10:05:33 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705625950] deletion - [2024-01-19T10:05:33.887Z] 2024-01-19 10:05:33 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705625950 [2024-01-19T10:05:33.887Z] INFO[2024-01-19 10:05:33] Verifying : Description : Verifying user [testuser1-1705625950] deletion [2024-01-19T10:05:33.887Z] INFO[2024-01-19 10:05:33] Actual:true, Expected: true [2024-01-19T10:05:34.816Z] 2024-01-19 10:05:34 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705625962 - e80e9f17-1795-419e-acbf-1dc483475046 [2024-01-19T10:05:35.072Z] 2024-01-19 10:05:35 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705625962] deletion - [2024-01-19T10:05:35.072Z] 2024-01-19 10:05:35 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705625962 [2024-01-19T10:05:35.072Z] INFO[2024-01-19 10:05:35] Verifying : Description : Verifying user [testuser1-1705625962] deletion [2024-01-19T10:05:35.072Z] INFO[2024-01-19 10:05:35] Actual:true, Expected: true [2024-01-19T10:05:36.439Z] 2024-01-19 10:05:36 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705625965 - 2d10b1f6-0632-4166-b5e3-c90e410ef610 [2024-01-19T10:05:36.695Z] 2024-01-19 10:05:36 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705625965] deletion - [2024-01-19T10:05:36.695Z] 2024-01-19 10:05:36 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705625965 [2024-01-19T10:05:36.695Z] INFO[2024-01-19 10:05:36] Verifying : Description : Verifying user [testuser1-1705625965] deletion [2024-01-19T10:05:36.695Z] INFO[2024-01-19 10:05:36] Actual:true, Expected: true [2024-01-19T10:05:38.061Z] 2024-01-19 10:05:37 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705625968 - 785699f6-b9ac-4002-acda-4fc838e1e130 [2024-01-19T10:05:38.318Z] 2024-01-19 10:05:38 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705625968] deletion - [2024-01-19T10:05:38.318Z] 2024-01-19 10:05:38 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705625968 [2024-01-19T10:05:38.318Z] INFO[2024-01-19 10:05:38] Verifying : Description : Verifying user [testuser1-1705625968] deletion [2024-01-19T10:05:38.318Z] INFO[2024-01-19 10:05:38] Actual:true, Expected: true [2024-01-19T10:05:39.697Z] 2024-01-19 10:05:39 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705628096 - cc430c49-9272-41cd-b95e-4be0248d69bb [2024-01-19T10:05:39.953Z] 2024-01-19 10:05:39 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705628096] deletion - [2024-01-19T10:05:39.953Z] 2024-01-19 10:05:39 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705628096 [2024-01-19T10:05:39.953Z] INFO[2024-01-19 10:05:39] Verifying : Description : Verifying user [testuser1-1705628096] deletion [2024-01-19T10:05:39.953Z] INFO[2024-01-19 10:05:39] Actual:true, Expected: true [2024-01-19T10:05:41.332Z] 2024-01-19 10:05:40 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705636817 - 3a8eee2e-3ae6-47b2-99f5-ae682fc2b14d [2024-01-19T10:05:41.588Z] 2024-01-19 10:05:41 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705636817] deletion - [2024-01-19T10:05:41.588Z] 2024-01-19 10:05:41 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705636817 [2024-01-19T10:05:41.588Z] INFO[2024-01-19 10:05:41] Verifying : Description : Verifying user [testuser1-1705636817] deletion [2024-01-19T10:05:41.588Z] INFO[2024-01-19 10:05:41] Actual:true, Expected: true [2024-01-19T10:05:42.526Z] 2024-01-19 10:05:42 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705636820 - 52e2390b-1c29-44bd-969c-e1fb4ceb50b3 [2024-01-19T10:05:43.087Z] 2024-01-19 10:05:42 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705636820] deletion - [2024-01-19T10:05:43.087Z] 2024-01-19 10:05:42 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705636820 [2024-01-19T10:05:43.087Z] INFO[2024-01-19 10:05:42] Verifying : Description : Verifying user [testuser1-1705636820] deletion [2024-01-19T10:05:43.087Z] INFO[2024-01-19 10:05:42] Actual:true, Expected: true [2024-01-19T10:05:44.487Z] 2024-01-19 10:05:44 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705636824 - 744d6f61-7c58-41c2-99b5-b79fe60034fa [2024-01-19T10:05:44.745Z] 2024-01-19 10:05:44 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705636824] deletion - [2024-01-19T10:05:44.745Z] 2024-01-19 10:05:44 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705636824 [2024-01-19T10:05:44.745Z] INFO[2024-01-19 10:05:44] Verifying : Description : Verifying user [testuser1-1705636824] deletion [2024-01-19T10:05:44.745Z] INFO[2024-01-19 10:05:44] Actual:true, Expected: true [2024-01-19T10:05:46.113Z] 2024-01-19 10:05:45 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637494 - 7ab1a700-5d59-4cf3-a59b-46256c94f61b [2024-01-19T10:05:46.369Z] 2024-01-19 10:05:46 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705637494] deletion - [2024-01-19T10:05:46.369Z] 2024-01-19 10:05:46 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705637494 [2024-01-19T10:05:46.369Z] INFO[2024-01-19 10:05:46] Verifying : Description : Verifying user [testuser1-1705637494] deletion [2024-01-19T10:05:46.369Z] INFO[2024-01-19 10:05:46] Actual:true, Expected: true [2024-01-19T10:05:47.733Z] 2024-01-19 10:05:47 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637497 - c3f9f3d9-1651-4a0d-800d-9afbf122ffa0 [2024-01-19T10:05:47.989Z] 2024-01-19 10:05:47 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705637497] deletion - [2024-01-19T10:05:47.989Z] 2024-01-19 10:05:47 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705637497 [2024-01-19T10:05:47.989Z] INFO[2024-01-19 10:05:47] Verifying : Description : Verifying user [testuser1-1705637497] deletion [2024-01-19T10:05:47.989Z] INFO[2024-01-19 10:05:47] Actual:true, Expected: true [2024-01-19T10:05:49.355Z] 2024-01-19 10:05:48 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637501 - ad3bd280-b2ab-44a8-8e03-cb49edc964e5 [2024-01-19T10:05:49.612Z] 2024-01-19 10:05:49 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705637501] deletion - [2024-01-19T10:05:49.612Z] 2024-01-19 10:05:49 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705637501 [2024-01-19T10:05:49.612Z] INFO[2024-01-19 10:05:49] Verifying : Description : Verifying user [testuser1-1705637501] deletion [2024-01-19T10:05:49.612Z] INFO[2024-01-19 10:05:49] Actual:true, Expected: true [2024-01-19T10:05:50.540Z] 2024-01-19 10:05:50 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637505 - b19c7b6e-9dcf-46db-8832-33dfc89f3250 [2024-01-19T10:05:51.102Z] 2024-01-19 10:05:50 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705637505] deletion - [2024-01-19T10:05:51.102Z] 2024-01-19 10:05:50 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705637505 [2024-01-19T10:05:51.102Z] INFO[2024-01-19 10:05:50] Verifying : Description : Verifying user [testuser1-1705637505] deletion [2024-01-19T10:05:51.102Z] INFO[2024-01-19 10:05:50] Actual:true, Expected: true [2024-01-19T10:05:52.474Z] 2024-01-19 10:05:52 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637509 - 74751a01-c85e-4c1e-9914-936792573b6c [2024-01-19T10:05:52.733Z] 2024-01-19 10:05:52 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705637509] deletion - [2024-01-19T10:05:52.733Z] 2024-01-19 10:05:52 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705637509 [2024-01-19T10:05:52.733Z] INFO[2024-01-19 10:05:52] Verifying : Description : Verifying user [testuser1-1705637509] deletion [2024-01-19T10:05:52.733Z] INFO[2024-01-19 10:05:52] Actual:true, Expected: true [2024-01-19T10:05:54.106Z] 2024-01-19 10:05:53 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705637513 - 42526588-0699-4a99-b785-affbe774f988 [2024-01-19T10:05:54.362Z] 2024-01-19 10:05:54 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705637513] deletion - [2024-01-19T10:05:54.362Z] 2024-01-19 10:05:54 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705637513 [2024-01-19T10:05:54.362Z] INFO[2024-01-19 10:05:54] Verifying : Description : Verifying user [testuser1-1705637513] deletion [2024-01-19T10:05:54.362Z] INFO[2024-01-19 10:05:54] Actual:true, Expected: true [2024-01-19T10:05:55.739Z] 2024-01-19 10:05:55 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705641006 - 8b3d5ca3-d403-499b-9712-afd37bc55594 [2024-01-19T10:05:55.995Z] 2024-01-19 10:05:55 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705641006] deletion - [2024-01-19T10:05:55.995Z] 2024-01-19 10:05:55 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705641006 [2024-01-19T10:05:55.995Z] INFO[2024-01-19 10:05:55] Verifying : Description : Verifying user [testuser1-1705641006] deletion [2024-01-19T10:05:55.995Z] INFO[2024-01-19 10:05:55] Actual:true, Expected: true [2024-01-19T10:05:57.362Z] 2024-01-19 10:05:56 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705641008 - 7ead8c09-d3d3-4fa5-bf3e-e6576a1154e5 [2024-01-19T10:05:57.618Z] 2024-01-19 10:05:57 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705641008] deletion - [2024-01-19T10:05:57.618Z] 2024-01-19 10:05:57 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705641008 [2024-01-19T10:05:57.618Z] INFO[2024-01-19 10:05:57] Verifying : Description : Verifying user [testuser1-1705641008] deletion [2024-01-19T10:05:57.618Z] INFO[2024-01-19 10:05:57] Actual:true, Expected: true [2024-01-19T10:05:58.545Z] 2024-01-19 10:05:58 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705641012 - dd66b4af-edba-49c4-b40f-f2328ceeb36d [2024-01-19T10:05:59.105Z] 2024-01-19 10:05:58 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705641012] deletion - [2024-01-19T10:05:59.105Z] 2024-01-19 10:05:58 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705641012 [2024-01-19T10:05:59.105Z] INFO[2024-01-19 10:05:58] Verifying : Description : Verifying user [testuser1-1705641012] deletion [2024-01-19T10:05:59.105Z] INFO[2024-01-19 10:05:58] Actual:true, Expected: true [2024-01-19T10:06:00.469Z] 2024-01-19 10:06:00 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser1-1705649083 - 7a8f5e17-45e5-4587-bc88-876237f707b2 [2024-01-19T10:06:00.725Z] 2024-01-19 10:06:00 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser1-1705649083] deletion - [2024-01-19T10:06:00.725Z] 2024-01-19 10:06:00 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser1-1705649083 [2024-01-19T10:06:00.725Z] INFO[2024-01-19 10:06:00] Verifying : Description : Verifying user [testuser1-1705649083] deletion [2024-01-19T10:06:00.725Z] INFO[2024-01-19 10:06:00] Actual:true, Expected: true [2024-01-19T10:06:02.093Z] 2024-01-19 10:06:01 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser2-1705628096 - 8f27b128-af90-4fa9-be36-35330680231b [2024-01-19T10:06:02.350Z] 2024-01-19 10:06:02 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser2-1705628096] deletion - [2024-01-19T10:06:02.350Z] 2024-01-19 10:06:02 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser2-1705628096 [2024-01-19T10:06:02.350Z] INFO[2024-01-19 10:06:02] Verifying : Description : Verifying user [testuser2-1705628096] deletion [2024-01-19T10:06:02.350Z] INFO[2024-01-19 10:06:02] Actual:true, Expected: true [2024-01-19T10:06:03.721Z] 2024-01-19 10:06:03 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser3-1705628096 - 7a6999d9-4319-4019-9086-498f3352e9c0 [2024-01-19T10:06:03.977Z] 2024-01-19 10:06:03 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser3-1705628096] deletion - [2024-01-19T10:06:03.977Z] 2024-01-19 10:06:03 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser3-1705628096 [2024-01-19T10:06:03.977Z] INFO[2024-01-19 10:06:03] Verifying : Description : Verifying user [testuser3-1705628096] deletion [2024-01-19T10:06:03.977Z] INFO[2024-01-19 10:06:03] Actual:true, Expected: true [2024-01-19T10:06:05.341Z] 2024-01-19 10:06:04 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser4-1705628096 - 60bed100-ea29-40e1-b3ca-633f06aed5f7 [2024-01-19T10:06:05.598Z] 2024-01-19 10:06:05 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser4-1705628096] deletion - [2024-01-19T10:06:05.598Z] 2024-01-19 10:06:05 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser4-1705628096 [2024-01-19T10:06:05.598Z] INFO[2024-01-19 10:06:05] Verifying : Description : Verifying user [testuser4-1705628096] deletion [2024-01-19T10:06:05.598Z] INFO[2024-01-19 10:06:05] Actual:true, Expected: true [2024-01-19T10:06:06.591Z] 2024-01-19 10:06:06 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser5-1705628096 - 4e3b1848-48b5-46d0-9906-bd5e240ecc62 [2024-01-19T10:06:07.154Z] 2024-01-19 10:06:07 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser5-1705628096] deletion - [2024-01-19T10:06:07.154Z] 2024-01-19 10:06:07 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser5-1705628096 [2024-01-19T10:06:07.154Z] INFO[2024-01-19 10:06:07] Verifying : Description : Verifying user [testuser5-1705628096] deletion [2024-01-19T10:06:07.154Z] INFO[2024-01-19 10:06:07] Actual:true, Expected: true [2024-01-19T10:06:08.524Z] 2024-01-19 10:06:08 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser6-1705628096 - 61c48609-132c-424f-a593-680e49e8a8cb [2024-01-19T10:06:08.784Z] 2024-01-19 10:06:08 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser6-1705628096] deletion - [2024-01-19T10:06:08.784Z] 2024-01-19 10:06:08 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser6-1705628096 [2024-01-19T10:06:08.784Z] INFO[2024-01-19 10:06:08] Verifying : Description : Verifying user [testuser6-1705628096] deletion [2024-01-19T10:06:08.784Z] INFO[2024-01-19 10:06:08] Actual:true, Expected: true [2024-01-19T10:06:10.155Z] 2024-01-19 10:06:09 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser7-1705628096 - fa58aa4c-f46b-454c-bd52-8370f548efa3 [2024-01-19T10:06:10.413Z] 2024-01-19 10:06:10 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser7-1705628096] deletion - [2024-01-19T10:06:10.413Z] 2024-01-19 10:06:10 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser7-1705628096 [2024-01-19T10:06:10.413Z] INFO[2024-01-19 10:06:10] Verifying : Description : Verifying user [testuser7-1705628096] deletion [2024-01-19T10:06:10.413Z] INFO[2024-01-19 10:06:10] Actual:true, Expected: true [2024-01-19T10:06:11.348Z] 2024-01-19 10:06:11 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser8-1705628096 - 04d90090-0cca-4b75-b326-74a6f278659d [2024-01-19T10:06:11.918Z] 2024-01-19 10:06:11 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser8-1705628096] deletion - [2024-01-19T10:06:11.918Z] 2024-01-19 10:06:11 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser8-1705628096 [2024-01-19T10:06:11.918Z] INFO[2024-01-19 10:06:11] Verifying : Description : Verifying user [testuser8-1705628096] deletion [2024-01-19T10:06:11.919Z] INFO[2024-01-19 10:06:11] Actual:true, Expected: true [2024-01-19T10:06:13.292Z] 2024-01-19 10:06:12 +0000:[INFO] [backup.FetchIDOfUser:#496] - Fetching ID of user testuser9-1705628096 - fa63e049-e0a3-40a4-8318-732a19e20499 [2024-01-19T10:06:13.549Z] 2024-01-19 10:06:13 +0000:[INFO] [backup.DeleteUser:#755] - Response for user [testuser9-1705628096] deletion - [2024-01-19T10:06:13.549Z] 2024-01-19 10:06:13 +0000:[INFO] [backup.DeleteUser:#759] - Deleted User - testuser9-1705628096 [2024-01-19T10:06:13.549Z] INFO[2024-01-19 10:06:13] Verifying : Description : Verifying user [testuser9-1705628096] deletion [2024-01-19T10:06:13.549Z] INFO[2024-01-19 10:06:13] Actual:true, Expected: true [2024-01-19T10:06:14.481Z] INFO[2024-01-19 10:06:14] Verifying : Description : Verifying cleaning up of all groups from keycloak [2024-01-19T10:06:14.481Z] INFO[2024-01-19 10:06:14] Actual:true, Expected: true [2024-01-19T10:06:14.481Z] 2024-01-19 10:06:14 +0000:[INFO] [backup.glob..func2:#399] - Group admin was not deleted [2024-01-19T10:06:14.481Z] 2024-01-19 10:06:14 +0000:[INFO] [backup.glob..func2:#399] - Group px-central-admin was not deleted [2024-01-19T10:06:14.481Z] INFO[2024-01-19 10:06:14] --------Test End------ [2024-01-19T10:06:14.481Z] INFO[2024-01-19 10:06:14] #Test: [2024-01-19T10:06:14.481Z] INFO[2024-01-19 10:06:14] #Description: [2024-01-19T10:06:14.481Z] INFO[2024-01-19 10:06:14] ------------------------ [2024-01-19T10:06:14.481Z] [2024-01-19T10:06:14.481Z] JUnit report was created: /testresults/junit_basic.xml [2024-01-19T10:06:14.481Z] [2024-01-19T10:06:14.481Z] [2024-01-19T10:06:14.481Z] Summarizing 42 Failures: [2024-01-19T10:06:14.481Z] [2024-01-19T10:06:14.481Z] [Fail] {AddMultipleNamespaceLabels} [It] Add multiple labels to namespaces, perform manual backup, schedule backup using namespace label and restore  [2024-01-19T10:06:14.481Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-19T10:06:14.481Z] [2024-01-19T10:06:14.482Z] [Fail] {DeleteNSDeleteClusterRestore} [It] Validates deleted namespace is restored when the application cluster is removed and re-added  [2024-01-19T10:06:14.482Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-19T10:06:14.482Z] [2024-01-19T10:06:14.482Z] [Fail] {AllNSBackupWithIncludeNewNSOption} [It] Validates schedule backups created with include new namespaces option includes newly created namespaces  [2024-01-19T10:06:14.482Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T10:06:14.482Z] [2024-01-19T10:06:14.482Z] [Fail] {BasicSelectiveRestore} [It] Selective Restore From a Basic Backup  [2024-01-19T10:06:14.482Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-19T10:06:14.482Z] [2024-01-19T10:06:14.482Z] [Fail] {DeleteAllBackupObjects} [It] Create backup objects and delete  [2024-01-19T10:06:14.482Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T10:06:14.482Z] [2024-01-19T10:06:14.482Z] [Fail] {DeleteUsersRole} [] Delete user and roles  [2024-01-19T10:06:14.482Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T10:06:14.482Z] [2024-01-19T10:06:14.482Z] [Fail] {CancelAllRunningBackupJobs} [] Cancel All Running Backup Jobs and validate  [2024-01-19T10:06:14.482Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T10:06:14.482Z] [2024-01-19T10:06:14.482Z] [Fail] {SetUnsetNSLabelDuringScheduleBackup} [] Create multiple namespaces and set unset namespace labels  [2024-01-19T10:06:14.482Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T10:06:14.482Z] [2024-01-19T10:06:14.482Z] [Fail] {NamespaceLabelledBackupOfEmptyNamespace} [It] Namespace labelled backup of empty namespace and restore it  [2024-01-19T10:06:14.482Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-19T10:06:14.482Z] [2024-01-19T10:06:14.482Z] [Fail] {UpdatesBackupOfUserFromAdmin} [] Updates Backups and Cluster of user from px-admin  [2024-01-19T10:06:14.482Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T10:06:14.482Z] [2024-01-19T10:06:14.482Z] [Fail] {ScheduleBackupCreationAllNS} [It] Schedule Backup Creation with all namespaces  [2024-01-19T10:06:14.482Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-19T10:06:14.482Z] [2024-01-19T10:06:14.482Z] [Fail] {CustomResourceRestore} [] Create custom resource restore  [2024-01-19T10:06:14.482Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T10:06:14.482Z] [2024-01-19T10:06:14.482Z] [Fail] {DeleteSameNameObjectsByMultipleUsersFromAdmin} [It] Deletes backups, backup schedules, restore and cluster objects created by multiple user with same name from the admin  [2024-01-19T10:06:14.482Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-19T10:06:14.482Z] [2024-01-19T10:06:14.482Z] [Fail] {SharedBackupDelete} [] Share the backups and delete  [2024-01-19T10:06:14.482Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T10:06:14.482Z] [2024-01-19T10:06:14.482Z] [Fail] {LicensingCountWithNodeLabelledBeforeClusterAddition} [It] Label the application cluster nodes before adding to backup and verify the license count  [2024-01-19T10:06:14.482Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-19T10:06:14.482Z] [2024-01-19T10:06:14.483Z] [Fail] {NamespaceLabelledBackupSharedWithDifferentAccessMode} [] Take namespace labelled backup and share with users having different access mode  [2024-01-19T10:06:14.483Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T10:06:14.483Z] [2024-01-19T10:06:14.483Z] [Fail] {CancelAllRunningRestoreJobs} [] Cancel All Running Restore Jobs and validate  [2024-01-19T10:06:14.483Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T10:06:14.483Z] [2024-01-19T10:06:14.483Z] [Fail] {VerifyRBACForAppAdmin} [It] Validates the RBAC operation for app-admin user  [2024-01-19T10:06:14.483Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-19T10:06:14.483Z] [2024-01-19T10:06:14.483Z] [Fail] {BackupLocationWithEncryptionKey} [It] Creating Backup Location with Encryption Keys  [2024-01-19T10:06:14.483Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T10:06:14.483Z] [2024-01-19T10:06:14.483Z] [Fail] {RestartBackupPodDuringBackupSharing} [It] Restart backup pod during backup sharing  [2024-01-19T10:06:14.483Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-19T10:06:14.483Z] [2024-01-19T10:06:14.483Z] [Fail] {NodeCountForLicensing} [It] Verify worker node on application cluster with label portworx.io/nobackup=true is not counted for licensing  [2024-01-19T10:06:14.483Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T10:06:14.483Z] [2024-01-19T10:06:14.483Z] [Fail] {MultipleInPlaceRestoreSameTime} [] Issue multiple in-place restores at the same time  [2024-01-19T10:06:14.483Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T10:06:14.483Z] [2024-01-19T10:06:14.483Z] [Fail] {SwapShareBackup} [] Share the backup with same name  [2024-01-19T10:06:14.483Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T10:06:14.483Z] [2024-01-19T10:06:14.483Z] [Fail] {ShareAndRemoveBackupLocation} [It] Share and remove backup location and add it back and check from other users if they show up  [2024-01-19T10:06:14.483Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-19T10:06:14.483Z] [2024-01-19T10:06:14.483Z] [Fail] {ManualAndScheduledBackupUsingNamespaceAndResourceLabel} [It] Manual, schedule backup and restore using namespace label and resource label  [2024-01-19T10:06:14.483Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-19T10:06:14.483Z] [2024-01-19T10:06:14.483Z] [Fail] {ShareBackupAndEdit} [It] Share the backup and edit  [2024-01-19T10:06:14.483Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T10:06:14.483Z] [2024-01-19T10:06:14.483Z] [Fail] {BackupMultipleNsWithSameLabel} [It] Taking backup and restoring multiple namespace having same labels  [2024-01-19T10:06:14.483Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T10:06:14.483Z] [2024-01-19T10:06:14.483Z] [Fail] {CancelClusterBackupShare} [It] Share all backups at cluster level with a user group and revoke it and validate  [2024-01-19T10:06:14.483Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T10:06:14.483Z] [2024-01-19T10:06:14.483Z] [Fail] {VerifyRBACForPxAdmin} [It] Validates the RBAC operation for px-admin group user.  [2024-01-19T10:06:14.483Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-19T10:06:14.483Z] [2024-01-19T10:06:14.483Z] [Fail] {DeleteBackupOfUserSharedRBAC} [It] Delete backups by user from admin with shared RBAC objects  [2024-01-19T10:06:14.483Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-19T10:06:14.483Z] [2024-01-19T10:06:14.483Z] [Fail] {ScheduleBackupDeleteAndRecreateNS} [JustBeforeEach] Validates schedule backups when namespaces are deleted and recreated  [2024-01-19T10:06:14.483Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T10:06:14.483Z] [2024-01-19T10:06:14.483Z] [Fail] {KubeAndPxNamespacesSkipOnAllNSBackup} [It] Verify if kube-system, kube-node-lease, kube-public and Px Namespace is skipped on all namespace backup  [2024-01-19T10:06:14.483Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T10:06:14.483Z] [2024-01-19T10:06:14.484Z] [Fail] {DeleteBackupSharedByMultipleUsersFromAdmin} [It] Delete backups of non admin user from px-admin  [2024-01-19T10:06:14.484Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-19T10:06:14.484Z] [2024-01-19T10:06:14.484Z] [Fail] {DuplicateSharedBackup} [] Validate shared backup is not duplicated without cluster  [2024-01-19T10:06:14.484Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T10:06:14.484Z] [2024-01-19T10:06:14.484Z] [Fail] {DeleteSharedBackupOfUserFromAdmin} [] Deletes backups shared by the user from the admin  [2024-01-19T10:06:14.484Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T10:06:14.484Z] [2024-01-19T10:06:14.484Z] [Fail] {ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress} [It] Scale down px-backup deployment when backups/restores are in progress and validate  [2024-01-19T10:06:14.484Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T10:06:14.484Z] [2024-01-19T10:06:14.484Z] [Fail] {LicensingCountBeforeAndAfterBackupPodRestart} [It] Verify the license count before and after the backup pod restarts  [2024-01-19T10:06:14.484Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T10:06:14.484Z] [2024-01-19T10:06:14.484Z] [Fail] {ShareBackupsAndClusterWithUser} [] Share Backup With Full Access Users and try to duplicate the backup  [2024-01-19T10:06:14.484Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T10:06:14.484Z] [2024-01-19T10:06:14.484Z] [Fail] {CustomResourceBackupAndRestore} [It] Create custom resource backup and restore  [2024-01-19T10:06:14.484Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-19T10:06:14.484Z] [2024-01-19T10:06:14.484Z] [Fail] {DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin} [] Delete failed and in-progress backups and restores of user from the admin  [2024-01-19T10:06:14.484Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T10:06:14.484Z] [2024-01-19T10:06:14.484Z] [Fail] {ScheduleBackupWithAdditionAndRemovalOfNS} [It] Perform schedule backup during which remove and add namespace and verify restoration of removed namespace  [2024-01-19T10:06:14.484Z] /go/src/github.com/portworx/torpedo/pkg/aetosutil/dashboardutil.go:427 [2024-01-19T10:06:14.484Z] [2024-01-19T10:06:14.484Z] [Fail] {BackupScheduleForOldAndNewNS} [] Schedule backup using namespace label for old and new namespaces  [2024-01-19T10:06:14.484Z] /go/src/github.com/portworx/torpedo/pkg/log/log.go:334 [2024-01-19T10:06:14.484Z] [2024-01-19T10:06:14.484Z] Ran 51 of 100 Specs in 66733.337 seconds [2024-01-19T10:06:14.484Z] FAIL! -- 9 Passed | 42 Failed | 0 Pending | 49 Skipped [2024-01-19T10:06:14.484Z] --- FAIL: TestBasic (66733.40s) [2024-01-19T10:06:14.484Z] FAIL [2024-01-19T10:06:14.484Z] [2024-01-19T10:06:14.484Z] You're using deprecated Ginkgo functionality: [2024-01-19T10:06:14.484Z] ============================================= [2024-01-19T10:06:14.484Z] Ginkgo 2.0 is under active development and will introduce several new features, improvements, and a small handful of breaking changes. [2024-01-19T10:06:14.484Z] A release candidate for 2.0 is now available and 2.0 should GA in Fall 2021. Please give the RC a try and send us feedback! [2024-01-19T10:06:14.484Z] - To learn more, view the migration guide at https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md [2024-01-19T10:06:14.484Z] - For instructions on using the Release Candidate visit https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md#using-the-beta [2024-01-19T10:06:14.484Z] - To comment, chime in at https://github.com/onsi/ginkgo/issues/711 [2024-01-19T10:06:14.484Z] [2024-01-19T10:06:14.485Z] You are using a custom reporter. Support for custom reporters will likely be removed in V2. Most users were using them to generate junit or teamcity reports and this functionality will be merged into the core reporter. In addition, Ginkgo 2.0 will support emitting a JSON-formatted report that users can then manipulate to generate custom reports. [2024-01-19T10:06:14.485Z] [2024-01-19T10:06:14.485Z] If this change will be impactful to you please leave a comment on https://github.com/onsi/ginkgo/issues/711 [2024-01-19T10:06:14.485Z] Learn more at: https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md#removed-custom-reporters [2024-01-19T10:06:14.485Z] [2024-01-19T10:06:14.485Z] To silence deprecations that can be silenced set the following environment variable: [2024-01-19T10:06:14.485Z] ACK_GINKGO_DEPRECATIONS=1.16.5 [2024-01-19T10:06:14.485Z] [2024-01-19T10:06:14.485Z] [2024-01-19T10:06:14.485Z] There were failures detected in the following suites: [2024-01-19T10:06:14.485Z] backup ./bin [2024-01-19T10:06:14.485Z] [2024-01-19T10:06:14.485Z] Ginkgo ran 1 suite in 18h32m13.675184027s [2024-01-19T10:06:14.485Z] Test Suite Failed [2024-01-19T10:06:14.485Z] [2024-01-19T10:06:14.485Z] Ginkgo 2.0 is coming soon! [2024-01-19T10:06:14.485Z] ========================== [2024-01-19T10:06:14.485Z] Ginkgo 2.0 is under active development and will introduce several new features, improvements, and a small handful of breaking changes. [2024-01-19T10:06:14.485Z] A release candidate for 2.0 is now available and 2.0 should GA in Fall 2021. Please give the RC a try and send us feedback! [2024-01-19T10:06:14.485Z] - To learn more, view the migration guide at https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md [2024-01-19T10:06:14.485Z] - For instructions on using the Release Candidate visit https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md#using-the-beta [2024-01-19T10:06:14.485Z] - To comment, chime in at https://github.com/onsi/ginkgo/issues/711 [2024-01-19T10:06:14.485Z] [2024-01-19T10:06:14.485Z] To silence this notice, set the environment variable: ACK_GINKGO_RC=true [2024-01-19T10:06:14.485Z] Alternatively you can: touch $HOME/.ack-ginkgo-rc [2024-01-19T10:06:16.390Z] Iteration: 66 [2024-01-19T10:06:17.321Z] Error: Torpedo finished with Error state [2024-01-19T10:06:17.321Z] Pod description: [2024-01-19T10:06:17.885Z] Name: torpedo [2024-01-19T10:06:17.885Z] Namespace: default [2024-01-19T10:06:17.885Z] Priority: 0 [2024-01-19T10:06:17.885Z] Node: 10.240.128.6/10.240.128.6 [2024-01-19T10:06:17.885Z] Start Time: Thu, 18 Jan 2024 15:32:34 +0000 [2024-01-19T10:06:17.885Z] Labels: app=torpedo [2024-01-19T10:06:17.885Z] Annotations: cni.projectcalico.org/containerID: 84a3c2ac6aed7feefe1e9ccbd323301bf9f2081b015f35f8e5a65f5b7b15e143 [2024-01-19T10:06:17.885Z] cni.projectcalico.org/podIP: [2024-01-19T10:06:17.885Z] cni.projectcalico.org/podIPs: [2024-01-19T10:06:17.885Z] k8s.v1.cni.cncf.io/network-status: [2024-01-19T10:06:17.885Z] [{ [2024-01-19T10:06:17.885Z] "name": "k8s-pod-network", [2024-01-19T10:06:17.885Z] "ips": [ [2024-01-19T10:06:17.885Z] "172.17.113.154" [2024-01-19T10:06:17.885Z] ], [2024-01-19T10:06:17.885Z] "default": true, [2024-01-19T10:06:17.885Z] "dns": {} [2024-01-19T10:06:17.885Z] }] [2024-01-19T10:06:17.885Z] Status: Failed [2024-01-19T10:06:17.885Z] IP: 172.17.113.154 [2024-01-19T10:06:17.885Z] IPs: [2024-01-19T10:06:17.885Z] IP: 172.17.113.154 [2024-01-19T10:06:17.885Z] Init Containers: [2024-01-19T10:06:17.885Z] init-sysctl: [2024-01-19T10:06:17.885Z] Container ID: cri-o://c4fc1e3ebd4ff34567e30110beade7a25ba7b4e43004beb7cce00b938b62652d [2024-01-19T10:06:17.885Z] Image: busybox [2024-01-19T10:06:17.885Z] Image ID: docker.io/library/busybox@sha256:ba76950ac9eaa407512c9d859cea48114eeff8a6f12ebaa5d32ce79d4a017dd8 [2024-01-19T10:06:17.885Z] Port: [2024-01-19T10:06:17.885Z] Host Port: [2024-01-19T10:06:17.885Z] Command: [2024-01-19T10:06:17.885Z] sh [2024-01-19T10:06:17.885Z] -c [2024-01-19T10:06:17.885Z] mkdir -p /mnt/testresults && chmod 777 /mnt/testresults/ [2024-01-19T10:06:17.885Z] State: Terminated [2024-01-19T10:06:17.885Z] Reason: Completed [2024-01-19T10:06:17.885Z] Exit Code: 0 [2024-01-19T10:06:17.885Z] Started: Thu, 18 Jan 2024 15:32:45 +0000 [2024-01-19T10:06:17.885Z] Finished: Thu, 18 Jan 2024 15:32:45 +0000 [2024-01-19T10:06:17.885Z] Ready: True [2024-01-19T10:06:17.885Z] Restart Count: 0 [2024-01-19T10:06:17.885Z] Environment: [2024-01-19T10:06:17.885Z] Mounts: [2024-01-19T10:06:17.885Z] /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-skw6c (ro) [2024-01-19T10:06:17.885Z] Containers: [2024-01-19T10:06:17.885Z] torpedo: [2024-01-19T10:06:17.886Z] Container ID: cri-o://424b9eeb15c11548272d3ba069f51828ed477df6a2c910b5b044e52e7ae4111d [2024-01-19T10:06:17.886Z] Image: portworx/torpedo-backup:9780382 [2024-01-19T10:06:17.886Z] Image ID: docker.io/portworx/torpedo-backup@sha256:5355b2875fd95a3d243cbdb33dc83ad23f2a29f28502418428de4288563ed4cb [2024-01-19T10:06:17.886Z] Port: [2024-01-19T10:06:17.886Z] Host Port: [2024-01-19T10:06:17.886Z] Command: [2024-01-19T10:06:17.886Z] ginkgo [2024-01-19T10:06:17.886Z] Args: [2024-01-19T10:06:17.886Z] --trace [2024-01-19T10:06:17.886Z] --timeout [2024-01-19T10:06:17.886Z] 720h0m0s [2024-01-19T10:06:17.886Z] -keepGoing [2024-01-19T10:06:17.886Z] --slowSpecThreshold [2024-01-19T10:06:17.886Z] 600 [2024-01-19T10:06:17.886Z] --focus={BasicBackupCreation}|{AddMultipleNamespaceLabels}|{AllNSBackupWithIncludeNewNSOption}|{BackupLocationWithEncryptionKey}|{BackupMultipleNsWithSameLabel}|{BasicSelectiveRestore}|{CancelAllRunningBackupJobs}|{CancelAllRunningRestoreJobs}|{CancelClusterBackupShare}|{CreateMultipleUsersAndGroups}|{CustomResourceBackupAndRestore}|{CustomResourceRestore}|{DeleteAllBackupObjects}|{DeleteUsersRole}|{DifferentAccessSameUser}|{DuplicateSharedBackup}|{LicensingCountBeforeAndAfterBackupPodRestart}|{LicensingCountWithNodeLabelledBeforeClusterAddition}|{ManualAndScheduleBackupUsingNamespaceLabel}|{MultipleInPlaceRestoreSameTime}|{NamespaceLabelledBackupSharedWithDifferentAccessMode}|{NodeCountForLicensing}|{RestartBackupPodDuringBackupSharing}|{RestoreEncryptedAndNonEncryptedBackups}|{ScaleDownPxBackupPodWhileBackupAndRestoreIsInProgress}|{ScheduleBackupCreationAllNS}|{ScheduleBackupCreationSingleNS}|{SetUnsetNSLabelDuringScheduleBackup}|{ShareAndRemoveBackupLocation}|{ShareBackupAndEdit}|{ShareBackupsAndClusterWithUser}|{ShareBackupWithUsersAndGroups}|{SharedBackupDelete}|{SwapShareBackup}|{ScheduleBackupDeleteAndRecreateNS}|{BackupScheduleForOldAndNewNS}|{DeleteNSDeleteClusterRestore}|{ManualAndScheduledBackupUsingNamespaceAndResourceLabel}|{ScheduleBackupWithAdditionAndRemovalOfNS}|{ManualAndScheduleBackupUsingNSLabelWithMaxCharLimit}|{DeleteBackupOfUserSharedRBAC}|{UpdatesBackupOfUserFromAdmin}|{DeleteBackupSharedByMultipleUsersFromAdmin}|{DeleteSameNameObjectsByMultipleUsersFromAdmin}|{DeleteSharedBackupOfUserFromAdmin}|{DeleteFailedInProgressBackupAndRestoreOfUserFromAdmin}|{DeleteObjectsByMultipleUsersFromNewAdmin}|{NamespaceLabelledBackupOfEmptyNamespace}|{VerifyRBACForPxAdmin}|{VerifyRBACForInfraAdmin}|{VerifyRBACForAppAdmin}|{VerifyRBACForAppUser}|{KubeAndPxNamespacesSkipOnAllNSBackup} [2024-01-19T10:06:17.886Z] [2024-01-19T10:06:17.886Z] bin/backup.test [2024-01-19T10:06:17.886Z] -- [2024-01-19T10:06:17.886Z] --spec-dir [2024-01-19T10:06:17.886Z] ../drivers/scheduler/k8s/specs [2024-01-19T10:06:17.886Z] --app-list [2024-01-19T10:06:17.886Z] mysql-ibm [2024-01-19T10:06:17.886Z] --deploy-pds-apps=false [2024-01-19T10:06:17.886Z] --pds-driver [2024-01-19T10:06:17.886Z] [2024-01-19T10:06:17.886Z] --secure-apps [2024-01-19T10:06:17.886Z] [2024-01-19T10:06:17.886Z] --repl1-apps [2024-01-19T10:06:17.886Z] [2024-01-19T10:06:17.886Z] --csi-app-list [2024-01-19T10:06:17.886Z] [2024-01-19T10:06:17.886Z] --scheduler [2024-01-19T10:06:17.886Z] k8s [2024-01-19T10:06:17.886Z] --max-storage-nodes-per-az [2024-01-19T10:06:17.886Z] 2 [2024-01-19T10:06:17.886Z] --backup-driver [2024-01-19T10:06:17.886Z] pxb [2024-01-19T10:06:17.886Z] --log-level [2024-01-19T10:06:17.886Z] debug [2024-01-19T10:06:17.886Z] --node-driver [2024-01-19T10:06:17.886Z] ssh [2024-01-19T10:06:17.886Z] --scale-factor [2024-01-19T10:06:17.886Z] 1 [2024-01-19T10:06:17.886Z] --hyper-converged=true [2024-01-19T10:06:17.886Z] --fail-on-px-pod-restartcount=false [2024-01-19T10:06:17.886Z] --minimun-runtime-mins [2024-01-19T10:06:17.886Z] 0 [2024-01-19T10:06:17.886Z] --driver-start-timeout [2024-01-19T10:06:17.886Z] 30m0s [2024-01-19T10:06:17.886Z] --chaos-level [2024-01-19T10:06:17.886Z] 5 [2024-01-19T10:06:17.886Z] --storagenode-recovery-timeout [2024-01-19T10:06:17.886Z] 35m0s [2024-01-19T10:06:17.886Z] --provisioner [2024-01-19T10:06:17.887Z] ibm [2024-01-19T10:06:17.887Z] --storage-driver [2024-01-19T10:06:17.887Z] ibm [2024-01-19T10:06:17.887Z] --config-map [2024-01-19T10:06:17.887Z] [2024-01-19T10:06:17.887Z] --custom-config [2024-01-19T10:06:17.887Z] [2024-01-19T10:06:17.887Z] --storage-upgrade-endpoint-url= [2024-01-19T10:06:17.887Z] --storage-upgrade-endpoint-version= [2024-01-19T10:06:17.887Z] --upgrade-storage-driver-endpoint-list= [2024-01-19T10:06:17.887Z] --enable-stork-upgrade=false [2024-01-19T10:06:17.887Z] --secret-type=k8s [2024-01-19T10:06:17.887Z] --pure-volumes=false [2024-01-19T10:06:17.887Z] --pure-fa-snapshot-restore-to-many-test=false [2024-01-19T10:06:17.887Z] --pure-san-type=ISCSI [2024-01-19T10:06:17.887Z] --vault-addr= [2024-01-19T10:06:17.887Z] --vault-token= [2024-01-19T10:06:17.887Z] --px-runtime-opts= [2024-01-19T10:06:17.887Z] --anthos-ws-node-ip= [2024-01-19T10:06:17.887Z] --anthos-inst-path= [2024-01-19T10:06:17.887Z] --autopilot-upgrade-version= [2024-01-19T10:06:17.887Z] --csi-generic-driver-config-map= [2024-01-19T10:06:17.887Z] --sched-upgrade-hops= [2024-01-19T10:06:17.887Z] --migration-hops= [2024-01-19T10:06:17.887Z] --license_expiry_timeout_hours=1h0m0s [2024-01-19T10:06:17.887Z] --metering_interval_mins=10m0s [2024-01-19T10:06:17.887Z] --testrail-milestone= [2024-01-19T10:06:17.887Z] --testrail-run-name= [2024-01-19T10:06:17.887Z] --testrail-run-id= [2024-01-19T10:06:17.887Z] --testrail-jeknins-build-url= [2024-01-19T10:06:17.887Z] --testrail-host= [2024-01-19T10:06:17.887Z] --testrail-username= [2024-01-19T10:06:17.887Z] --testrail-password= [2024-01-19T10:06:17.887Z] --jira-username= [2024-01-19T10:06:17.887Z] --jira-token= [2024-01-19T10:06:17.887Z] --jira-account-id= [2024-01-19T10:06:17.887Z] --user= [2024-01-19T10:06:17.887Z] --enable-dash=true [2024-01-19T10:06:17.887Z] --test-desc=px-backup-system-test-4cac76dc [2024-01-19T10:06:17.887Z] --test-type=system-tests [2024-01-19T10:06:17.887Z] --test-tags=flavour:iks,provisioner:iks,cadence:on-demand [2024-01-19T10:06:17.887Z] --testset-id=0 [2024-01-19T10:06:17.887Z] --branch=ibm-mysql [2024-01-19T10:06:17.887Z] --product=px-backup [2024-01-19T10:06:17.887Z] --torpedo-job-name=torpedo-daily-job [2024-01-19T10:06:17.887Z] --torpedo-job-type=functional [2024-01-19T10:06:17.887Z] [2024-01-19T10:06:17.887Z] State: Terminated [2024-01-19T10:06:17.887Z] Reason: Error [2024-01-19T10:06:17.887Z] Exit Code: 1 [2024-01-19T10:06:17.887Z] Started: Thu, 18 Jan 2024 15:34:00 +0000 [2024-01-19T10:06:17.887Z] Finished: Fri, 19 Jan 2024 10:06:14 +0000 [2024-01-19T10:06:17.887Z] Ready: False [2024-01-19T10:06:17.887Z] Restart Count: 0 [2024-01-19T10:06:17.887Z] Environment: [2024-01-19T10:06:17.887Z] NODE_NAME: (v1:spec.nodeName) [2024-01-19T10:06:17.887Z] K8S_VENDOR: [2024-01-19T10:06:17.887Z] TORPEDO_SSH_USER: [2024-01-19T10:06:17.887Z] TORPEDO_SSH_PASSWORD: **** [2024-01-19T10:06:17.887Z] TORPEDO_SSH_KEY: [2024-01-19T10:06:17.887Z] AZURE_TENANT_ID: [2024-01-19T10:06:17.887Z] VOLUME_PROVIDER: none [2024-01-19T10:06:17.887Z] OBJECT_STORE_PROVIDER: aws [2024-01-19T10:06:17.887Z] AZURE_CLIENT_ID: [2024-01-19T10:06:17.887Z] AZURE_CLIENT_SECRET: [2024-01-19T10:06:17.887Z] AZURE_ACCOUNT_NAME: [2024-01-19T10:06:17.887Z] AZURE_ACCOUNT_KEY: [2024-01-19T10:06:17.887Z] AZURE_SUBSCRIPTION_ID: [2024-01-19T10:06:17.887Z] AZURE_CLUSTER_NAME: [2024-01-19T10:06:17.887Z] AWS_ACCESS_KEY_ID: **** [2024-01-19T10:06:17.887Z] AWS_SECRET_ACCESS_KEY: **** [2024-01-19T10:06:17.887Z] AWS_REGION: us-east-1 [2024-01-19T10:06:17.887Z] KUBECONFIGS: source-config,destination-config [2024-01-19T10:06:17.887Z] S3_ENDPOINT: s3.us-east.cloud-object-storage.appdomain.cloud [2024-01-19T10:06:17.887Z] S3_AWS_ACCESS_KEY_ID: [2024-01-19T10:06:17.887Z] S3_AWS_SECRET_ACCESS_KEY: [2024-01-19T10:06:17.887Z] S3_REGION: us-east-1 [2024-01-19T10:06:17.887Z] BUCKET_NAME: px-backup-system-test-4cac76dc [2024-01-19T10:06:17.887Z] LOCKED_BUCKET_NAME: [2024-01-19T10:06:17.887Z] S3_DISABLE_SSL: FALSE [2024-01-19T10:06:17.887Z] DIAGS_BUCKET: [2024-01-19T10:06:17.887Z] PROVIDERS: aws [2024-01-19T10:06:17.887Z] CLUSTER_PROVIDER: ibm [2024-01-19T10:06:17.887Z] INTERNAL_DOCKER_REGISTRY: [2024-01-19T10:06:17.887Z] IMAGE_PULL_SERVER: [2024-01-19T10:06:17.887Z] IMAGE_PULL_USERNAME: [2024-01-19T10:06:17.887Z] IMAGE_PULL_PASSWORD: [2024-01-19T10:06:17.888Z] VSPHERE_USER: [2024-01-19T10:06:17.888Z] VSPHERE_PWD: [2024-01-19T10:06:17.888Z] VSPHERE_HOST_IP: [2024-01-19T10:06:17.888Z] IBMCLOUD_API_KEY: **** [2024-01-19T10:06:17.888Z] CONTROL_PLANE_URL: [2024-01-19T10:06:17.888Z] DS_VERSION: [2024-01-19T10:06:17.888Z] DS_BUILD: [2024-01-19T10:06:17.888Z] NAMESPACE: [2024-01-19T10:06:17.888Z] NO_OF_NODES: [2024-01-19T10:06:17.888Z] DATA_SERVICE: [2024-01-19T10:06:17.888Z] DEPLOY_ALL_VERSIONS: [2024-01-19T10:06:17.888Z] DEPLOY_ALL_IMAGES: [2024-01-19T10:06:17.888Z] DEPLOY_ALL_DATASERVICE: [2024-01-19T10:06:17.888Z] GCP_PROJECT_ID: [2024-01-19T10:06:17.888Z] PDS_USERNAME: [2024-01-19T10:06:17.888Z] PDS_PASSWORD: [2024-01-19T10:06:17.888Z] PDS_CLIENT_SECRET: [2024-01-19T10:06:17.888Z] PDS_CLIENT_ID: [2024-01-19T10:06:17.888Z] PDS_PARAM_CM: [2024-01-19T10:06:17.888Z] PDS_ISSUER_URL: [2024-01-19T10:06:17.888Z] CLUSTER_TYPE: [2024-01-19T10:06:17.888Z] TARGET_KUBECONFIG: [2024-01-19T10:06:17.888Z] TARGET_CLUSTER_NAME: [2024-01-19T10:06:17.888Z] PX_ORACLE_user_ocid: [2024-01-19T10:06:17.888Z] PX_ORACLE_fingerprint: [2024-01-19T10:06:17.888Z] PX_ORACLE_private_key_path: [2024-01-19T10:06:17.888Z] INSTANCE_GROUP: [2024-01-19T10:06:17.888Z] LOGGLY_API_TOKEN: [2024-01-19T10:06:17.888Z] PODMETRIC_METERING_INTERVAL_MINUTES: [2024-01-19T10:06:17.888Z] TARGET_PXBACKUP_VERSION: [2024-01-19T10:06:17.888Z] TARGET_STORK_VERSION: [2024-01-19T10:06:17.888Z] PX_BACKUP_HELM_REPO_BRANCH: [2024-01-19T10:06:17.888Z] BACKUP_TYPE: native_csi [2024-01-19T10:06:17.888Z] NFS_SERVER_ADDR: [2024-01-19T10:06:17.888Z] NFS_SUB_PATH: [2024-01-19T10:06:17.888Z] NFS_MOUNT_OPTION: [2024-01-19T10:06:17.888Z] NFS_PATH: [2024-01-19T10:06:17.888Z] SKIP_PX_OPERATOR_UPGRADE: false [2024-01-19T10:06:17.888Z] VOLUME_SNAPSHOT_CLASS: ibmc-vpcblock-snapshot [2024-01-19T10:06:17.888Z] S3_SSE_TYPE: [2024-01-19T10:06:17.888Z] S3_POLICY_SID: [2024-01-19T10:06:17.888Z] S3_ENCRYPTION_POLICY: [2024-01-19T10:06:17.888Z] Mounts: [2024-01-19T10:06:17.888Z] /testresults/ from testresults (rw) [2024-01-19T10:06:17.888Z] /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-skw6c (ro) [2024-01-19T10:06:17.888Z] Conditions: [2024-01-19T10:06:17.888Z] Type Status [2024-01-19T10:06:17.888Z] Initialized True [2024-01-19T10:06:17.888Z] Ready False [2024-01-19T10:06:17.888Z] ContainersReady False [2024-01-19T10:06:17.888Z] PodScheduled True [2024-01-19T10:06:17.888Z] Volumes: [2024-01-19T10:06:17.888Z] testresults: [2024-01-19T10:06:17.888Z] Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) [2024-01-19T10:06:17.888Z] ClaimName: testresults [2024-01-19T10:06:17.888Z] ReadOnly: false [2024-01-19T10:06:17.888Z] aws-volume: [2024-01-19T10:06:17.888Z] Type: ConfigMap (a volume populated by a ConfigMap) [2024-01-19T10:06:17.888Z] Name: aws-cm [2024-01-19T10:06:17.888Z] Optional: false [2024-01-19T10:06:17.888Z] kube-api-access-skw6c: [2024-01-19T10:06:17.888Z] Type: Projected (a volume that contains injected data from multiple sources) [2024-01-19T10:06:17.888Z] TokenExpirationSeconds: 3607 [2024-01-19T10:06:17.888Z] ConfigMapName: kube-****-ca.crt [2024-01-19T10:06:17.888Z] ConfigMapOptional: [2024-01-19T10:06:17.888Z] DownwardAPI: true [2024-01-19T10:06:17.888Z] ConfigMapName: openshift-service-ca.crt [2024-01-19T10:06:17.888Z] ConfigMapOptional: [2024-01-19T10:06:17.888Z] QoS Class: BestEffort [2024-01-19T10:06:17.888Z] Node-Selectors: [2024-01-19T10:06:17.888Z] Tolerations: apps=false:NoSchedule [2024-01-19T10:06:17.888Z] node-role.kubernetes.io/control-plane op=Exists [2024-01-19T10:06:17.888Z] node-role.kubernetes.io/controlplane=true [2024-01-19T10:06:17.888Z] node-role.kubernetes.io/etcd=true [2024-01-19T10:06:17.888Z] node-role.kubernetes.io/master:NoSchedule [2024-01-19T10:06:17.888Z] node.kubernetes.io/not-ready:NoExecute op=Exists for 300s [2024-01-19T10:06:17.888Z] node.kubernetes.io/unreachable:NoExecute op=Exists for 300s [2024-01-19T10:06:17.888Z] Events: [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Declarative: Post Actions) [Pipeline] script [Pipeline] { [Pipeline] echo [2024-01-19T10:06:18.519Z] Warning: A secret was passed to "echo" using Groovy String interpolation, which is insecure. [2024-01-19T10:06:18.519Z] Affected argument(s) used the following variable(s): [PX_BACKUP_MONGODB_USERNAME, GKE_JSON_KEY] [2024-01-19T10:06:18.520Z] See https://jenkins.io/redirect/groovy-string-interpolation for details. [2024-01-19T10:06:18.520Z] Docker Command: docker run --net=host --rm -t -v /var/run/docker.sock:/var/run/docker.sock -e KUBECONFIG=/kubeconfig/kubeconfig -v /****/workspace/px-backup-system-tests/iks/iks/4cac76dc:/****/workspace/px-backup-system-tests/iks/iks/4cac76dc -v /****/workspace/px-backup-system-tests/iks/iks/4cac76dc/clusters/kubeconfig_src_spawn_gke:/cluster/cloud-source-config -v /****/workspace/px-backup-system-tests/iks/iks/4cac76dc/px-backup/system-tests/gke:/gcp -v /****/workspace/px-backup-system-tests/iks/iks/4cac76dc/gcp.json:/mnt/gcloud/application_default_credentials.json -e KUBECONFIG=/cluster/cloud-source-config -e SCALE_FACTOR=1 -e GKE_CRED=**** -e GOOGLE_APPLICATION_CREDENTIALS=/****/workspace/px-backup-system-tests/iks/iks/4cac76dc/gcp.json portworx/spawn:master kubectl apply -f /****/workspace/px-backup-system-tests/iks/iks/4cac76dc/px-backup/system-tests/busybox-downloader.yaml [Pipeline] sh [2024-01-19T10:06:18.822Z] + docker run --net=host --rm -t -v /var/run/docker.sock:/var/run/docker.sock -e KUBECONFIG=/kubeconfig/kubeconfig -v /****/workspace/px-backup-system-tests/iks/iks/4cac76dc:/****/workspace/px-backup-system-tests/iks/iks/4cac76dc -v /****/workspace/px-backup-system-tests/iks/iks/4cac76dc/clusters/kubeconfig_src_spawn_gke:/cluster/cloud-source-config -v /****/workspace/px-backup-system-tests/iks/iks/4cac76dc/px-backup/system-tests/gke:/gcp -v /****/workspace/px-backup-system-tests/iks/iks/4cac76dc/gcp.json:/mnt/gcloud/application_default_credentials.json -e KUBECONFIG=/cluster/cloud-source-config -e SCALE_FACTOR=1 -e GKE_CRED=**** -e GOOGLE_APPLICATION_CREDENTIALS=/****/workspace/px-backup-system-tests/iks/iks/4cac76dc/gcp.json portworx/spawn:master kubectl apply -f /****/workspace/px-backup-system-tests/iks/iks/4cac76dc/px-backup/system-tests/busybox-downloader.yaml [2024-01-19T10:06:18.822Z] error: error loading config file "/cluster/cloud-source-config": read /cluster/cloud-source-config: is a directory [Pipeline] } [Pipeline] // script Error when executing always post condition: Also: org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: f5893e5c-6227-44e9-ada8-74c92395dfc0 hudson.AbortException: script returned exit code 1 at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.handleExit(DurableTaskStep.java:668) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:614) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:555) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [Pipeline] script [Pipeline] { [Pipeline] echo [2024-01-19T10:06:19.014Z] pxBackupScripts.collectDiags(['path/to/instances.json', 'path/to/instances.json']) [Pipeline] script [Pipeline] { [Pipeline] sh [2024-01-19T10:06:19.388Z] + set -x [2024-01-19T10:06:19.389Z] + curl -s -X POST 10.13.112.242/api/short_url --header Accept: */* --header kbn-xsrf: true --header Content-Type: application/json --header Authorization: Basic --data-raw {"locatorId": "LEGACY_SHORT_URL_LOCATOR","params": {"url": "/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-3d%2Fd,to:now))&_a=(columns:!(message),filters:!(),index:%270db7ea41-3b01-4658-8ae0-6446dbc31a53%27,interval:auto,query:(language:kuery,query:%27system-test.id%20:%204cac76dc%27),sort:!(!(%27@timestamp%27,desc)))"}} [2024-01-19T10:06:19.389Z] + jq -r .slug [Pipeline] } [Pipeline] // script [Pipeline] sh [2024-01-19T10:06:23.064Z] + curl -s https://jenkins.pwx.dev.purestorage.com/job/portworx-backup/job/system-tests/job/byoc/job/Cloud_BYOC/42/testReport/api/json [2024-01-19T10:06:23.064Z] + jq .passCount [2024-01-19T10:06:23.064Z] parse error: Invalid numeric literal at line 1, column 18 [Pipeline] } [Pipeline] // script Error when executing failure post condition: Also: org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 537db689-d111-4156-8ab9-192d30e8a8c3 hudson.AbortException: script returned exit code 4 at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.handleExit(DurableTaskStep.java:668) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:614) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:555) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timestamps [Pipeline] } [Pipeline] // ansiColor [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // ws [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline ERROR: script returned exit code 1 Finished: FAILURE