Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Flake] Scylla Manager integration [It] should register cluster and sync repair tasks #1694

Closed
Tracked by #1897
zimnx opened this issue Jan 18, 2024 · 30 comments · Fixed by #1850
Closed
Tracked by #1897

[Flake] Scylla Manager integration [It] should register cluster and sync repair tasks #1694

zimnx opened this issue Jan 18, 2024 · 30 comments · Fixed by #1850
Assignees
@rzetelskik
Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.

Comments

@zimnx
Copy link
Collaborator

zimnx commented Jan 18, 2024

Link to the job that flaked.

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1691/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1747926847733108736

Snippet of what failed.

 Scylla Manager integration [It] should discover cluster and sync tasks
github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllamanager.go:23
  Timeline >>
  STEP: Creating a new namespace @ 01/18/24 10:36:49.96
  Jan 18 10:36:50.156: INFO: Created namespace "e2e-test-scyllacluster-pgwg6-tvn8t".
  STEP: Waiting for service account token Secret "e2e-user-token" in namespace "e2e-test-scyllacluster-pgwg6-tvn8t". @ 01/18/24 10:36:50.448
  STEP: Waiting for default ServiceAccount in namespace "e2e-test-scyllacluster-pgwg6-tvn8t". @ 01/18/24 10:36:51.262
  STEP: Waiting for kube-root-ca.crt in namespace "e2e-test-scyllacluster-pgwg6-tvn8t". @ 01/18/24 10:36:52.008
  STEP: Creating a ScyllaCluster @ 01/18/24 10:36:52.154
  STEP: Waiting for the ScyllaCluster to rollout (RV=6490) @ 01/18/24 10:36:52.205
  Jan 18 10:41:27.531: INFO: ScyllaCluster e2e-test-scyllacluster-pgwg6-tvn8t/basic-9lr8s (RV=12525) is rolled out
  STEP: Verifying the ScyllaCluster @ 01/18/24 10:41:27.532
  Jan 18 10:41:27.756: INFO: Found 1 pvc(s) in namespace "e2e-test-scyllacluster-pgwg6-tvn8t"
  Jan 18 10:41:27.756: INFO: Found 1 pvc(s) for ScyllaCluster "e2e-test-scyllacluster-pgwg6-tvn8t/basic-9lr8s"
  STEP: Waiting for the ScyllaCluster(s) to reach consistency ALL @ 01/18/24 10:41:27.902
  Jan 18 10:41:28.039: INFO: ScyllaDB nodes have reached status consistency.
  STEP: Inserting data @ 01/18/24 10:41:28.076
  Jan 18 10:41:28.076: INFO: Creating CQL session (hosts="10.17.109.246")
  STEP: Inserting data @ 01/18/24 10:41:28.083
  Jan 18 10:41:28.083: INFO: Creating keyspace "257x4fqc" with RF "'replication_factor': 1"
  Jan 18 10:41:28.106: INFO: Creating table "257x4fqc"."test"
  Jan 18 10:41:28.178: INFO: Inserting data into table "257x4fqc"."test"
  Jan 18 10:41:28.192: INFO: Awaiting schema agreement
  Jan 18 10:41:28.193: INFO: Schema agreement reached
  STEP: Verifying the data @ 01/18/24 10:41:28.193
  Jan 18 10:41:28.193: INFO: Reading data from table "257x4fqc"."test"
  STEP: Waiting for the cluster sync with Scylla Manager @ 01/18/24 10:41:28.196
  [FAILED] in [It] - github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllamanager.go:91 @ 01/18/24 10:45:28.197
  STEP: Collecting events from namespace "e2e-test-scyllacluster-pgwg6-tvn8t". @ 01/18/24 10:45:28.197
  STEP: Found 37 events. @ 01/18/24 10:45:28.259
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:36:52 +0000 UTC - event for basic-9lr8s-auth-token: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-pgwg6-tvn8t/basic-9lr8s-auth-token created
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:36:52 +0000 UTC - event for basic-9lr8s-member: {scyllacluster-controller } RoleBindingCreated: RoleBinding e2e-test-scyllacluster-pgwg6-tvn8t/basic-9lr8s-member created
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:36:52 +0000 UTC - event for basic-9lr8s-member: {scyllacluster-controller } ServiceAccountCreated: ServiceAccount e2e-test-scyllacluster-pgwg6-tvn8t/basic-9lr8s-member created
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:37:21 +0000 UTC - event for basic-9lr8s-local-client-ca: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-pgwg6-tvn8t/basic-9lr8s-local-client-ca created
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:37:21 +0000 UTC - event for basic-9lr8s-local-client-ca: {scyllacluster-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-pgwg6-tvn8t/basic-9lr8s-local-client-ca created
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:37:53 +0000 UTC - event for basic-9lr8s-local-user-admin: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-pgwg6-tvn8t/basic-9lr8s-local-user-admin created
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:38:26 +0000 UTC - event for basic-9lr8s-local-serving-ca: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-pgwg6-tvn8t/basic-9lr8s-local-serving-ca created
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:38:26 +0000 UTC - event for basic-9lr8s-local-serving-ca: {scyllacluster-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-pgwg6-tvn8t/basic-9lr8s-local-serving-ca created
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:38:55 +0000 UTC - event for basic-9lr8s-us-east-1-us-east-1a: {scyllacluster-controller } StatefulSetCreated: StatefulSet e2e-test-scyllacluster-pgwg6-tvn8t/basic-9lr8s-us-east-1-us-east-1a created
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:38:55 +0000 UTC - event for basic-9lr8s-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Claim data-basic-9lr8s-us-east-1-us-east-1a-0 Pod basic-9lr8s-us-east-1-us-east-1a-0 in StatefulSet basic-9lr8s-us-east-1-us-east-1a success
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:38:55 +0000 UTC - event for basic-9lr8s-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Pod basic-9lr8s-us-east-1-us-east-1a-0 in StatefulSet basic-9lr8s-us-east-1-us-east-1a successful
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:38:55 +0000 UTC - event for data-basic-9lr8s-us-east-1-us-east-1a-0: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "local.csi.scylladb.com" or manually created by system administrator
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:38:55 +0000 UTC - event for data-basic-9lr8s-us-east-1-us-east-1a-0: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:38:55 +0000 UTC - event for data-basic-9lr8s-us-east-1-us-east-1a-0: {local.csi.scylladb.com_local-csi-driver-vw75k_077e1607-d57f-4bf0-9ea9-c018698718b1 } Provisioning: External provisioner is provisioning volume for claim "e2e-test-scyllacluster-pgwg6-tvn8t/data-basic-9lr8s-us-east-1-us-east-1a-0"
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:38:55 +0000 UTC - event for data-basic-9lr8s-us-east-1-us-east-1a-0: {local.csi.scylladb.com_local-csi-driver-vw75k_077e1607-d57f-4bf0-9ea9-c018698718b1 } ProvisioningSucceeded: Successfully provisioned volume pvc-551c8aa9-b04f-400b-a85a-7669e987425f
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:38:56 +0000 UTC - event for basic-9lr8s-us-east-1-us-east-1a-0: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-pgwg6-tvn8t/basic-9lr8s-us-east-1-us-east-1a-0 to gke-so-60dbe5af-d46f-471e-b31-workers-fe3df9b2-lkf0
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:38:56 +0000 UTC - event for basic-9lr8s-us-east-1-us-east-1a-0: {kubelet gke-so-60dbe5af-d46f-471e-b31-workers-fe3df9b2-lkf0} FailedMount: MountVolume.SetUp failed for volume "scylladb-serving-certs" : secret "basic-9lr8s-local-serving-certs" not found
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:38:56 +0000 UTC - event for nodeconfig-podinfo-026dd60d-0e98-41fd-a59d-ec5f89cd5583: {NodeConfigCM-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-pgwg6-tvn8t/nodeconfig-podinfo-026dd60d-0e98-41fd-a59d-ec5f89cd5583 created
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:39:05 +0000 UTC - event for basic-9lr8s: {scyllacluster-controller } PodDisruptionBudgetCreated: PodDisruptionBudget e2e-test-scyllacluster-pgwg6-tvn8t/basic-9lr8s created
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:39:05 +0000 UTC - event for basic-9lr8s-client: {scyllacluster-controller } ServiceCreated: Service e2e-test-scyllacluster-pgwg6-tvn8t/basic-9lr8s-client created
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:39:05 +0000 UTC - event for basic-9lr8s-us-east-1-us-east-1a-0: {scyllacluster-controller } ServiceCreated: Service e2e-test-scyllacluster-pgwg6-tvn8t/basic-9lr8s-us-east-1-us-east-1a-0 created
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:39:32 +0000 UTC - event for basic-9lr8s-local-cql-connection-configs-admin: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-pgwg6-tvn8t/basic-9lr8s-local-cql-connection-configs-admin created
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:39:32 +0000 UTC - event for basic-9lr8s-local-serving-certs: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-pgwg6-tvn8t/basic-9lr8s-local-serving-certs created
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:40:03 +0000 UTC - event for basic-9lr8s-us-east-1-us-east-1a-0: {kubelet gke-so-60dbe5af-d46f-471e-b31-workers-fe3df9b2-lkf0} Pulled: Container image "quay.io/scylladb-dev/ci:scylla-operator-pr-1691-a21ec5c750d37946e622d7622825285af5f0ad7d" already present on machine
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:40:14 +0000 UTC - event for basic-9lr8s-us-east-1-us-east-1a-0: {kubelet gke-so-60dbe5af-d46f-471e-b31-workers-fe3df9b2-lkf0} Created: Created container sidecar-injection
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:40:15 +0000 UTC - event for basic-9lr8s-us-east-1-us-east-1a-0: {kubelet gke-so-60dbe5af-d46f-471e-b31-workers-fe3df9b2-lkf0} Started: Started container sidecar-injection
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:40:16 +0000 UTC - event for basic-9lr8s-local-serving-certs: {scyllacluster-controller } SecretUpdated: Secret e2e-test-scyllacluster-pgwg6-tvn8t/basic-9lr8s-local-serving-certs updated
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:40:32 +0000 UTC - event for basic-9lr8s-us-east-1-us-east-1a-0: {kubelet gke-so-60dbe5af-d46f-471e-b31-workers-fe3df9b2-lkf0} Created: Created container scylla-manager-agent
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:40:32 +0000 UTC - event for basic-9lr8s-us-east-1-us-east-1a-0: {kubelet gke-so-60dbe5af-d46f-471e-b31-workers-fe3df9b2-lkf0} Pulled: Container image "docker.io/scylladb/scylla-manager-agent:3.2.5" already present on machine
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:40:32 +0000 UTC - event for basic-9lr8s-us-east-1-us-east-1a-0: {kubelet gke-so-60dbe5af-d46f-471e-b31-workers-fe3df9b2-lkf0} Started: Started container scylla
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:40:32 +0000 UTC - event for basic-9lr8s-us-east-1-us-east-1a-0: {kubelet gke-so-60dbe5af-d46f-471e-b31-workers-fe3df9b2-lkf0} Created: Created container scylla
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:40:32 +0000 UTC - event for basic-9lr8s-us-east-1-us-east-1a-0: {kubelet gke-so-60dbe5af-d46f-471e-b31-workers-fe3df9b2-lkf0} Pulled: Container image "docker.io/scylladb/scylla:5.4.0" already present on machine
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:40:33 +0000 UTC - event for basic-9lr8s-us-east-1-us-east-1a-0: {kubelet gke-so-60dbe5af-d46f-471e-b31-workers-fe3df9b2-lkf0} Started: Started container scylla-manager-agent
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:40:35 +0000 UTC - event for nodeconfig-podinfo-026dd60d-0e98-41fd-a59d-ec5f89cd5583: {NodeConfigCM-controller } ConfigMapUpdated: ConfigMap e2e-test-scyllacluster-pgwg6-tvn8t/nodeconfig-podinfo-026dd60d-0e98-41fd-a59d-ec5f89cd5583 updated
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:40:41 +0000 UTC - event for basic-9lr8s-us-east-1-us-east-1a-0: {kubelet gke-so-60dbe5af-d46f-471e-b31-workers-fe3df9b2-lkf0} Unhealthy: Readiness probe failed: HTTP probe failed with statuscode: 500
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:40:50 +0000 UTC - event for basic-9lr8s-us-east-1-us-east-1a-0: {kubelet gke-so-60dbe5af-d46f-471e-b31-workers-fe3df9b2-lkf0} Unhealthy: Readiness probe failed: HTTP probe failed with statuscode: 503
  Jan 18 10:45:28.259: INFO: At 2024-01-18 10:41:17 +0000 UTC - event for basic-9lr8s-us-east-1-us-east-1a-0: {scyllacluster-controller } ServiceUpdated: Service e2e-test-scyllacluster-pgwg6-tvn8t/basic-9lr8s-us-east-1-us-east-1a-0 updated
  STEP: Collecting dumps from namespace "e2e-test-scyllacluster-pgwg6-tvn8t". @ 01/18/24 10:45:28.259
  STEP: Destroying namespace "e2e-test-scyllacluster-pgwg6-tvn8t". @ 01/18/24 10:45:28.893
  STEP: Waiting for namespace "e2e-test-scyllacluster-pgwg6-tvn8t" to be removed. @ 01/18/24 10:45:28.938
  << Timeline
  [FAILED] Unexpected error:
      <wait.errInterrupted>: 
      timed out waiting for the condition
      {
          cause: <*errors.errorString | 0xc000431420>{
              s: "timed out waiting for the condition",
          },
      }
  occurred
  In [It] at: github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllamanager.go:91 @ 01/18/24 10:45:28.197
  Full Stack Trace
    github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster.glob..func18.1()
    	github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllamanager.go:91 +0xb4b
@zimnx zimnx added the kind/flake Categorizes issue or PR as related to a flaky test. label Jan 18, 2024
@scylla-operator-bot scylla-operator-bot bot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Jan 18, 2024
@zimnx zimnx mentioned this issue Jan 18, 2024
Merged
@zimnx zimnx added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Jan 18, 2024
@scylla-operator-bot scylla-operator-bot bot removed the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Jan 18, 2024
@tnozicka
Copy link
Member

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1700/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1750073547096592384#1:test-build-log.txt%3A996

@tnozicka tnozicka mentioned this issue Jan 24, 2024
Merged
@zimnx
Copy link
Collaborator Author

zimnx commented Jan 26, 2024

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1703/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1750468988573323264

@zimnx zimnx mentioned this issue Jan 26, 2024
Merged
@zimnx
Copy link
Collaborator Author

zimnx commented Jan 31, 2024

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1707/pull-scylla-operator-master-e2e-gke-parallel/1752375983240908800

@zimnx zimnx mentioned this issue Jan 31, 2024
Merged
@zimnx
Copy link
Collaborator Author

zimnx commented Feb 6, 2024

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1712/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1754867749869326336

@zimnx zimnx mentioned this issue Feb 6, 2024
Merged
@zimnx
Copy link
Collaborator Author

zimnx commented Feb 20, 2024

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1738/pull-scylla-operator-master-e2e-gke-parallel/1759928957475491840

@zimnx zimnx mentioned this issue Feb 20, 2024
Merged
@zimnx
Copy link
Collaborator Author

zimnx commented Feb 20, 2024

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1740/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1759939699566186496

@zimnx zimnx mentioned this issue Feb 20, 2024
Merged
@tnozicka
Copy link
Member

tnozicka commented Mar 7, 2024

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1795/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1765693251337064448#1:test-build-log.txt%3A2254

@tnozicka tnozicka mentioned this issue Mar 7, 2024
Merged
@zimnx
Copy link
Collaborator Author

zimnx commented Mar 12, 2024

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1804/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1767240191401005056

@zimnx zimnx mentioned this issue Mar 12, 2024
Merged
@tnozicka
Copy link
Member

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1818/pull-scylla-operator-master-e2e-gke-parallel/1768248040348454912#1:test-build-log.txt%3A1965

@tnozicka tnozicka mentioned this issue Mar 14, 2024
Merged
@tnozicka
Copy link
Member

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1846/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1770098564370796544#1:test-build-log.txt%3A1259

@tnozicka tnozicka mentioned this issue Mar 20, 2024
Merged
@rzetelskik
Copy link
Member

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1754/pull-scylla-operator-master-e2e-gke-parallel/1769731412795068416#1:test-build-log.txt%3A1073

@rzetelskik rzetelskik mentioned this issue Mar 20, 2024
Merged
@zimnx
Copy link
Collaborator Author

zimnx commented Mar 26, 2024

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1824/pull-scylla-operator-master-e2e-gke-parallel/1772307332387049472

@zimnx zimnx mentioned this issue Mar 26, 2024
Merged
@zimnx
Copy link
Collaborator Author

zimnx commented Mar 26, 2024

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1824/pull-scylla-operator-master-e2e-gke-parallel/1772635224547528704

@tnozicka tnozicka mentioned this issue Apr 3, 2024
Merged
@rzetelskik
Copy link
Member

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1880/pull-scylla-operator-master-e2e-gke-parallel/1775876717043257344#1:test-build-log.txt%3A1620

@rzetelskik rzetelskik mentioned this issue Apr 4, 2024
Merged
@rzetelskik
Copy link
Member

Since I have the questionable pleasure of working on the manager controller now, I volunteer.

/assign

@rzetelskik
Copy link
Member

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1881/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1775852197246930944#1:test-build-log.txt%3A1481

@rzetelskik rzetelskik mentioned this issue Apr 11, 2024
Merged
@rzetelskik
Copy link
Member

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1891/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1779805444886237184#1:test-build-log.txt%3A948

@rzetelskik
Copy link
Member

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1891/pull-scylla-operator-master-e2e-gke-parallel/1779805444856877056#1:test-build-log.txt%3A1604

@rzetelskik rzetelskik mentioned this issue Apr 15, 2024
Merged
@rzetelskik
Copy link
Member

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1891/pull-scylla-operator-master-e2e-gke-parallel/1779828276999491584#1:test-build-log.txt%3A1480

@rzetelskik
Copy link
Member

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1891/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1779832848220426240#1:test-build-log.txt%3A1366

@rzetelskik
Copy link
Member

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1868/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1779871557737254912#1:test-build-log.txt%3A1584

This was referenced Apr 15, 2024
Merged
Closed
@zimnx zimnx mentioned this issue Apr 17, 2024
Open
@rzetelskik
Copy link
Member

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1754/pull-scylla-operator-master-e2e-gke-parallel/1780249855176740864#1:test-build-log.txt%3A1441

@rzetelskik rzetelskik changed the title Flake - Scylla Manager integration [It] should discover cluster and sync tasks [Flake] Scylla Manager integration [It] should register cluster and sync repair tasks Apr 18, 2024
@rzetelskik rzetelskik mentioned this issue Apr 18, 2024
Closed
@tnozicka
Copy link
Member

@rzetelskik have you managed to take a look? Do we know what needs to be fixed / does this depend on the manager annotation support?

@rzetelskik
Copy link
Member

@rzetelskik have you managed to take a look? Do we know what needs to be fixed

It's generally caused by an entirely broken reconciliation logic. See e.g. #1752. Most of the recent flakes were due to the deletions not being propagated to status, which is not surprising if you look into the code - reconciliation loop iterates over the state taken from the manager and schedules actions based on the differences between it and the spec/status. So if it successfully deletes a task in manager, but hits a conflict when updating the status, well, that's just that, the task is not going to be removed from status any time later, because it's now gone from manager.

#1850 should address most of the flakiness. It's now waiting for #1851, #1853.

does this depend on the manager annotation support?

Maybe, but the above needs to be fixed first anyway. I was told to touch as little of the existing logic as possible, so I didn't touch anything related to cluster creation/deletion logic e.g., or updating the tasks for that matter.

@tnozicka
Copy link
Member

no worries and thanks. (I primarily asked to see have a list of PR to prioritize.)

@rzetelskik
Copy link
Member

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1851/pull-scylla-operator-master-e2e-gke-parallel/1780935955910234112#1:test-build-log.txt%3A1463

@rzetelskik rzetelskik mentioned this issue Apr 19, 2024
Merged
@tnozicka
Copy link
Member

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1855/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1781240963608350720#1:test-build-log.txt%3A1352

@rzetelskik rzetelskik mentioned this issue May 6, 2024
Merged
@rzetelskik rzetelskik linked a pull request May 6, 2024 that will close this issue
Fix manager task reconciliation and status synchronisation #1850
Merged
@tnozicka
Copy link
Member

tnozicka commented May 7, 2024

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1475/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1787755121241755648#1:test-build-log.txt%3A1178

@tnozicka tnozicka mentioned this issue May 7, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rzetelskik rzetelskik

Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix manager task reconciliation and status synchronisation rzetelskik/scylla-operator
3 participants
@zimnx @tnozicka @rzetelskik