BAS deployment breaks lite-core-roks playbook

[durera]

Please deliver fixes to this branch: https://github.com/ibm-mas/ansible-devops/tree/bugfixes2411

1st run:

TASK [ibm.mas_devops.bas_install : Wait for FullDeployment to be ready (60s delay)] ********************************************************************************************************************************************************
Wednesday 24 November 2021  15:52:40 +0000 (0:00:02.010)       0:03:43.775 ****
FAILED - RETRYING: Wait for FullDeployment to be ready (60s delay) (40 retries left).
FAILED - RETRYING: Wait for FullDeployment to be ready (60s delay) (39 retries left).
... <snip> ...
FAILED - RETRYING: Wait for FullDeployment to be ready (60s delay) (3 retries left).
FAILED - RETRYING: Wait for FullDeployment to be ready (60s delay) (2 retries left).
FAILED - RETRYING: Wait for FullDeployment to be ready (60s delay) (1 retries left).
fatal: [localhost]: FAILED! => {"api_found": true, "attempts": 40, "changed": false, "resources": [{"apiVersion": "bas.ibm.com/v1", "kind": "FullDeployment", "metadata": {"creationTimestamp": "2021-11-24T15:52:39Z", "generation": 1, "managedFields": [{"apiVersion": "bas.ibm.com/v1", "fieldsType": "FieldsV1", "fieldsV1": {"f:spec": {".": {}, "f:airgapped": {".": {}, "f:backup_deletion_frequency": {}, "f:backup_retention_period": {}, "f:enabled": {}}, "f:allowed_domains": {}, "f:db_archive": {".": {}, "f:frequency": {}, "f:persistent_storage": {".": {}, "f:storage_class": {}, "f:storage_size": {}}, "f:retention_age": {}}, "f:env_type": {}, "f:event_scheduler_frequency": {}, "f:ibmproxyurl": {}, "f:image_pull_secret": {}, "f:kafka": {".": {}, "f:storage_class": {}, "f:storage_size": {}, "f:zookeeper_storage_class": {}, "f:zookeeper_storage_size": {}}, "f:postgres": {".": {}, "f:storage_class": {}, "f:storage_size": {}}, "f:prometheus_metrics": {}, "f:prometheus_scheduler_frequency": {}}, "f:status": {"f:phase": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2021-11-24T15:53:49Z"}, {"apiVersion": "bas.ibm.com/v1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {".": {}, "f:conditions": {}}}, "manager": "ansible-operator", "operation": "Update", "time": "2021-11-24T16:01:43Z"}], "name": "fulldeployment", "namespace": "ibm-bas", "resourceVersion": "133659", "selfLink": "/apis/bas.ibm.com/v1/namespaces/ibm-bas/fulldeployments/fulldeployment", "uid": "299ab75b-61f5-4751-97d7-b50840e5c1a7"}, "spec": {"airgapped": {"backup_deletion_frequency": "@daily", "backup_retention_period": 7, "enabled": "false"}, "allowed_domains": "*", "db_archive": {"frequency": "@monthly", "persistent_storage": {"storage_class": "", "storage_size": "10G"}, "retention_age": 6}, "env_type": "lite", "event_scheduler_frequency": "*/10 * * * *", "ibmproxyurl": "https://iaps.ibm.com", "image_pull_secret": "bas-images-pull-secret", "kafka": {"storage_class": "", "storage_size": "5G", "zookeeper_storage_class": "", "zookeeper_storage_size": "5G"}, "postgres": {"storage_class": "", "storage_size": "10G"}, "prometheus_metrics": [], "prometheus_scheduler_frequency": "@daily"}, "status": {"conditions": [{"ansibleResult": {"changed": 12, "completion": "2021-11-24T16:01:43.152198", "failures": 1, "ok": 17, "skipped": 3}, "lastTransitionTime": "2021-11-24T16:01:43Z", "message": "Failed to find exact match for kafka.strimzi.io/v1beta2.Kafka by [kind, name, singularName, shortNames]", "reason": "Failed", "status": "False", "type": "Failure"}, {"lastTransitionTime": "2021-11-24T16:01:43Z", "message": "Running reconciliation", "reason": "Running", "status": "True", "type": "Running"}], "phase": "Installing"}}]}

NO MORE HOSTS LEFT *************************************************************************************************************************************************************************************************************************

PLAY RECAP *********************************************************************************************************************************************************************************************************************************
localhost                  : ok=98   changed=22   unreachable=0    failed=1    skipped=32   rescued=0    ignored=0

Wednesday 24 November 2021  16:33:58 +0000 (0:41:17.379)       0:45:01.155 ****
===============================================================================

2nd run the next day, different failure at the same place:

FAILED - RETRYING: Wait for FullDeployment to be ready (60s delay) (1 retries left).
fatal: [localhost]: FAILED! => {"api_found": true, "attempts": 40, "changed": false, "resources": [{"apiVersion": "bas.ibm.com/v1", "kind": "FullDeployment", "metadata": {"creationTimestamp": "2021-11-24T15:52:39Z", "generation": 1, "managedFields": [{"apiVersion": "bas.ibm.com/v1", "fieldsType": "FieldsV1", "fieldsV1": {"f:spec": {".": {}, "f:airgapped": {".": {}, "f:backup_deletion_frequency": {}, "f:backup_retention_period": {}, "f:enabled": {}}, "f:allowed_domains": {}, "f:db_archive": {".": {}, "f:frequency": {}, "f:persistent_storage": {".": {}, "f:storage_class": {}, "f:storage_size": {}}, "f:retention_age": {}}, "f:env_type": {}, "f:event_scheduler_frequency": {}, "f:ibmproxyurl": {}, "f:image_pull_secret": {}, "f:kafka": {".": {}, "f:storage_class": {}, "f:storage_size": {}, "f:zookeeper_storage_class": {}, "f:zookeeper_storage_size": {}}, "f:postgres": {".": {}, "f:storage_class": {}, "f:storage_size": {}}, "f:prometheus_metrics": {}, "f:prometheus_scheduler_frequency": {}}, "f:status": {"f:phase": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2021-11-24T15:53:49Z"}, {"apiVersion": "bas.ibm.com/v1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {".": {}, "f:conditions": {}}}, "manager": "ansible-operator", "operation": "Update", "time": "2021-11-25T12:03:28Z"}], "name": "fulldeployment", "namespace": "ibm-bas", "resourceVersion": "846474", "selfLink": "/apis/bas.ibm.com/v1/namespaces/ibm-bas/fulldeployments/fulldeployment", "uid": "299ab75b-61f5-4751-97d7-b50840e5c1a7"}, "spec": {"airgapped": {"backup_deletion_frequency": "@daily", "backup_retention_period": 7, "enabled": "false"}, "allowed_domains": "*", "db_archive": {"frequency": "@monthly", "persistent_storage": {"storage_class": "", "storage_size": "10G"}, "retention_age": 6}, "env_type": "lite", "event_scheduler_frequency": "*/10 * * * *", "ibmproxyurl": "https://iaps.ibm.com", "image_pull_secret": "bas-images-pull-secret", "kafka": {"storage_class": "", "storage_size": "5G", "zookeeper_storage_class": "", "zookeeper_storage_size": "5G"}, "postgres": {"storage_class": "", "storage_size": "10G"}, "prometheus_metrics": [], "prometheus_scheduler_frequency": "@daily"}, "status": {"conditions": [{"ansibleResult": {"changed": 1, "completion": "2021-11-25T12:03:28.405458", "failures": 1, "ok": 18, "skipped": 4}, "lastTransitionTime": "2021-11-25T12:03:28Z", "message": "unknown playbook failure", "reason": "Failed", "status": "False", "type": "Failure"}, {"lastTransitionTime": "2021-11-25T12:03:28Z", "message": "Running reconciliation", "reason": "Running", "status": "True", "type": "Running"}], "phase": "Installing"}}]}

NO MORE HOSTS LEFT *************************************************************************************************************************************************************************************************************************

PLAY RECAP *********************************************************************************************************************************************************************************************************************************
localhost                  : ok=98   changed=13   unreachable=0    failed=1    skipped=32   rescued=0    ignored=0

ibm-bas namespace is left in this state:

/mnt/c/Users/DaveParker$ oc get pods -n ibm-bas
NAME                                                      READY   STATUS      RESTARTS   AGE
amq-streams-cluster-operator-v1.7.3-6dc4b4d74d-fn62k      1/1     Running     0          20h
backrest-backup-instrumentationdb-r9c8c                   0/1     Completed   0          15h
behavior-analytics-services-operator-d5f6f5899-s8qvd      2/2     Running     0          20h
createcluster-2zg6f                                       0/1     Completed   0          20h
dashboard-deployment-66c874dbbb-wksj4                     2/2     Running     0          20h
instrumentationdb-7b69964d8c-9cjbb                        1/1     Running     0          20h
instrumentationdb-backrest-shared-repo-75566dfd69-wz44f   1/1     Running     0          20h
instrumentationdb-stanza-create-crv5s                     0/1     Completed   0          20h
kafka-zookeeper-0                                         0/1     Pending     0          20h
pgo-deploy-ln6lx                                          0/1     Completed   0          20h
postgres-operator-6c585b8c78-mzz8v                        4/4     Running     0          20h

[andrercm]

@durera are you running which ocp version? ocp 4.8?

[andrercm]

@durera should be fixed in #71

[andrercm]

@durera thanks for confirming, I'll close this issue then.