investigator reports kafka issues in thoth-infra-stage

[fridex]

Describe the bug

%4|1648816923.918|CONFWARN|rdkafka#producer-1| [thrd:app]: Configuration property enable.auto.commit is a consumer property and will be ignored by this producer instance
%3|1648816923.923|FAIL|rdkafka#producer-1| [thrd:ssl://thoth-kafka-bootstrap-aicoe.apps.ocp4.prod.psi.redhat.com]: ssl://thoth-kafka-bootstrap-aicoe.apps.ocp4.prod.psi.redhat.com:443/bootstrap: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 1ms in state SSL_HANDSHAKE)
%3|1648816924.920|FAIL|rdkafka#producer-1| [thrd:ssl://thoth-kafka-bootstrap-aicoe.apps.ocp4.prod.psi.redhat.com]: ssl://thoth-kafka-bootstrap-aicoe.apps.ocp4.prod.psi.redhat.com:443/bootstrap: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 0ms in state SSL_HANDSHAKE, 1 identical error(s) suppressed)
%3|1648816954.926|FAIL|rdkafka#producer-1| [thrd:ssl://thoth-kafka-bootstrap-aicoe.apps.ocp4.prod.psi.redhat.com]: ssl://thoth-kafka-bootstrap-aicoe.apps.ocp4.prod.psi.redhat.com:443/bootstrap: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 0ms in state SSL_HANDSHAKE, 30 identical error(s) suppressed)
%3|1648816984.927|FAIL|rdkafka#producer-1| [thrd:ssl://thoth-kafka-bootstrap-aicoe.apps.ocp4.prod.psi.redhat.com]: ssl://thoth-kafka-bootstrap-aicoe.apps.ocp4.prod.psi.redhat.com:443/bootstrap: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 0ms in state SSL_HANDSHAKE, 30 identical error(s) suppressed)
%3|1648817014.929|FAIL|rdkafka#producer-1| [thrd:ssl://thoth-kafka-bootstrap-aicoe.apps.ocp4.prod.psi.redhat.com]: ssl://thoth-kafka-bootstrap-aicoe.apps.ocp4.prod.psi.redhat.com:443/bootstrap: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 0ms in state SSL_HANDSHAKE, 30 identical error(s) suppressed)
%3|1648817044.934|FAIL|rdkafka#producer-1| [thrd:ssl://thoth-kafka-bootstrap-aicoe.apps.ocp4.prod.psi.redhat.com]: ssl://thoth-kafka-bootstrap-aicoe.apps.ocp4.prod.psi.redhat.com:443/bootstrap: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 0ms in state SSL_HANDSHAKE, 30 identical error(s) suppressed)
%3|1648817074.936|FAIL|rdkafka#producer-1| [thrd:ssl://thoth-kafka-bootstrap-aicoe.apps.ocp4.prod.psi.redhat.com]: ssl://thoth-kafka-bootstrap-aicoe.apps.ocp4.prod.psi.redhat.com:443/bootstrap: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 0ms in state SSL_HANDSHAKE, 30 identical error(s) suppressed)
%3|1648817104.939|FAIL|rdkafka#producer-1| [thrd:ssl://thoth-kafka-bootstrap-aicoe.apps.ocp4.prod.psi.redhat.com]: ssl://thoth-kafka-bootstrap-aicoe.apps.ocp4.prod.psi.redhat.com:443/bootstrap: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 0ms in state SSL_HANDSHAKE, 30 identical error(s) suppressed

To Reproduce
Steps to reproduce the behavior:

1. Schedule an advise in ocp4-stage environment
2. See the advise keeps in pending state
3. See advise gets never triggered

Expected behavior

Advises should work in ocp4-stage

[harshad16]

/triage accepted

[harshad16]

/lifecycle active
/assign

seems like the kafka is crashlooping.

[harshad16]

The amq streams operator is updated to new version.
we need to check on the requirements and upgrade our Kafka to fit to these necessities

[fridex]

Perfect, thanks for digging into this. CC @mayaCostantini

[harshad16]

CRDs have changed, the whole kafka manifest requires upgrade.
https://access.redhat.com/documentation/en-us/red_hat_amq/2021.q2/html/deploying_and_upgrading_amq_streams_on_openshift/assembly-upgrade-str#assembly-upgrade-resources-str

also version needed to be bumped to:
https://access.redhat.com/documentation/en-us/red_hat_amq_streams/2.0/html/deploying_and_upgrading_amq_streams_on_openshift/assembly-upgrade-str#kafka-versions

[harshad16]

This is all set however, there are sometimes strange happening in the infra namespace:
The quota seems to be full, however, the pods are not running.

[harshad16]

All fixed and the kafka setup is back 👍

[fridex]

@harshad16 perfect, thanks a lot 💯