CI Failure (RpkException - SASL authentication failed) in `SchemaRegistryAutoAuthTest.test_schema_id_validation` #11203

andijcr · 2023-06-05T11:54:59Z

https://buildkite.com/redpanda/vtools/builds/7916#0188856f-e4e9-4ee3-944e-95a01dd43a36

Module: rptest.tests.schema_registry_test
Class:  SchemaRegistryAutoAuthTest
Method: test_schema_id_validation
Arguments:
{
  "client_type": 1,
  "payload_class": "com.redpanda.A.B.C.D.NestedPayload",
  "protocol": 2,
  "subject_name_strategy": "io.confluent.kafka.serializers.subject.TopicRecordNameStrategy",
  "validate_schema_id": true
}

====================================================================================================
test_id:    rptest.tests.schema_registry_test.SchemaRegistryAutoAuthTest.test_schema_id_validation.protocol=PROTOBUF.client_type=Python.validate_schema_id=True.subject_name_strategy=SubjectNameStrategyCompat.TOPIC_RECORD_NAME.payload_class=com.redpanda.A.B.C.D.NestedPayload
status:     FAIL
run time:   7.094 seconds


    RpkException('command /opt/redpanda/bin/rpk topic --brokers ip-172-31-8-110:9092,ip-172-31-4-136:9092,ip-172-31-11-200:9092 --user admin --password admin --sasl-mechanism SCRAM-SHA-256 create serde-topic-PROTOBUF-Python --partitions 1 --replicas 1 --topic-config confluent.value.schema.validation:true --topic-config confluent.value.subject.name.strategy:io.confluent.kafka.serializers.subject.TopicNameStrategy returned 1, output: ', 'unable to create topics [serde-topic-PROTOBUF-Python]: SASL authentication failed: security: Invalid credentials: SASL_AUTHENTICATION_FAILED: SASL Authentication failed.\n', 1)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 481, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 79, in wrapped
    r = f(self, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/tests/schema_registry_test.py", line 1295, in test_schema_id_validation
    self._create_topic(
  File "/home/ubuntu/redpanda/tests/rptest/tests/schema_registry_test.py", line 228, in _create_topic
    rpk_tools.create_topic(topic=topic,
  File "/home/ubuntu/redpanda/tests/rptest/clients/rpk.py", line 232, in create_topic
    wait_until_result(create_topic,
  File "/home/ubuntu/redpanda/tests/rptest/util.py", line 88, in wait_until_result
    wait_until(wrapped_condition, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 53, in wait_until
    raise e
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 44, in wait_until
    if condition():
  File "/home/ubuntu/redpanda/tests/rptest/util.py", line 75, in wrapped_condition
    cond = condition()
  File "/home/ubuntu/redpanda/tests/rptest/clients/rpk.py", line 229, in create_topic
    raise e
  File "/home/ubuntu/redpanda/tests/rptest/clients/rpk.py", line 223, in create_topic
    output = self._run_topic(cmd)
  File "/home/ubuntu/redpanda/tests/rptest/clients/rpk.py", line 739, in _run_topic
    return self._execute(cmd, stdin=stdin, timeout=timeout)
  File "/home/ubuntu/redpanda/tests/rptest/clients/rpk.py", line 857, in _execute
    raise RpkException(
rptest.clients.rpk.RpkException: RpkException<command /opt/redpanda/bin/rpk topic --brokers ip-172-31-8-110:9092,ip-172-31-4-136:9092,ip-172-31-11-200:9092 --user admin --password admin --sasl-mechanism SCRAM-SHA-256 create serde-topic-PROTOBUF-Python --partitions 1 --replicas 1 --topic-config confluent.value.schema.validation:true --topic-config confluent.value.subject.name.strategy:io.confluent.kafka.serializers.subject.TopicNameStrategy returned 1, output:  error: unable to create topics [serde-topic-PROTOBUF-Python]: SASL authentication failed: security: Invalid credentials: SASL_AUTHENTICATION_FAILED: SASL Authentication failed.
 returncode: 1>

The text was updated successfully, but these errors were encountered:

andijcr · 2023-06-05T11:57:15Z

similar errors from another testcase:
test_serde_client:
https://buildkite.com/redpanda/redpanda/builds/30521#01887ff8-4af2-4fbd-9cbc-f16a02ab2397
https://buildkite.com/redpanda/redpanda/builds/30451#01887c7a-3456-40d7-a82f-6b9c33a97b06

andijcr · 2023-06-07T14:29:29Z

https://buildkite.com/redpanda/redpanda/builds/30741#0188934a-2fa5-4f28-9cfb-91b20654fe8c
https://buildkite.com/redpanda/redpanda/builds/30747#018893b5-39f7-4283-ab66-361a421f333b

andijcr · 2023-06-09T13:40:29Z

https://buildkite.com/redpanda/redpanda/builds/30935#01889ec8-335b-4bc2-8161-311ac4533641

andijcr · 2023-06-09T13:42:40Z

https://buildkite.com/redpanda/redpanda/builds/30940#01889ed5-156c-4534-a1b9-74f5ab2b6d07

NyaliaLui · 2023-06-09T14:23:24Z

May be related to #11141

BenPope · 2023-06-09T15:15:50Z

Similar errors exist in SchemaRegistryAutoAuthTest.test_serde_client

michael-redpanda · 2023-06-12T19:46:42Z

NyaliaLui · 2023-06-12T21:02:55Z

So I sync'd with @BenPope and we both agreed that this failure is not happening because of a problem with AutoAuth or ephemeral credentials. Instead, this problem is happening because of a timing issue between RedpandaService auto-creating the admin user and RPK issuing a CreateTopic request.

I continued the investigation to see if using a different client to create topics solves the problem, so I tried using KafkaCliTools.
The result was that all test runs passed.

I continued my experiments by forcing the RpkTool to retry on SASL_AUTHENTICATION_FAILED. The result was that all test runs passed.

This tells me that our RpkTool may be issuing a CreateTopics request before the admin user finishes replicating to other nodes or something like that.

Nevertheless, this seems like a test issue. Therefore, I am assigning sev/low tag and I am looking into ways to have our tests "wait-to-issue-kafka-requests" until the admin user has replicated across the cluster. A simple solution would be to merge the retry on CreateTopics; however, I would like to see if there is another solution that may be more robust.

In tests that enable SASL auth, it is possible for RPK to issue a CreateTopics requests before the user has replicated across the cluster. The solution is to retry the request. Fixes redpanda-data#11203

andijcr added kind/bug Something isn't working ci-failure labels Jun 5, 2023

piyushredpanda added the area/kafka label Jun 5, 2023

michael-redpanda assigned michael-redpanda and unassigned michael-redpanda Jun 7, 2023

BenPope mentioned this issue Jun 8, 2023

schema_registry: Schema ID Validation low-level tests #11297

Merged

7 tasks

NyaliaLui self-assigned this Jun 9, 2023

NyaliaLui added the sev/low Bugs which are non-functional paper cuts, e.g. typos, issues in log messages label Jun 12, 2023

BenPope mentioned this issue Jun 13, 2023

schema_registry: Switch Schema ID Validation enablement to cluster config #11284

Merged

7 tasks

NyaliaLui mentioned this issue Jun 13, 2023

tests/rpk: auto-retry on SASL auth fail #11390

Merged

7 tasks

dotnwat closed this as completed in #11390 Jun 14, 2023

andijcr mentioned this issue Jun 14, 2023

[cloud_storage] segment_meta_cstore::materialize: revert find to lower_bound #11404

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI Failure (RpkException - SASL authentication failed) in `SchemaRegistryAutoAuthTest.test_schema_id_validation` #11203

CI Failure (RpkException - SASL authentication failed) in `SchemaRegistryAutoAuthTest.test_schema_id_validation` #11203

andijcr commented Jun 5, 2023

andijcr commented Jun 5, 2023

andijcr commented Jun 7, 2023 •

edited

Loading

andijcr commented Jun 9, 2023

andijcr commented Jun 9, 2023

NyaliaLui commented Jun 9, 2023

BenPope commented Jun 9, 2023

michael-redpanda commented Jun 12, 2023

NyaliaLui commented Jun 12, 2023 •

edited

Loading

CI Failure (RpkException - SASL authentication failed) in SchemaRegistryAutoAuthTest.test_schema_id_validation #11203

CI Failure (RpkException - SASL authentication failed) in SchemaRegistryAutoAuthTest.test_schema_id_validation #11203

Comments

andijcr commented Jun 5, 2023

andijcr commented Jun 5, 2023

andijcr commented Jun 7, 2023 • edited Loading

andijcr commented Jun 9, 2023

andijcr commented Jun 9, 2023

NyaliaLui commented Jun 9, 2023

BenPope commented Jun 9, 2023

michael-redpanda commented Jun 12, 2023

NyaliaLui commented Jun 12, 2023 • edited Loading

CI Failure (RpkException - SASL authentication failed) in `SchemaRegistryAutoAuthTest.test_schema_id_validation` #11203

CI Failure (RpkException - SASL authentication failed) in `SchemaRegistryAutoAuthTest.test_schema_id_validation` #11203

andijcr commented Jun 7, 2023 •

edited

Loading

NyaliaLui commented Jun 12, 2023 •

edited

Loading