Security: SASL compatibility

[simplesteph]

It seems that this tool can't be used in a secure cluster, as we need to be authenticated to the zookeeper cluster using either a super user (using a digest) or using the principal that kafka client is using.

We need to have options (not exactly sure which) to deal with secure kafka clusters

Example of stacktrace:

root@ip-10-13-80-10:/# kafka-assigner -z zoo1:2181/kafka-xfs -e set-replication-factor --topic __consumer_offsets --replication-factor 3
[INFO] Connecting to zookeeper zoo1:2181/kafka-xfs
[INFO] Getting partition list from Zookeeper
[INFO] Closing connection to zookeeper
[INFO] Partition moves required: 50
[INFO] Number of batches: 5
[INFO] Executing partition reassignment 1/5: {"version": 1, "partitions": [{"topic": "__consumer_offsets", "partition": 0, "replicas": [2, 1, 3]}, {"topic": "__consumer_offsets", "partition": 1, "replicas": [2, 1, 3]}, {"topic": "__consumer_offsets", "partition": 2, "replicas": [2, 1, 3]}, {"topic": "__consumer_offsets", "partition": 3, "replicas": [2, 1, 3]}, {"topic": "__consumer_offsets", "partition": 4, "replicas": [2, 1, 3]}, {"topic": "__consumer_offsets", "partition": 5, "replicas": [2, 1, 3]}, {"topic": "__consumer_offsets", "partition": 6, "replicas": [2, 1, 3]}, {"topic": "__consumer_offsets", "partition": 7, "replicas": [2, 1, 3]}, {"topic": "__consumer_offsets", "partition": 8, "replicas": [2, 1, 3]}, {"topic": "__consumer_offsets", "partition": 9, "replicas": [2, 1, 3]}]}
Traceback (most recent call last):
  File "/usr/local/bin/kafka-assigner", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/kafka/tools/assigner/__main__.py", line 124, in main
    batch.execute(i + 1, len(batches), args.zookeeper, tools_path, plugins, dry_run)
  File "/usr/local/lib/python2.7/dist-packages/kafka/tools/assigner/models/reassignment.py", line 63, in execute
    remaining_partitions = check_reassignment_completion(zookeeper, tools_path, assignfile.name)
  File "/usr/local/lib/python2.7/dist-packages/kafka/tools/assigner/models/reassignment.py", line 92, in check_reassignment_completion
    raise ReassignmentFailedException("The reassignment in progress failed with the following verification output:\n{0}".format(lines))
kafka.tools.assigner.exceptions.ReassignmentFailedException: The reassignment in progress failed with the following verification output:
['Status of partition reassignment: \n', "ERROR: Assigned replicas (2) don't match the list of replicas for reassignment (2,1,3) for partition [__consumer_offsets,7]\n", "ERROR: Assigned replicas (2) don't match the list of replicas for reassignment (2,1,3) for partition [__consumer_offsets,4]\n", "ERROR: Assigned replicas (2) don't match the list of replicas for reassignment (2,1,3) for partition [__consumer_offsets,3]\n", "ERROR: Assigned replicas (2) don't match the list of replicas for reassignment (2,1,3) for partition [__consumer_offsets,0]\n", "ERROR: Assigned replicas (2) don't match the list of replicas for reassignment (2,1,3) for partition [__consumer_offsets,8]\n", "ERROR: Assigned replicas (2) don't match the list of replicas for reassignment (2,1,3) for partition [__consumer_offsets,5]\n", "ERROR: Assigned replicas (2) don't match the list of replicas for reassignment (2,1,3) for partition [__consumer_offsets,6]\n", "ERROR: Assigned replicas (2) don't match the list of replicas for reassignment (2,1,3) for partition [__consumer_offsets,2]\n", "ERROR: Assigned replicas (2) don't match the list of replicas for reassignment (2,1,3) for partition [__consumer_offsets,9]\n", "ERROR: Assigned replicas (2) don't match the list of replicas for reassignment (2,1,3) for partition [__consumer_offsets,1]\n", 'Reassignment of partition [__consumer_offsets,7] failed\n', 'Reassignment of partition [__consumer_offsets,4] failed\n', 'Reassignment of partition [__consumer_offsets,3] failed\n', 'Reassignment of partition [__consumer_offsets,0] failed\n', 'Reassignment of partition [__consumer_offsets,8] failed\n', 'Reassignment of partition [__consumer_offsets,5] failed\n', 'Reassignment of partition [__consumer_offsets,6] failed\n', 'Reassignment of partition [__consumer_offsets,2] failed\n', 'Reassignment of partition [__consumer_offsets,9] failed\n', 'Reassignment of partition [__consumer_offsets,1] failed\n']

[simplesteph]

Actually if you prefix your command with
KAFKA_OPTS="-Djava.security.krb5.conf=/etc/kafka/krb5.conf -Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf"
it works. That's because the Kafka Tool is using the java binaries