KAFKA-5173: Log jaas.config if broker fails to start

[rajinisivaram]

No description provided.

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/3567/
Test FAILed (JDK 7 and Scala 2.10).

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/3569/
Test PASSed (JDK 7 and Scala 2.10).

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/3565/
Test FAILed (JDK 8 and Scala 2.12).

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/3575/
Test FAILed (JDK 8 and Scala 2.11).

[ijuma]

The additional logging:

2017-05-05 22:08:40,058] ERROR Jaas config /tmp/kafka9139469497082854641.tmp exists? true, KafkaServer context is: null (kafka.utils.TestUtils$:99)
[2017-05-05 22:08:40,060] ERROR Jaas config file contents: KafkaServer {
  com.sun.security.auth.module.Krb5LoginModule required
  debug=true
  serviceName="kafka"
  keyTab="/tmp/kafka3843243500191588648.tmp"
  principal="kafka/localhost@EXAMPLE.COM"
  storeKey="true"
  useKeyTab="true";

};
KafkaClient {
  com.sun.security.auth.module.Krb5LoginModule required
  debug=true
  serviceName="kafka"
  keyTab="/tmp/kafka4115104151315724578.tmp"
  principal="client2@EXAMPLE.COM"
  storeKey="true"
  useKeyTab="true";

};
Server {
  org.apache.zookeeper.server.auth.DigestLoginModule required
  debug=false
  user_super="adminpasswd"
  user_fpj="fpjsecret";

};
Client {
  org.apache.zookeeper.server.auth.DigestLoginModule required
  debug=false
  username="fpj"
  password="fpjsecret";

}; (kafka.utils.TestUtils$:99)

[ijuma]

I think it's a case of not clearing the cached Configuration before the relevant code is executed.

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/3712/
Test PASSed (JDK 7 and Scala 2.10).

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/3718/
Test PASSed (JDK 8 and Scala 2.11).

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/3708/
Test PASSed (JDK 8 and Scala 2.12).

[rajinisivaram]

retest this please

[rajinisivaram]

@ijuma Can you take a look at this? Thanks.

While running tests locally, I found three threads being left behind by the broker after shutdown. This could be one cause for failures in time-sensitve tests like the quota tests that are failing fairly frequently now. As far as I can tell, these threads should never be reloading JAAS config, so they are unlikely to be causing the jaas.config issue.

The PR logs JAAS sections and thread stack traces on failure. The last set of tests passed, I am waiting for a failure to see the thread dump.

[ijuma]

@rajinisivaram, your thread leak changes look good. Maybe submit them as a separate PR?

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/3723/
Test PASSed (JDK 8 and Scala 2.11).

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/3717/
Test PASSed (JDK 7 and Scala 2.10).

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/3713/
Test PASSed (JDK 8 and Scala 2.12).

[rajinisivaram]

@ijuma Yes, will do that under KAFKA-5182. Thanks.

[rajinisivaram]

retest this please

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/3727/
Test FAILed (JDK 8 and Scala 2.11).

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/3717/
Test PASSed (JDK 8 and Scala 2.12).

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/3721/
Test PASSed (JDK 7 and Scala 2.10).

[rajinisivaram]

retest this please

[rajinisivaram]

Closing this PR. Thread leak was fixed under #3014. JAAS config logging isn't necessary since none of the issues found were related to JAAS config being incorrect. Instead of thread dump on failure, PR #3026 fails if broker controller has not been shutdown. This is better since thread dump at the time of failure will not provide any indication about which tests may have caused the thread leak.

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/3779/
Test PASSed (JDK 7 and Scala 2.11).