METRON-819: Document kafka console producer parameter for sensors with kerberos

[mmiklavc]

Addresses https://issues.apache.org/jira/browse/METRON-819

Adds a note about adding the security protocol property to sensors leveraging the Kafka console producer.

Pull Request Checklist

Thank you for submitting a contribution to Apache Metron (Incubating).
Please refer to our Development Guidelines for the complete guide to follow for contributions.
Please refer also to our Build Verification Guidelines for complete smoke testing guides.

In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:

For all changes:

• Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
• Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
• Has your PR been rebased against the latest commit within the target branch (typically master)?

For documentation related changes:

• Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html:

  cd site-book
  bin/generate-md.sh
  mvn site:site
  

Note:

Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommened that travis-ci is set up for your personal repository such that your branches are built there before submitting a pull request.

[nickwallen]

I went through your instructions and all seemed well with the world. But then I tried to use the kafka-console-producer to actually write data to Kafka and it fails Any ideas what the problem might be?

[metron@node1 ~]$ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
[metron@node1 ~]$ echo "foo" | kafka-console-producer.sh --broker-list node1:6667 --topic yaf --security-protocol SASL_PLAINTEXT
[2017-04-07 16:29:00,639] WARN The TGT cannot be renewed beyond the next expiry date: Sat Apr 08 16:28:58 UTC 2017.This process will not be able to authenticate new SASL connections after that time (for example, it will not be able to authenticate a new connection with a Kafka Broker).  Ask your system administrator to either increase the 'renew until' time by doing : 'modprinc -maxrenewlife null ' within kadmin, or instead, to generate a keytab for null. Because the TGT's expiry cannot be further extended by refreshing, exiting refresh thread now. (org.apache.kafka.common.security.kerberos.KerberosLogin)
[2017-04-07 16:29:00,897] WARN Error while fetching metadata with correlation id 0 : {yaf=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
[2017-04-07 16:29:00,897] ERROR Error when sending message to topic yaf with key: null, value: 3 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [yaf]

I then tried to go back and check the Kafka ACLs and am now getting an error. I was able to set the ACLs, but now I cannot see them.

[metron@node1 ~]$ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
[metron@node1 ~]$ kafka-acls.sh --list --topic yaf --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181
[2017-04-07 16:24:47,794] WARN Could not login: the client is being asked for a password, but the Zookeeper client code does not currently support obtaining a password from the user. Make sure that the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' and restart the client. If you still get this message after that, the TGT in the ticket cache has expired and must be manually refreshed. To do so, first determine if you are using a password or a keytab. If the former, run kinit in a Unix shell in the environment of the user who is running this Zookeeper client using the command 'kinit <princ>' (where <princ> is the name of the client's Kerberos principal). If the latter, do 'kinit -k -t <keytab> <princ>' (where <princ> is the name of the Kerberos principal, and <keytab> is the location of the keytab file). After manually refreshing your cache, restart this client. If you continue to see this message after manually refreshing your cache, ensure that your KDC host's clock is in sync with this host's clock. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2017-04-07 16:24:47,796] WARN SASL configuration failed: javax.security.auth.login.LoginException: No password provided Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn)
Error while executing ACL command: Authentication failure
org.I0Itec.zkclient.exception.ZkAuthFailedException: Authentication failure
	at org.I0Itec.zkclient.ZkClient.waitForKeeperState(ZkClient.java:946)

[nickwallen]

@mmiklavc Can you try listing and applying acls with the root user instead of metron?

Ok, sure.

1. As root, I can see the ACLs. But oddly there are none set.

[root@node1 ~]# kafka-acls.sh --list --topic yaf --authorizer-properties zookeeper.connect=node1:2181
Current ACLs for resource `Topic:yaf`:

2. Then I set the ACLs again. So this looks pretty good. Now it looks like they were set.

[root@node1 ~]# kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:metron --group yaf_parser;
Adding ACLs for resource `Group:yaf_parser`:
 	User:metron has Allow permission for operations: All from hosts: *

Current ACLs for resource `Group:yaf_parser`:
 	User:metron has Allow permission for operations: All from hosts: *

3. But then if I immediately check them again and they do not exist. Weird.

[root@node1 ~]# kafka-acls.sh --list --topic yaf --authorizer-properties zookeeper.connect=node1:2181 --authorizer kafka.security.auth.SimpleAclAuthorizer
Current ACLs for resource `Topic:yaf`:

[cestella]

@nickwallen Anything going on in the kafka broker logs in /var/log/...?

[nickwallen]

The issues that I am having currently are with Quick Dev. But I have actually been able to do this on a separate cluster in a slightly different way. On the other cluster, I did not use the --group option when setting the ACL. If I did set the group, then I had to ensure that the group matched what was used by the kafka-console-producer.

So as a test, I granted access without the --group on Quick Dev.

1. Grant access. Look ma, no group.

[root@node1 ~]# kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:metron --topic yaf
Adding ACLs for resource `Topic:yaf`:
 	User:metron has Allow permission for operations: All from hosts: *

Current ACLs for resource `Topic:yaf`:
 	User:metron has Allow permission for operations: All from hosts: *

2. Validate the ACL. Looks good this time.

[root@node1 ~]# kafka-acls.sh --list --topic yaf --authorizer-properties zookeeper.connect=node1:2181 --authorizer kafka.security.auth.SimpleAclAuthorizer
Current ACLs for resource `Topic:yaf`:
 	User:metron has Allow permission for operations: All from hosts: *

3. And now I can send data successfully.

[root@node1 ~]# echo "foo" | kafka-console-producer.sh --broker-list node1:6667 --topic yaf --security-protocol SASL_PLAINTEXT
[2017-04-07 17:05:28,830] WARN The TGT cannot be renewed beyond the next expiry date: Sat Apr 08 16:11:26 UTC 2017.This process will not be able to authenticate new SASL connections after that time (for example, it will not be able to authenticate a new connection with a Kafka Broker).  Ask your system administrator to either increase the 'renew until' time by doing : 'modprinc -maxrenewlife null ' within kadmin, or instead, to generate a keytab for null. Because the TGT's expiry cannot be further extended by refreshing, exiting refresh thread now. (org.apache.kafka.common.security.kerberos.KerberosLogin)

[nickwallen]

Anything going on in the kafka broker logs in /var/log/...?

@cestella Nothing interesting that I can find in the logs, unfortunately.

[nickwallen]

I think I am confusing steps (12) and (13) from your instructions or something. But something else weird is going on. I'm just not sure what.

It seems like the ACLs were set and then at some point they got unset somehow. I'm going to start over and walk through it all again. Maybe I made a mistake.

[mattf-apache]

Is Ambari in the mix

Urm, sorry. That was old behavior, they changed it so Ambari doesn't do that any more. It prevented too many useful config tweaks. Now, ambari-agent only resets config state to match the ambari database, at startup time and when configs in Ambari are changed.

[nickwallen]

FYI I was able to get this working. Mike's docs are 100% correct, there were just a few minor steps that tripped me up (like using relative paths instead of absolute paths.) I updated those just to help others avoid the same stupid user mistakes.

[nickwallen]

Should we call out the need to kinit beforehand?

[mmiklavc]

Added the kinit

[mmiklavc]

@nickwallen Thanks for testing the Kerberos instructions out!

[nickwallen]

+1 Rockin!

[mmiklavc]

Merge with master again

[mmiklavc]

Don't merge this - need to move this doc change to the new manual kerberos doc in metron-deployment dir. A fix for a bad merge is pending.

[nickwallen]

I also remember (after fighting with this for a while this morning) that if you don't have a JAAS config file defined then it won't work. Should this be part of the docs in this PR??

Without it, you just get this error.

[root@y136 ~]# kafka-console-consumer.sh  --zookeeper y113:2181 --topic bro --security-protocol SASL_PLAINTEXT
[2017-04-28 16:47:20,596] WARN Could not login: the client is being asked for a password, but the Zookeeper client code does not currently support obtaining a password from the user. Make sure that the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' and restart the client. If you still get this message after that, the TGT in the ticket cache has expired and must be manually refreshed. To do so, first determine if you are using a password or a keytab. If the former, run kinit in a Unix shell in the environment of the user who is running this Zookeeper client using the command 'kinit <princ>' (where <princ> is the name of the client's Kerberos principal). If the latter, do 'kinit -k -t <keytab> <princ>' (where <princ> is the name of the Kerberos principal, and <keytab> is the location of the keytab file). After manually refreshing your cache, restart this client. If you continue to see this message after manually refreshing your cache, ensure that your KDC host's clock is in sync with this host's clock. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2017-04-28 16:47:20,599] WARN SASL configuration failed: javax.security.auth.login.LoginException: No password provided Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn)
No brokers found in ZK.

After doing the following, then it works for me.

1. Define ~/.java.login.config

[root@y137 ~]# cat ~/.java.login.config
KafkaClient {
  com.sun.security.auth.module.Krb5LoginModule required
  useTicketCache=false
  useKeyTab=true
  principal="yaf/y137...@EXAMPLE.COM"
  keyTab="/etc/security/keytabs/yaf.service.keytab"
  renewTicket=true
  debug=true
  serviceName="kafka"
  storeKey=true;
};

2. Tell the JVM where to find your JAAS file.

[root@y137 ~]# cat /usr/lib/jvm/java-1.8.0-openjdk/jre/lib/security/java.security | grep "login.config.url.1"
login.config.url.1=file:${user.home}/.java.login.config

[mmiklavc]

I didn't have any problems using the Kafka shell commands without a JAAS file. That was on single node Vagrant, but I'm not sure there should be much of a difference in this case.

[mmiklavc]

@nickwallen can you elaborate on this fix/config a bit? I think we should definitely add this detail to the doc. It looks like you've created a yaf user and principal here. Any additional setup or configuration required? I'm also curious about the difference in setup that required the jaas file here versus the original setup that allowed you to use the console consumer successfully per the comments above.

[nickwallen]

I'm also curious about the difference in setup that required the jaas file here versus the original setup that allowed you to use the console consumer successfully per the comments above.

In the original test, the JAAS stuff was configured as a by-product of something else I was working on. So I have never gotten this to work without configuring JAAS.

[nickwallen]

can you elaborate on this fix/config a bit? I think we should definitely add this detail to the doc. It looks like you've created a yaf user and prin

In the text that I pasted, I was switching between two different use cases. So I accidentally pasted the JAAS setup for my 'yaf' principal, versus the 'bro' principal. So ignore that.

But I cannot think of anything beyond the two steps I defined above. I just create a JAAS file telling it which keytab and principal to use, then update the JVM's security config so that it knows where to find the JAAS file, then it works for me.

This is just the one way I have gotten it to work. There may be a better/simpler way that I am not aware of.

[justinleet]

@mmiklavc Can you take care of the conflicts?

[mmiklavc]

@justinleet - I'm changing this entirely per @nickwallen's comments about the jaas file. Coming shortly.

[mmiklavc]

Ok, I grabbed the latest master branch and force pushed out a new commit with the doc changes to this branch.

@nickwallen @justinleet can you take a peek at this and see if there isn't anything else I've missed? Tested this by virtue of the internal performance evaluation we ran through recently.

[justinleet]

I'm +1, by inspection. I think this covers it. I'll let @nickwallen double check if there's anything else.

[nickwallen]

+1 look great. Thanks