METRON-1266 Profiler - SASL Authentication Failed

[nickwallen]

When running the Profiler on a cluster that has multiple nodes and is secured by Kerberos, it was observed that the HBaseBolt was unable to write to HBase. The Storm worker running the HBaseBolt logged the following exception. This does not occur all the time and does not occur in all environments.

2017-10-19 14:51:00.146 o.a.h.h.i.AbstractRpcClient [ERROR] SASL authentication failed. 
The most likely cause is missing or invalid credentials. Consider 'kinit'.
javax.security.sasl.SaslException: GSS initiate failed
...

Changes

To fix this, the topology.auto-credentials property needs to be set on the Profiler topology when running in a Kerberized environment. This is similar to how the other topologies, like Enrichment, are already configured.

After finding this, the big mystery for me was why this bug did not cause this issue in all kerberized environments, all the time. Surely this miss should break the Profiler when running in any Kerberized environment and so should have been caught sooner.

The problem is obviously that a ticket cannot be found to authenticate when attempting to flush profile measurements to HBase. Due to this configuration miss, the Profiler topology itself is not able to generate Kerberos tickets for authentication. At the same time, if the ticket cache on the worker node is already populated with a valid ticket, then this issue will not occur. The ticket cache can be populated if another process generates a ticket or a user manually kinits on the same node.

This explains why the problem occurs sporadically and only in some environments. This issue is less likely to occur in an environment, like Full Dev, where there are fewer, more active nodes. In this case, it is likely that some other process or user already pre-populated the ticket cache. In a larger, multi-node cluster, the ticket cache is less likely to be populated.

That's my working theory at least. Feel free to refute.

Testing

I tested this by applying the fix in a 12 node Metron cluster. This fixed the problem and allowed the Profiler to write to HBase. I also tested this in Full Dev on both plain vanilla mode and after kerberization.

To test this change, follow these steps.

1. Stand-up Full Dev and run the Metron Service Check.
2. Using Ambari change the profiler duration from 15 minutes to 1 minute. Then restart the Profiler.
3. Create a simple Profile following these instructions.
4. Wait a few minutes for the Profiler to gather data and flush.
5. In the Stellar REPL, run PROFILE_GET to retrieve the data from HBase. Ensure that data can be retrieved.
6. Kerberize the cluster and run the Metron Service Check.
7. Update the Bro sensor stub at /opt/sensor-stubs/bin/start-bro-stub and include --security-protocol=SASL_PLAINTEXT as an argument to the kafka-console-producer.sh command. Then run the script in a terminal so that data is flowing through Metron. Also see this for more information.
8. Wait a few minutes for the Profiler to gather data and flush.
9. In the Stellar REPL, run PROFILE_GET to retrieve the data from HBase. Ensure that new data is being written, post-kerberization.

Pull Request Checklist

• Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
• Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
• Has your PR been rebased against the latest commit within the target branch (typically master)?
• Have you included steps to reproduce the behavior or problem that is being changed or addressed?
• Have you included steps or a guide to how the change may be verified and tested manually?
• Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
• Have you written or updated unit tests and or integration tests to verify your changes?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

[cestella]

oh god, that's weird. +1 by inspection, let's get this in master asap