METRON-1571 Correct KAFKA_TAIL Seek to End Logic

[nickwallen]

Changes

1. KAFKA_TAIL now performs manual partition assignment and correctly uses the Kafka API to seek to the end of the topic.

   Previously, the function messed with the 'group.id' to effectively seek to end. This is an abuse of the API. Also, a fixed group.id is required for using the KAFKA_* functions in a Kerberized cluster.

2. The unit tests for KAFKA_TAIL were improved.

3. KAKFA_GET and KAFKA_TAIL now use a more accurate mechanism for adhering to the user's requested max wait time.

   Previously, the max wait was divided by the max poll timeout to estimate how many times Kafka should be polled before returning to the user. This method is not always accurate when some poll requests return values and others do not.

4. The MPack now defines a global property bootstrap.servers which allow the KAFKA_* functions to work out-of-the-box after a Metron installation.

   Previously, a user had to manually define this property before using the KAKFA_* functions. If the property is not defined correctly, they would have to wait an excessive amount of time for the connection to timeout since there is no means in the REPL to interrupt a function call.

Testing

1. Launch the development environment.

2. Launch the REPL in the development environment.

3. Tail some messages from Kafka

   KAFKA_TAIL("bro")
   

4. Get some messages from Kafka.

   KAFKA_GET("bro")
   

Pull Request Checklist

• Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
• Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
• Has your PR been rebased against the latest commit within the target branch (typically master)?
• Have you included steps to reproduce the behavior or problem that is being changed or addressed?
• Have you included steps or a guide to how the change may be verified and tested manually?
• Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
• Have you written or updated unit tests and or integration tests to verify your changes?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

[merrimanr]

Documentation is very clear and everything worked perfectly. +1