From 8f44e2732159bcf615f5883a9c48e7776954ac98 Mon Sep 17 00:00:00 2001 From: Mohan Venkateshaiah Date: Tue, 31 Jul 2018 15:20:31 +0530 Subject: [PATCH 1/2] Update steps to run the pycapa on kerberized cluster. --- metron-sensors/pycapa/README.md | 56 +++++++++++++++++++++++---------- 1 file changed, 39 insertions(+), 17 deletions(-) diff --git a/metron-sensors/pycapa/README.md b/metron-sensors/pycapa/README.md index 7e688a8fca..dfd08e8f11 100644 --- a/metron-sensors/pycapa/README.md +++ b/metron-sensors/pycapa/README.md @@ -34,9 +34,9 @@ Pycapa performs network packet capture, both off-the-wire and from a Kafka topic Installation ============ -General notes on the installation of Pycapa. +General notes on the installation of Pycapa. * Python 2.7 is required. -* The following package dependencies are required and can be installed automatically with `pip`. +* The following package dependencies are required and can be installed automatically with `pip`. The requirements are installed as part of step 4 * [confluent-kafka-python](https://github.com/confluentinc/confluent-kafka-python) * [pcapy](https://github.com/CoreSecurity/pcapy) * These instructions can be used directly on CentOS 7+. @@ -54,8 +54,8 @@ General notes on the installation of Pycapa. ``` export PREFIX=/usr - wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O - | tar -xz - cd librdkafka-0.9.4/ + wget https://github.com/edenhill/librdkafka/archive/v0.11.5.tar.gz -O - | tar -xz + cd librdkafka-0.11.5/ ./configure --prefix=$PREFIX make make install @@ -239,10 +239,27 @@ The probe can be used in a Kerberized environment. Follow these additional step * The keytab used is located at `/etc/security/keytabs/metron.headless.keytab` * The service principal is `metron@EXAMPLE.COM` + If it is not, ensure that you have `libsasl` or `libsasl2` installed. On CentOS, this can be installed with the following command. + ``` + yum install -y cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi + ``` + + 1. Use the non-binary wheel to install confluent-kafka-python. + + The Python client README (https://github.com/confluentinc/confluent-kafka-python) has an important note for Kerberos case: + ``` + **NOTE:** The pre-built Linux wheels do NOT contain SASL Kerberos support. + If you need SASL Kerberos support you must install librdkafka and + its dependencies using the above repositories and then build + confluent-kafka from source. + ``` + + so you will need to use the non-binary wheel to install confluent-kafka-python and build/install librdkafka separately + 1. Build Librdkafka with SASL support (` --enable-sasl`) and install at your chosen $PREFIX. ``` - wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O - | tar -xz - cd librdkafka-0.9.4/ + wget https://github.com/edenhill/librdkafka/archive/v0.11.5.tar.gz -O - | tar -xz + cd librdkafka-0.11.5/ ./configure --prefix=$PREFIX --enable-sasl make make install @@ -250,14 +267,19 @@ The probe can be used in a Kerberized environment. Follow these additional step 1. Validate Librdkafka does indeed support SASL. Run the following command and ensure that `sasl` is returned as a built-in feature. ``` - $ examples/rdkafka_example -X builtin.features - builtin.features = gzip,snappy,ssl,sasl,regex - ``` - - If it is not, ensure that you have `libsasl` or `libsasl2` installed. On CentOS, this can be installed with the following command. - ``` - yum install -y cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi + $ examples/rdkafka_example -X builtin.features + builtin.features = gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins ``` +1. The source install of confluent-kafka: + + ``` + // If you have already installed, remove the binary wheel python client first, repeat until it says no longer installed + pip uninstall -y confluent-kafka + ``` + + ``` + pip install --no-binary :all: confluent-kafka + ``` 1. Grant access to your Kafka topic. In this example the topic is simply named `pcap`. ``` @@ -279,8 +301,8 @@ The probe can be used in a Kerberized environment. Follow these additional step * `security.protocol` * `sasl.kerberos.keytab` * `sasl.kerberos.principal` - - ``` + + ``` $ pycapa --producer \ --interface eth0 \ --kafka-broker kafka1:6667 \ @@ -292,8 +314,8 @@ The probe can be used in a Kerberized environment. Follow these additional step INFO:root:Starting packet capture INFO:root:Waiting for '1' message(s) to flush INFO:root:'10' packet(s) in, '10' packet(s) out - ``` - + ``` + FAQs ==== From 15b11b0cd8b1040cb0db847616cae0bd43dbea93 Mon Sep 17 00:00:00 2001 From: Mohan Venkateshaiah Date: Wed, 1 Aug 2018 18:22:13 +0530 Subject: [PATCH 2/2] incorporated review comments. --- metron-sensors/pycapa/README.md | 29 +++++++++-------------------- 1 file changed, 9 insertions(+), 20 deletions(-) diff --git a/metron-sensors/pycapa/README.md b/metron-sensors/pycapa/README.md index dfd08e8f11..9f01d336ed 100644 --- a/metron-sensors/pycapa/README.md +++ b/metron-sensors/pycapa/README.md @@ -231,30 +231,19 @@ Capturing on 'Standard input' ### Kerberos -The probe can be used in a Kerberized environment. Follow these additional steps to use Pycapa with Kerberos. The following assumptions have been made. These may need altered to fit your environment. +The probe can be used in a Kerberized environment. The Python client README (https://github.com/confluentinc/confluent-kafka-python) has an important note for Kerberos case that the pre-built Linux wheels do NOT contain SASL Kerberos support. You will need to use the non-binary wheel to install confluent-kafka-python and build/install librdkafka separately. Follow these additional steps to use Pycapa with Kerberos. The following assumptions have been made. These may need altered to fit your environment. * The Kafka broker is at `kafka1:6667` * Zookeeper is at `zookeeper1:2181` * The Kafka security protocol is `SASL_PLAINTEXT` * The keytab used is located at `/etc/security/keytabs/metron.headless.keytab` * The service principal is `metron@EXAMPLE.COM` - - If it is not, ensure that you have `libsasl` or `libsasl2` installed. On CentOS, this can be installed with the following command. - ``` - yum install -y cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi - ``` - - 1. Use the non-binary wheel to install confluent-kafka-python. - The Python client README (https://github.com/confluentinc/confluent-kafka-python) has an important note for Kerberos case: - ``` - **NOTE:** The pre-built Linux wheels do NOT contain SASL Kerberos support. - If you need SASL Kerberos support you must install librdkafka and - its dependencies using the above repositories and then build - confluent-kafka from source. + +1. If it is not, ensure that you have `libsasl` or `libsasl2` installed. On CentOS, this can be installed with the following command. + ``` + yum install -y cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi ``` - - so you will need to use the non-binary wheel to install confluent-kafka-python and build/install librdkafka separately 1. Build Librdkafka with SASL support (` --enable-sasl`) and install at your chosen $PREFIX. ``` @@ -270,10 +259,10 @@ The probe can be used in a Kerberized environment. Follow these additional step $ examples/rdkafka_example -X builtin.features builtin.features = gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins ``` -1. The source install of confluent-kafka: - - ``` - // If you have already installed, remove the binary wheel python client first, repeat until it says no longer installed +1. The source install of confluent-kafka. + + If you have already installed, remove the binary wheel python client first, repeat until it says no longer installed + ``` pip uninstall -y confluent-kafka ```