Skip to content
Kafka connector for Venafi Security Events
Java
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
config
logos
src
.gitignore
.travis.yml
LICENSE
README.md
pom.xml

README.md

kafka-connect-venafi

Build Status

kafka-connect-venafi is a Kafka connector for Venafi Trust Protection Platform security events.

Connector Version Source Technology Version Confluent Platorm Version
0.9.X Venafi Trust Protection Platform 18.X Compatible Confluent Platform Version - ≥5.0.X

Description

This connector connects via HTTP to your instance of the Venafi and pulls your Log events into Kafka, allowing you to do any filtering/transforming/processing you'd like to do within a comfortable Kafka environment.
N.B. Currently the connector starts from the beginning of time (i.e. processes all past events first), a future release will allow the option of starting from now (i.e. skipping all previous events).

EventLog object produced by this source Connector as defined by the Venafi WebSDK
  • ClientTimestamp: The time that the client generated the event.
  • Component: A string that identifies a component DN. For events that occur in a subsystem other than Config, such as a Secret Store, the Distinguished Name (DN) component may be blank.
  • ComponentId: The component ID that originated the event.
  • ComponentSubsystem: The component subsytem that originated the event.
  • Grouping: An integer for the Component ID or Group Id that correlates to a set of events.
  • Id: A hexadecimal value that corresponds to an Event ID.
  • Name: The event name.
  • ServerTimestamp: The time the Trust Protection Platform server received the event.
  • Severity: A string value for one of these event severities: Emergency, Alert, Critical, Error, Warning, Notice, Info, or Debug
  • SourceIP: The IP address of the host that originated the event.
  • Text1: A string variable that contains all or part of an event message.
  • Text2: An additional string, if present, that appears as part of an event message.
  • Value1: An integer variable, if present, that appears as part of an event message.
  • Value2: An additional integer variable, if present, that appears as part of an event message.

Development

To manually install the connector:

  1. Build the JAR with mvn package
  2. Find the JAR in your target folder called venafi-log-connector-<version you're building>-fat.jar
  3. Create a connect property file
name=venafi
connector.class=com.opencredo.connect.venafi.tpp.log.TppLogSourceConnector
tasks.max=1
venafi.base.url=<your_base_url>
venafi.username=<your_api_username>
venafi.password=<your_api_password>

This is filled with the minimum values required, any default values are provided by the config definition class. This can also be looked at for more information on configuration, or look at the wiki on the config definitions. Make sure to replace the items with the <> brackets with what's needed to connect to your Venafi VEDSDK instance.
There is a quickstart properties file included with the code here which has some dummy values for your base Url, username & password but can be used as an example.
4. Create a directory and place the JAR file in it, e.g. <path-to-confluent>/share/java/kafka-connect-venafi-tpp.
5. Then call: <path-to-confluent>/bin/confluent load venafi -d venafi.properties

For more information please look at the Confluent instructions on manually installing connectors.


If you need to unload/reload the connector, use: <path-to-confluent>/bin/confluent unload venafi
If you intend to change the JAR please stop, change the JAR, then start the cluster.

Useful commands while developing

sudo bin/confluent start  
sudo bin/confluent status
sudo bin/confluent load venafi -d ~/venafi.properties
sudo bin/confluent status venafi
sudo bin/kafka-topics --list --zookeeper localhost:2181
sudo bin/kafka-console-consumer --bootstrap-server localhost:9092 --topic connect-offsets --from-beginning
sudo bin/kafka-avro-console-consumer --bootstrap-server localhost:9092 --topic VENAFI-LOGS
sudo bin/confluent log connect

Config Definitions explained.

venafi.base.url URL to API

  • Type: string
  • Valid Values: non-empty string and no ISO control characters
  • Importance: high

venafi.password The password to use with the API.

  • Type: string
  • Importance: high

venafi.username The username to use with the API.

  • Type: string
  • Importance: high

venafi.topic Topic to publish Venafi log data to.

  • Type: string
  • Default: VENAFI-LOGS
  • Valid Values: non-empty string and no ISO control characters
  • Importance: high

venafi.batch.size Window of data to pull from log API.

  • Type: int
  • Default: 100
  • Valid Values: [2,...,10000]
  • Importance: low

venafi.poll.interval Poll interval in milliseconds.

  • Type: int
  • Default: 1000
  • Importance: low
You can’t perform that action at this time.