HSDP Kafka module

Module to create an Apache kafka cluster deployed on the HSDP Container Host infrastructure. This module serves as a blueprint for future HSDP Container Host modules. Example usage

module "kafka" {
  source = "github.com/philips-labs/terraform-hsdp-kafka"

  nodes             = 3
  bastion_host      = "bastion.host"
  user              = "ronswanson"
  private_key       = file("~/.ssh/dec.key")
  user_groups       = ["ronswanson", "poc"]
  zookeeper_connect = "zookeeper-server:2181"
  kafka_trust_store   = {
    truststore = "./kafkatruststore.jks"
    password   = "somepass"
  }
  kafka_key_store     = {
    keystore   = "./kafkakeystore.jks"
    password   = "somepass"
  }
  zoo_trust_store   = {
    truststore = "./zootruststore.jks"
    password   = "somepass"
  }
  zoo_key_store     = {
    keystore   = "./zookeystore.jks"
    password   = "somepass"
  }
  message_max_bytes = 1048576
}

IMPORTANT SECURITY INFORMATION

This module currently enables only mTLS-SSL between Kafka, Zookeeper or any connecting client apps. Operating and maintaining applications on Container Host is always your responsibility. This includes ensuring any security measures are in place in case you need them.

Requirements

Name	Version
terraform	>= 0.13.0
hsdp	>= 0.9.4
random	>= 2.2.1

Providers

Name	Version
hsdp	>= 0.9.4
random	>= 2.2.1

Inputs

Name	Description	Type	Default	Required
auto_create_topics_enable	turn on or off auto-create-topics, defaults to true	bool	true	no
bastion_host	Bastion host to use for SSH connections	string	n/a	yes
default_replication_factor	default kafka server replication factor	number	1	no
enable_exporters	Indicates whether jmx exporter and kafka exporter is enabled or not	bool	false	no
host_name	The middlename for your host default is a random number	string	""	no
image	The docker image to use	string	"bitnami/kafka:latest"	no
instance_type	The instance type to use	string	"t3.large"	no
iops	IOPS to provision for EBS storage	number	500	no
kafka_ca_root	CA root store for SSL (only applicable when exporter is required, so only when 'enable_exporters==true')	string	""	no
kafka_key_store	A list of key stores one for each nore	object( { keystore = string, password = string } )	n/a	yes
kafka_private_key	Private Key for SSL (only applicable when exporter is required, so only when 'enable_exporters==true')	string	""	no
kafka_public_key	Public Key for SSL (only applicable when exporter is required, so only when 'enable_exporters==true')	string	""	no
kafka_trust_store	Trust store for SSL	object( { truststore = string, password = string } )	n/a	yes
nodes	Number of nodes	number	1	no
private_key	Private key for SSH access	string	n/a	yes
retention_hours	Retention hours for Kakfa topics	string	"-1"	no
tld	The tld for your host default is a dev	string	"dev"	no
user	LDAP user to use for connections	string	n/a	yes
user_groups	User groups to assign to cluster	list(string)	[]	no
volume_size	The volume size to use in GB	number	50	no
zoo_key_store	Zookeeper Key store for SSL	object( { keystore = string, password = string } )	n/a	yes
zoo_trust_store	Zookeeper Trust store for SSL	object( { truststore = string, password = string } )	n/a	yes
zookeeper_connect	Zookeeper connect string to use	string	n/a	yes
message_max_bytes	Maximum request size of a message supported in broker	number	n/a	no
max_partition_fetch_bytes	The maximum amount of data per-partition the broker will return	number	n/a	no

Incase you are wondering why we need zookeeper key store, its required by bitnami please refer to bitnami documentation.

Key Store object

This object has two properties that needs to be filled

Name	Description	Type	Default	Required
keystore	The path of the keystore file in JKS format	string	none	yes
password	The password to be used for the key store	string	none	yes

Trust Store object

This object has two properties that needs to be filled

Name	Description	Type	Default	Required
truststore	The path of the truststore file in JKS format	string	none	yes
password	The password to be used for the trust store	string	none	yes

Generate caroot (PEM format)

Kafka exporter needs a caroot file. This can be extracted from the truststore file. In order to that use the following script:

inputJksFilename=$1
outputPublicKeyFilename=$2
inputKeyStorePassword=$3

# Exports the pub cert from the truststore
keytool -export  -rfc -alias caroot -keystore $inputJksFilename -file $outputPublicKeyFilename -storepass $inputKeyStorePassword -noprompt

Generate public/private keys (PEM format)

Kafka exporter needs the certificate in PEM format. In order to get these files use the following script:

inputJksFilename=$1
outputPublicKeyFilename=$2
outputPrivateKeyFilename=$3
inputKeyStorePassword=$4

#Export the public key with the certificate format
openssl pkcs12 -in $inputJksFilename -out $outputPublicKeyFilename -clcerts -nokeys -passin pass:$inputKeyStorePassword

#Export the private key with the PEM format
keytool -importkeystore -srckeystore $inputJksFilename -destkeystore tempKeyStore.p12 -deststoretype PKCS12 -srcstorepass $inputKeyStorePassword -deststorepass $inputKeyStorePassword -noprompt
openssl pkcs12 -in tempKeyStore.p12 -nodes -nocerts -out $outputPrivateKeyFilename  -passin pass:$inputKeyStorePassword -passout pass:$inputKeyStorePassword

Outputs

Name	Description
kafka_name_nodes	Container Host DNS names of Kafka instances
kafka_nodes	Container Host IP addresses of Kafka instances
kafka_port	Port where you can reach Kafka

Contact / Getting help

Andy Lo-A-Foe andy.lo-a-foe@philips.com

License

License is MIT