Confluent Cookbook

Installs the Confluent package and can run its services,

• Kafka
• Kafka REST
• Schema Registry
• Kafka Connect

View the Change Log to see what has changed.

Getting Started

Install Confluent Package

If you include the recipe[confluent] this will install the Confluent package and nothing else.

You can find the package installed under /opt/confluent (by default) with the name confluent-VERSION.

You can also find the configuration under,

• /etc/kafka
• /etc/kafka-rest
• /etc/schema-registry
• /etc/kafka-connect

Kafka Service

If you include the recipe[confluent::kafka] this will install the Confluent package, configure and start the Kafka service.

You can configure the service using the attribtues node["confluent"]["kafka"]["server.properties"][...] = .... Use Confluent's Kafka doc to figure out the appropriate configuration for yourself.

You can find the SysV script at /etc/init.d/kafka-rest or service kafka-rest [start|stop|restart|status].

You can find the logs at /var/log/confluent/kafka.log.

Kafka REST Service

If you include the recipe[confluent::kafka-rest] this will install the Confluent package, configure and start the Kafka REST service.

You can configure the service using the attribtues node["confluent"]["kafka-rest"]["kafka-rest.properties"][...] = .... Use Confluent's Kafka REST doc to figure out the appropriate configuration for yourself.

You can find the SysV script at /etc/init.d/kafka-rest or service kafka-rest [start|stop|restart|status].

You can find the logs at /var/log/confluent/kafka-rest.log.

Schema Registry Service

If you include the recipe[confluent::schema-registry] this will install the Confluent package, configure and start the Schema Registry service.

You can configure the service using the attribtues node["confluent"]["schema-registry"]["schema-registry.properties"][...] = .... Use Confluent's Schema Registry doc to figure out the appropriate configuration for yourself.

You can find the SysV script at /etc/init.d/schema-registry or service schema-registry [start|stop|restart|status].

You can find the logs at /var/log/confluent/schema-registry.log.

Kafka Connect

If you include the recipe[confluent::kafka-connect] this will install the Confluent package, configure and start the Kafka connector. This will listen on port 8083 exposing its rest api to control spinning up new connectors.

You can configure the service using the attribtues node["confluent"]["kafka-connect"]["worker.properties"][...] = .... Use Confluent's Kafka Connect doc to figure out the appropriate configuration for yourself.

You can find the SysV script at /etc/init.d/kafka-connect or service kafka-connect [start|stop|restart|status].

You can find the logs at /var/log/confluent/kafka-connect.log.

Zookeeper (For development purposes only)

If you include the recipe[confluent::zookeeper] this will install the Confluent package and start a zookeeper process. It will listen on port 2181. This is a single zookeeper worker and is not recomended for production use. The primary purpose of this recipe is to get everything running inside vagrant as a self contained system without having to run process on the host machine.

Attributes

Generic

• node["confluent"]["version"] : The version of the Confluent package to install (default=2.0.1)
• node["confluent"]["scala_version"] : The scala version of the Confluent package to install (default=2.11.7)
• node["confluent"]["artifact_url"] : The URL to the Confluent package to install. This is generated using the version and scala_version attributes. It downloads from packages.confluent.io.
• node["confluent"]["install_dir"] : The directory to install the Confluent package (default=/opt/confluent)
• node["confluent"]["user"] : The user that owns the Confluent package files and runs the services (default=confluent)
• node["confluent"]["uid"] : optional staticly assign a uid for above user (default=unset picks form system config)
• node["confluent"]["group"] : The group that owns the Confluent package files and runs the services (default=confluent)
• node["confluent"]["gid"] : optional staticly assign a gid for above group (default=unset picks form system config)

Kafka

• node["confluent"]["kafka"]["server.properties"] : A Hash of properties that configure the Kafka service (default={})
• node["confluent"]["kafka"]["env_vars"] : A Hash of environment variables applied when running the service
• node["confluent"]["kafka"]["log4j.properties"] : A Hash of properties that configure log4j for the Kafka service (see attributes for defaults)
• node['confluent']['kafka']['brokers'] : A single broker String or List of brokers by hostname, fqdn, or ipaddress
• node['confluent']['kafka']['zookeepers'] : A list of zookeeper hostname:port's to add to kafka config (default=nil)
• node['confluent']['kafka']['zookeeper_chroot'] : An optional chroot path for zookeeper hostname:port/chroot's to add to kafka config (default=nil)

Kafka REST

• node["confluent"]["kafka-rest"]["kafka-rest.properties"] : A Hash of properties that configure the Kafka REST service (default={})
• node["confluent"]["kafka-rest"]["env_vars"] : A Hash of environment variables applied when running the service
• node["confluent"]["kafka-rest"]["log4j.properties"] : A Hash of properties that configure log4j for the Kafka REST service (see attributes for defaults)

Schema Registry

• node["confluent"]["schema-registry"]["schema-registry.properties"] : A Hash of properties that configure the Schema Registry service (default={})
• node["confluent"]["schema-registry"]["env_vars"] : A Hash of environment variables applied when running the service
• node["confluent"]["schema-registry"]["log4j.properties"] : A Hash of properties that configure log4j for the Schema Registry service (see attributes for defaults)

Kafka Connect

• node["confluent"]["kafka-connect"]["jar_urls"] : an array of urls to remote files to download and install in the directory share/java/kafka-connect-all located in the extracted confluent directory which is where connect looks by default.
• node["confluent"]["kafka-connect"]["properties_files"] : a hash where the key is a property file name, and the value is a hash of keys/values for the property file. Used to drop in property files via chef config as opposed to the rest api.
• node["confluent"]["kafka-connect"]["distributed_mode"] : Boolean used to decide if it should launch in standalone or distributed mode. Defaults to true
• node["confluent"]["kafka-connect"]["worker_properties_file_name"] : The name of the properties file to use when starting the connect service.
• node["confluent"]["kafka-connect"]["worker.properties"] : hash of properties to configure the connect properties with.
• node["confluent"]["kafka-connect"]["env_vars"] : A Hash of environment variables applied when running the service
• node["confluent"]["kafka-connect"]["log4j.properties"] : A Hash of properties that configure log4j for the Schema Registry service (see attributes for defaults)

Zookeeper

• node["confluent"]["zookeeper"]["zookeeper.properties"] : a hash of properties to configure the zookeeper server with

Testing

Style

• rake style : runs foodcritic and rubocop
  • todo in .rubocop.yml
• rake unit : runs chefspec tests form ./spec
  • todo from untouched resources
• rake kitchen : runs kitchen tests
  • problem with gem version conflict in Rakefile. Run kitchen from command line: kitchen test