Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

added a "bin_dir" parameter to configure where the Kafka scripts are #147

Closed
wants to merge 11 commits into from
Closed

added a "bin_dir" parameter to configure where the Kafka scripts are #147

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
f43ac28
issue#145: added the "bin_dir" parameter to configure where the Kafka…
LionelCons May 17, 2017
20f3791
modulesync 0.21.3
bastelfreak Jun 11, 2017
af44357
fix open ended dependency version requirement
bastelfreak Jun 11, 2017
c6c44ae
delete Guardfile
bastelfreak Jun 11, 2017
fff27dc
bump to minimal recommended puppet version
bastelfreak Jun 11, 2017
2819c0d
bump to minimal recommended stdlib version
bastelfreak Jun 11, 2017
34b72a2
test against latest dependencies
bastelfreak Jun 11, 2017
8942fca
only test for working URLs
bastelfreak Jun 11, 2017
23e8192
rebased from upstream
LionelCons Jun 29, 2017
232f96b
add support to kafka app logs dir
jolivares May 22, 2017
65e92b9
reuse params log_dir
jolivares May 24, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 5 additions & 9 deletions .fixtures.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,9 @@
fixtures:
repositories:
archive: "https://github.com/voxpupuli/puppet-archive.git"
java: "https://github.com/puppetlabs/puppetlabs-java.git"
stdlib:
repo: "https://github.com/puppetlabs/puppetlabs-stdlib.git"
ref: "4.11.0"
zookeeper: "https://github.com/deric/puppet-zookeeper.git"
systemd:
repo: "git://github.com/camptocamp/puppet-systemd"
ref: "0.2.2"
archive: 'https://github.com/voxpupuli/puppet-archive.git'
java: 'https://github.com/puppetlabs/puppetlabs-java.git'
stdlib: 'https://github.com/puppetlabs/puppetlabs-stdlib.git'
zookeeper: 'https://github.com/deric/puppet-zookeeper.git'
systemd: 'https://github.com/camptocamp/puppet-systemd.git'
symlinks:
kafka: "#{source_dir}"
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,4 @@ Puppetfile.lock
*.iml
.*.sw?
.yardoc/
Guardfile
2 changes: 1 addition & 1 deletion .msync.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
modulesync_config_version: '0.21.0'
modulesync_config_version: '0.21.3'
1 change: 1 addition & 0 deletions .rspec_parallel
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
--format progress
8 changes: 8 additions & 0 deletions .rubocop.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -503,6 +503,10 @@ Style/ClosingParenthesisIndentation:

# RSpec

RSpec/BeforeAfterAll:
Exclude:
- spec/acceptance/**/*

# We don't use rspec in this way
RSpec/DescribeClass:
Enabled: False
Expand All @@ -522,6 +526,10 @@ RSpec/RepeatedDescription:
RSpec/NestedGroups:
Enabled: False

# this is broken on ruby1.9
Style/IndentHeredoc:
Enabled: False

# disable Yaml safe_load. This is needed to support ruby2.0.0 development envs
Security/YAMLLoad:
Enabled: false
1 change: 1 addition & 0 deletions Gemfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ group :test do
gem 'github_changelog_generator', '~> 1.13.0', :require => false if RUBY_VERSION < '2.2.2'
gem 'rack', '~> 1.0', :require => false if RUBY_VERSION < '2.2.2'
gem 'github_changelog_generator', :require => false if RUBY_VERSION >= '2.2.2'
gem 'parallel_tests', :require => false
end

group :development do
Expand Down
5 changes: 0 additions & 5 deletions Guardfile
View file
Open in desktop

This file was deleted.

53 changes: 25 additions & 28 deletions manifests/broker.pp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,9 @@
# [*config_dir*]
# The directory to create the kafka config files to
#
# [*bin_dir*]
# The directory where the kafka scripts are
#
# === Examples
#
# Create a single broker instance which talks to a local zookeeper instance.
Expand All @@ -49,36 +52,30 @@
# }
#
class kafka::broker (
$version = $kafka::params::version,
$scala_version = $kafka::params::scala_version,
$install_dir = $kafka::params::install_dir,
$mirror_url = $kafka::params::mirror_url,
$config = {},
$config_defaults = $kafka::params::broker_config_defaults,
$install_java = $kafka::params::install_java,
$package_dir = $kafka::params::package_dir,
$service_install = $kafka::params::broker_service_install,
$service_ensure = $kafka::params::broker_service_ensure,
$service_restart = $kafka::params::service_restart,
$service_requires_zookeeper = $kafka::params::service_requires_zookeeper,
$jmx_opts = $kafka::params::broker_jmx_opts,
$heap_opts = $kafka::params::broker_heap_opts,
$log4j_opts = $kafka::params::broker_log4j_opts,
$opts = $kafka::params::broker_opts,
$group_id = $kafka::params::group_id,
$user_id = $kafka::params::user_id,
$config_dir = $kafka::params::config_dir,
$version = $kafka::params::version,
$scala_version = $kafka::params::scala_version,
$install_dir = $kafka::params::install_dir,
Stdlib::HTTPUrl $mirror_url = $kafka::params::mirror_url,
Hash $config = {},
$config_defaults = $kafka::params::broker_config_defaults,
Boolean $install_java = $kafka::params::install_java,
Integer $limit_nofile = $kafka::params::limit_nofile,
Stdlib::Absolutepath $package_dir = $kafka::params::package_dir,
Boolean $service_install = $kafka::params::broker_service_install,
Enum['running', 'stopped'] $service_ensure = $kafka::params::broker_service_ensure,
Boolean $service_restart = $kafka::params::service_restart,
$service_requires_zookeeper = $kafka::params::service_requires_zookeeper,
$jmx_opts = $kafka::params::broker_jmx_opts,
$heap_opts = $kafka::params::broker_heap_opts,
$log4j_opts = $kafka::params::broker_log4j_opts,
$opts = $kafka::params::broker_opts,
$group_id = $kafka::params::group_id,
$user_id = $kafka::params::user_id,
$config_dir = $kafka::params::config_dir,
$bin_dir = $kafka::params::bin_dir,
$log_dir = $kafka::params::log_dir,
) inherits kafka::params {

validate_re($::osfamily, 'RedHat|Debian\b', "${::operatingsystem} not supported")
validate_re($mirror_url, $kafka::params::mirror_url_regex, "${mirror_url} is not a valid url")
validate_hash($config)
validate_bool($install_java)
validate_absolute_path($package_dir)
validate_bool($service_install)
validate_re($service_ensure, '^(running|stopped)$')
validate_bool($service_restart)

class { '::kafka::broker::install': }
-> class { '::kafka::broker::config': }
-> class { '::kafka::broker::service': }
Expand Down
3 changes: 3 additions & 0 deletions manifests/broker/service.pp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,13 @@
$service_ensure = $kafka::broker::service_ensure,
$service_requires_zookeeper = $kafka::broker::service_requires_zookeeper,
$jmx_opts = $kafka::broker::jmx_opts,
$limit_nofile = $kafka::broker::limit_nofile,
$log4j_opts = $kafka::broker::log4j_opts,
$heap_opts = $kafka::broker::heap_opts,
$opts = $kafka::broker::opts,
$config_dir = $kafka::broker::config_dir,
$bin_dir = $kafka::broker::bin_dir,
$log_dir = $kafka::broker::log_dir,
) {

if $caller_module_name != $module_name {
Expand Down
41 changes: 20 additions & 21 deletions manifests/consumer.pp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,9 @@
# [*config_dir*]
# The directory to create the kafka config files to
#
# [*bin_dir*]
# The directory where the kafka scripts are
#
# === Examples
#
# Create the consumer service connecting to a local zookeeper
Expand All @@ -48,29 +51,25 @@
# config => { 'client.id' => '0', 'zookeeper.connect' => 'localhost:2181' }
# }
class kafka::consumer (
$version = $kafka::params::version,
$scala_version = $kafka::params::scala_version,
$install_dir = $kafka::params::install_dir,
$mirror_url = $kafka::params::mirror_url,
$config = {},
$config_defaults = $kafka::params::consumer_config_defaults,
$service_config = {},
$service_defaults = $kafka::params::consumer_service_defaults,
$install_java = $kafka::params::install_java,
$package_dir = $kafka::params::package_dir,
$service_restart = $kafka::params::service_restart,
$service_requires_zookeeper = $kafka::params::service_requires_zookeeper,
$consumer_jmx_opts = $kafka::params::consumer_jmx_opts,
$consumer_log4j_opts = $kafka::params::consumer_log4j_opts,
$config_dir = $kafka::params::config_dir,
$version = $kafka::params::version,
$scala_version = $kafka::params::scala_version,
$install_dir = $kafka::params::install_dir,
Stdlib::HTTPUrl $mirror_url = $kafka::params::mirror_url,
$config = {},
$config_defaults = $kafka::params::consumer_config_defaults,
$service_config = {},
$service_defaults = $kafka::params::consumer_service_defaults,
Boolean $install_java = $kafka::params::install_java,
Integer $limit_nofile = $kafka::params::limit_nofile,
Stdlib::Absolutepath $package_dir = $kafka::params::package_dir,
Boolean $service_restart = $kafka::params::service_restart,
$service_requires_zookeeper = $kafka::params::service_requires_zookeeper,
$consumer_jmx_opts = $kafka::params::consumer_jmx_opts,
$consumer_log4j_opts = $kafka::params::consumer_log4j_opts,
$config_dir = $kafka::params::config_dir,
$bin_dir = $kafka::params::bin_dir,
) inherits kafka::params {

validate_re($::osfamily, 'RedHat|Debian\b', "${::operatingsystem} not supported")
validate_re($mirror_url, $kafka::params::mirror_url_regex, "${mirror_url} is not a valid url")
validate_bool($install_java)
validate_absolute_path($package_dir)
validate_bool($service_restart)

class { '::kafka::consumer::install': }
-> class { '::kafka::consumer::service': }
-> Class['kafka::consumer']
Expand Down
2 changes: 2 additions & 0 deletions manifests/consumer/service.pp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@
$service_config = $kafka::consumer::service_config,
$service_defaults = $kafka::consumer::service_defaults,
$service_requires_zookeeper = $kafka::consumer::service_requires_zookeeper,
$limit_nofile = $kafka::consumer::limit_nofile,
$bin_dir = $kafka::consumer::bin_dir,
) {

if $caller_module_name != $module_name {
Expand Down
36 changes: 18 additions & 18 deletions manifests/init.pp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,33 +52,33 @@
# [*config_dir*]
# The directory to create the kafka config files to
#
# [*bin_dir*]
# The directory where the kafka scripts are
#
# [*log_dir*]
# The directory for kafka log files
#
# === Examples
#
#
class kafka (
$version = $kafka::params::version,
$scala_version = $kafka::params::scala_version,
$install_dir = $kafka::params::install_dir,
$mirror_url = $kafka::params::mirror_url,
$install_java = $kafka::params::install_java,
$package_dir = $kafka::params::package_dir,
$package_name = $kafka::params::package_name,
$package_ensure = $kafka::params::package_ensure,
$group_id = $kafka::params::group_id,
$user_id = $kafka::params::user_id,
$user = $kafka::params::user,
$group = $kafka::params::group,
$config_dir = $kafka::params::config_dir,
$log_dir = $kafka::params::log_dir,
$version = $kafka::params::version,
$scala_version = $kafka::params::scala_version,
$install_dir = $kafka::params::install_dir,
Stdlib::HTTPUrl $mirror_url = $kafka::params::mirror_url,
Boolean $install_java = $kafka::params::install_java,
Stdlib::Absolutepath $package_dir = $kafka::params::package_dir,
$package_name = $kafka::params::package_name,
$package_ensure = $kafka::params::package_ensure,
$group_id = $kafka::params::group_id,
$user_id = $kafka::params::user_id,
$user = $kafka::params::user,
$group = $kafka::params::group,
$config_dir = $kafka::params::config_dir,
$bin_dir = $kafka::params::bin_dir,
$log_dir = $kafka::params::log_dir,
) inherits kafka::params {

validate_re($::osfamily, 'RedHat|Debian\b', "${::operatingsystem} not supported")
validate_bool($install_java)
validate_absolute_path($package_dir)

$basefilename = "kafka_${scala_version}-${version}.tgz"
$package_url = "${mirror_url}/kafka/${version}/${basefilename}"

Expand Down
57 changes: 26 additions & 31 deletions manifests/mirror.pp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,9 @@
# [*config_dir*]
# The directory to create the kafka config files to
#
# [*bin_dir*]
# The directory where the kafka scripts are
#
# === Examples
#
# Create the mirror service connecting to a local zookeeper
Expand All @@ -64,39 +67,31 @@
# }
#
class kafka::mirror (
$version = $kafka::params::version,
$scala_version = $kafka::params::scala_version,
$install_dir = $kafka::params::install_dir,
$mirror_url = $kafka::params::mirror_url,
$consumer_config = {},
$consumer_config_defaults = $kafka::params::consumer_config_defaults,
$producer_config = {},
$producer_config_defaults = $kafka::params::producer_config_defaults,
$num_streams = $kafka::params::num_streams,
$num_producers = $kafka::params::num_producers,
$abort_on_send_failure = $kafka::params::abort_on_send_failure,
$install_java = $kafka::params::install_java,
$whitelist = $kafka::params::whitelist,
$blacklist = $kafka::params::blacklist,
$max_heap = $kafka::params::mirror_max_heap,
$package_dir = $kafka::params::package_dir,
$service_restart = $kafka::params::service_restart,
$service_requires_zookeeper = $kafka::params::service_requires_zookeeper,
$mirror_jmx_opts = $kafka::params::mirror_jmx_opts,
$mirror_log4j_opts = $kafka::params::mirror_log4j_opts,
$config_dir = $kafka::params::config_dir,
$version = $kafka::params::version,
$scala_version = $kafka::params::scala_version,
$install_dir = $kafka::params::install_dir,
Stdlib::HTTPUrl $mirror_url = $kafka::params::mirror_url,
$consumer_config = {},
$consumer_config_defaults = $kafka::params::consumer_config_defaults,
$producer_config = {},
$producer_config_defaults = $kafka::params::producer_config_defaults,
Integer $num_streams = $kafka::params::num_streams,
Integer $num_producers = $kafka::params::num_producers,
Boolean $abort_on_send_failure = $kafka::params::abort_on_send_failure,
Boolean $install_java = $kafka::params::install_java,
Integer $limit_nofile = $kafka::params::limit_nofile,
$whitelist = $kafka::params::whitelist,
$blacklist = $kafka::params::blacklist,
Pattern[/\d+[g|G|m|M|k|K]/] $max_heap = $kafka::params::mirror_max_heap,
Stdlib::Absolutepath $package_dir = $kafka::params::package_dir,
Boolean $service_restart = $kafka::params::service_restart,
$service_requires_zookeeper = $kafka::params::service_requires_zookeeper,
$mirror_jmx_opts = $kafka::params::mirror_jmx_opts,
$mirror_log4j_opts = $kafka::params::mirror_log4j_opts,
$config_dir = $kafka::params::config_dir,
$bin_dir = $kafka::params::bin_dir,
) inherits kafka::params {

validate_re($::osfamily, 'RedHat|Debian\b', "${::operatingsystem} not supported")
validate_re($mirror_url, $kafka::params::mirror_url_regex, "${mirror_url} is not a valid url")
validate_integer($num_streams)
validate_integer($num_producers)
validate_bool($abort_on_send_failure)
validate_bool($install_java)
validate_re($max_heap, '\d+[g|G|m|M|k|K]', "${max_heap} is not a valid heap size")
validate_absolute_path($package_dir)
validate_bool($service_restart)

class { '::kafka::mirror::install': }
-> class { '::kafka::mirror::config': }
-> class { '::kafka::mirror::service': }
Expand Down
1 change: 1 addition & 0 deletions manifests/mirror/service.pp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
$blacklist = $kafka::mirror::blacklist,
$max_heap = $kafka::mirror::max_heap,
$config_dir = $kafka::params::config_dir,
$bin_dir = $kafka::params::bin_dir,
) inherits ::kafka::params {

if $caller_module_name != $module_name {
Expand Down
9 changes: 9 additions & 0 deletions manifests/params.pp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,17 @@
# It sets variables according to platform
#
class kafka::params {

# this is all only tested on Debian and RedHat
# params gets included everywhere so we can do the validation here
unless $facts['os']['family'] =~ /(RedHat|Debian)/ {
warning("${facts['os']['family']} is not supported")
}
$version = '0.9.0.1'
$scala_version = '2.11'
$install_dir = "/opt/kafka-${scala_version}-${version}"
$config_dir = '/opt/kafka/config'
$bin_dir = '/opt/kafka/bin'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we're on puppet4 (or even 5 \o/) now, can you please use the datatype Stdlib::Absolutepath here (and in the other classes where you use the $bin_dir)?

$log_dir = '/var/log/kafka'
$mirror_url = 'http://mirrors.ukfast.co.uk/sites/ftp.apache.org'
$install_java = true
Expand Down Expand Up @@ -47,6 +54,8 @@
-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9993'
$consumer_log4j_opts = $broker_log4j_opts

$limit_nofile = 65536

$service_restart = true

#http://kafka.apache.org/documentation.html#brokerconfigs
Expand Down
Loading