From 8305bdd9ca6f9fdd053165aae080091086743547 Mon Sep 17 00:00:00 2001 From: Konstantinos Tsakalozos Date: Thu, 2 Jun 2016 16:18:37 +0300 Subject: [PATCH 1/7] BIGTOP-2486: Add Kafka Charm Squashed changes: * Advertise hostname in server.properties * Opening kafka ports * lint; close ports * Fixed up tests. * Tests were failing due to some ambiguities in the version of zookeeper that we are deploying. Fixed them so that they work with my setup (juju 2.0, installing zookeeper from the charms store), and have some checks that should make them work with other setups. * Also fixed a mistake, where the deploy test was calling self.unit, but had set self.kafka. * Added ability to override host.name (bind address) for kafka * Got rid of try-catch in tests, and fixed defaults. * config.yaml should default to null, rather than 0.0.0.0 (They both do the same thing, basically, but 'null' results in nothing being written to server.properties.) --- .../src/charm/kafka/layer-kafka/LICENSE | 177 ++++++++++++++++++ .../src/charm/kafka/layer-kafka/README.md | 171 +++++++++++++++++ .../src/charm/kafka/layer-kafka/actions.yaml | 44 +++++ .../kafka/layer-kafka/actions/create-topic | 50 +++++ .../kafka/layer-kafka/actions/kafkautils.py | 38 ++++ .../kafka/layer-kafka/actions/list-topics | 42 +++++ .../charm/kafka/layer-kafka/actions/list-zks | 36 ++++ .../kafka/layer-kafka/actions/read-topic | 55 ++++++ .../kafka/layer-kafka/actions/smoke-test | 64 +++++++ .../kafka/layer-kafka/actions/write-topic | 54 ++++++ .../src/charm/kafka/layer-kafka/config.yaml | 7 + .../src/charm/kafka/layer-kafka/copyright | 16 ++ .../src/charm/kafka/layer-kafka/icon.svg | 90 +++++++++ .../src/charm/kafka/layer-kafka/layer.yaml | 20 ++ .../lib/charms/layer/bigtop_kafka.py | 88 +++++++++ .../src/charm/kafka/layer-kafka/metadata.yaml | 32 ++++ .../charm/kafka/layer-kafka/reactive/kafka.py | 81 ++++++++ .../kafka/layer-kafka/tests/01-deploy.py | 58 ++++++ .../kafka/layer-kafka/tests/02-smoke-test.py | 59 ++++++ .../charm/kafka/layer-kafka/tests/tests.yaml | 3 + 20 files changed, 1185 insertions(+) create mode 100644 bigtop-packages/src/charm/kafka/layer-kafka/LICENSE create mode 100644 bigtop-packages/src/charm/kafka/layer-kafka/README.md create mode 100644 bigtop-packages/src/charm/kafka/layer-kafka/actions.yaml create mode 100755 bigtop-packages/src/charm/kafka/layer-kafka/actions/create-topic create mode 100644 bigtop-packages/src/charm/kafka/layer-kafka/actions/kafkautils.py create mode 100755 bigtop-packages/src/charm/kafka/layer-kafka/actions/list-topics create mode 100755 bigtop-packages/src/charm/kafka/layer-kafka/actions/list-zks create mode 100755 bigtop-packages/src/charm/kafka/layer-kafka/actions/read-topic create mode 100755 bigtop-packages/src/charm/kafka/layer-kafka/actions/smoke-test create mode 100755 bigtop-packages/src/charm/kafka/layer-kafka/actions/write-topic create mode 100644 bigtop-packages/src/charm/kafka/layer-kafka/config.yaml create mode 100644 bigtop-packages/src/charm/kafka/layer-kafka/copyright create mode 100644 bigtop-packages/src/charm/kafka/layer-kafka/icon.svg create mode 100644 bigtop-packages/src/charm/kafka/layer-kafka/layer.yaml create mode 100755 bigtop-packages/src/charm/kafka/layer-kafka/lib/charms/layer/bigtop_kafka.py create mode 100644 bigtop-packages/src/charm/kafka/layer-kafka/metadata.yaml create mode 100644 bigtop-packages/src/charm/kafka/layer-kafka/reactive/kafka.py create mode 100755 bigtop-packages/src/charm/kafka/layer-kafka/tests/01-deploy.py create mode 100755 bigtop-packages/src/charm/kafka/layer-kafka/tests/02-smoke-test.py create mode 100644 bigtop-packages/src/charm/kafka/layer-kafka/tests/tests.yaml diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/LICENSE b/bigtop-packages/src/charm/kafka/layer-kafka/LICENSE new file mode 100644 index 000000000..f433b1a53 --- /dev/null +++ b/bigtop-packages/src/charm/kafka/layer-kafka/LICENSE @@ -0,0 +1,177 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/README.md b/bigtop-packages/src/charm/kafka/layer-kafka/README.md new file mode 100644 index 000000000..a87d3d8ca --- /dev/null +++ b/bigtop-packages/src/charm/kafka/layer-kafka/README.md @@ -0,0 +1,171 @@ + +## Overview + +Apache Kafka is an open-source message broker project developed by the Apache +Software Foundation written in Scala. The project aims to provide a unified, +high-throughput, low-latency platform for handling real-time data feeds. Learn +more at [kafka.apache.org](http://kafka.apache.org/). + +This charm deploys the Kafka component of the Apache Bigtop platform. + +## Usage + +Kafka requires the Zookeeper distributed coordination service. Deploy and +relate them as follows: + + juju deploy zookeeper + juju deploy kafka + juju add-relation kafka zookeeper + +Once deployed, we can list the zookeeper servers that our kafka brokers +are connected to. The following will list `:` information for each +zookeeper unit in the environment (e.g.: `10.0.3.221:2181`). + + juju run-action kafka/0 list-zks + juju show-action-output # <-- id from above command + +We can create a Kafka topic with: + + juju run-action kafka/0 create-topic topic= \ + partitions=<#> replication=<#> + juju show-action-output # <-- id from above command + +We can list topics with: + + juju run-action kafka/0 list-topics + juju show-action-output # <-- id from above command + +We can write to a topic with: + + juju run-action kafka/0 write-topic topic= data= + juju show-action-output # <-- id from above command + +We can read from a topic with: + + juju run-action kafka/0 read-topic topic= partition=<#> + juju show-action-output # <-- id from above command + +_**Note**: The above assumes Juju 2.0 or greater. If using an earlier version +of Juju, the action syntax is:_ + + juju action do kafka/0 + juju action fetch # <-- id from above command + + +## Status and Smoke Test + +Kafka provides extended status reporting to indicate when it is ready: + + juju status + +This is particularly useful when combined with `watch` to track the on-going +progress of the deployment: + + watch -n 0.5 juju status + +The message for each unit will provide information about that unit's state. +Once they all indicate that they are ready, you can perform a "smoke test" +to verify that Kafka is working as expected using the built-in `smoke-test` +action: + + juju run-action kafka/0 smoke-test + +_**Note**: The above assumes Juju 2.0 or greater. If using an earlier version +of Juju, the syntax is `juju action do kafka/0 smoke-test`._ + +After a minute or so, you can check the results of the smoke test: + + juju show-action-status + +_**Note**: The above assumes Juju 2.0 or greater. If using an earlier version +of Juju, the syntax is `juju action status`._ + + +You will see `status: completed` if the smoke test was successful, or +`status: failed` if it was not. You can get more information on why it failed +via: + + juju show-action-output + +_**Note**: The above assumes Juju 2.0 or greater. If using an earlier version +of Juju, the syntax is `juju action fetch `._ + + +## Scaling + +Creating a cluster with many brokers is as easy as adding more Kafka units: + + juju add-unit kafka + +After adding additional brokers, you will be able to create topics with +replication up to the number of kafka units. + +To verify replication is working you can do the following: + + juju add-unit kafka -n 2 + juju run-action kafka/0 create-topic topic=my-replicated-topic \ + partitions=1 replication=2 + +_**Note**: The above assumes Juju 2.0 or greater. If using an earlier version +of Juju, the syntax is `juju action do kafka/0 create-topic `._ + +Query for the description of the just created topic: + + juju run --unit kafka/0 'kafka-topics.sh --describe \ + --topic my-replicated-topic --zookeeper :2181' + +You should get a response similar to: + + Topic: my-replicated-topic PartitionCount:1 ReplicationFactor:2 Configs: + Topic: my-replicated-topic Partition: 0 Leader: 2 Replicas: 2,0 Isr: 2,0 + + +## Connecting External Clients + +By default, this charm does not expose Kafka outside of the provider's network. +To allow external clients to connect to Kafka, first expose the service: + + juju expose kafka + +Next, ensure the external client can resolve the short hostname of the kafka +unit. A simple way to do this is to add an `/etc/hosts` entry on the external +kafka client machine. Gather the needed info from juju: + + user@juju-client$ juju run --unit kafka/0 'hostname -s' + kafka-0 + user@juju-client$ juju status --format=yaml kafka/0 | grep public-address + public-address: 40.784.149.135 + +Update `/etc/hosts` on the external kafka client: + + user@kafka-client$ echo "40.784.149.135 kafka-0" | sudo tee -a /etc/hosts + +The external kafka client should now be able to access Kafka by using +`kafka-0:9092` as the broker. + + + +## Contact Information +- + + +## Help +- [Apache Kafka home page](http://kafka.apache.org/) +- [Apache Kafka issue tracker](https://issues.apache.org/jira/browse/KAFKA) +- [Juju mailing list](https://lists.ubuntu.com/mailman/listinfo/juju) +- [Juju community](https://jujucharms.com/community) diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/actions.yaml b/bigtop-packages/src/charm/kafka/layer-kafka/actions.yaml new file mode 100644 index 000000000..5db0d5814 --- /dev/null +++ b/bigtop-packages/src/charm/kafka/layer-kafka/actions.yaml @@ -0,0 +1,44 @@ +create-topic: + description: Create a new Kafka topic + params: + topic: + type: string + description: Topic name + partitions: + type: integer + description: Number of partitions for the topic being created + replication: + type: integer + description: Replication factor for each partition in the topic + required: [topic, partitions, replication] + additionalProperties: false +list-topics: + description: List all Kafka topics +list-zks: + description: List ip:port info for connected Zookeeper servers +read-topic: + description: Consume an existing kafka topic + params: + topic: + type: string + description: Topic name + partition: + type: integer + description: Partition to consume + required: [topic, partition] + additionalProperties: false +smoke-test: + description: > + Verify that Kafka is working as expected by listing zookeepers, then + creating/listing/deleting a topic +write-topic: + description: Write to a kafka topic + params: + topic: + type: string + description: Topic name + data: + type: string + description: Data to write to topic + required: [topic, data] + additionalProperties: false diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/actions/create-topic b/bigtop-packages/src/charm/kafka/layer-kafka/actions/create-topic new file mode 100755 index 000000000..b402fe893 --- /dev/null +++ b/bigtop-packages/src/charm/kafka/layer-kafka/actions/create-topic @@ -0,0 +1,50 @@ +#!/usr/bin/env python3 + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import kafkautils +import subprocess + +from charmhelpers.core import hookenv, host +from charms.reactive import is_state +from jujubigdata.utils import run_as + + +if not is_state('kafka.started'): + kafkautils.fail('Kafka service not yet ready') + + +# Grab the business +topic_name = hookenv.action_get('topic') +topic_partitions = hookenv.action_get('partitions') +topic_replication = hookenv.action_get('replication') + +# Create the topic if kafka is running +if host.service_available('kafka-server') and host.service_running('kafka-server'): + zookeepers = kafkautils.get_zookeepers() + try: + output = run_as('kafka', 'kafka-topics.sh', + '--zookeeper', zookeepers, '--create', + '--topic', topic_name, + '--partitions', topic_partitions, + '--replication-factor', topic_replication, + capture_output=True) + except subprocess.CalledProcessError as e: + kafkautils.fail('Kafka command failed', e.output) + else: + hookenv.action_set({'output': output}) +else: + kafkautils.fail('Kafka service is not running', 'Please start kafka-server') diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/actions/kafkautils.py b/bigtop-packages/src/charm/kafka/layer-kafka/actions/kafkautils.py new file mode 100644 index 000000000..52e529901 --- /dev/null +++ b/bigtop-packages/src/charm/kafka/layer-kafka/actions/kafkautils.py @@ -0,0 +1,38 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import re +import sys + +from charmhelpers.core import hookenv + + +def fail(msg, output=''): + hookenv.action_set({'output': output}) + hookenv.action_fail(msg) + sys.exit() + + +def get_zookeepers(): + cfg = '/etc/kafka/conf/server.properties' + print(cfg) + file = open(cfg, "r") + + for line in file: + if re.search('^zookeeper.connect=.*', line): + zks = line.split("=")[1].strip('\n') + return zks + + return None diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/actions/list-topics b/bigtop-packages/src/charm/kafka/layer-kafka/actions/list-topics new file mode 100755 index 000000000..0ef3a3f02 --- /dev/null +++ b/bigtop-packages/src/charm/kafka/layer-kafka/actions/list-topics @@ -0,0 +1,42 @@ +#!/usr/bin/env python3 + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import kafkautils +import subprocess + +from charmhelpers.core import hookenv, host +from charms.reactive import is_state +from jujubigdata.utils import run_as + + +if not is_state('kafka.started'): + kafkautils.fail('Kafka service not yet ready') + + +# List topics if kafka is running +if host.service_available('kafka-server') and host.service_running('kafka-server'): + zookeepers = kafkautils.get_zookeepers() + try: + output = run_as('kafka', '/usr/lib/kafka/bin/kafka-topics.sh', + '--zookeeper', zookeepers, '--list', + capture_output=True) + except subprocess.CalledProcessError as e: + kafkautils.fail('Kafka command failed', e.output) + else: + hookenv.action_set({'output': output}) +else: + kafkautils.fail('Kafka service is not running') diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/actions/list-zks b/bigtop-packages/src/charm/kafka/layer-kafka/actions/list-zks new file mode 100755 index 000000000..517d60c0b --- /dev/null +++ b/bigtop-packages/src/charm/kafka/layer-kafka/actions/list-zks @@ -0,0 +1,36 @@ +#!/usr/bin/env python3 + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import kafkautils + +from charmhelpers.core import hookenv, host +from charms.reactive import is_state + + +if not is_state('kafka.started'): + kafkautils.fail('Kafka service not yet ready', 'Please deploy kafka and required relations') + + +# List zookeepers if kafka is running +if host.service_available('kafka-server') and host.service_running('kafka-server'): + zookeepers = kafkautils.get_zookeepers() + if zookeepers: + hookenv.action_set({'output': zookeepers}) + else: + kafkautils.fail('No zookeeper.connect string found', 'Please relate kafka to zookeeper') +else: + kafkautils.fail('Kafka service is not running', 'Please start kafka-server') diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/actions/read-topic b/bigtop-packages/src/charm/kafka/layer-kafka/actions/read-topic new file mode 100755 index 000000000..b385f668e --- /dev/null +++ b/bigtop-packages/src/charm/kafka/layer-kafka/actions/read-topic @@ -0,0 +1,55 @@ +#!/usr/bin/env python3 + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys +sys.path.append('lib') + +import kafkautils +import subprocess + +from charmhelpers.core import hookenv, host +from charms.layer.apache_bigtop_base import get_layer_opts +from charms.reactive import is_state +from jujubigdata import utils + + +if not is_state('kafka.started'): + kafkautils.fail('Kafka service not yet ready') + + +# Grab the business +topic_name = hookenv.action_get('topic') +topic_partition = hookenv.action_get('partition') + +# Read the topic if kafka is running +if host.service_available('kafka-server') and host.service_running('kafka-server'): + host = subprocess.check_output(['hostname', '-s']).decode('utf8').strip() + port = get_layer_opts().port('kafka') + zookeepers = kafkautils.get_zookeepers() + try: + output = utils.run_as('kafka', '/usr/lib/kafka/bin/kafka-simple-consumer-shell.sh', + '--broker-list', '{}:{}'.format(host, port), + '--topic', topic_name, + '--partition', topic_partition, + '--no-wait-at-logend', + capture_output=True) + except subprocess.CalledProcessError as e: + kafkautils.fail('Kafka command failed', e.output) + else: + hookenv.action_set({'output': output}) +else: + kafkautils.fail('Kafka service is not running') diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/actions/smoke-test b/bigtop-packages/src/charm/kafka/layer-kafka/actions/smoke-test new file mode 100755 index 000000000..5c1435a19 --- /dev/null +++ b/bigtop-packages/src/charm/kafka/layer-kafka/actions/smoke-test @@ -0,0 +1,64 @@ +#!/usr/bin/env python3 + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import kafkautils +import subprocess + +from charmhelpers.core import hookenv, host +from charms.reactive import is_state +from jujubigdata.utils import run_as + + +if not is_state('kafka.started'): + kafkautils.fail('Kafka service not yet ready') + + +# Define smoke test params +topic_name = "smoketest" +topic_partitions = 1 +topic_replication = 1 + +# Smoke only when kafka is running +if host.service_available('kafka-server') and host.service_running('kafka-server'): + # List ZKs + zookeepers = kafkautils.get_zookeepers() + if not zookeepers: + kafkautils.fail('No zookeeper.connect string found') + + # Create a topic + try: + output = run_as('kafka', 'kafka-topics.sh', + '--zookeeper', zookeepers, '--create', + '--topic', topic_name, + '--partitions', topic_partitions, + '--replication-factor', topic_replication, + capture_output=True) + except subprocess.CalledProcessError as e: + kafkautils.fail('Kafka command failed', e.output) + + # List topics + try: + output = run_as('kafka', 'kafka-topics.sh', + '--zookeeper', zookeepers, '--list', + capture_output=True) + except subprocess.CalledProcessError as e: + kafkautils.fail('Kafka command failed', e.output) + + # If we haven't failed yet, we passed + hookenv.action_set({'outcome': 'success'}) +else: + kafkautils.fail('Kafka service is not running') diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/actions/write-topic b/bigtop-packages/src/charm/kafka/layer-kafka/actions/write-topic new file mode 100755 index 000000000..b879b654d --- /dev/null +++ b/bigtop-packages/src/charm/kafka/layer-kafka/actions/write-topic @@ -0,0 +1,54 @@ +#!/usr/bin/env python3 + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys +sys.path.append('lib') + +import kafkautils +import subprocess + +from charmhelpers.core import hookenv, host +from charms.layer.apache_bigtop_base import get_layer_opts +from charms.reactive import is_state +from jujubigdata import utils + + +if not is_state('kafka.started'): + kafkautils.fail('Kafka service not yet ready') + + +# Grab the business +topic_name = hookenv.action_get('topic') +data = hookenv.action_get('data') + +# Write to the topic if kafka is running +if host.service_available('kafka-server') and host.service_running('kafka-server'): + host = subprocess.check_output(['hostname', '-s']).decode('utf8').strip() + port = get_layer_opts().port('kafka') + zookeepers = kafkautils.get_zookeepers() + try: + output = utils.run_as('kafka', 'kafka-console-producer.sh', + '--broker-list', '{}:{}'.format(host, port), + '--topic', topic_name, + capture_output=True, + input=bytes(data, 'UTF-8')) + except subprocess.CalledProcessError as e: + kafkautils.fail('Kafka command failed', e.output) + else: + hookenv.action_set({'output': output}) +else: + kafkautils.fail('Kafka service is not running') diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/config.yaml b/bigtop-packages/src/charm/kafka/layer-kafka/config.yaml new file mode 100644 index 000000000..0922423e1 --- /dev/null +++ b/bigtop-packages/src/charm/kafka/layer-kafka/config.yaml @@ -0,0 +1,7 @@ +options: + bind_addr: + default: null + type: string + description: | + IP address of the interface to listen on, if something other than the + default is desired. \ No newline at end of file diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/copyright b/bigtop-packages/src/charm/kafka/layer-kafka/copyright new file mode 100644 index 000000000..e900b97c4 --- /dev/null +++ b/bigtop-packages/src/charm/kafka/layer-kafka/copyright @@ -0,0 +1,16 @@ +Format: http://dep.debian.net/deps/dep5/ + +Files: * +Copyright: Copyright 2015, Canonical Ltd., All Rights Reserved. +License: Apache License 2.0 + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/icon.svg b/bigtop-packages/src/charm/kafka/layer-kafka/icon.svg new file mode 100644 index 000000000..1564f9941 --- /dev/null +++ b/bigtop-packages/src/charm/kafka/layer-kafka/icon.svg @@ -0,0 +1,90 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/layer.yaml b/bigtop-packages/src/charm/kafka/layer-kafka/layer.yaml new file mode 100644 index 000000000..daa7f5851 --- /dev/null +++ b/bigtop-packages/src/charm/kafka/layer-kafka/layer.yaml @@ -0,0 +1,20 @@ +repo: git@github.com:juju-solutions/layer-apache-bigtop-kafka.git +includes: + - 'layer:apache-bigtop-base' + - 'interface:zookeeper' + - 'interface:kafka' +options: + apache-bigtop-base: + groups: + - kafka + users: + kafka: + groups: ['kafka'] + ports: + # Ports that need to be exposed, overridden, or manually specified. + # Only expose ports serving a UI or external API (i.e., namenode and + # resourcemanager). Communication among units within the cluster does + # not need ports to be explicitly opened. + kafka: + port: 9092 + exposed_on: 'kafka' diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/lib/charms/layer/bigtop_kafka.py b/bigtop-packages/src/charm/kafka/layer-kafka/lib/charms/layer/bigtop_kafka.py new file mode 100755 index 000000000..77925c83b --- /dev/null +++ b/bigtop-packages/src/charm/kafka/layer-kafka/lib/charms/layer/bigtop_kafka.py @@ -0,0 +1,88 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +from charmhelpers.core import hookenv +from charmhelpers.core import host +from jujubigdata import utils +from charms.layer.apache_bigtop_base import Bigtop +from charms import layer +from subprocess import check_output + + +class Kafka(object): + """ + This class manages Kafka. + """ + def __init__(self): + self.dist_config = utils.DistConfig( + data=layer.options('apache-bigtop-base')) + + def open_ports(self): + for port in self.dist_config.exposed_ports('kafka'): + hookenv.open_port(port) + + def close_ports(self): + for port in self.dist_config.exposed_ports('kafka'): + hookenv.close_port(port) + + def configure_kafka(self, zk_units): + # Get ip:port data from our connected zookeepers + zks = [] + for unit in zk_units: + ip = utils.resolve_private_address(unit['host']) + zks.append("%s:%s" % (ip, unit['port'])) + zks.sort() + zk_connect = ",".join(zks) + service, unit_num = os.environ['JUJU_UNIT_NAME'].split('/', 1) + kafka_port = self.dist_config.port('kafka') + + roles = ['kafka-server'] + override = { + 'kafka::server::broker_id': unit_num, + 'kafka::server::port': kafka_port, + 'kafka::server::zookeeper_connection_string': zk_connect, + } + bind_addr = hookenv.config().get('bind_addr') + if bind_addr: + override['kafka::server::bind_addr'] = bind_addr + + bigtop = Bigtop() + bigtop.render_site_yaml(roles=roles, overrides=override) + bigtop.trigger_puppet() + self.set_advertise() + self.restart() + + def restart(self): + self.stop() + self.start() + + def start(self): + host.service_start('kafka-server') + + def stop(self): + host.service_stop('kafka-server') + + def set_advertise(self): + short_host = check_output(['hostname', '-s']).decode('utf8').strip() + + # Configure server.properties + # NB: We set the advertised.host.name below to our short hostname + # to kafka (admin will still have to expose kafka and ensure the + # external client can resolve the short hostname to our public ip). + kafka_server_conf = '/etc/kafka/conf/server.properties' + utils.re_edit_in_place(kafka_server_conf, { + r'^#?advertised.host.name=.*': 'advertised.host.name=%s' % short_host, + }) diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/metadata.yaml b/bigtop-packages/src/charm/kafka/layer-kafka/metadata.yaml new file mode 100644 index 000000000..b8a777ca9 --- /dev/null +++ b/bigtop-packages/src/charm/kafka/layer-kafka/metadata.yaml @@ -0,0 +1,32 @@ +name: kafka +summary: High-performance distributed messaging system +maintainer: Juju Big Data +description: | + Fast + A single Kafka broker can handle hundreds of megabytes of reads and writes per + second from thousands of clients. + + Scalable + Kafka is designed to allow a single cluster to serve as the central data + backbone for a large organization. It can be elastically and transparently + expanded without downtime. Data streams are partitioned and spread over a + cluster of machines to allow data streams larger than the capability of any + single machine and to allow clusters of co-ordinated consumers. + + Durable + Messages are persisted on disk and replicated within the cluster to prevent + data loss. Each broker can handle terabytes of messages without performance + impact. + + Distributed by Design + Kafka has a modern cluster-centric design that offers strong durability and + fault-tolerance guarantees. + + This charm provides Kafka via Apache Bigtop. +tags: ["apache", "bigdata", "bigtop"] +provides: + client: + interface: kafka +requires: + zookeeper: + interface: zookeeper diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/reactive/kafka.py b/bigtop-packages/src/charm/kafka/layer-kafka/reactive/kafka.py new file mode 100644 index 000000000..1b31ad1bf --- /dev/null +++ b/bigtop-packages/src/charm/kafka/layer-kafka/reactive/kafka.py @@ -0,0 +1,81 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from charmhelpers.core import hookenv +from charms.layer.apache_bigtop_base import get_layer_opts +from charms.layer.bigtop_kafka import Kafka +from charms.reactive import set_state, remove_state, when, when_not +from charms.reactive.helpers import data_changed + + +@when('bigtop.available') +@when_not('zookeeper.joined') +def waiting_for_zookeeper(): + hookenv.status_set('blocked', 'waiting for relation to zookeeper') + + +@when('bigtop.available', 'zookeeper.joined') +@when_not('kafka.started', 'zookeeper.ready') +def waiting_for_zookeeper_ready(zk): + hookenv.status_set('waiting', 'waiting for zookeeper to become ready') + + +@when('bigtop.available', 'zookeeper.ready') +@when_not('kafka.started') +def configure_kafka(zk): + hookenv.status_set('maintenance', 'setting up kafka') + kafka = Kafka() + zks = zk.zookeepers() + kafka.configure_kafka(zks) + kafka.open_ports() + set_state('kafka.started') + hookenv.status_set('active', 'ready') + + +@when('kafka.started', 'zookeeper.ready') +def configure_kafka_zookeepers(zk): + """Configure ready zookeepers and restart kafka if needed. + + As zks come and go, server.properties will be updated. When that file + changes, restart Kafka and set appropriate status messages. + """ + zks = zk.zookeepers() + if not data_changed('zookeepers', zks): + return + + hookenv.log('Checking Zookeeper configuration') + hookenv.status_set('maintenance', 'updating zookeeper instances') + kafka = Kafka() + kafka.configure_kafka(zks) + hookenv.status_set('active', 'ready') + + +@when('kafka.started') +@when_not('zookeeper.ready') +def stop_kafka_waiting_for_zookeeper_ready(): + hookenv.status_set('maintenance', 'zookeeper not ready, stopping kafka') + kafka = Kafka() + kafka.close_ports() + kafka.stop() + remove_state('kafka.started') + hookenv.status_set('waiting', 'waiting for zookeeper to become ready') + + +@when('client.joined', 'zookeeper.ready') +def serve_client(client, zookeeper): + kafka_port = get_layer_opts().port('kafka') + client.send_port(kafka_port) + client.send_zookeepers(zookeeper.zookeepers()) + hookenv.log('Sent Kafka configuration to client') diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/tests/01-deploy.py b/bigtop-packages/src/charm/kafka/layer-kafka/tests/01-deploy.py new file mode 100755 index 000000000..af11f9c80 --- /dev/null +++ b/bigtop-packages/src/charm/kafka/layer-kafka/tests/01-deploy.py @@ -0,0 +1,58 @@ +#!/usr/bin/env python3 + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import unittest +import amulet + + +class TestDeploy(unittest.TestCase): + """ + Trivial deployment test for Apache Kafka. + """ + @classmethod + def setUpClass(cls): + cls.d = amulet.Deployment(series='trusty') + cls.d.add('kafka', 'kafka') + cls.d.add('openjdk', 'openjdk') + cls.d.add('zk', 'zookeeper') + + cls.d.configure('openjdk', {'java-type': 'jdk', + 'java-major': '8'}) + + cls.d.relate('kafka:zookeeper', 'zk:zookeeper') + cls.d.relate('kafka:java', 'openjdk:java') + try: + cls.d.relate('zk:java', 'openjdk:java') + except ValueError: + # No need to related older versions of the zookeeper charm + # to java. + pass + + cls.d.setup(timeout=900) + cls.d.sentry.wait_for_messages({'kafka': 'ready'}, timeout=1800) + cls.kafka = cls.d.sentry['kafka'][0] + + def test_deploy(self): + """ + Simple test to make sure the Kafka java process is running. + """ + output, retcode = self.kafka.run("pgrep -a java") + assert 'Kafka' in output, "Kafka daemon is not started" + + +if __name__ == '__main__': + unittest.main() diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/tests/02-smoke-test.py b/bigtop-packages/src/charm/kafka/layer-kafka/tests/02-smoke-test.py new file mode 100755 index 000000000..330cde9ff --- /dev/null +++ b/bigtop-packages/src/charm/kafka/layer-kafka/tests/02-smoke-test.py @@ -0,0 +1,59 @@ +#!/usr/bin/env python3 + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import unittest +import amulet + + +class TestDeploy(unittest.TestCase): + """ + Smoke test of Apache Kafka. + """ + @classmethod + def setUpClass(cls): + cls.d = amulet.Deployment(series='trusty') + cls.d.add('kafka', 'kafka') + cls.d.add('openjdk', 'openjdk') + cls.d.add('zk', 'zookeeper') + + cls.d.configure('openjdk', {'java-type': 'jdk', + 'java-major': '8'}) + + cls.d.relate('kafka:zookeeper', 'zk:zookeeper') + cls.d.relate('kafka:java', 'openjdk:java') + try: + cls.d.relate('zk:java', 'openjdk:java') + except ValueError: + # No need to related older versions of the zookeeper charm + # to java. + pass + + cls.d.setup(timeout=900) + cls.d.sentry.wait_for_messages({'kafka': 'ready'}, timeout=1800) + cls.kafka = cls.d.sentry['kafka'][0] + + def test_kafka(self): + """ + Validate Kafka by running the smoke-test action. + """ + smk_uuid = self.kafka.action_do("smoke-test") + output = self.d.action_fetch(smk_uuid, full_output=True) + assert "completed" in output['status'] + + +if __name__ == '__main__': + unittest.main() diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/tests/tests.yaml b/bigtop-packages/src/charm/kafka/layer-kafka/tests/tests.yaml new file mode 100644 index 000000000..3b6ce3e5f --- /dev/null +++ b/bigtop-packages/src/charm/kafka/layer-kafka/tests/tests.yaml @@ -0,0 +1,3 @@ +reset: false +packages: + - amulet From a524829accf7f2c21537be62950dee3ec795d3ef Mon Sep 17 00:00:00 2001 From: Pete Vander Giessen Date: Thu, 28 Jul 2016 11:52:45 -0400 Subject: [PATCH 2/7] Fixed up bind address for kafka. (#32) Changed name to "network_interface". You may now pass an interface name or a CIDR range, and juju will figure out the correct ip address to drop into the server.properties config. --- .../src/charm/kafka/layer-kafka/config.yaml | 11 +++++++---- .../layer-kafka/lib/charms/layer/bigtop_kafka.py | 7 ++++--- .../src/charm/kafka/layer-kafka/reactive/kafka.py | 10 +++++++++- 3 files changed, 20 insertions(+), 8 deletions(-) diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/config.yaml b/bigtop-packages/src/charm/kafka/layer-kafka/config.yaml index 0922423e1..c8e723a6d 100644 --- a/bigtop-packages/src/charm/kafka/layer-kafka/config.yaml +++ b/bigtop-packages/src/charm/kafka/layer-kafka/config.yaml @@ -1,7 +1,10 @@ options: - bind_addr: - default: null + network_interface: + default: "" type: string description: | - IP address of the interface to listen on, if something other than the - default is desired. \ No newline at end of file + Network interface to bind Kafka to. Defaults to accepting + connections on all interfaces. Accepts either the name of an + interface (e.g., 'eth0'), or a CIDR range. If the latter, we\'ll + bind to the first interface that we find with an IP address in + that range. diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/lib/charms/layer/bigtop_kafka.py b/bigtop-packages/src/charm/kafka/layer-kafka/lib/charms/layer/bigtop_kafka.py index 77925c83b..3d3c07927 100755 --- a/bigtop-packages/src/charm/kafka/layer-kafka/lib/charms/layer/bigtop_kafka.py +++ b/bigtop-packages/src/charm/kafka/layer-kafka/lib/charms/layer/bigtop_kafka.py @@ -55,9 +55,10 @@ def configure_kafka(self, zk_units): 'kafka::server::port': kafka_port, 'kafka::server::zookeeper_connection_string': zk_connect, } - bind_addr = hookenv.config().get('bind_addr') - if bind_addr: - override['kafka::server::bind_addr'] = bind_addr + network_interface = hookenv.config().get('network_interface') + if network_interface: + ip = Bigtop().get_ip_for_interface(network_interface) + override['kafka::server::bind_addr'] = ip bigtop = Bigtop() bigtop.render_site_yaml(roles=roles, overrides=override) diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/reactive/kafka.py b/bigtop-packages/src/charm/kafka/layer-kafka/reactive/kafka.py index 1b31ad1bf..7e26194e7 100644 --- a/bigtop-packages/src/charm/kafka/layer-kafka/reactive/kafka.py +++ b/bigtop-packages/src/charm/kafka/layer-kafka/reactive/kafka.py @@ -36,6 +36,8 @@ def waiting_for_zookeeper_ready(zk): @when_not('kafka.started') def configure_kafka(zk): hookenv.status_set('maintenance', 'setting up kafka') + data_changed( # Prime data changed for network interface + 'kafka.network_interface', hookenv.config().get('network_interface')) kafka = Kafka() zks = zk.zookeepers() kafka.configure_kafka(zks) @@ -50,9 +52,15 @@ def configure_kafka_zookeepers(zk): As zks come and go, server.properties will be updated. When that file changes, restart Kafka and set appropriate status messages. + + This method also handles the restart if our network_interface + config has changed. + """ zks = zk.zookeepers() - if not data_changed('zookeepers', zks): + network_interface = hookenv.config().get('network_interface') + if not data_changed('zookeepers', zks) and not data_changed( + 'kafka.network_interface', network_interface): return hookenv.log('Checking Zookeeper configuration') From 5cbf8b418a0e38f66bf84784ee17a1e7e4c8f4d2 Mon Sep 17 00:00:00 2001 From: Pete Vander Giessen Date: Fri, 29 Jul 2016 15:11:12 -0400 Subject: [PATCH 3/7] Cleaning up "loose ends" from previous PR. Added documentation on the network_interface config value to README, added tests, and made the way that variables get passed to configure_kafka more consistent. --- .../src/charm/kafka/layer-kafka/README.md | 30 ++++++ .../lib/charms/layer/bigtop_kafka.py | 3 +- .../charm/kafka/layer-kafka/reactive/kafka.py | 2 +- .../layer-kafka/tests/10-config-changed.py | 98 +++++++++++++++++++ 4 files changed, 130 insertions(+), 3 deletions(-) create mode 100755 bigtop-packages/src/charm/kafka/layer-kafka/tests/10-config-changed.py diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/README.md b/bigtop-packages/src/charm/kafka/layer-kafka/README.md index a87d3d8ca..8566e83a6 100644 --- a/bigtop-packages/src/charm/kafka/layer-kafka/README.md +++ b/bigtop-packages/src/charm/kafka/layer-kafka/README.md @@ -159,6 +159,36 @@ The external kafka client should now be able to access Kafka by using `kafka-0:9092` as the broker. +## Network Interfaces + +In some network environments, you may want to restrict kafka to +listening for incoming connections on a specific network interface +(for example, for security reasons). To do so, you may pass either a +network interface name or a CIDR range specifying a subnet to the +``network_interface`` configuration variable. For example: + + juju set-config kafka network_interface=eth0 + +or + + juju set-config kafka network_interface=10.0.2.0/24 + +Each kafka machine in your cluster will lookup the IP address of that +network interface, or find the first network interface with an IP +address in the specified subnet, and bind kafka to that address. + +If you make a mistake, and pass in an invalid name for a network +interface, you may recover by passing the correct name to set-config, +and then running "juju resolved" on each unit: + + juju set-config kafka network_interface=eth0 + juju resolved kafka/0 + +If you want to go back to listening on any network interface on the +machine, simply pass ``0.0.0.0`` to ``network_interface``. + + juju set-config kafka network_interface=0.0.0.0 + ## Contact Information - diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/lib/charms/layer/bigtop_kafka.py b/bigtop-packages/src/charm/kafka/layer-kafka/lib/charms/layer/bigtop_kafka.py index 3d3c07927..c2f1fc480 100755 --- a/bigtop-packages/src/charm/kafka/layer-kafka/lib/charms/layer/bigtop_kafka.py +++ b/bigtop-packages/src/charm/kafka/layer-kafka/lib/charms/layer/bigtop_kafka.py @@ -38,7 +38,7 @@ def close_ports(self): for port in self.dist_config.exposed_ports('kafka'): hookenv.close_port(port) - def configure_kafka(self, zk_units): + def configure_kafka(self, zk_units, network_interface=None): # Get ip:port data from our connected zookeepers zks = [] for unit in zk_units: @@ -55,7 +55,6 @@ def configure_kafka(self, zk_units): 'kafka::server::port': kafka_port, 'kafka::server::zookeeper_connection_string': zk_connect, } - network_interface = hookenv.config().get('network_interface') if network_interface: ip = Bigtop().get_ip_for_interface(network_interface) override['kafka::server::bind_addr'] = ip diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/reactive/kafka.py b/bigtop-packages/src/charm/kafka/layer-kafka/reactive/kafka.py index 7e26194e7..6e7d325ed 100644 --- a/bigtop-packages/src/charm/kafka/layer-kafka/reactive/kafka.py +++ b/bigtop-packages/src/charm/kafka/layer-kafka/reactive/kafka.py @@ -66,7 +66,7 @@ def configure_kafka_zookeepers(zk): hookenv.log('Checking Zookeeper configuration') hookenv.status_set('maintenance', 'updating zookeeper instances') kafka = Kafka() - kafka.configure_kafka(zks) + kafka.configure_kafka(zks, network_interface) hookenv.status_set('active', 'ready') diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/tests/10-config-changed.py b/bigtop-packages/src/charm/kafka/layer-kafka/tests/10-config-changed.py new file mode 100755 index 000000000..dd20c53d9 --- /dev/null +++ b/bigtop-packages/src/charm/kafka/layer-kafka/tests/10-config-changed.py @@ -0,0 +1,98 @@ +#!/usr/bin/env python3 + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import unittest +import amulet +import re + + +class TestConfigChanged(unittest.TestCase): + """ + Test to verify that we update network interface bindings successfully. + + """ + @classmethod + def setUpClass(cls): + cls.d = amulet.Deployment(series='trusty') + cls.d.log.debug("foo!") + cls.d.add('kafka', 'kafka') + cls.d.add('openjdk', 'openjdk') + cls.d.add('zk', 'zookeeper') + + cls.d.configure('openjdk', {'java-type': 'jdk', + 'java-major': '8'}) + + cls.d.relate('kafka:zookeeper', 'zk:zookeeper') + cls.d.relate('kafka:java', 'openjdk:java') + try: + cls.d.relate('zk:java', 'openjdk:java') + except ValueError: + # No need to related older versions of the zookeeper charm + # to java. + pass + + cls.d.setup(timeout=900) + cls.d.sentry.wait_for_messages({'kafka': 'ready'}, timeout=1800) + cls.kafka = cls.d.sentry['kafka'][0] + + def test_bind_network_interface(self): + """ + Test to verify that we update network interface bindings successfully. + + """ + self.d.configure('kafka', {'network_interface': 'eth0'}) + self.d.sentry.wait_for_messages({'kafka': 'updating zookeeper instances'}, timeout=600) + + self.d.sentry.wait_for_messages({'kafka': 'ready'}, timeout=600) + ret = self.kafka.run( + 'grep host.name /etc/kafka/conf/server.properties')[0] + # Correct line should start with host.name (no comment hash + # mark), followed by an equals sign and something that looks + # like an IP address (we aren't too strict about it being a + # valid ip address.) + matcher = re.compile("^host\.name=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}.*") + + self.assertTrue('host.name' in ret) + self.assertTrue(matcher.match(ret)) + + # Verify that smoke tests still run + smk_uuid = self.kafka.action_do("smoke-test") + output = self.d.action_fetch(smk_uuid, full_output=True) + assert "completed" in output['status'] + + def test_reset_network_interface(self): + """ + Verify that we can reset the network interface to 0. + + """ + self.d.configure('kafka', {'network_interface': '0.0.0.0'}) + self.d.sentry.wait_for_messages({'kafka': 'updating zookeeper instances'}, timeout=600) + self.d.sentry.wait_for_messages({'kafka': 'ready'}, timeout=600) + ret = self.kafka.run( + 'grep host.name /etc/kafka/conf/server.properties')[0] + + matcher = re.compile("^host\.name=0\.0\.0\.0.*") + self.assertTrue(matcher.match(ret)) + + # Verify that smoke tests still run + smk_uuid = self.kafka.action_do("smoke-test") + output = self.d.action_fetch(smk_uuid, full_output=True) + assert "completed" in output['status'] + + +if __name__ == '__main__': + unittest.main() From 9239883b3be77ef7403197e01015086bb22f4fb4 Mon Sep 17 00:00:00 2001 From: Pete Vander Giessen Date: Mon, 8 Aug 2016 09:32:50 -0400 Subject: [PATCH 4/7] Added -r to resolved instructions in README. Without -r flag, hook won't auto retry. --- bigtop-packages/src/charm/kafka/layer-kafka/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/README.md b/bigtop-packages/src/charm/kafka/layer-kafka/README.md index 8566e83a6..cee0ecf38 100644 --- a/bigtop-packages/src/charm/kafka/layer-kafka/README.md +++ b/bigtop-packages/src/charm/kafka/layer-kafka/README.md @@ -182,7 +182,7 @@ interface, you may recover by passing the correct name to set-config, and then running "juju resolved" on each unit: juju set-config kafka network_interface=eth0 - juju resolved kafka/0 + juju resolved -r kafka/0 If you want to go back to listening on any network interface on the machine, simply pass ``0.0.0.0`` to ``network_interface``. From fd9605c3e5554dfa46ff8581d3656a0560f96a57 Mon Sep 17 00:00:00 2001 From: Kevin W Monroe Date: Sat, 8 Oct 2016 15:49:24 +0000 Subject: [PATCH 5/7] README tweaks for unified bigtop messaging and juju 2.0 instructions --- .../src/charm/kafka/layer-kafka/README.md | 154 ++++++++++-------- 1 file changed, 90 insertions(+), 64 deletions(-) diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/README.md b/bigtop-packages/src/charm/kafka/layer-kafka/README.md index cee0ecf38..1db627eec 100644 --- a/bigtop-packages/src/charm/kafka/layer-kafka/README.md +++ b/bigtop-packages/src/charm/kafka/layer-kafka/README.md @@ -14,7 +14,7 @@ See the License for the specific language governing permissions and limitations under the License. --> -## Overview +# Overview Apache Kafka is an open-source message broker project developed by the Apache Software Foundation written in Scala. The project aims to provide a unified, @@ -23,53 +23,60 @@ more at [kafka.apache.org](http://kafka.apache.org/). This charm deploys the Kafka component of the Apache Bigtop platform. -## Usage + +# Deploying / Using + +A working Juju installation is assumed to be present. If Juju is not yet set +up, please follow the +[getting-started](https://jujucharms.com/docs/2.0/getting-started) +instructions prior to deploying this charm. Kafka requires the Zookeeper distributed coordination service. Deploy and relate them as follows: - juju deploy zookeeper juju deploy kafka + juju deploy zookeeper juju add-relation kafka zookeeper -Once deployed, we can list the zookeeper servers that our kafka brokers +Once deployed, there are a number of actions available in this charm. +> **Note**: Actions described below assume Juju 2.0 or greater. If using an +earlier version of Juju, the action syntax is: +`juju action do kafka/0 ; juju action fetch `. + +List the zookeeper servers that our kafka brokers are connected to. The following will list `:` information for each zookeeper unit in the environment (e.g.: `10.0.3.221:2181`). juju run-action kafka/0 list-zks juju show-action-output # <-- id from above command -We can create a Kafka topic with: +Create a Kafka topic with: juju run-action kafka/0 create-topic topic= \ partitions=<#> replication=<#> juju show-action-output # <-- id from above command -We can list topics with: +List topics with: juju run-action kafka/0 list-topics juju show-action-output # <-- id from above command -We can write to a topic with: +Write to a topic with: juju run-action kafka/0 write-topic topic= data= juju show-action-output # <-- id from above command -We can read from a topic with: +Read from a topic with: juju run-action kafka/0 read-topic topic= partition=<#> juju show-action-output # <-- id from above command -_**Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the action syntax is:_ - - juju action do kafka/0 - juju action fetch # <-- id from above command +# Verifying -## Status and Smoke Test - -Kafka provides extended status reporting to indicate when it is ready: +## Status +Apache Bigtop charms provide extended status reporting to indicate when they +are ready: juju status @@ -78,64 +85,66 @@ progress of the deployment: watch -n 0.5 juju status -The message for each unit will provide information about that unit's state. -Once they all indicate that they are ready, you can perform a "smoke test" -to verify that Kafka is working as expected using the built-in `smoke-test` -action: +The message column will provide information about a given unit's state. +This charm is ready for use once the status message indicates that it is +ready. - juju run-action kafka/0 smoke-test +## Smoke Test +This charm provides a `smoke-test` action that can be used to verify the +application is functioning as expected. The test will verify connectivity +between Kafka and Zookeeper, and will test creation and listing of Kafka +topics. Run the action as follows: -_**Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju action do kafka/0 smoke-test`._ + juju run-action slave/0 smoke-test -After a minute or so, you can check the results of the smoke test: +> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version +of Juju, the syntax is `juju action do kafka/0 smoke-test`. - juju show-action-status +Watch the progress of the smoke test actions with: -_**Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju action status`._ + watch -n 0.5 juju show-action-status +> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version +of Juju, the syntax is `juju action status`. -You will see `status: completed` if the smoke test was successful, or -`status: failed` if it was not. You can get more information on why it failed -via: +Eventually, the action should settle to `status: completed`. If it +reports `status: failed`, the application is not working as expected. Get +more information about a specific smoke test with: juju show-action-output -_**Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju action fetch `._ +> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version +of Juju, the syntax is `juju action fetch `. -## Scaling +# Scaling -Creating a cluster with many brokers is as easy as adding more Kafka units: +Expanding a cluster with many brokers is as easy as adding more Kafka units: juju add-unit kafka -After adding additional brokers, you will be able to create topics with -replication up to the number of kafka units. - -To verify replication is working you can do the following: +After adding additional brokers, topics may be created with +replication up to the number of ready units. For example, if there are two +ready units, create a replicated topic as follows: - juju add-unit kafka -n 2 juju run-action kafka/0 create-topic topic=my-replicated-topic \ partitions=1 replication=2 -_**Note**: The above assumes Juju 2.0 or greater. If using an earlier version -of Juju, the syntax is `juju action do kafka/0 create-topic `._ +> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version +of Juju, the syntax is `juju action do kafka/0 create-topic `. -Query for the description of the just created topic: +Query the description of the recently created topic: juju run --unit kafka/0 'kafka-topics.sh --describe \ --topic my-replicated-topic --zookeeper :2181' -You should get a response similar to: +An expected response should be similar to: Topic: my-replicated-topic PartitionCount:1 ReplicationFactor:2 Configs: Topic: my-replicated-topic Partition: 0 Leader: 2 Replicas: 2,0 Isr: 2,0 -## Connecting External Clients +# Connecting External Clients By default, this charm does not expose Kafka outside of the provider's network. To allow external clients to connect to Kafka, first expose the service: @@ -159,43 +168,60 @@ The external kafka client should now be able to access Kafka by using `kafka-0:9092` as the broker. -## Network Interfaces +# Network Interfaces -In some network environments, you may want to restrict kafka to -listening for incoming connections on a specific network interface -(for example, for security reasons). To do so, you may pass either a -network interface name or a CIDR range specifying a subnet to the -``network_interface`` configuration variable. For example: +In some network environments, kafka may need to be restricted to +listen for incoming connections on a specific network interface +(e.g.: for security reasons). To do so, configure kafka with either a +network interface name or a CIDR range specifying a subnet. For example: - juju set-config kafka network_interface=eth0 + juju config kafka network_interface=eth0 + juju config kafka network_interface=10.0.2.0/24 -or +> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version +of Juju, the syntax is `juju set-config kafka network_interface=eth0`. - juju set-config kafka network_interface=10.0.2.0/24 - -Each kafka machine in your cluster will lookup the IP address of that +Each kafka machine in the cluster will lookup the IP address of that network interface, or find the first network interface with an IP address in the specified subnet, and bind kafka to that address. -If you make a mistake, and pass in an invalid name for a network -interface, you may recover by passing the correct name to set-config, -and then running "juju resolved" on each unit: +If a mistake is made and an invalid name for the network interface is +configured, recover by re-configuring with the correct name and then +run "juju resolved" on each unit: + + juju config kafka network_interface=eth0 + juju resolved kafka/0 - juju set-config kafka network_interface=eth0 - juju resolved -r kafka/0 +> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version +of Juju, the syntax is `juju set-config kafka network_interface=eth0; +juju resolved -r kafka/0`. -If you want to go back to listening on any network interface on the +To go back to listening on any network interface on the machine, simply pass ``0.0.0.0`` to ``network_interface``. - juju set-config kafka network_interface=0.0.0.0 + juju config kafka network_interface=0.0.0.0 + + +# Network-Restricted Environments +Charms can be deployed in environments with limited network access. To deploy +in this environment, configure a Juju model with appropriate +proxy and/or mirror options. See +[Configuring Models](https://jujucharms.com/docs/2.0/models-config) for more +information. + + +# Contact Information -## Contact Information - -## Help +# Resources + +- [Apache Bigtop](http://bigtop.apache.org/) home page +- [Apache Bigtop mailing lists](http://bigtop.apache.org/mail-lists.html) - [Apache Kafka home page](http://kafka.apache.org/) - [Apache Kafka issue tracker](https://issues.apache.org/jira/browse/KAFKA) +- [Juju Bigtop charms](https://jujucharms.com/q/apache/bigtop) - [Juju mailing list](https://lists.ubuntu.com/mailman/listinfo/juju) - [Juju community](https://jujucharms.com/community) From 17fa10c49311d12ce5fc81d0f09b144edc984f7c Mon Sep 17 00:00:00 2001 From: Kevin W Monroe Date: Sat, 8 Oct 2016 15:59:38 +0000 Subject: [PATCH 6/7] put Bigtop in the summary; inherit tags from bigtop base layer --- bigtop-packages/src/charm/kafka/layer-kafka/metadata.yaml | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/metadata.yaml b/bigtop-packages/src/charm/kafka/layer-kafka/metadata.yaml index b8a777ca9..5a1139487 100644 --- a/bigtop-packages/src/charm/kafka/layer-kafka/metadata.yaml +++ b/bigtop-packages/src/charm/kafka/layer-kafka/metadata.yaml @@ -1,5 +1,5 @@ name: kafka -summary: High-performance distributed messaging system +summary: High-performance distributed messaging system from Apache Bigtop maintainer: Juju Big Data description: | Fast @@ -21,9 +21,7 @@ description: | Distributed by Design Kafka has a modern cluster-centric design that offers strong durability and fault-tolerance guarantees. - - This charm provides Kafka via Apache Bigtop. -tags: ["apache", "bigdata", "bigtop"] +tags: [] provides: client: interface: kafka From 74ebf49d0e4176afea13f6faa08806b396d8bbd5 Mon Sep 17 00:00:00 2001 From: Kevin W Monroe Date: Sat, 8 Oct 2016 17:54:17 +0000 Subject: [PATCH 7/7] remove LICENSE (available in the base source tree) --- .../src/charm/kafka/layer-kafka/LICENSE | 177 ------------------ 1 file changed, 177 deletions(-) delete mode 100644 bigtop-packages/src/charm/kafka/layer-kafka/LICENSE diff --git a/bigtop-packages/src/charm/kafka/layer-kafka/LICENSE b/bigtop-packages/src/charm/kafka/layer-kafka/LICENSE deleted file mode 100644 index f433b1a53..000000000 --- a/bigtop-packages/src/charm/kafka/layer-kafka/LICENSE +++ /dev/null @@ -1,177 +0,0 @@ - - Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS