diff --git a/README.md b/README.md index 08f8aeb..9c9d41a 100644 --- a/README.md +++ b/README.md @@ -2,25 +2,39 @@ IBM Message Hub is a scalable, distributed, high throughput message bus to unite your on-premise and off-premise cloud technologies. You can wire micro-services together using open protocols, connect stream data to analytics to realise powerful insight and feed event data to multiple applications to react in real time. This repository is for samples which interact with the Message Hub service. -Currently, there are samples for the Kafka, MQ Light and REST APIs. +Currently, there are samples for the Kafka and MQ Light APIs. Information and instructions regarding the use of these samples can be found in their respective directories. +## Aim of the Samples +The aim of the samples is to help you get started with Message Hub within minutes. They are not production-ready applications but should give you useful pointers at how to build, package and deploy applications as well as how to make basic API calls against us with error handling. We did all the heavy lifting so you can focus on developing exciting code with value! + +## Provisioning your Message Hub Cluster +In order to provision a Message Hub cluster, please visit the [IBM Cloud® catalog](https://console.stage1.bluemix.net/catalog/). Please also familiarise yourself with Message Hub and Apache Kafka basics and terminology. [Our documentation](https://console.bluemix.net/docs/services/MessageHub/) is a good starting point. + + +### Pricing plans +IBM Message Hub can be provisioned on IBM Cloud® in various pricing plans. Please refer to our [documentation](https://console.bluemix.net/docs/services/MessageHub/messagehub085.html#plan_choose) to help choose a plan that works for you. + +__Important Note__: Provisioning a Message Hub service in IBM Cloud® incurs a fee. Please review pricing before provisioning. The samples in this repository will create topic(s) on your behalf - creating a topic might also incur a fee. For more information, please consult the IBM Cloud® documentation if necessary. + + +## Connecting to your Message Hub Cluster +In each sample, we demonstrate a single connection path for both our Standard and Enterprise plans respectively. The aim was to get you started quickly. However your client's needs might be different. Therefore we wrote a [guide](https://console.bluemix.net/docs/services/MessageHub/messagehub127.html#connect_messagehub) that discusses credential generation in detail and showing you all possible ways of doing this. + +## Our APIs and Sample Applications + ### Kafka API (recommended): -* kafka-java-console-sample : Sample Java console application using the Message Hub Kafka API -* kafka-java-liberty-sample : Sample IBM Websphere Liberty profile application using the Message Hub Kafka API -* kafka-nodejs-console-sample : Sample Node.js console application using the Message Hub Kafka API -* kafka-python-console-sample : Sample Python console application using the Message Hub Kafka API +* [kafka-java-console-sample](/kafka-java-console-sample/README.md) : Sample Java console application using the Message Hub Kafka API +* [kafka-java-liberty-sample](/kafka-java-liberty-sample/README.md) : Sample IBM Websphere Liberty profile application using the Message Hub Kafka API +* [kafka-nodejs-console-sample](kafka-nodejs-console-sample/README.md) : Sample Node.js console application using the Message Hub Kafka API +* [kafka-python-console-sample](/kafka-python-console-sample/README.md) : Sample Python console application using the Message Hub Kafka API ### MQ Light API: -* mqlight : MQ Light samples in Java, Python, Node.js and Ruby +* [mqlight](/mqlight/README.md) : MQ Light samples in Java, Python, Node.js and Ruby -### REST API: -* rest-nodejs-express-sample : Sample Node.js/Express application using the Message Hub REST API +## Get Further Assistance If you have any issues, just ask us a question (tagged with message-hub) on [StackOverflow.com](http://stackoverflow.com/questions/tagged/message-hub). -For more information regarding IBM Message Hub, [view the documentation on Bluemix](https://www.ng.bluemix.net/docs/services/MessageHub/index.html). - -__Important Note__: The samples in this repository will create topic(s) on your behalf - creating a topic incurs a fee. For more information, view the README files in each part of the -repository and consult the Bluemix documentation if necessary. +For more information regarding IBM Message Hub, [view the documentation on IBM Cloud](https://www.console.ng.bluemix.net/docs/services/MessageHub/index.html). \ No newline at end of file diff --git a/connecting_to_mhub.md b/connecting_to_mhub.md new file mode 100644 index 0000000..fe3d519 --- /dev/null +++ b/connecting_to_mhub.md @@ -0,0 +1,133 @@ +# Connecting to Message Hub + +**TODO - Karen** ==> Charlie said: *`"Rename the 'Connecting to your cluster' section to 'Connecting to Message Hub' in the stage1 docs, and move down one level to being under About Message Hub"`* + +How you connect to your cluster varies according to whether you're using the Standard or Enterprise plan and also whether you're connecting from a Cloud Foundry application or from any other external client. You will need two pieces of information to connect to any of our APIs: + +- The APIs' endpoint URLs +- An api key for authentication + +The steps to obtain these details can vary subtly and you should follow the appropriate steps below for your instance: + +## Prerequisites: +### Provision a Message Hub instance + +It is a pre-requisite to [provision a Message Hub service instance](https://console.stage1.bluemix.net/docs/services/MessageHub/messagehub085.html#plan_choose). Please make sure you have already done so, noting your plan of choice. *Provisioning a Message Hub instance might incur a charge.* Afterwards, please refer to the sections below to obtain our API connection details: + +--- + +## Standard Plan: + +Services provisioned using the Standard Plan are Cloud Foundry services. This means they are deployed into a Cloud Foundry organisation and space, and are grouped in the dashboard under the heading 'Cloud Foundry Services'. To connect an application, the method used depends on where the application is deployed: + +### Cloud Foundry applications + +For applications running inside Cloud Foundry, the application must be bound to the Message Hub service instance. Once bound, the connection details are then made available to the application in JSON format in the VCAP_SERVICES environment variable. Binding an application and service can be achieved using either the [web portal](https://console.bluemix.net/) or the [IBM Cloud CLI](https://console.bluemix.net/docs/cli/reference/bluemix_cli/get_started.html). + +**Web Portal** + +1) Make sure you're in the desired Cloud Foundry organization and space +2) Locate your Cloud Foundry Application on the [Dashboard](https://console.stage1.bluemix.net/dashboard/apps/) or create one if you have not got one already by clicking on the "Create Resource" button +3) Click on your application tile +4) Select "Connections" +5) Click "Create Connection" +6) Select the Message Hub service tile you would like to bind to and click "Connect". You might need to restage your application for the changes to take effect. +7) Click the "Runtime" tab on the left and select the "Environment variables" tab in the center. You can now verify your VCAP_SERVICES information. Your app can now access these as environment variables. + +**IBM Cloud CLI** + +1) Make sure you're in the desired Cloud Foundry organization and space, you can navigate interactively by issuing `ibmcloud target -cf` +2) Find your app: `ibmcloud app list`. If you have a manifest file, you can create a new app by running `ibmcloud app push` +3) Find your service: `ibmcloud service list` +4) Bind your app to the service: `ibmcloud service bind ` +5) Verify that the VCAP_SERVICES environment variables are available in your application runtime. You can do this by calling `ibmcloud app env `. +6) Pass these credentials to your application. + +You might need to restage your application for the changes to take effect. + +### External Applications + +For applications running outside of Cloud Foundry, credentials are generated by creating a Service Key. Once obtained, the details of the key then need to be manually passed to your application via mechanisms of your own choice: + +**Portal** + +1) Make sure you're in the desired Cloud Foundry organization and space +2) Locate your Cloud Foundry Message Hub service on the [Dashboard](https://console.stage1.bluemix.net/dashboard) +3) Click on your service tile +4) Select "Service Credentials" +5) Click "New Credential" +6) Fill in the details for your new credential such as name and click "Add". A new credential appears in the credentials list. +7) By clicking on this credential (view credential), the details are revealed in JSON format. +8) Pass these credentials to your application. + +**IBM Cloud CLI** + +1) Make sure you're in the desired Cloud Foundry organization and space, you can navigate interactively by issuing `ibmcloud target -cf` +2) Find your service: `ibmcloud service list` +3) Either create a service key: `ibmcloud service key-create ` or use an existing service key: `ibmcloud service keys ` +4) Get the details for the key: `ibmcloud service key-show ` . This returns the service key details in JSON format. +5) Pass these credentials to your application. + +--- + +## Enterprise Plan: + +Services provisioned using the Enterprise Plan are grouped in the dashboard under the heading 'Services'. The Enterprise Plan is [IAM enabled](https://console.bluemix.net/docs/iam/quickstart.html#getstarted). You don't need to understand IAM to get started but some knowledge is recommended, should you wish to [secure your Message Hub service](https://console.bluemix.net/docs/services/MessageHub/messagehub124.html#security). + +In this section, we're going to explain the steps involved in binding your application and/or obtaining Service Keys for your service. *Please note that in order to be authorized to create topics, your application or Service Key has to be of `Manager` access role.* + +To connect an application, the method used depends on where the application is deployed: + +### Cloud Foundry Applications + +Your application must be bound to the Message Hub service instance. In order to bind a Cloud Foundry Application to a non-Cloud Foundry Service with One Cloud, a Cloud Foundry Service Alias needs to be created first. This alias is then referenced from your Cloud Foundry Application when binding. + +Once bound, the connection details are then made available to the application in JSON format through the VCAP_SERVICES environment variable. Binding an application and service can be achieved using either the [web portal](https://console.bluemix.net/) or the [IBM Cloud CLI](https://console.bluemix.net/docs/cli/reference/bluemix_cli/get_started.html). + +**Portal** +1) Make sure you're in the desired Cloud Foundry organization and space +2) Locate your Cloud Foundry Application on the [Dashboard](https://console.stage1.bluemix.net/dashboard/apps/) or create one if you have not got one already by clicking on the "Create Resource" button +3) Click on your application tile +4) Select "Connections" +5) Click "Create Connection" +6) Select the Message Hub service tile you would like to bind to and click "Connect" +7) The above step will create a Cloud Foundry Service Alias for your Message Hub service and bind your application to this alias afterwards. On the portal, this happens automatically but on the IBM Cloud CLI, a separate manual step is required. You might need to restage your application for the changes to take effect. +8) Click the "Runtime" tab on the left and select the "Environment variables" tab in the center. You can now verify your VCAP_SERVICES information. Your app can now access these as environment variables. + +**IBM Cloud CLI** +1) Make sure you're in the desired Cloud Foundry organization and space, you can navigate interactively by issuing `ibmcloud target -cf` +2) Find your app: `ibmcloud app list`. If you have a manifest file, you can create a new app by running `ibmcloud app push`. As the app is not bound to Message Hub yet, it will not be able to establish a connection. Therefore it is recommended to push the application with the ` --no-start` parameter to avoid unnecessary connection failures. +3) Find your service: `ibmcloud resource service-instances` +4) Create a Cloud Foundry Service Alias: `ibmcloud resource service-alias-create --instance-name ` +5) Bind your app to the Service Alias created above: `ibmcloud service bind `. Alternatively you can update your manifest file and re-push the application. +6) Verify that the VCAP_SERVICES environment variable is available in your application runtime. You can do this by calling `ibmcloud app env `. +7) Pass these credentials to your application. + +You might need to restage your application for the changes to take effect. + + +### External Applications + +For applications running outside of Cloud Foundry, credentials are generated by creating a Service Key. Once obtained, the details of the key then need to be manually passed to your application via mechanisms of your own choice: + +**Portal** +1) Locate your Message Hub service on the [Dashboard](https://console.stage1.bluemix.net/dashboard) +2) Click on your service tile +3) Select "Service Credentials" +4) Click "New Credential" +5) Fill in the details for your new credential and click "Add". A new credential will appear in the credentials list. +6) By clicking on this credential (view credential), the details are revealed in JSON format. +7) Pass these credentials to your application. Make sure your application parses the details. + +**IBM Cloud CLI** +1) Find your service: `ibmcloud resource service-instances` +2) Create a Service Key: `ibmcloud resource service-key-create --instance-name ` +3) Print the Service Key: `ibmcloud resource service-key ` +4) Pass these credentials to your application. Make sure your application parses the details. + +**TODO - Karen** ==> Charlie gave me the instructions below, however, I'm not sure it's supposed to be in this section as it's not about how to use IBM Cloud but about how to use Kafka? Maybe the section he highlights could be copied over and referenced/linked from here? I think this is too specific... + +Charlie said: + +*`"Add some info describing how to take the information made available from above e.g. like the info in the Connecting a client to the Kafka API section of the alpha docs on stage 1? https://console.stage1.bluemix.net/docs/services/MessageHub/messagehub122.html#alpha_about "`* + diff --git a/java/message-hub-liberty-sample/lib-message-hub/README.md b/java/message-hub-liberty-sample/lib-message-hub/README.md deleted file mode 100644 index 57513be..0000000 --- a/java/message-hub-liberty-sample/lib-message-hub/README.md +++ /dev/null @@ -1,5 +0,0 @@ -# message-hub-login-library - -This library is required to connect to IBM Message Hub __only__ when using the Kafka __0.9.x__ Java client. - -It is not needed any more to run the Java samples included in this repository. diff --git a/java/message-hub-liberty-sample/lib-message-hub/README.md b/java/message-hub-liberty-sample/lib-message-hub/README.md new file mode 120000 index 0000000..6a0a832 --- /dev/null +++ b/java/message-hub-liberty-sample/lib-message-hub/README.md @@ -0,0 +1 @@ +../../../kafka-0.9/message-hub-login-library/README.md \ No newline at end of file diff --git a/java/message-hub-liberty-sample/lib-message-hub/messagehub.login-1.0.0.jar b/java/message-hub-liberty-sample/lib-message-hub/messagehub.login-1.0.0.jar deleted file mode 100644 index 00ae55b..0000000 Binary files a/java/message-hub-liberty-sample/lib-message-hub/messagehub.login-1.0.0.jar and /dev/null differ diff --git a/java/message-hub-liberty-sample/lib-message-hub/messagehub.login-1.0.0.jar b/java/message-hub-liberty-sample/lib-message-hub/messagehub.login-1.0.0.jar new file mode 120000 index 0000000..776d3f2 --- /dev/null +++ b/java/message-hub-liberty-sample/lib-message-hub/messagehub.login-1.0.0.jar @@ -0,0 +1 @@ +../../../kafka-0.9/message-hub-login-library/messagehub.login-1.0.0.jar \ No newline at end of file diff --git a/java/message-hub-liberty-sample/lib-message-hub/messagehub.login-1.0.0.jar.asc b/java/message-hub-liberty-sample/lib-message-hub/messagehub.login-1.0.0.jar.asc deleted file mode 100644 index 1700780..0000000 --- a/java/message-hub-liberty-sample/lib-message-hub/messagehub.login-1.0.0.jar.asc +++ /dev/null @@ -1,11 +0,0 @@ ------BEGIN PGP SIGNATURE----- -Version: GnuPG v1 - -iQEcBAABAgAGBQJWXHVJAAoJEEv2uTXLGdqSTEgIAIz/vgFHbiuPH8y7FkdjPov6 -agiRXP98cxUTq2NZiO8Wm4EasC96eYTdr6SvfVbF59RBwGepUSMe9PgsLL1QYFew -kU+Jv2dNV0te2gRWfyrnJEqwjiyMdIzxjtbWSxMx2MsMoyZegXUsBahKtI9D0DnJ -goaaYp/i3owAHFAQSWFrVHzP6yynEd1Uy2VcmkbjUFkK7YcVcNvJBTLPlaKs+3CK -pNrC5TF0iRaP26WcU+SZ6YcngAPeigQsaKaTeFBgBzAPPxg2918qiYk0w/9H813Q -8E7+/ljnR1AntXhwjJWu0cQSF9/hzxc3WqisOj+MNdoYs/6i45zAYm6xso3VSJo= -=HOc2 ------END PGP SIGNATURE----- diff --git a/java/message-hub-liberty-sample/lib-message-hub/messagehub.login-1.0.0.jar.asc b/java/message-hub-liberty-sample/lib-message-hub/messagehub.login-1.0.0.jar.asc new file mode 120000 index 0000000..6b3d10d --- /dev/null +++ b/java/message-hub-liberty-sample/lib-message-hub/messagehub.login-1.0.0.jar.asc @@ -0,0 +1 @@ +../../../kafka-0.9/message-hub-login-library/messagehub.login-1.0.0.jar.asc \ No newline at end of file diff --git a/kafka-java-console-sample/README.md b/kafka-java-console-sample/README.md index 52a899e..4836f6f 100644 --- a/kafka-java-console-sample/README.md +++ b/kafka-java-console-sample/README.md @@ -1,91 +1,39 @@ # IBM Message Hub Kafka Java console sample application This Java console application demonstrates how to connect to [IBM Message Hub](https://console.ng.bluemix.net/docs/services/MessageHub/index.html), send and receive messages using the [Kafka](https://kafka.apache.org) Java API. It also shows how to create and list topics using the Message Hub Admin REST API. -It can be run locally on your machine or deployed into [IBM Bluemix](https://console.ng.bluemix.net/). - -__Important Note__: This sample creates on your behalf a topic named `kafka-java-console-sample-topic` with one partition - this will incur a fee if the topic does not already exist on your account. - -## Global Prerequisites -To build and run the sample, you must have the following installed: -* [git](https://git-scm.com/) -* [Gradle](https://gradle.org/) -* Java 7+ -* [Message Hub Service Instance](https://console.ng.bluemix.net/catalog/services/message-hub/) provisioned in [IBM Bluemix](https://console.ng.bluemix.net/) - -## Prerequisites (Bluemix) -* [Cloud Foundry Command Line Interface](https://github.com/cloudfoundry/cli/releases) installed - -## Running the Build Script -Run the following commands on your local machine, after the prerequisites for your environment have been completed: -```shell -gradle clean && gradle build - ``` - -## Running the Sample (Local) -Once built, to run the sample, execute the following command: -```shell -java -jar build/libs/kafka-java-console-sample-2.0.jar -``` - -To find the values for ``, `` and ``, access your Message Hub instance in Bluemix, go to the `Service Credentials` tab and select the `Credentials` you want to use. If your user value is `token`, specify that with the password seperated by a `:`. - -__Note__: `` must be a single string enclosed in quotes. For example: `"host1:port1,host2:port2"`. We recommend using all the Kafka hosts listed in the `Credentials` you selected. - -Alternatively, you can run only the producer or only the consumer by respectively adding `-producer` or `-consumer` to the command above. - -You can also specify the Kafka topic to use by adding the `-topic` argument. For -example specifying: `-topic mytopic` would use Kafka topic `mytopic`. - -The sample will run indefinitely until interrupted. To stop the process, use `Ctrl+C`, for example. - -## Running the Sample (Bluemix) -Ensure that the previous `gradle build` command has produced a zip file artifact under `build/distributions`. - -Open the `manifest.yml` file and rename the `"Message Hub-CHANGEME"` entry to that of your own -Message Hub Service Instance name. - -Connect to Bluemix with the Cloud Foundry Command Line Interface, then run the following command in -the same directory as the `manifest.yml` file: -```shell -cf push -``` - -__Note:__ The Bluemix distribution will automatically update the required files for you at runtime, -using the `VCAP_SERVICES` information provided by Bluemix. - -## Running the Sample (Docker) -You can also build the sample into a docker image using the following command: -```shell -docker build --tag kafka-java-console-sample . -``` - -The container can then be run as follows: -```shell -docker run -it kafka-java-console-sample -``` - -The values for ``, ``, and `` are -as above. - -## Sample Output -Below is a snippet of the output generated by the sample: - -``` -[2016-11-30 17:30:53,492] INFO Running in local mode. (com.messagehub.samples.MessageHubConsoleSample) -[2016-11-30 17:30:53,492] INFO Updating JAAS configuration (com.messagehub.samples.MessageHubConsoleSample) -[2016-11-30 17:30:53,506] INFO Kafka Endpoints: kafka01-prod01.messagehub.services.us-south.bluemix.net:9093,kafka02-prod01.messagehub.services.us-south.bluemix.net:9093,kafka03-prod01.messagehub.services.us-south.bluemix.net:9093,kafka04-prod01.messagehub.services.us-south.bluemix.net:9093,kafka05-prod01.messagehub.services.us-south.bluemix.net:9093 (com.messagehub.samples.MessageHubConsoleSample) -[2016-11-30 17:30:53,506] INFO Admin REST Endpoint: https://kafka-admin-prod01.messagehub.services.us-south.bluemix.net:443 (com.messagehub.samples.MessageHubConsoleSample) -[2016-11-30 17:30:53,506] INFO Creating the topic kafka-java-console-sample-topic (com.messagehub.samples.MessageHubConsoleSample) - (com.messagehub.samples.MessageHubConsoleSample)e :{} -[2016-11-30 17:30:54,947] INFO Admin REST Listing Topics: [{"name":"kafka-java-console-sample-topic","partitions":1,"retentionMs":"86400000","markedForDeletion":false}] (com.messagehub.samples.MessageHubConsoleSample) -[2016-11-30 17:30:55,952] INFO [Partition(topic = kafka-java-console-sample-topic, partition = 0, leader = 0, replicas = [0,1,4,], isr = [0,4,1,]] (com.messagehub.samples.ConsumerRunnable) -[2016-11-30 17:30:55,953] INFO class com.messagehub.samples.ConsumerRunnable is starting. (com.messagehub.samples.ConsumerRunnable) -[2016-11-30 17:30:57,023] INFO [Partition(topic = kafka-java-console-sample-topic, partition = 0, leader = 0, replicas = [0,1,4,], isr = [0,4,1,]] (com.messagehub.samples.ProducerRunnable) -[2016-11-30 17:30:57,024] INFO MessageHubConsoleSample will run until interrupted. (com.messagehub.samples.MessageHubConsoleSample) -[2016-11-30 17:30:57,024] INFO class com.messagehub.samples.ProducerRunnable is starting. (com.messagehub.samples.ProducerRunnable) -[2016-11-30 17:30:58,018] INFO Message produced, offset: 0 (com.messagehub.samples.ProducerRunnable) -[2016-11-30 17:30:58,956] INFO No messages consumed (com.messagehub.samples.ConsumerRunnable) -[2016-11-30 17:31:00,301] INFO Message consumed: ConsumerRecord(topic = kafka-java-console-sample-topic, partition = 0, offset = 1, CreateTime = 1480527060022, checksum = 1906962734, serialized key size = 3, serialized value size = 25, key = key, value = This is a test message #1) (com.messagehub.samples.ConsumerRunnable) -[2016-11-30 17:31:00,397] INFO Message produced, offset: 1 (com.messagehub.samples.ProducerRunnable) -[2016-11-30 17:31:02,550] INFO Message consumed: ConsumerRecord(topic = kafka-java-console-sample-topic, partition = 0, offset = 2, CreateTime = 1480527062401, checksum = 3801731428, serialized key size = 3, serialized value size = 25, key = key, value = This is a test message #2) (com.messagehub.samples.ConsumerRunnable) -``` +It can be run locally on your machine or deployed into [IBM Cloud](https://console.ng.bluemix.net/). The readme focuses on running the application on a local console. + +For help with additional deployment modes, please refer to our [connection guide](https://console.stage1.bluemix.net/docs/services/MessageHub/messagehub127.html#connect_messagehub). + +__Important Note__: This sample creates a topic named `kafka-java-console-sample-topic` on your behalf with one partition - this will incur a fee on our Standard Plan if the topic does not already exist on your account. + +## Prerequisites +To build and run the sample, you must have the done the following: + +* Obtain this repository's contents, either use `git` or just download the samples as a ZIP +* Install [Gradle](https://gradle.org/) +* Install Java 7+ +* Install the [IBM Cloud CLI](https://console.bluemix.net/docs/cli/reference/bluemix_cli/download_cli.html) +* Provision a [Message Hub Service Instance](https://console.ng.bluemix.net/catalog/services/message-hub/) in [IBM Cloud®](https://console.ng.bluemix.net/) + +## Standard or Enterprise Plan? + +**It's important to know which Message Hub plan you're using as the sample deployment steps are subtly different on each plan respectively.** + +By this point, you should have a Message Hub instance provisioned. If you haven't done this step yet, please refer to the main [readme](/README.md). + +If you are not sure what type of Message Hub instance you have then you can find this information out by visiting IBM Cloud's web console [dashboard](https://console.bluemix.net/dashboard). + +*Please make sure you are in the appropriate Region, Account, Organization and Space where you provisioned your Message Hub instance!* + +* Message Hub Standard plan services are "Cloud Foundry Services" with the plan column showing "Standard". +* Message Hub Enterprise plan services are "Services" with the plan column showing "Enterprise". + + +## Deploy the Application + +As the Standard and Enterprise Plan deployment steps are subtly different, we split the deployment steps into separate sections. Please navigate to the appropriate page(s): + +### [Standard Plan Deployment Guide](./standard_plan.md) + +### [Enterprise plan Deployment Guide](./enterprise_plan.md) diff --git a/kafka-java-console-sample/enterprise_plan.md b/kafka-java-console-sample/enterprise_plan.md new file mode 100644 index 0000000..9c978c0 --- /dev/null +++ b/kafka-java-console-sample/enterprise_plan.md @@ -0,0 +1,89 @@ +# IBM Message Hub Kafka Java console sample application deployment to an Enterprise Plan Message Hub + +## Overview + +To run the samples, you will: + +* Set up a Cloud Foundry Service Alias for your Service +* Setup your `manifest.yml` with your service details +* Build and package the sample application +* Use `ibmcloud cf push` to deploy the app to IBM Cloud® +* Create a binding in between your application and your Cloud Foundry Service Alias +* Start the application +* View the application's logs + +## Prerequisites +* You logged in to IBM Cloud via the command line tool, [IBM Cloud CLI](https://console.bluemix.net/docs/cli/reference/bluemix_cli/get_started.html#getting-started) + +## Set up a cloud Foundry alias for your Service +The Enterprise plan is IAM enabled. Therefore the following extra step is required to create a Cloud Foundry alias for your Service: + +Create a Cloud Foundry alias for your service's associated CRN (your selected organization and space will be where your Service Alias will be created. If you wish to change your organization and space, you can use `ibmcloud target -cf`): + +```shell +ibmcloud resource service-alias-create --instance-name +``` + +Having created this alias associated your Service with a Cloud Foundry Organization and Space, thereby enabling your Cloud Foundry application to referrence it and connect to it. + +## Setup the manifest.yml +To deploy the sample as a Cloud Foundry application, you need to edit `manifest.yml`. + +You need to point the application at your Message Hub Cloud Foundry Service **Alias** service by editing: + +``` + services: + - "" +``` + +You can optionally rename your application, eg.: + +``` +name: kafka-java-console-sample +``` + +## Build the sample +Run the following commands on your local machine to build and package the sample application + +```shell +gradle clean && gradle build + ``` + +Ensure that the `gradle build` command has produced a zip file artifact under `build/distributions`. + +Navigate to the appropriate Cloud Foundry Account, Organization and Space of your choice. If you're new to IBM Cloud then just navigate to the place where your Message Hub instance has been provisioned. *(If you're following this guide from the start then you can skip this step safely as you're already at the location where your Service Alias has been created.)* + +## Push the application into IBM Cloud® +Push the app without starting it immediately by running the following command in the same directory as the `manifest.yml` file: +``` +ibmcloud app push --no-start +``` + +## Re-configure the binding +A binding between your app and service-alias is created for you automatically, but by default does not have permissions to create topics. This means that we need to create the existing binding and create a new one with the correct role: + +``` +ibmcloud resource service-binding-delete +ibmcloud resource service-binding-create Manager +``` + +## Start the app + +```shell +ibmcloud app start +``` + +The sample application should have created the default sample topic and started producing and consuming messages in an infinte loop. View the logs to verify this: + +``` +ibmcloud app logs +``` + +__Note:__ The IBM Cloud distribution will automatically update the required files for you at runtime, +using the `VCAP_SERVICES` information provided by IBM Cloud. + +## Further references + +If you are unfamiliar with Cloud Foundry applications then you might find the following document useful: + +[Cloud Foundry manifest documentation](http://docs.cloudfoundry.org/devguide/deploy-apps/manifest.html) \ No newline at end of file diff --git a/kafka-java-console-sample/manifest.yml b/kafka-java-console-sample/manifest.yml index e384a6b..007f052 100644 --- a/kafka-java-console-sample/manifest.yml +++ b/kafka-java-console-sample/manifest.yml @@ -7,4 +7,4 @@ applications: no-route: true health-check-type: none services: - - "Message Hub-CHANGEME" + - "" diff --git a/kafka-java-console-sample/src/com/messagehub/samples/MessageHubConsoleSample.java b/kafka-java-console-sample/src/com/messagehub/samples/MessageHubConsoleSample.java index 4476312..f80a607 100644 --- a/kafka-java-console-sample/src/com/messagehub/samples/MessageHubConsoleSample.java +++ b/kafka-java-console-sample/src/com/messagehub/samples/MessageHubConsoleSample.java @@ -123,7 +123,7 @@ public static void main(String args[]) { MessageHubCredentials credentials = BluemixEnvironment.getMessageHubCredentials(); bootstrapServers = stringArrayToCSV(credentials.getKafkaBrokersSasl()); - adminRestURL = credentials.getKafkaRestUrl(); + adminRestURL = credentials.getKafkaAdminUrl(); apiKey = credentials.getApiKey(); user = credentials.getUser(); password = credentials.getPassword(); diff --git a/kafka-java-console-sample/src/com/messagehub/samples/bluemix/MessageHubCredentials.java b/kafka-java-console-sample/src/com/messagehub/samples/bluemix/MessageHubCredentials.java index 2b559a6..99c5c75 100644 --- a/kafka-java-console-sample/src/com/messagehub/samples/bluemix/MessageHubCredentials.java +++ b/kafka-java-console-sample/src/com/messagehub/samples/bluemix/MessageHubCredentials.java @@ -25,7 +25,7 @@ @JsonIgnoreProperties(ignoreUnknown=true) public class MessageHubCredentials { - private String apiKey, kafkaRestUrl, user, password; + private String apiKey, kafkaAdminUrl, user, password; private String[] kafkaBrokersSasl; @JsonProperty("api_key") @@ -38,14 +38,14 @@ public void setLabel(String apiKey) { this.apiKey = apiKey; } - @JsonProperty("kafka_rest_url") - public String getKafkaRestUrl() { - return kafkaRestUrl; + @JsonProperty("kafka_admin_url") + public String getKafkaAdminUrl() { + return kafkaAdminUrl; } - @JsonProperty("kafka_rest_url") - public void setKafkaRestUrl(String kafkaRestUrl) { - this.kafkaRestUrl = kafkaRestUrl; + @JsonProperty("kafka_admin_url") + public void setKafkaAdminUrl(String kafkaAdminUrl) { + this.kafkaAdminUrl = kafkaAdminUrl; } @JsonProperty diff --git a/kafka-java-console-sample/standard_plan.md b/kafka-java-console-sample/standard_plan.md new file mode 100644 index 0000000..88e3192 --- /dev/null +++ b/kafka-java-console-sample/standard_plan.md @@ -0,0 +1,59 @@ +# IBM Message Hub Kafka Java console sample application deployment to a Standard Plan Message Hub + + +## Overview +To run the samples, you will: +* Setup your `manifest.yml` with your service details +* Build and package the sample application +* Use `ibmcloud cf push` to deploy the app to IBM Cloud® +* View the application's logs + +## Setup the manifest.yml +To deploy the sample as a Cloud Foundry application, you need to edit `manifest.yml`. + +You need to point the application at your Message Hub service by editing: + +``` + services: + - "" +``` + +You can optionally rename your application, eg.: + +``` +name: kafka-java-console-sample +``` + +## Running the Build Script +Run the following commands on your local machine, after the prerequisites for your environment have been completed: +```shell +gradle clean && gradle build + ``` + +## Running the Sample on IBM Cloud +1) Ensure that the previous `gradle build` command has produced a zip file artifact under `build/distributions`. + +2) Before continuing, connect to IBM Cloud with the [IBM Cloud command line interface](https://console.bluemix.net/docs/cli/reference/bluemix_cli/get_started.html#getting-started). + +3) Navigate to the appropriate Cloud Foundry Account, Organization and Space of your choice. If you're new to IBM Cloud then just navigate to the place where your Message Hub instance has been provisioned. + +4) Push and start the application in IBM Cloud: + +``` +ibmcloud app push +``` + +5) The sample application should have created the default sample topic and started producing and consuming messages in an infinte loop. View the logs to verify this: + +``` +ibmcloud app logs +``` + +__Note:__ The IBM Cloud distribution will automatically update the required files for you at runtime, +using the `VCAP_SERVICES` information provided by IBM Cloud. + +## Further references + +If you are unfamiliar with Cloud Foundry applications then you might find the following document useful: + +[Cloud Foundry manifest documentation](http://docs.cloudfoundry.org/devguide/deploy-apps/manifest.html) \ No newline at end of file diff --git a/kafka-java-liberty-sample/README.md b/kafka-java-liberty-sample/README.md index 5bbf445..68885db 100644 --- a/kafka-java-liberty-sample/README.md +++ b/kafka-java-liberty-sample/README.md @@ -1,88 +1,40 @@ # IBM Message Hub Kafka Liberty sample application -IBM® Message Hub is a scalable, distributed, high throughput messaging service built on the top of Apache Kafka. It underpins the integration of your on-premise and off-premise cloud services and technologies. You can wire micro-services together using open protocols, connect stream data to analytics to realise powerful insight and feed event data to multiple applications to react in real time. -**Liberty for Java™ applications on IBM® Bluemix®** are powered by the IBM WebSphere® Liberty Buildpack. The Liberty profile is a highly composable, fast-to-start, dynamic application server runtime environment. It is part of IBM WebSphere Application Server v8.5.5. +**Liberty for Java™ applications on IBM Cloud®** are powered by the IBM WebSphere® Liberty Buildpack. The Liberty profile is a highly composable, fast-to-start, dynamic application server runtime environment. It is part of IBM WebSphere Application Server v8.5.5. This repository holds a sample cloud web application that was built using Liberty for Java™. The app will interact with a bound Message Hub service to produce and consume messages. -For more information regarding IBM Message Hub, [see the documentation on Bluemix](https://www.ng.bluemix.net/docs/services/MessageHub/index.html). +For more information regarding IBM Message Hub, [see the documentation on IBM Cloud®](https://www.ng.bluemix.net/docs/services/MessageHub/index.html). __Important Note__: This sample creates a topic on your behalf with one partition - this will incur a fee if the topic does not already exist on your account. -## General Prerequisites -To build and run the sample, you must have the following installed: - -* [git](https://git-scm.com/) -* [Gradle](https://gradle.org/) -* Java 7+ -* [Message Hub Service Instance](https://console.ng.bluemix.net/catalog/services/message-hub/) provisioned in [IBM Bluemix](https://console.ng.bluemix.net/) - -If you are not familiar with Liberty for Java on Bluemix, please consult the [documentation](https://console.ng.bluemix.net/docs/starters/liberty/index.html#liberty). - -## General Steps -To deploy and run the sample: -* Create a Message Hub service -* Add in JVM and LIBERTY licenses -* Run `gradle build war` -* Use `cf push` to deploy the app to Bluemix -* Open a browser and navigate to the app's URL -* Press the button to produce message(s) to Kafka, you can then see the consumed messages. - -## Building the Sample -Install the project using gradle: -```shell -gradle build war - ``` - -You should see a directory called `target` created in your project home directory. A WAR file is created under `target/defaultServer`, as well as a copy of the server.xml file. - -## Deployment Prerequisites -To deploy applications using the IBM WebSphere Application Server Liberty Buildpack, you are required to accept the IBM Liberty license and IBM JRE license by following the instructions below: - -1. Read the current IBM [Liberty-License][] and the current IBM [JVM-License][]. -2. Extract the `D/N: ` from the Liberty-License and JVM-License. -3. Add the following environment variables and extracted license codes to the `manifest.yml` file in the directory from which you push your application. For further information on the format of the `manifest.yml` file refer to the [manifest documentation][]. - -```yaml -env: - IBM_JVM_LICENSE: - IBM_LIBERTY_LICENSE: -``` - -__Note:__ Please use domain *eu-gb.mybluemix.net* within the manifest.yml if you are using Bluemix within London (console.eu-gb.bluemix.net). You may also need to use a unique hostname e.g. *host: JohnsSampleLibertyApp* - -## Deploy the Sample to Bluemix -Now we can push the app to Bluemix: -```shell -cf push - ``` - -The `cf push` command deploys the sample into Bluemix with a random route and binds the Message Hub service. You can find out the hostname of the deployed app from the output of this command. For example: -```shell -requested state: started -instances: 1/1 -usage: 512M x 1 instances -urls: messagehublibertyapp.mybluemix.net -last uploaded: Tue Aug 22 12:49:10 UTC 2017 -stack: cflinuxfs2 -buildpack: liberty-for-java - ``` - -## Produce and Consume Messages -Once the sample has been successfully deployed, navigate to the URL stated in the deployment logs. Once here, you can produce a message by clicking on the `Produce a Message` button. - -If the message was successfully produced and then consumed, you will then see the prompted message: - -##### Already consumed messages: -```shell -Message: [{"value":"This is a test message, msgId=0"}]. Offset: 0 -``` - -##### We have produced a message: ```Message produced:This is a test message, msgId=0``` -```shell -Consumed messages: [{"value":"This is a test message, msgId=0"}]. Offset: 0 -``` - -[Liberty-License]: http://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/wasdev/downloads/wlp/8.5.5.7/lafiles/runtime/en.html -[JVM-License]: http://www14.software.ibm.com/cgi-bin/weblap/lap.pl?la_formnum=&li_formnum=L-JWOD-9SYNCP&title=IBM%C2%AE+SDK%2C+Java+Technology+Edition%2C+Version+8.0&l=en -[manifest documentation]: http://docs.cloudfoundry.org/devguide/deploy-apps/manifest.html +## Prerequisites +To build and run the sample, you must have the done the following: + +* Obtain this repository's contents, either use `git` or just download the samples as a ZIP +* Install [Gradle](https://gradle.org/) +* Install Java 7+ +* Install the [IBM Cloud CLI](https://console.bluemix.net/docs/cli/reference/bluemix_cli/download_cli.html) +* Provision a [Message Hub Service Instance](https://console.ng.bluemix.net/catalog/services/message-hub/) in [IBM Cloud®](https://console.ng.bluemix.net/) + +## Standard or Enterprise Plan? + +**It's important to know which Message Hub plan you're using as the sample deployment steps are subtly different on each plan respectively.** + +By this point, you should have a Message Hub instance provisioned. If you haven't done this step yet, please refer to the main [readme](/README.md). + +If you are not sure what type of Message Hub instance you have then you can find this information out by visiting IBM Cloud's web console [dashboard](https://console.bluemix.net/dashboard). + +*Please make sure you are in the appropriate Region, Account, Organization and Space where you provisioned your Message Hub instance!* + +* Message Hub Standard plan services are "Cloud Foundry Services" with the plan column showing "Standard". +* Message Hub Enterprise plan services are "Services" with the plan column showing "Enterprise". + + +## Deploy the Application + +As the Standard and Enterprise Plan deployment steps are subtly different, we split the deployment steps into separate sections. Please navigate to the appropriate page(s): + +### [Standard Plan Deployment Guide](./standard_plan.md) + +### [Enterprise plan Deployment Guide](./enterprise_plan.md) diff --git a/kafka-java-liberty-sample/build.gradle b/kafka-java-liberty-sample/build.gradle index 5b9ad32..800b5c6 100644 --- a/kafka-java-liberty-sample/build.gradle +++ b/kafka-java-liberty-sample/build.gradle @@ -44,7 +44,7 @@ repositories { dependencies { compile 'org.apache.geronimo.specs:geronimo-servlet_3.0_spec:1.0' - compile 'org.apache.kafka:kafka-clients:0.10.2.+' + compile 'org.apache.kafka:kafka-clients:1.1.+' compile 'log4j:log4j:1.2.17' compile 'org.slf4j:slf4j-log4j12:1.7.21' compile 'org.slf4j:slf4j-api:1.7.21' diff --git a/kafka-java-liberty-sample/enterprise_plan.md b/kafka-java-liberty-sample/enterprise_plan.md new file mode 100644 index 0000000..2a15754 --- /dev/null +++ b/kafka-java-liberty-sample/enterprise_plan.md @@ -0,0 +1,109 @@ +# IBM Message Hub Kafka Liberty sample application deployment to an Enterprise Plan Message Hub + +## Overview + +To deploy and run the sample: +* Create a Cloud Foundry Service Alias for your Enterprise Service +* Setup your `manifest.yml` with your service details +* Use `ibmcloud cf push --no-start` to deploy the app to IBM Cloud® +* Re-configure binding with Manager role +* Start the app +* Inspect the application's logs + +## Set up a Cloud Foundry Service Alias +Before continuing, connect to IBM Cloud with the [IBM Cloud command line interface](https://console.bluemix.net/docs/cli/reference/bluemix_cli/get_started.html#getting-started). + +The Enterprise plan is IAM enabled. Therefore the following extra step is required to create a Cloud Foundry alias for your Service: + +Create a Cloud Foundry alias for your service's associated CRN: +```shell +ibmcloud resource service-alias-create --instance-name +``` + +Having created this alias associated your Service with a Cloud Foundry Organization and Space, thereby enabling your Cloud Foundry application to referrence it and connect to it. + +## Setup the manifest.yml +To deploy applications using the IBM WebSphere Application Server Liberty Buildpack, you are required to accept the IBM Liberty license and IBM JRE license by following the instructions below: + +1. Read the current IBM [Liberty-License](http://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/wasdev/downloads/wlp/8.5.5.7/lafiles/runtime/en.html) and the current IBM [JVM-License](http://www14.software.ibm.com/cgi-bin/weblap/lap.pl?la_formnum=&li_formnum=L-JWOD-9SYNCP&title=IBM%C2%AE+SDK%2C+Java+Technology+Edition%2C+Version+8.0&l=en). +2. Select the Message Hub service you would like to bind your application to. Do this by replacing `` with your actual service's name in `manifest.yml`: +```yaml + services: + - "" +``` +3. Consider your domain: You might need to change this in the `manifest.yml` as the domain varies by IBM Cloud region. If unsure, just delete the domain line and IBM Cloud will pick the domain for you. +4. Extract the `D/N: ` from the Liberty-License and JVM-License. +5. Add the following environment variables and extracted license codes to the `manifest.yml` file in the directory from which you push your application. For further information on the format of +the `manifest.yml` file refer to the [manifest documentation]. + +```yaml +env: + IBM_JVM_LICENSE: + IBM_LIBERTY_LICENSE: +``` + +__Note:__ You may need to use a unique hostname e.g. *host: JohnsSampleLibertyApp* + +## Build the Sample +Build the project using gradle: +```shell +gradle build war + ``` + +You should see a directory called `target` created in your project home directory. A WAR file is created under `target/defaultServer`, as well as a copy of the server.xml file. + +## Deploy the Application + +Push the app without starting it immediately by running the following command in the same directory as the `manifest.yml` file: +``` +ibmcloud app push --no-start +``` + +## Re-configure the binding +A binding between your app and service-alias is created for you automatically, but by default does not have permissions to create topics. This means that we need to create the existing binding and create a new one with the correct role: + +``` +ibmcloud resource service-binding-delete +ibmcloud resource service-binding-create Manager +``` + +## Start the app +Now it should be safe to start the application, **make sure you capture this output as it display's your application's URL binding**: +``` +ibmcloud app start +``` +You can optionally inspect the app's logs (The app only logs when the UI button gets hit): +``` +ibmcloud app logs +``` + +## Produce and Consume Messages +Once the sample has been successfully deployed, navigate to the URL **stated in the start logs of your `ibmcloud app start` command above**. This url is made up from the app name and domain that were specified in the manifest.yml. Once here, you can produce a message by clicking on the `Post Message` button. + +If the message was successfully produced and then consumed, you will then see the prompted message: + +##### Already consumed messages: +```shell +Message: [{"value":"This is a test message, msgId=0"}]. Offset: 1 +``` + +##### We have produced a message: ```This is a test message, msgId=0``` +```shell +Consumed messages: [{"value":"This is a test message, msgId=0"}]. Offset: 1 +``` + + +## Further references + +If you are unfamiliar with Cloud Foundry applications or Liberty then you might find the following documents useful: + +[Cloud Foundry manifest documentation](http://docs.cloudfoundry.org/devguide/deploy-apps/manifest.html) + +[Liberty documentation](https://console.ng.bluemix.net/docs/starters/liberty/index.html#liberty) + + +## Licenses + +[Liberty-License](http://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/wasdev/downloads/wlp/8.5.5.7/lafiles/runtime/en.html) + +[JVM-License](http://www14.software.ibm.com/cgi-bin/weblap/lap.pl?la_formnum=&li_formnum=L-JWOD-9SYNCP&title=IBM%C2%AE+SDK%2C+Java+Technology+Edition%2C+Version+8.0&l=en) \ No newline at end of file diff --git a/kafka-java-liberty-sample/manifest.yml b/kafka-java-liberty-sample/manifest.yml index 0cedf06..55aeacc 100644 --- a/kafka-java-liberty-sample/manifest.yml +++ b/kafka-java-liberty-sample/manifest.yml @@ -3,9 +3,12 @@ applications: memory: 512M instances: 1 buildpack: liberty-for-java - domain: mybluemix.net + domain: mybluemix.net name: MessageHubLibertyApp disk_quota: 1024M random-route: true services: - - "message-hub-service" + - "" +env: + IBM_JVM_LICENSE: + IBM_LIBERTY_LICENSE: diff --git a/kafka-java-liberty-sample/resources/consumer.properties b/kafka-java-liberty-sample/resources/consumer.properties index 7c4ed3d..24772b6 100644 --- a/kafka-java-liberty-sample/resources/consumer.properties +++ b/kafka-java-liberty-sample/resources/consumer.properties @@ -1,6 +1,7 @@ key.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer client.id=message-hub-sample +group.id=message-hub-sample enable.auto.commit=false auto.offset.reset=latest security.protocol=SASL_SSL diff --git a/kafka-java-liberty-sample/src/main/java/com/messagehub/samples/env/MessageHubCredentials.java b/kafka-java-liberty-sample/src/main/java/com/messagehub/samples/env/MessageHubCredentials.java index ef18044..33fcf18 100644 --- a/kafka-java-liberty-sample/src/main/java/com/messagehub/samples/env/MessageHubCredentials.java +++ b/kafka-java-liberty-sample/src/main/java/com/messagehub/samples/env/MessageHubCredentials.java @@ -25,7 +25,7 @@ @JsonIgnoreProperties(ignoreUnknown=true) public class MessageHubCredentials { - private String apiKey, kafkaRestUrl, user, password; + private String apiKey, kafkaAdminUrl, user, password; private String[] kafkaBrokers, kafkaBrokersSasl; @JsonProperty("api_key") @@ -38,14 +38,14 @@ public void setLabel(String apiKey) { this.apiKey = apiKey; } - @JsonProperty("kafka_rest_url") - public String getKafkaRestUrl() { - return kafkaRestUrl; + @JsonProperty("kafka_admin_url") + public String getKafkaAdminUrl() { + return kafkaAdminUrl; } - @JsonProperty("kafka_rest_url") - public void setKafkaRestUrl(String kafkaRestUrl) { - this.kafkaRestUrl = kafkaRestUrl; + @JsonProperty("kafka_admin_url") + public void setKafkaAdminUrl(String kafkaAdminUrl) { + this.kafkaAdminUrl = kafkaAdminUrl; } @JsonProperty diff --git a/kafka-java-liberty-sample/src/main/java/com/messagehub/samples/servlet/KafkaServlet.java b/kafka-java-liberty-sample/src/main/java/com/messagehub/samples/servlet/KafkaServlet.java index 1ba4db8..e3a701e 100644 --- a/kafka-java-liberty-sample/src/main/java/com/messagehub/samples/servlet/KafkaServlet.java +++ b/kafka-java-liberty-sample/src/main/java/com/messagehub/samples/servlet/KafkaServlet.java @@ -80,7 +80,7 @@ public class KafkaServlet extends HttpServlet { + File.separator + "resources"; private KafkaProducer kafkaProducer; private ConsumerRunnable consumerRunnable; - private String kafkaHost, restHost, apiKey, topic = "testTopic"; + private String kafkaHost, adminHost, apiKey, topic = "testTopic"; private String user, password; private String producedMessage, currentConsumedMessage; private int producedMessages = 0; @@ -148,7 +148,7 @@ public void init() throws ServletException { MessageHubCredentials credentials = messageHubEnvironment.getCredentials(); kafkaHost = credentials.getKafkaBrokersSasl()[0]; - restHost = credentials.getKafkaRestUrl(); + adminHost = credentials.getKafkaAdminUrl(); apiKey = credentials.getApiKey(); user = credentials.getUser(); password = credentials.getPassword(); @@ -162,7 +162,7 @@ public void init() throws ServletException { } // Check topic - RESTRequest restApi = new RESTRequest(restHost, apiKey); + RESTRequest restApi = new RESTRequest(adminHost, apiKey); // Create a topic, ignore a 422 response - this means that the // topic name already exists. restApi.post("/admin/topics", "{ \"name\": \"" + topic + "\" }", new int[]{422}); diff --git a/kafka-java-liberty-sample/src/main/webapp/index.html b/kafka-java-liberty-sample/src/main/webapp/index.html index 1af2d5a..65e1330 100644 --- a/kafka-java-liberty-sample/src/main/webapp/index.html +++ b/kafka-java-liberty-sample/src/main/webapp/index.html @@ -31,7 +31,7 @@
- IBM® Message Hub is a scalable, distributed, high throughput messaging service built on the top of Apache Kafka. + IBM Cloud® Message Hub is a scalable, distributed, high throughput messaging service built on the top of Apache Kafka. It underpins the integration of your on-premise and off-premise cloud services and technologies. You can wire micro-services together using open protocols, connect stream data to analytics to realise powerful insight and feed event data to multiple applications to react in real time.

diff --git a/kafka-java-liberty-sample/src/main/wlp/server.xml b/kafka-java-liberty-sample/src/main/wlp/server.xml index 5a98dd2..2632d72 100644 --- a/kafka-java-liberty-sample/src/main/wlp/server.xml +++ b/kafka-java-liberty-sample/src/main/wlp/server.xml @@ -27,7 +27,6 @@ servlet-3.1 appSecurity-2.0 - wsSecurity-1.1 ` with your actual service's name in `manifest.yml`: +```yaml + services: + - "" +``` +3. Consider your domain: You might need to change this in the `manifest.yml` as the domain varies by IBM Cloud region. If unsure, just delete the domain line and IBM Cloud will pick the domain for you. +4. Extract the `D/N: ` from the Liberty-License and JVM-License. +5. Add the following environment variables and extracted license codes to the `manifest.yml` file in the directory from which you push your application. For further information on the format of +the `manifest.yml` file refer to the [manifest documentation]. + +```yaml +env: + IBM_JVM_LICENSE: + IBM_LIBERTY_LICENSE: +``` + +__Note:__ You may need to use a unique hostname e.g. *host: JohnsSampleLibertyApp* + +## Build the Sample +Build the project using gradle: +```shell +gradle build war + ``` + +You should see a directory called `target` created in your project home directory. A WAR file is created under `target/defaultServer`, as well as a copy of the server.xml file. + +## Deploy the Sample to IBM Cloud +Before continuing, connect to IBM Cloud with the [IBM Cloud command line interface](https://console.bluemix.net/docs/cli/reference/bluemix_cli/get_started.html#getting-started). + + +Once connected to IBM Cloud, push the app, **make sure you capture this output as it display's your application's URL binding**: + +``` +ibmcloud app push +``` + +## Produce and Consume Messages +Once the sample has been successfully deployed, navigate to the URL **stated in the start logs of your `ibmcloud app push` command above**. This url is made up from the app name and domain that were specified in the manifest.yml. Once here, you can produce a message by clicking on the `Post Message` button. + +If the message was successfully produced and then consumed, you will then see the prompted message: + +##### Already consumed messages: +```shell +Message: [{"value":"This is a test message, msgId=0"}]. Offset: 1 +``` + +##### We have produced a message: ```This is a test message, msgId=0``` +```shell +Consumed messages: [{"value":"This is a test message, msgId=0"}]. Offset: 1 +``` + + +## Further references + +If you are unfamiliar with Cloud Foundry applications or Liberty then you might find the following documents useful: + +[Cloud Foundry manifest documentation](http://docs.cloudfoundry.org/devguide/deploy-apps/manifest.html) + +[Liberty documentation](https://console.ng.bluemix.net/docs/starters/liberty/index.html#liberty) + + +## Licenses + +[Liberty-License](http://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/wasdev/downloads/wlp/8.5.5.7/lafiles/runtime/en.html) + +[JVM-License](http://www14.software.ibm.com/cgi-bin/weblap/lap.pl?la_formnum=&li_formnum=L-JWOD-9SYNCP&title=IBM%C2%AE+SDK%2C+Java+Technology+Edition%2C+Version+8.0&l=en) \ No newline at end of file diff --git a/kafka-nodejs-console-sample/Dockerfile b/kafka-nodejs-console-sample/Dockerfile index 5bb20a5..f969716 100644 --- a/kafka-nodejs-console-sample/Dockerfile +++ b/kafka-nodejs-console-sample/Dockerfile @@ -1,11 +1,11 @@ # Dockerfile to run the sample under current Node LTS # -# docker build -t node-rdkafka +# docker build . -t node-rdkafka # docker run --rm -it -e VCAP_SERVICES=${VCAP_SERVICES} node-rdkafka # OR -# docker run --rm -it node-rdkafka +# docker run --rm -it node-rdkafka /etc/ssl/certs # -FROM node:carbon-stretch +FROM ubuntu RUN apt-get update -qqy \ && apt-get install -y --no-install-recommends \ @@ -17,13 +17,14 @@ RUN apt-get update -qqy \ libssl-dev \ make \ python \ + nodejs npm ca-certificates \ && rm -rf /var/cache/apt/* /var/lib/apt/lists/* WORKDIR /usr/src/app COPY *.js *.json *.md /usr/src/app/ -RUN npm install --unsafe-perm - +RUN npm install -d +ENV LD_LIBRARY_PATH=/usr/src/app/node_modules/node-rdkafka/build/deps ENTRYPOINT [ "node", "app.js" ] CMD [ "" ] diff --git a/kafka-nodejs-console-sample/README.md b/kafka-nodejs-console-sample/README.md index 4f9b969..03c1a8f 100644 --- a/kafka-nodejs-console-sample/README.md +++ b/kafka-nodejs-console-sample/README.md @@ -1,93 +1,40 @@ # IBM Message Hub Kafka Node.js console sample application This Node.js console application demonstrates how to connect to [IBM Message Hub](https://console.ng.bluemix.net/docs/services/MessageHub/index.html), send and receive messages using the [node-rdkafka](https://github.com/Blizzard/node-rdkafka) module. It also shows how to create and list topics using the Message Hub Admin REST API. -It can be run locally on your machine or deployed into [IBM Bluemix](https://console.ng.bluemix.net/). +It can be run locally on your machine or deployed into [IBM Cloud®](https://console.ng.bluemix.net/). + +In this tutorial, we'll focus on deploying the application into the IBM Cloud, therefore you don't have to build it locally, it will be done on your behalf on IBM Cloud. __Important Note__: This sample creates on your behalf a topic named `kafka-nodejs-console-sample-topic` with one partition - this will incur a fee if the topic does not already exist on your account. ## Global Prerequisites -To build and run the sample, you must have the following installed: -* [git](https://git-scm.com/) -* [Message Hub Service Instance](https://console.ng.bluemix.net/catalog/services/message-hub/) provisioned in [IBM Bluemix](https://console.ng.bluemix.net/) - -## Prerequisites (Local - macOS and Linux only) -* [Node.js](https://nodejs.org/en/) 6.X LTS -* [node-gyp] (https://www.npmjs.com/package/node-gyp) - -Node-rdkafka will build librdkafka automatically. You must ensure you have the dependencies listed below installed. For more details, see [librdakfka's instructions](../docs/librdkafka.md). - -##### Linux -* openssl-dev -* libsasl2-dev -* libsasl2-modules -* C++ toolchain - -##### macOS -* [Brew](http://brew.sh/) -* [Apple Xcode command line tools](https://developer.apple.com/xcode/) -* `openssl` via Brew -* Export `CPPFLAGS=-I/usr/local/opt/openssl/include` and `LDFLAGS=-L/usr/local/opt/openssl/lib` -* Open Keychain Access, export all certificates in System Roots to a single .pem file - -## Prerequisites (Bluemix) +To run the sample, you must have done the following: +* Obtain this repository's contents, either use `git` or just download the samples as a ZIP +* [Message Hub Service Instance](https://console.ng.bluemix.net/catalog/services/message-hub/) provisioned in [IBM Cloud®](https://console.ng.bluemix.net/) + +## Prerequisites (IBM Cloud®) * [Cloud Foundry Command Line Interface](https://github.com/cloudfoundry/cli/releases) installed -## Installing dependencies (Local) -Run the following commands on your local machine, after the prerequisites for your environment have been completed: -```shell -npm install -``` - -## Running the Sample (Local - macOS and Linux only) -Once built, to run the sample, execute the following command: -```shell -node app.js -``` - -To find the values for ``, `` and ``, access your Message Hub instance in Bluemix, go to the `Service Credentials` tab and select the `Credentials` you want to use. If your user value is `token`, specify that with the password seperated by a `:`. - -`` is the path where the trusted SSL certificates are stored on your machine and is therefore system dependent. -For example: -* Ubuntu: /etc/ssl/certs -* RedHat: /etc/pki/tls/cert.pem -* macOS: The .pem file you created in the prerequisite section - -__Note__: `` must be a single string enclosed in quotes. For example: `"host1:port1,host2:port2"`. We recommend using all the Kafka hosts listed in the `Credentials` you selected. - -Alternatively, you can run only the producer or only the consumer by respectively appending the switches `-producer` or `-consumer` to the command above. - -The sample will run indefinitely until interrupted. To stop the process, use `Ctrl+C`, for example. - -## Running the Sample (Bluemix) - -Open the `manifest.yml` file and rename the `"Message Hub-CHANGEME"` entry to that of your own Message Hub Service Instance name. - -Connect to Bluemix with the Cloud Foundry Command Line Interface, then run the following command in the same directory as the `manifest.yml` file: -```shell -cf push -``` - -## Sample Output -Below is a snippet of the output generated by the sample: - -``` -Topic mh-nodejs-console-sample-topic created -Existing topics: -[ { name: 'mh-nodejs-console-sample-topic', - partitions: 1, - retentionMs: '86400000', - markedForDeletion: false } ] -The consumer has started -The producer has started -Topic object created with opts {"request.required.acks":-1} -Consumer obtained metadata: {"orig_broker_id":0,"orig_broker_name":"sasl_ssl://kafka01-prod01.messagehub.services.us-south.bluemix.net:9093/0","topics":[{"name":"mh-nodejs-console-sample-topic","partitions":[{"id":0,"leader":0,"replicas":[0,1,4],"isrs":[null,null,null,1]}]}],"brokers":[{"id":2,"host":"kafka03-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":4,"host":"kafka05-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":1,"host":"kafka02-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":3,"host":"kafka04-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":0,"host":"kafka01-prod01.messagehub.services.us-south.bluemix.net","port":9093}]} -Producer obtained metadata: {"orig_broker_id":0,"orig_broker_name":"sasl_ssl://kafka01-prod01.messagehub.services.us-south.bluemix.net:9093/0","topics":[{"name":"mh-nodejs-console-sample-topic","partitions":[{"id":0,"leader":0,"replicas":[0,1,4],"isrs":[null,null,null,1]}]}],"brokers":[{"id":2,"host":"kafka03-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":4,"host":"kafka05-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":1,"host":"kafka02-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":3,"host":"kafka04-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":0,"host":"kafka01-prod01.messagehub.services.us-south.bluemix.net","port":9093}]} -Message produced, offset: 1 -No messages consumed -Message produced, offset: 2 -Message consumed: topic=mh-nodejs-console-sample-topic, partition=0, offset=1, key=key, value=This is a test message #1 -Message produced, offset: 3 -Message consumed: topic=mh-nodejs-console-sample-topic, partition=0, offset=2, key=key, value=This is a test message #2 -Message produced, offset: 4 -Message consumed: topic=mh-nodejs-console-sample-topic, partition=0, offset=3, key=key, value=This is a test message #3 -``` +## Standard or Enterprise Plan? + +**It's important to know which Message Hub plan you're using as the sample deployment steps are subtly different on each plan respectively.** + +By this point, you should have a Message Hub instance provisioned. If you haven't done this step yet, please refer to the main [readme](/README.md). + +If you are not sure what type of Message Hub instance you have then you can find this information out by visiting IBM Cloud's web console [dashboard](https://console.bluemix.net/dashboard). + +*Please make sure you are in the appropriate Region, Account, Organization and Space where you provisioned your Message Hub instance!* + +* Message Hub Standard plan services are "Cloud Foundry Services" with the plan column showing "Standard". +* Message Hub Enterprise plan services are "Services" with the plan column showing "Enterprise". + + +## Deploy the Application + +As the Standard and Enterprise Plan deployment steps are subtly different, we split the deployment steps into separate sections. Please navigate to the appropriate page(s): + +### [Standard Plan Deployment Guide](./standard_plan.md) + +### [Enterprise plan Deployment Guide](./enterprise_plan.md) + +### [Local Deployment](./local.md) diff --git a/kafka-nodejs-console-sample/app.js b/kafka-nodejs-console-sample/app.js index 5eea5a2..7123a47 100644 --- a/kafka-nodejs-console-sample/app.js +++ b/kafka-nodejs-console-sample/app.js @@ -55,7 +55,7 @@ if (process.env.VCAP_SERVICES) { if (process.argv.length < 6) { console.log('ERROR: It appears the application is running outside of Bluemix but the arguments are incorrect for local mode.'); console.log('\nUsage:\n' + - 'node ' + process.argv[1] + ' { | : } [ -consumer | -producer ]\n'); + 'node ' + process.argv[1] + ' [ -consumer | -producer ]\n'); process.exit(-1); } @@ -78,7 +78,7 @@ if (process.env.VCAP_SERVICES) { "label": "messagehub", "credentials": { "api_key": apiKey, - "kafka_rest_url": restEndpoint, + "kafka_admin_url": restEndpoint, } } ] @@ -200,14 +200,14 @@ function runLoops() { } // Start the clients - if (runProducer) { - producer = ProducerLoop.buildProducer(Kafka, producer_opts, topicName, shutdown); - producer.connect(); - } - if (runConsumer) { consumer = ConsumerLoop.buildConsumer(Kafka, consumer_opts, topicName, shutdown); consumer.connect(); } + + if (runProducer) { + producer = ProducerLoop.buildProducer(Kafka, producer_opts, topicName, shutdown); + producer.connect(); + } }; diff --git a/kafka-nodejs-console-sample/consumerLoop.js b/kafka-nodejs-console-sample/consumerLoop.js index f4e2f8f..160ee7d 100644 --- a/kafka-nodejs-console-sample/consumerLoop.js +++ b/kafka-nodejs-console-sample/consumerLoop.js @@ -45,6 +45,7 @@ exports.buildConsumer = function(Kafka, consumer_opts, topicName, shutdown) { // Register error listener consumer.on('event.error', function(err) { console.error('Error from consumer:' + JSON.stringify(err)); + consumer.consume(); }); var consumedMessages = [] diff --git a/kafka-nodejs-console-sample/enterprise_plan.md b/kafka-nodejs-console-sample/enterprise_plan.md new file mode 100644 index 0000000..99270eb --- /dev/null +++ b/kafka-nodejs-console-sample/enterprise_plan.md @@ -0,0 +1,99 @@ +# IBM Message Hub Kafka Node.js console sample application: Enterprise Plan deployment into IBM Cloud + +## Prerequisites (IBM Cloud®) +* [Cloud Foundry Command Line Interface](https://github.com/cloudfoundry/cli/releases) installed + +## Overview +To deploy and run the sample: +* Create a Cloud Foundry Service Alias for your Enterprise Service +* Setup your `manifest.yml` with your service details +* Use `ibmcloud cf push --no-start` to deploy the app to IBM Cloud® +* Re-configure binding with Manager role +* Start the app +* View the application's logs + +## Set up a Cloud Foundry Service Alias +Before continuing, connect to IBM Cloud with the [IBM Cloud command line interface](https://console.bluemix.net/docs/cli/reference/bluemix_cli/get_started.html#getting-started). + +The Enterprise plan is IAM enabled. Therefore the following extra step is required to create a Cloud Foundry alias for your Service: + +Create a Cloud Foundry alias for your service's associated CRN: +```shell +ibmcloud resource service-alias-create --instance-name +``` + +Having created this alias associated your Service with a Cloud Foundry Organization and Space, thereby enabling your Cloud Foundry application to referrence it and connect to it. + + +## Setup the manifest.yml +Open the `manifest.yml` file and rename the `"Message Hub-CHANGEME"` entry to that of your own Message Hub Service Instance *Alias* name: + +``` + services: + - "" +``` + +Optionally rename the service: + +``` +- name: kafka-nodejs-console-sample +``` + +## Push the application into IBM Cloud® +Connect to IBM Cloud® with the Cloud Foundry Command Line Interface, then run the following command in the same directory as the `manifest.yml` file: +```shell +ibmcloud cf push --no-start +``` + +This will push your application but will not start it just yet. We'll need to re-configure the security settings such that the application is allowed to create topics: + +## Re-configure the binding +The NodeJS build pack tries to be helpful by auto-creating a binding in between the Cloud Foundry Service Alias and the application pushed. This is however with a role that doesn't allow topic creation within Message Hub. As such, we need to re-bind the app: + +``` +ibmcloud resource service-binding-delete +ibmcloud resource service-binding-create Manager +``` + +## Start the app +Now it should be safe to start the application: +``` +ibmcloud app start +``` + +## Inspect the app's logs +You can optionally inspect the app's logs (The app only logs when the UI button gets hit): +``` +ibmcloud app logs +``` + +## Sample Output +Below is a snippet of the output generated by the sample: + +``` +Topic mh-nodejs-console-sample-topic created +Existing topics: +[ { name: 'mh-nodejs-console-sample-topic', + partitions: 1, + retentionMs: '86400000', + markedForDeletion: false } ] +The consumer has started +The producer has started +Topic object created with opts {"request.required.acks":-1} +Consumer obtained metadata: {"orig_broker_id":0,"orig_broker_name":"sasl_ssl://kafka01-prod01.messagehub.services.us-south.bluemix.net:9093/0","topics":[{"name":"mh-nodejs-console-sample-topic","partitions":[{"id":0,"leader":0,"replicas":[0,1,4],"isrs":[null,null,null,1]}]}],"brokers":[{"id":2,"host":"kafka03-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":4,"host":"kafka05-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":1,"host":"kafka02-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":3,"host":"kafka04-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":0,"host":"kafka01-prod01.messagehub.services.us-south.bluemix.net","port":9093}]} +Producer obtained metadata: {"orig_broker_id":0,"orig_broker_name":"sasl_ssl://kafka01-prod01.messagehub.services.us-south.bluemix.net:9093/0","topics":[{"name":"mh-nodejs-console-sample-topic","partitions":[{"id":0,"leader":0,"replicas":[0,1,4],"isrs":[null,null,null,1]}]}],"brokers":[{"id":2,"host":"kafka03-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":4,"host":"kafka05-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":1,"host":"kafka02-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":3,"host":"kafka04-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":0,"host":"kafka01-prod01.messagehub.services.us-south.bluemix.net","port":9093}]} +Message produced, offset: 1 +No messages consumed +Message produced, offset: 2 +Message consumed: topic=mh-nodejs-console-sample-topic, partition=0, offset=1, key=key, value=This is a test message #1 +Message produced, offset: 3 +Message consumed: topic=mh-nodejs-console-sample-topic, partition=0, offset=2, key=key, value=This is a test message #2 +Message produced, offset: 4 +Message consumed: topic=mh-nodejs-console-sample-topic, partition=0, offset=3, key=key, value=This is a test message #3 +``` + +## Further references + +If you are unfamiliar with Cloud Foundry applications or Liberty then you might find the following documents useful: + +[Cloud Foundry manifest documentation](http://docs.cloudfoundry.org/devguide/deploy-apps/manifest.html) diff --git a/kafka-nodejs-console-sample/local.md b/kafka-nodejs-console-sample/local.md new file mode 100644 index 0000000..13ebf07 --- /dev/null +++ b/kafka-nodejs-console-sample/local.md @@ -0,0 +1,74 @@ +# IBM Message Hub Kafka Node.js console sample application: Local Development guide +As pushing the application into IBM Cloud® does not require you to build the application locally, this guide is here to guide you through the process, should you wish to build the application locally. + +We will not discuss establishing a connection from your laptop to Message Hub. This is described in the [ connection guide](https://console.bluemix.net/docs/services/MessageHub/messagehub127.html#connect_messagehub). + +## Prerequisites (Local deployment only, for macOS and Linux) +* [Node.js](https://nodejs.org/en/) 6.X LTS +* [node-gyp] (https://www.npmjs.com/package/node-gyp) + +Node-rdkafka will build librdkafka automatically. You must ensure you have the dependencies listed below installed. For more details, see [librdakfka's instructions](../docs/librdkafka.md). + +##### Linux +* openssl-dev +* libsasl2-dev +* libsasl2-modules +* C++ toolchain + +##### macOS +* [Brew](http://brew.sh/) +* [Apple Xcode command line tools](https://developer.apple.com/xcode/) +* `openssl` via Brew +* Export `CPPFLAGS=-I/usr/local/opt/openssl/include` and `LDFLAGS=-L/usr/local/opt/openssl/lib` +* Open Keychain Access, export all certificates in System Roots to a single .pem file + +## Installing dependencies (Local) +Run the following commands on your local machine, after the prerequisites for your environment have been completed: +```shell +npm install +``` + +## Running the Sample (Local - macOS and Linux only) +Once built, to run the sample, execute the following command: +```shell +node app.js +``` + +To find the values for ``, `` and ``, access your Message Hub instance in IBM Cloud®, go to the `Service Credentials` tab and select the `Credentials` you want to use. If your user value is `token`, specify that with the password seperated by a `:`. + +`` is the path where the trusted SSL certificates are stored on your machine and is therefore system dependent. +For example: +* Ubuntu: /etc/ssl/certs +* RedHat: /etc/pki/tls/cert.pem +* macOS: The .pem file you created in the prerequisite section + +__Note__: `` must be a single string enclosed in quotes. For example: `"host1:port1,host2:port2"`. We recommend using all the Kafka hosts listed in the `Credentials` you selected. + +Alternatively, you can run only the producer or only the consumer by respectively appending the switches `-producer` or `-consumer` to the command above. + +The sample will run indefinitely until interrupted. To stop the process, use `Ctrl+C`, for example. + +## Sample Output +Below is a snippet of the output generated by the sample: + +``` +Topic mh-nodejs-console-sample-topic created +Existing topics: +[ { name: 'mh-nodejs-console-sample-topic', + partitions: 1, + retentionMs: '86400000', + markedForDeletion: false } ] +The consumer has started +The producer has started +Topic object created with opts {"request.required.acks":-1} +Consumer obtained metadata: {"orig_broker_id":0,"orig_broker_name":"sasl_ssl://kafka01-prod01.messagehub.services.us-south.bluemix.net:9093/0","topics":[{"name":"mh-nodejs-console-sample-topic","partitions":[{"id":0,"leader":0,"replicas":[0,1,4],"isrs":[null,null,null,1]}]}],"brokers":[{"id":2,"host":"kafka03-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":4,"host":"kafka05-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":1,"host":"kafka02-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":3,"host":"kafka04-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":0,"host":"kafka01-prod01.messagehub.services.us-south.bluemix.net","port":9093}]} +Producer obtained metadata: {"orig_broker_id":0,"orig_broker_name":"sasl_ssl://kafka01-prod01.messagehub.services.us-south.bluemix.net:9093/0","topics":[{"name":"mh-nodejs-console-sample-topic","partitions":[{"id":0,"leader":0,"replicas":[0,1,4],"isrs":[null,null,null,1]}]}],"brokers":[{"id":2,"host":"kafka03-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":4,"host":"kafka05-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":1,"host":"kafka02-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":3,"host":"kafka04-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":0,"host":"kafka01-prod01.messagehub.services.us-south.bluemix.net","port":9093}]} +Message produced, offset: 1 +No messages consumed +Message produced, offset: 2 +Message consumed: topic=mh-nodejs-console-sample-topic, partition=0, offset=1, key=key, value=This is a test message #1 +Message produced, offset: 3 +Message consumed: topic=mh-nodejs-console-sample-topic, partition=0, offset=2, key=key, value=This is a test message #2 +Message produced, offset: 4 +Message consumed: topic=mh-nodejs-console-sample-topic, partition=0, offset=3, key=key, value=This is a test message #3 +``` diff --git a/kafka-nodejs-console-sample/package.json b/kafka-nodejs-console-sample/package.json index c90e77d..e249f36 100644 --- a/kafka-nodejs-console-sample/package.json +++ b/kafka-nodejs-console-sample/package.json @@ -33,7 +33,7 @@ "node-rdkafka" ], "dependencies": { - "message-hub-rest": "^1.2.0", + "message-hub-rest": "^2.0.0-alpha.0", "node-rdkafka": "2.3.3" }, "engines": { diff --git a/kafka-nodejs-console-sample/standard_plan.md b/kafka-nodejs-console-sample/standard_plan.md new file mode 100644 index 0000000..c460213 --- /dev/null +++ b/kafka-nodejs-console-sample/standard_plan.md @@ -0,0 +1,76 @@ +# IBM Message Hub Kafka Node.js console sample application: Standard Plan deployment into IBM Cloud + +## Prerequisites (IBM Cloud®) +* [Cloud Foundry Command Line Interface](https://github.com/cloudfoundry/cli/releases) installed + +## Overview +To run the samples, you will: +* Setup your `manifest.yml` with your service details +* Use `ibmcloud cf push` to deploy and start the app to IBM Cloud® +* View the application's logs + + +## Prepare the `manifest.yml` file + +Open the `manifest.yml` file and rename the `"Message Hub-CHANGEME"` entry to that of your own Message Hub Service Instance name. + +``` + services: + - "" +``` + +You can optionally change the application's name: + +``` +- name: kafka-nodejs-console-sample +``` + +## Deploying and running the sample on the IBM Cloud + +Connect to IBM Cloud® with the Cloud Foundry Command Line Interface. You might need to navigate to the Cloud Foundry Organization and Space your Message Hub instance has been provisoned in. Use `ibmcloud target -cf` to do this interactively. + +When in the right Organisation and Space, run the following command in the same directory as the `manifest.yml` file, run the following to push out the application into the IBM Cloud: + +```shell +ibmcloud cf push +``` + +You should see the application built and started. + +## View the logs: Sample Output + +You can view the application by running, don't forget to change the application's name, should you have renamed it earlier: + +```shell +ibmcloud app logs kafka-nodejs-console-sample +``` + +Below is a snippet of the output generated by the sample: + +``` +Topic mh-nodejs-console-sample-topic created +Existing topics: +[ { name: 'mh-nodejs-console-sample-topic', + partitions: 1, + retentionMs: '86400000', + markedForDeletion: false } ] +The consumer has started +The producer has started +Topic object created with opts {"request.required.acks":-1} +Consumer obtained metadata: {"orig_broker_id":0,"orig_broker_name":"sasl_ssl://kafka01-prod01.messagehub.services.us-south.bluemix.net:9093/0","topics":[{"name":"mh-nodejs-console-sample-topic","partitions":[{"id":0,"leader":0,"replicas":[0,1,4],"isrs":[null,null,null,1]}]}],"brokers":[{"id":2,"host":"kafka03-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":4,"host":"kafka05-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":1,"host":"kafka02-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":3,"host":"kafka04-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":0,"host":"kafka01-prod01.messagehub.services.us-south.bluemix.net","port":9093}]} +Producer obtained metadata: {"orig_broker_id":0,"orig_broker_name":"sasl_ssl://kafka01-prod01.messagehub.services.us-south.bluemix.net:9093/0","topics":[{"name":"mh-nodejs-console-sample-topic","partitions":[{"id":0,"leader":0,"replicas":[0,1,4],"isrs":[null,null,null,1]}]}],"brokers":[{"id":2,"host":"kafka03-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":4,"host":"kafka05-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":1,"host":"kafka02-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":3,"host":"kafka04-prod01.messagehub.services.us-south.bluemix.net","port":9093},{"id":0,"host":"kafka01-prod01.messagehub.services.us-south.bluemix.net","port":9093}]} +Message produced, offset: 1 +No messages consumed +Message produced, offset: 2 +Message consumed: topic=mh-nodejs-console-sample-topic, partition=0, offset=1, key=key, value=This is a test message #1 +Message produced, offset: 3 +Message consumed: topic=mh-nodejs-console-sample-topic, partition=0, offset=2, key=key, value=This is a test message #2 +Message produced, offset: 4 +Message consumed: topic=mh-nodejs-console-sample-topic, partition=0, offset=3, key=key, value=This is a test message #3 +``` + +## Further references + +If you are unfamiliar with Cloud Foundry applications or Liberty then you might find the following documents useful: + +[Cloud Foundry manifest documentation](http://docs.cloudfoundry.org/devguide/deploy-apps/manifest.html) diff --git a/kafka-python-console-sample/Dockerfile b/kafka-python-console-sample/Dockerfile index 45543f3..dad571d 100644 --- a/kafka-python-console-sample/Dockerfile +++ b/kafka-python-console-sample/Dockerfile @@ -5,7 +5,7 @@ # OR # docker run --rm -it python-rdkafka # -FROM python:3 +FROM python:3.6-stretch RUN apt-get update -qqy \ && apt-get install -y --no-install-recommends \ @@ -17,7 +17,7 @@ RUN apt-get update -qqy \ libssl-dev \ make \ python3 \ - && curl -sL https://github.com/edenhill/librdkafka/archive/v0.9.5.tar.gz \ + && curl -sL https://github.com/edenhill/librdkafka/archive/v0.11.4.tar.gz \ | tar zx -C /usr/src --strip-components=1 \ && cd /usr/src && ./configure --prefix=/usr && make && make install \ && rm -rf /var/cache/apt/* /var/lib/apt/lists/* diff --git a/kafka-python-console-sample/README.md b/kafka-python-console-sample/README.md index e154ed2..b4c1576 100644 --- a/kafka-python-console-sample/README.md +++ b/kafka-python-console-sample/README.md @@ -1,100 +1,35 @@ # IBM Message Hub Kafka Python console sample application This Python console application demonstrates how to connect to [IBM Message Hub](https://console.ng.bluemix.net/docs/services/MessageHub/index.html), send and receive messages using the [confluent-kafka-python](https://github.com/confluentinc/confluent-kafka-python) library. It also shows how to create and list topics using the Message Hub Admin REST API. -It can be run locally on your machine or deployed into [IBM Bluemix](https://console.ng.bluemix.net/). +This tutorial will explain how to run the sample app in [IBM Cloud®](https://console.ng.bluemix.net/) using a Message Hub Service. It is also possible to host the application locally and still target your Message Hub Service running in IBM Cloud but that is not covered during this tutorial. __Important Note__: This sample creates on your behalf a topic named `kafka-python-console-sample-topic` with one partition - this will incur a fee if the topic does not already exist on your account. -## Global Prerequisites -To build and run the sample, you must have the following installed: -* [git](https://git-scm.com/) -* [Message Hub Service Instance](https://console.ng.bluemix.net/catalog/services/message-hub/) provisioned in [IBM Bluemix](https://console.ng.bluemix.net/) +## Prerequisites +To build and run the sample, you must have the following: -## Prerequisites (Local - macOS and Linux only) -* [Python](https://www.python.org/) >= 3.4.3 -* [librdkafka](https://github.com/edenhill/librdkafka) >= 0.9.3 +* Obtain this repository's contents, either use `git` or just download the samples as a ZIP +* Install the [IBM Cloud CLI](https://console.bluemix.net/docs/cli/reference/bluemix_cli/download_cli.html) +* Provision a [Message Hub Service Instance](https://console.ng.bluemix.net/catalog/services/message-hub/) in [IBM Cloud®](https://console.ng.bluemix.net/) -For more details regarding librdkafka installation, see [librdkafka's instructions](../docs/librdkafka.md). +## Standard or Enterprise Plan? -##### Linux -* openssl-dev -* libsasl2-dev -* libsasl2-modules -* C++ toolchain -* python3-dev +**It's important to know which Message Hub plan you're using as the sample deployment steps are subtly different on each plan respectively.** -##### macOS -* [Brew](http://brew.sh/) -* [Apple Xcode command line tools](https://developer.apple.com/xcode/) -* `openssl`, `python3` and `librdkafka` via Brew -* Export `CPPFLAGS=-I/usr/local/opt/openssl/include` and `LDFLAGS=-L/usr/local/opt/openssl/lib` -* Open Keychain Access, export all certificates in System Roots to a single .pem file +By this point, you should have a Message Hub instance provisioned. If you haven't done this step yet, please refer to the main [readme](/README.md). -## Prerequisites (Bluemix) -* [Cloud Foundry Command Line Interface](https://github.com/cloudfoundry/cli/releases) installed +If you are not sure what type of Message Hub instance you have then you can find this information out by visiting IBM Cloud's web console [dashboard](https://console.bluemix.net/dashboard). -## Installing dependencies (Local) -Run the following commands on your local machine, after the prerequisites for your environment have been completed: -```shell -pip3 install -r requirements/dev.txt -``` +*Please make sure you are in the appropriate Region, Account, Organization and Space where you provisioned your Message Hub instance!* -## Running the Sample (Local - macOS and Linux only) -Once built, to run the sample, execute the following command: -```shell -python ./app.py -``` +* Message Hub Standard plan services are "Cloud Foundry Services" with the plan column showing "Standard". +* Message Hub Enterprise plan services are "Services" with the plan column showing "Enterprise". -To find the values for ``, `` and ``, access your Message Hub instance in Bluemix, go to the `Service Credentials` tab and select the `Credentials` you want to use. If your user value is `token`, specify that with the password seperated by a `:`. -`` is the path where the trusted SSL certificates are stored on your machine and is therefore system dependent. -For example: -* Ubuntu: /etc/ssl/certs -* RedHat: /etc/pki/tls/cert.pem -* macOS: The .pem file you created in the prerequisite section +## Deploy the Application -__Note__: `` must be a single string enclosed in quotes. For example: `"host1:port1,host2:port2"`. We recommend using all the Kafka hosts listed in the `Credentials` you selected. +As the Standard and Enterprise Plan deployment steps are subtly different, we split the deployment steps into separate sections. Please navigate to the appropriate page(s): -Alternatively, you can run only the producer or only the consumer by respectively appending the switches `-producer` or `-consumer` to the command above. +### [Standard Plan Deployment Guide](./standard_plan.md#running-the-sample-ibm-cloud) -The sample will run indefinitely until interrupted. To stop the process, use `Ctrl+C`, for example. - -## Running the Sample (Bluemix) - -Open the `manifest.yml` file and rename the `"Message Hub-CHANGEME"` entry to that of your own Message Hub Service Instance name. - -Connect to Bluemix with the Cloud Foundry Command Line Interface, then run the following command in the same directory as the `manifest.yml` file: -```shell -cf push -``` - -## Sample Output -Below is a snippet of the output generated by the sample: - -``` -Running in local mode. -Kafka Endpoints: kafka01-prod01.messagehub.services.us-south.bluemix.net:9093 -Admin REST Endpoint: https://kafka-admin-prod01.messagehub.services.us-south.bluemix.net:443 -Creating the topic kafka-python-console-sample-topic with Admin REST API -{} -Admin REST Listing Topics: -[{"name":"kafka-python-console-sample-topic","partitions":1,"retentionMs":"86400000","markedForDeletion":false}] -This sample app will run until interrupted. -The producer has started -The consumer has started -Message produced, offset: 0 -Message produced, offset: 2 -No messages consumed -Message produced, offset: 3 -No messages consumed -Message produced, offset: 4 -Message consumed: topic=kafka-python-console-sample-topic, partition=0, offset=4, key=key, value=This is a test message #4 -Message produced, offset: 5 -No messages consumed -Message produced, offset: 6 -Message consumed: topic=kafka-python-console-sample-topic, partition=0, offset=5, key=key, value=This is a test message #5 -Message produced, offset: 7 -No messages consumed -Message produced, offset: 8 -Message consumed: topic=kafka-python-console-sample-topic, partition=0, offset=6, key=key, value=This is a test message #6 -``` +### [Enterprise plan Deployment Guide](./enterprise_plan.md#running-the-sample-ibm-cloud) \ No newline at end of file diff --git a/kafka-python-console-sample/app.py b/kafka-python-console-sample/app.py index 50f3bb7..1008dbd 100644 --- a/kafka-python-console-sample/app.py +++ b/kafka-python-console-sample/app.py @@ -62,7 +62,6 @@ def __init__(self, args): self.opts['brokers'] = args[1] self.opts['rest_endpoint'] = args[2] - if ":" in args[3]: credentials_list = args[3].split(":") self.opts['api_key'] = credentials_list[1] diff --git a/kafka-python-console-sample/enterprise_plan.md b/kafka-python-console-sample/enterprise_plan.md new file mode 100644 index 0000000..2b8f603 --- /dev/null +++ b/kafka-python-console-sample/enterprise_plan.md @@ -0,0 +1,65 @@ +# IBM Message Hub Kafka Python console sample application: Enterprise Plan +This Python console application demonstrates how to connect to [IBM Message Hub](https://console.ng.bluemix.net/docs/services/MessageHub/index.html), send and receive messages using the [confluent-kafka-python](https://github.com/confluentinc/confluent-kafka-python) library. It also shows how to create and list topics using the Message Hub Admin REST API. + +This tutorial will explain how to run the sample app in [Docker](https://docs.docker.com/) using a Message Hub Service in [IBM Cloud®](https://console.ng.bluemix.net/). We package the sample in docker this way to control the environment and allow SNI headers to be supported. It is also possible to host the application locally and still target your Message Hub Service running in IBM Cloud but that is not covered during this tutorial. + +## Prerequisites +To build and run the sample, you must have the following: +* [Docker](https://docs.docker.com/) installed +* Provision a [Message Hub Service Instance](https://console.ng.bluemix.net/catalog/services/message-hub/) in [IBM Cloud®](https://console.ng.bluemix.net/ss) + +## Get Service Credentials from your Service +You will need to obtain the values for ``, ``, `` and ``, so that they can be passed into the application later. To do this, access your Message Hub instance in the [Bluemix UI](console.bluemix.net/dashboard/apps), go to the Service Credentials tab and select the Credentials you want to use. + +**Note:** must be a single string enclosed in quotes. For example: "host1:port1,host2:port2". We recommend using all the Kafka hosts listed in the Credentials you selected. + +## Build and run the sample using Docker +Build the docker container from the same directory as the `Dockerfile`: +``` +docker build -t kafka-python-console-sample . +``` + +Issue the following command with your service credentials to run the sample: +``` +docker run --rm -it : /etc/ssl/certs +``` + +Alternatively, you can run only the producer or only the consumer by respectively appending the switches -producer or -consumer to the command above. + +The sample will run indefinitely until interrupted. To stop the process, use Ctrl+C, for example. + +**Note:** The final parameter that is passed in is the ``. This is the path where the trusted SSL certificates are stored on your machine and is therefore system dependent. For example: + +* Ubuntu (Used for this example): /etc/ssl/certs +* RedHat: /etc/pki/tls/cert.pem +* macOS: The .pem file you created in the prerequisite section + +## Sample Output +Below is a snippet of the output generated by the sample: + +``` +Kafka Endpoints: kafka01-prod01.messagehub.services.us-south.bluemix.net:9093 +Admin REST Endpoint: https://kafka-admin-prod01.messagehub.services.us-south.bluemix.net:443 +Creating the topic kafka-python-console-sample-topic with Admin REST API +{} +Admin REST Listing Topics: +[{"name":"kafka-python-console-sample-topic","partitions":1,"retentionMs":"86400000","markedForDeletion":false}] +This sample app will run until interrupted. +The producer has started +The consumer has started +Message produced, offset: 0 +Message produced, offset: 2 +No messages consumed +Message produced, offset: 3 +No messages consumed +Message produced, offset: 4 +Message consumed: topic=kafka-python-console-sample-topic, partition=0, offset=4, key=key, value=This is a test message #4 +Message produced, offset: 5 +No messages consumed +Message produced, offset: 6 +Message consumed: topic=kafka-python-console-sample-topic, partition=0, offset=5, key=key, value=This is a test message #5 +Message produced, offset: 7 +No messages consumed +Message produced, offset: 8 +Message consumed: topic=kafka-python-console-sample-topic, partition=0, offset=6, key=key, value=This is a test message #6 +``` \ No newline at end of file diff --git a/kafka-python-console-sample/standard_plan.md b/kafka-python-console-sample/standard_plan.md new file mode 100644 index 0000000..5be4a90 --- /dev/null +++ b/kafka-python-console-sample/standard_plan.md @@ -0,0 +1,68 @@ +# IBM Message Hub Kafka Python console sample application: Standard Plan +This Python console application demonstrates how to connect to [IBM Message Hub](https://console.ng.bluemix.net/docs/services/MessageHub/index.html), send and receive messages using the [confluent-kafka-python](https://github.com/confluentinc/confluent-kafka-python) library. It also shows how to create and list topics using the Message Hub Admin REST API. + +This tutorial will explain how to run the sample app in [IBM Cloud®](https://console.ng.bluemix.net/) using a Message Hub Service. It is also possible to host the application locally and still target your Message Hub Service running in IBM Cloud but that is not covered during this tutorial. + +__Important Note__: This sample creates on your behalf a topic named `kafka-python-console-sample-topic` with one partition - this will incur a fee if the topic does not already exist on your account. + +## Prerequisites +To build and run the sample, you must have the following: +* Install the [IBM Cloud CLI](https://console.bluemix.net/docs/cli/reference/bluemix_cli/download_cli.html) +* Provision a [Message Hub Service Instance](https://console.ng.bluemix.net/catalog/services/message-hub/) in [IBM Cloud®](https://console.ng.bluemix.net/) + +## Overview +To deploy and run the sample: +* Setup your `manifest.yml` with your service details +* Use `ibmcloud cf push` to deploy the app to IBM Cloud® +* Check the app logs to see messages being produced and consumed + +## Running the Sample (IBM Cloud®) +Open the `manifest.yml` file and rename the `"Message Hub-CHANGEME"` entry to that of your own Message Hub Service Instance name. + +Connect to IBM Cloud® with the Cloud Foundry Command Line Interface, making sure you are in the same org and space as your service, and run the following command in the same directory as the `manifest.yml` file: +```shell +ibmcloud cf push +``` + +Once the app has has started, you can run: +``` +ibmcloud cf logs kafka-python-console-sample --recent +``` + +This will show the app producing and consuming messages. + +## Sample Output +Below is a snippet of the output generated by the sample: + +``` +Kafka Endpoints: kafka01-prod01.messagehub.services.us-south.bluemix.net:9093 +Admin REST Endpoint: https://kafka-admin-prod01.messagehub.services.us-south.bluemix.net:443 +Creating the topic kafka-python-console-sample-topic with Admin REST API +{} +Admin REST Listing Topics: +[{"name":"kafka-python-console-sample-topic","partitions":1,"retentionMs":"86400000","markedForDeletion":false}] +This sample app will run until interrupted. +The producer has started +The consumer has started +Message produced, offset: 0 +Message produced, offset: 2 +No messages consumed +Message produced, offset: 3 +No messages consumed +Message produced, offset: 4 +Message consumed: topic=kafka-python-console-sample-topic, partition=0, offset=4, key=key, value=This is a test message #4 +Message produced, offset: 5 +No messages consumed +Message produced, offset: 6 +Message consumed: topic=kafka-python-console-sample-topic, partition=0, offset=5, key=key, value=This is a test message #5 +Message produced, offset: 7 +No messages consumed +Message produced, offset: 8 +Message consumed: topic=kafka-python-console-sample-topic, partition=0, offset=6, key=key, value=This is a test message #6 +``` + +## Further references + +If you are unfamiliar with Cloud Foundry applications or Liberty then you might find the following documents useful: + +[Cloud Foundry manifest documentation](http://docs.cloudfoundry.org/devguide/deploy-apps/manifest.html) \ No newline at end of file diff --git a/mqlight/README.md b/mqlight/README.md index 477302d..6bdcdc7 100644 --- a/mqlight/README.md +++ b/mqlight/README.md @@ -1,3 +1,6 @@ +## Using the MQ Light API in Message Hub +The MQ Light API provides a simple AMQP-based messaging interface for Java™, Node.js, Python, and Ruby. The API is provided for backward compatibility with the earlier MQ Light service. The MQ Light API is available as part of the Standard plan only. For more information, please refer to our [documentation](https://console.bluemix.net/docs/services/MessageHub/messagehub075.html#mql_using). + ## MQ Light Samples You can find samples that demonstrate using the MQ Light API with IBM Message