Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade Hazelcast example with the latest hazelcast-kubernetes-bootst… #39580

Merged
merged 1 commit into from
Feb 8, 2017
Merged

Upgrade Hazelcast example with the latest hazelcast-kubernetes-bootst… #39580

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion examples/examples_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -241,7 +241,7 @@ func TestExampleObjectSchemas(t *testing.T) {
"pod": &api.Pod{},
},
"../examples/storage/hazelcast": {
"hazelcast-controller": &api.ReplicationController{},
"hazelcast-deployment": &extensions.Deployment{},
"hazelcast-service": &api.Service{},
},
"../examples/meteor": {
Expand Down
218 changes: 102 additions & 116 deletions examples/storage/hazelcast/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ The following document describes the development of a _cloud native_ [Hazelcast]

Any topology changes are communicated and handled by Hazelcast nodes themselves.

This document also attempts to describe the core components of Kubernetes: _Pods_, _Services_, and _Replication Controllers_.
This document also attempts to describe the core components of Kubernetes: _Pods_, _Services_, and _Deployments_.

### Prerequisites

Expand All @@ -30,7 +30,7 @@ In this case, we shall not run a single Hazelcast pod, because the discovery mec

### Adding a Hazelcast Service

In Kubernetes a _[Service](../../../docs/user-guide/services.md)_ describes a set of Pods that perform the same task. For example, the set of nodes in a Hazelcast cluster. An important use for a Service is to create a load balancer which distributes traffic across members of the set. But a _Service_ can also be used as a standing query which makes a dynamically changing set of Pods available via the Kubernetes API. This is actually how our discovery mechanism works, by relying on the service to discover other Hazelcast pods.
In Kubernetes a _[Service](../../../docs/user-guide/services.md)_ describes a set of Pods that perform the same task. For example, the set of nodes in a Hazelcast cluster. An important use for a Service is to create a load balancer which distributes traffic across members of the set. But a _Service_ can also be used as a standing query which makes a dynamically changing set of Pods available via the Kubernetes API. This is actually how our discovery mechanism works, by relying on the service to discover other Hazelcast pods.

Here is the service description:

Expand Down Expand Up @@ -65,52 +65,41 @@ $ kubectl create -f examples/storage/hazelcast/hazelcast-service.yaml

The real power of Kubernetes and Hazelcast lies in easily building a replicated, resizable Hazelcast cluster.

In Kubernetes a _[Replication Controller](../../../docs/user-guide/replication-controller.md)_ is responsible for replicating sets of identical pods. Like a _Service_ it has a selector query which identifies the members of it's set. Unlike a _Service_ it also has a desired number of replicas, and it will create or delete _Pods_ to ensure that the number of _Pods_ matches up with it's desired state.
In Kubernetes a _[_Deployment_](../../../docs/user-guide/deployments.md)_ is responsible for replicating sets of identical pods. Like a _Service_ it has a selector query which identifies the members of its set. Unlike a _Service_ it also has a desired number of replicas, and it will create or delete _Pods_ to ensure that the number of _Pods_ matches up with its desired state.

Replication Controllers will "adopt" existing pods that match their selector query, so let's create a Replication Controller with a single replica to adopt our existing Hazelcast Pod.
Deployments will "adopt" existing pods that match their selector query, so let's create a Deployment with a single replica to adopt our existing Hazelcast Pod.

<!-- BEGIN MUNGE: EXAMPLE hazelcast-controller.yaml -->

```yaml
apiVersion: v1
kind: ReplicationController
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: hazelcast
labels:
name: hazelcast
name: hazelcast
spec:
replicas: 1
selector:
name: hazelcast
template:
metadata:
labels:
name: hazelcast
spec:
containers:
- resources:
limits:
cpu: 0.1
image: quay.io/pires/hazelcast-kubernetes:0.6.1
name: hazelcast
env:
- name: "DNS_DOMAIN"
value: "cluster.local"
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- containerPort: 5701
name: hazelcast
- name: hazelcast
image: quay.io/pires/hazelcast-kubernetes:0.8.0
imagePullPolicy: Always
env:
- name: "DNS_DOMAIN"
value: "cluster.local"
ports:
- name: hazelcast
containerPort: 5701
```

[Download example](hazelcast-controller.yaml?raw=true)
[Download example](hazelcast-deployment.yaml?raw=true)
<!-- END MUNGE: EXAMPLE hazelcast-controller.yaml -->

There are a few things to note in this description. First is that we are running the `quay.io/pires/hazelcast-kubernetes` image, tag `0.5`. This is a `busybox` installation with JRE 8 Update 45. However it also adds a custom [`application`](https://github.com/pires/hazelcast-kubernetes-bootstrapper) that finds any Hazelcast nodes in the cluster and bootstraps an Hazelcast instance accordingly. The `HazelcastDiscoveryController` discovers the Kubernetes API Server using the built in Kubernetes discovery service, and then uses the Kubernetes API to find new nodes (more on this later).

You may also note that we tell Kubernetes that the container exposes the `hazelcast` port. Finally, we tell the cluster manager that we need 1 cpu core.
You may note that we tell Kubernetes that the container exposes the `hazelcast` port.

The bulk of the replication controller config is actually identical to the Hazelcast pod declaration above, it simply gives the controller a recipe to use when creating new pods. The other parts are the `selector` which contains the controller's selector query, and the `replicas` parameter which specifies the desired number of replicas, in this case 1.

Expand All @@ -119,127 +108,124 @@ Last but not least, we set `DNS_DOMAIN` environment variable according to your K
Create this controller:

```sh
$ kubectl create -f examples/storage/hazelcast/hazelcast-controller.yaml
$ kubectl create -f examples/storage/hazelcast/hazelcast-deployment.yaml
```

After the controller provisions successfully the pod, you can query the service endpoints:

```sh
$ kubectl get endpoints hazelcast -o json
{
"kind": "Endpoints",
"apiVersion": "v1",
"metadata": {
"name": "hazelcast",
"namespace": "default",
"selfLink": "/api/v1/namespaces/default/endpoints/hazelcast",
"uid": "094e507a-2700-11e5-abbc-080027eae546",
"resourceVersion": "4094",
"creationTimestamp": "2015-07-10T12:34:41Z",
"labels": {
"name": "hazelcast"
}
},
"subsets": [
{
"addresses": [
{
"ip": "10.244.37.3",
"targetRef": {
"kind": "Pod",
"namespace": "default",
"name": "hazelcast-nsyzn",
"uid": "f57eb6b0-2706-11e5-abbc-080027eae546",
"resourceVersion": "4093"
}
}
],
"ports": [
{
"port": 5701,
"protocol": "TCP"
}
]
}
]
}
$ kubectl get endpoints hazelcast -o yaml
apiVersion: v1
kind: Endpoints
metadata:
creationTimestamp: 2016-12-16T08:57:27Z
labels:
name: hazelcast
name: hazelcast
namespace: default
resourceVersion: "11360"
selfLink: /api/v1/namespaces/default/endpoints/hazelcast
uid: 46447198-70eb-11e6-940c-0800278ab84d
subsets:
- addresses:
- ip: 10.244.37.2
targetRef:
kind: Pod
name: hazelcast-1790698550-3heau
namespace: default
resourceVersion: "11359"
uid: c9c3febd-70eb-11e6-940c-0800278ab84d
ports:
- port: 5701
protocol: TCP
```

You can see that the _Service_ has found the pod created by the replication controller.

Now it gets even more interesting.

Let's scale our cluster to 2 pods:

Now it gets even more interesting. Let's scale our cluster to 2 pods:
```sh
$ kubectl scale rc hazelcast --replicas=2
$ kubectl scale deployment hazelcast --replicas 2
```

Now if you list the pods in your cluster, you should see two hazelcast pods:

```sh
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
hazelcast-nanfb 1/1 Running 0 40s
hazelcast-nsyzn 1/1 Running 0 2m
kube-dns-xudrp 3/3 Running 0 1h
$ kubectl get deployment,pods
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deploy/hazelcast 2 2 2 2 1m

NAME READY STATUS RESTARTS AGE
po/hazelcast-3980717115-k1xsk 1/1 Running 0 1m
po/hazelcast-3980717115-pbhbq 1/1 Running 0 22s
```

To prove that this all works, you can use the `log` command to examine the logs of one pod, for example:

```sh
$ kubectl log hazelcast-nanfb hazelcast
2015-07-10 13:26:34.443 INFO 5 --- [ main] com.github.pires.hazelcast.Application : Starting Application on hazelcast-nanfb with PID 5 (/bootstrapper.jar started by root in /)
2015-07-10 13:26:34.535 INFO 5 --- [ main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@42cfcf1: startup date [Fri Jul 10 13:26:34 GMT 2015]; root of context hierarchy
2015-07-10 13:26:35.888 INFO 5 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Registering beans for JMX exposure on startup
2015-07-10 13:26:35.924 INFO 5 --- [ main] c.g.p.h.HazelcastDiscoveryController : Asking k8s registry at https://kubernetes.default.svc.cluster.local..
2015-07-10 13:26:37.259 INFO 5 --- [ main] c.g.p.h.HazelcastDiscoveryController : Found 2 pods running Hazelcast.
2015-07-10 13:26:37.404 INFO 5 --- [ main] c.h.instance.DefaultAddressPicker : [LOCAL] [someGroup] [3.5] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [10.244.77.3, 10.244.37.3]
2015-07-10 13:26:37.405 INFO 5 --- [ main] c.h.instance.DefaultAddressPicker : [LOCAL] [someGroup] [3.5] Prefer IPv4 stack is true.
2015-07-10 13:26:37.415 INFO 5 --- [ main] c.h.instance.DefaultAddressPicker : [LOCAL] [someGroup] [3.5] Picked Address[10.244.77.3]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true
2015-07-10 13:26:37.852 INFO 5 --- [ main] com.hazelcast.spi.OperationService : [10.244.77.3]:5701 [someGroup] [3.5] Backpressure is disabled
2015-07-10 13:26:37.879 INFO 5 --- [ main] c.h.s.i.o.c.ClassicOperationExecutor : [10.244.77.3]:5701 [someGroup] [3.5] Starting with 2 generic operation threads and 2 partition operation threads.
2015-07-10 13:26:38.531 INFO 5 --- [ main] com.hazelcast.system : [10.244.77.3]:5701 [someGroup] [3.5] Hazelcast 3.5 (20150617 - 4270dc6) starting at Address[10.244.77.3]:5701
2015-07-10 13:26:38.532 INFO 5 --- [ main] com.hazelcast.system : [10.244.77.3]:5701 [someGroup] [3.5] Copyright (c) 2008-2015, Hazelcast, Inc. All Rights Reserved.
2015-07-10 13:26:38.533 INFO 5 --- [ main] com.hazelcast.instance.Node : [10.244.77.3]:5701 [someGroup] [3.5] Creating TcpIpJoiner
2015-07-10 13:26:38.534 INFO 5 --- [ main] com.hazelcast.core.LifecycleService : [10.244.77.3]:5701 [someGroup] [3.5] Address[10.244.77.3]:5701 is STARTING
2015-07-10 13:26:38.672 INFO 5 --- [ cached1] com.hazelcast.nio.tcp.SocketConnector : [10.244.77.3]:5701 [someGroup] [3.5] Connecting to /10.244.37.3:5701, timeout: 0, bind-any: true
2015-07-10 13:26:38.683 INFO 5 --- [ cached1] c.h.nio.tcp.TcpIpConnectionManager : [10.244.77.3]:5701 [someGroup] [3.5] Established socket connection between /10.244.77.3:59951
2015-07-10 13:26:45.699 INFO 5 --- [ration.thread-1] com.hazelcast.cluster.ClusterService : [10.244.77.3]:5701 [someGroup] [3.5]
kubectl logs -f hazelcast-39807171
15-k1xsk
2017-01-30 12:42:50.774 INFO 6 --- [ main] com.github.pires.hazelcast.Application : Starting Application on hazelcast-3980717115-k1xsk with PID 6 (/bootstrapper.jar started by root in /)
2017-01-30 12:42:50.781 INFO 6 --- [ main] com.github.pires.hazelcast.Application : No active profile set, falling back to default profiles: default
2017-01-30 12:42:50.852 INFO 6 --- [ main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@14514713: startup date [Mon Jan 30 12:42:50 GMT 2017]; root of context hierarchy
2017-01-30 12:42:52.304 INFO 6 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Registering beans for JMX exposure on startup
2017-01-30 12:42:52.323 INFO 6 --- [ main] c.g.p.h.HazelcastDiscoveryController : Asking k8s registry at https://kubernetes.default.svc.cluster.local..
2017-01-30 12:42:52.857 INFO 6 --- [ main] c.g.p.h.HazelcastDiscoveryController : Found 1 pods running Hazelcast.
2017-01-30 12:42:52.990 INFO 6 --- [ main] c.h.instance.DefaultAddressPicker : [LOCAL] [someGroup] [3.7.5] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [10.244.9.2]
2017-01-30 12:42:52.990 INFO 6 --- [ main] c.h.instance.DefaultAddressPicker : [LOCAL] [someGroup] [3.7.5] Prefer IPv4 stack is true.
2017-01-30 12:42:53.002 INFO 6 --- [ main] c.h.instance.DefaultAddressPicker : [LOCAL] [someGroup] [3.7.5] Picked [10.244.9.2]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true
2017-01-30 12:42:53.032 INFO 6 --- [ main] com.hazelcast.system : [10.244.9.2]:5701 [someGroup] [3.7.5] Hazelcast 3.7.5 (20170124 - 111f332) starting at [10.244.9.2]:5701
2017-01-30 12:42:53.032 INFO 6 --- [ main] com.hazelcast.system : [10.244.9.2]:5701 [someGroup] [3.7.5] Copyright (c) 2008-2016, Hazelcast, Inc. All Rights Reserved.
2017-01-30 12:42:53.032 INFO 6 --- [ main] com.hazelcast.system : [10.244.9.2]:5701 [someGroup] [3.7.5] Configured Hazelcast Serialization version : 1
2017-01-30 12:42:53.343 INFO 6 --- [ main] c.h.s.i.o.impl.BackpressureRegulator : [10.244.9.2]:5701 [someGroup] [3.7.5] Backpressure is disabled
2017-01-30 12:42:54.273 INFO 6 --- [ main] com.hazelcast.instance.Node : [10.244.9.2]:5701 [someGroup] [3.7.5] Creating TcpIpJoiner
2017-01-30 12:42:54.507 INFO 6 --- [ main] c.h.s.i.o.impl.OperationExecutorImpl : [10.244.9.2]:5701 [someGroup] [3.7.5] Starting 2 partition threads
2017-01-30 12:42:54.508 INFO 6 --- [ main] c.h.s.i.o.impl.OperationExecutorImpl : [10.244.9.2]:5701 [someGroup] [3.7.5] Starting 3 generic threads (1 dedicated for priority tasks)
2017-01-30 12:42:54.525 INFO 6 --- [ main] com.hazelcast.core.LifecycleService : [10.244.9.2]:5701 [someGroup] [3.7.5] [10.244.9.2]:5701 is STARTING
2017-01-30 12:42:54.529 INFO 6 --- [ main] c.h.n.t.n.NonBlockingIOThreadingModel : [10.244.9.2]:5701 [someGroup] [3.7.5] TcpIpConnectionManager configured with Non Blocking IO-threading model: 3 input threads and 3 output threads
2017-01-30 12:42:54.578 INFO 6 --- [ main] com.hazelcast.cluster.impl.TcpIpJoiner : [10.244.9.2]:5701 [someGroup] [3.7.5]


Members [1] {
Member [10.244.9.2]:5701 - f9cae801-59da-49d9-b8de-7719abb53844 this
}

2017-01-30 12:42:54.660 INFO 6 --- [ main] com.hazelcast.core.LifecycleService : [10.244.9.2]:5701 [someGroup] [3.7.5] [10.244.9.2]:5701 is STARTED
2017-01-30 12:42:54.662 INFO 6 --- [ main] com.github.pires.hazelcast.Application : Started Application in 5.078 seconds (JVM running for 5.771)
2017-01-30 12:44:08.780 INFO 6 --- [thread-Acceptor] c.h.nio.tcp.SocketAcceptorThread : [10.244.9.2]:5701 [someGroup] [3.7.5] Accepting socket connection from /10.244.93.3:45945
2017-01-30 12:44:08.814 INFO 6 --- [cached.thread-1] c.h.nio.tcp.TcpIpConnectionManager : [10.244.9.2]:5701 [someGroup] [3.7.5] Established socket connection between /10.244.9.2:5701 and /10.244.93.3:45945
2017-01-30 12:44:15.785 INFO 6 --- [ration.thread-0] c.h.internal.cluster.ClusterService : [10.244.9.2]:5701 [someGroup] [3.7.5]

Members [2] {
Member [10.244.37.3]:5701
Member [10.244.77.3]:5701 this
Member [10.244.9.2]:5701 - f9cae801-59da-49d9-b8de-7719abb53844 this
Member [10.244.93.3]:5701 - 4e15667b-ce17-40c2-b045-abe3fb25d48b
}

2015-07-10 13:26:47.722 INFO 5 --- [ main] com.hazelcast.core.LifecycleService : [10.244.77.3]:5701 [someGroup] [3.5] Address[10.244.77.3]:5701 is STARTED
2015-07-10 13:26:47.723 INFO 5 --- [ main] com.github.pires.hazelcast.Application : Started Application in 13.792 seconds (JVM running for 14.542)
```

Now let's scale our cluster to 4 nodes:

```sh
$ kubectl scale rc hazelcast --replicas=4
$ kubectl scale deployment hazelcast --replicas 4
```

Examine the status again by checking the logs and you should see the 4 members connected.
Examine the status again by checking a node's logs and you should see the 4 members connected. Something like:
```
(...)

Members [4] {
Member [10.244.9.2]:5701 - f9cae801-59da-49d9-b8de-7719abb53844 this
Member [10.244.93.3]:5701 - 4e15667b-ce17-40c2-b045-abe3fb25d48b
Member [10.244.9.3]:5701 - e0f36fa4-16bf-4009-a034-d4e7a4105003
Member [10.244.93.4]:5701 - 7ac96b48-aa47-4410-885f-1ad0fc3690f0
}
```

### tl; dr;

For those of you who are impatient, here is the summary of the commands we ran in this tutorial.

```sh
# create a service to track all hazelcast nodes
kubectl create -f examples/storage/hazelcast/hazelcast-service.yaml

# create a replication controller to replicate hazelcast nodes
kubectl create -f examples/storage/hazelcast/hazelcast-controller.yaml

# scale up to 2 nodes
kubectl scale rc hazelcast --replicas=2

# scale up to 4 nodes
kubectl scale rc hazelcast --replicas=4
kubectl create -f service.yaml
kubectl create -f deployment.yaml
kubectl scale deployment hazelcast --replicas 2
kubectl scale deployment hazelcast --replicas 4
```

### Hazelcast Discovery Source
Expand Down
31 changes: 0 additions & 31 deletions examples/storage/hazelcast/hazelcast-controller.yaml
View file
Open in desktop

This file was deleted.

22 changes: 22 additions & 0 deletions examples/storage/hazelcast/hazelcast-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: hazelcast
labels:
name: hazelcast
spec:
template:
metadata:
labels:
name: hazelcast
spec:
containers:
- name: hazelcast
image: quay.io/pires/hazelcast-kubernetes:0.8.0
imagePullPolicy: Always
env:
- name: "DNS_DOMAIN"
value: "cluster.local"
ports:
- name: hazelcast
containerPort: 5701