This operator manages NATS clusters atop Kubernetes, automating their creation and administration.
Clone or download
wallyqs Merge pull request #93 from onesolpark/master
Added option to modify server image to custom image
Latest commit 42fbfbd Nov 12, 2018

README.md

NATS Operator

License Apache 2.0 Build Status Version

NATS Operator manages NATS clusters atop Kubernetes, automating their creation and administration.

Requirements

  • Kubernetes v1.8+

Getting Started

The current version of the operator creates a NatsCluster Custom Resources Definition (CRD) under the nats.io API group, to which you can make requests to create NATS clusters.

To add the NatsCluster and NATS Operator to your cluster you can run:

$ kubectl apply -f https://raw.githubusercontent.com/nats-io/nats-operator/master/deploy/default-rbac.yaml

$ kubectl apply -f https://raw.githubusercontent.com/nats-io/nats-operator/master/deploy/deployment.yaml

You will then be able to confirm that there is a new CRD registered in the cluster:

$ kubectl get crd

NAME                                    AGE
natsclusters.nats.io                    1s

An example of creating a 3 node cluster is below. The NATS operator will be responsible of assembling the cluster and replacing pods in case of failures.

echo '
apiVersion: "nats.io/v1alpha2"
kind: "NatsCluster"
metadata:
  name: "example-nats-cluster"
spec:
  size: 3
  version: "1.3.0"
' | kubectl apply -f -

To list all the NATS clusters:

$ kubectl get natsclusters.nats.io

NAME                   AGE
example-nats-cluster   1s

Helm support

There is an alternative way to install NATS Operator for Helm users. You can go to helm/nats-operator and use the Helm charts found in the repo.

Installing NATS Operator on a Custom Namespace with RBAC enabled

In order to install the NATS Operator in a custom namespace, if RBAC is enabled it is then necessary to create a ServiceAccount and ClusterRoleBinding referring to the custom namespace in order for the operator to be able run properly.

The example below will deploy the nats-operator under the nats-io namespace, using the nats-operator ServiceAccount:

$ kubectl create namespace nats-io

$ echo '
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nats-operator
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: nats-operator-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nats-operator
subjects:
- kind: ServiceAccount
  name: nats-operator
  namespace: nats-io
' | kubectl -n nats-io apply -f -

$ kubectl -n nats-io apply -f https://raw.githubusercontent.com/nats-io/nats-operator/master/deploy/role.yaml

$ kubectl -n nats-io apply -f https://raw.githubusercontent.com/nats-io/nats-operator/master/deploy/deployment.yaml

$ kubectl -n nats-io logs deployment/nats-operator
time="2018-10-25T03:04:58Z" level=info msg="nats-operator Version: 0.2.3-v1alpha2+git"

Note that the NATS operator only monitors the NatsCluster resources which are created in the namespace where it was deployed, so if you want to create a cluster you have to specify the same one as the operator:

$ kubectl -n nats-io apply -f example/example-nats-cluster.yaml
natscluster "example-nats-1" created

$ kubectl -n nats-io get natsclusters
NAME             AGE
example-nats-1   6m

$ kubectl -n nats-io get pods -l nats_cluster=example-nats-1
NAME               READY     STATUS    RESTARTS   AGE
example-nats-1-1   1/1       Running   0          7m
example-nats-1-2   1/1       Running   0          7m
example-nats-1-3   1/1       Running   0          6m

TLS support

By using a pair of opaque secrets (one for the clients and then another for the routes), it is possible to set TLS for the communication between the clients and also for the transport between the routes:

apiVersion: "nats.io/v1alpha2"
kind: "NatsCluster"
metadata:
  name: "nats"
spec:
  # Number of nodes in the cluster
  size: 3
  version: "1.3.0"

  tls:
    # Certificates to secure the NATS client connections:
    serverSecret: "nats-clients-tls"

    # Certificates to secure the routes.
    routesSecret: "nats-routes-tls"

In order for TLS to be properly established between the nodes, it is necessary to create a wildcard certificate that matches the subdomain created for the service from the clients and the one for the routes.

The routesSecret has to provide the files: ca.pem, route-key.pem, route.pem, for the CA, server private and public key respectively.

$ kubectl create secret generic nats-routes-tls --from-file=ca.pem --from-file=route-key.pem --from-file=route.pem

Similarly, the serverSecret has to provide the files: ca.pem, server-key.pem, and server.pem for the CA, server private key and public key used to secure the connection with the clients.

$ kubectl create secret generic nats-clients-tls --from-file=ca.pem --from-file=server-key.pem --from-file=server.pem

Authorization

Using ServiceAccounts

The NATS Operator can define permissions based on Roles by using any present ServiceAccount in a namespace. To use this feature, it is necessary to use a Kubernetes +v1.10 cluster with the TokenRequest and PodShareProcessNamespace feature flags enabled. To try this feature using minikube you can configure it to start as follows:

minikube start \
  --feature-gates="TokenRequest=true,PodShareProcessNamespace=true" \
  --extra-config=apiserver.service-account-signing-key-file=/var/lib/localkube/certs/apiserver.key \
  --extra-config=apiserver.service-account-issuer=api \
  --extra-config=apiserver.service-account-api-audiences=api \
  --extra-config=apiserver.service-account-key-file=/var/lib/localkube/certs/sa.pub

ServiceAccounts integration can then be enabled by setting the enableServiceAccounts flag to true in the NatsCluster configuration.

---
apiVersion: nats.io/v1alpha2
kind: NatsCluster
metadata:
  name: example-nats
spec:
  size: 3
  version: "1.3.0"

  pod:
    enableConfigReload: true
  auth:
    enableServiceAccounts: true

Permissions for a ServiceAccount can be set by creating a NatsServiceRole for that account. In the example below, there are two accounts, one is an admin user that has more permissions.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nats-admin-user
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nats-user
---
apiVersion: nats.io/v1alpha2
kind: NatsServiceRole
metadata:
  name: nats-user
  namespace: nats-io

  # Specifies which NATS cluster will be mapping this account.
  labels:
    nats_cluster: example-nats
spec:
  permissions:
    publish: ["foo.*", "foo.bar.quux"]
    subscribe: ["foo.bar"]
---
apiVersion: nats.io/v1alpha2
kind: NatsServiceRole
metadata:
  name: nats-admin-user
  namespace: nats-io
  labels:
    nats_cluster: example-nats
spec:
  permissions:
    publish: [">"]
    subscribe: [">"]

The above will create two different Secrets which can then be mounted as volumes for a Pod.

$ kubectl -n nats-io get secrets
NAME                                       TYPE          DATA      AGE
...
nats-admin-user-example-nats-bound-token   Opaque        1         43m
nats-user-example-nats-bound-token         Opaque        1         43m

An example of mounting the secret in a Pod can be found below:

---
apiVersion: v1
kind: Pod
metadata:
  name: nats-user-pod
  labels:
    nats_cluster: example-nats
spec:
  volumes:
    - name: "token"
      projected:
        sources:
        - secret:
            name: "nats-user-example-nats-bound-token"
            items:
              - key: token
                path: "token"
  restartPolicy: Never
  containers:
    - name: nats-ops
      command: ["/bin/sh"]
      image: "wallyqs/nats-ops:latest"
      tty: true
      stdin: true
      stdinOnce: true
      volumeMounts:
      - name: "token"
        mountPath: "/var/run/secrets/nats.io"

Then within the Pod the token can be used to authenticate against the server using the created token.

$ kubectl -n nats-io attach -it nats-user-pod

/go # nats-sub -s nats://nats-user:`cat /var/run/secrets/nats.io/token`@example-nats:4222 hello.world
Listening on [hello.world]
^C
/go # nats-sub -s nats://nats-admin-user:`cat /var/run/secrets/nats.io/token`@example-nats:4222 hello.world
Can't connect: nats: authorization violation

Using a single secret with the explicit configuration.

Authorization can also be set for the server by using a secret where the permissions are defined in JSON:

{
  "users": [
    { "username": "user1", "password": "secret1" },
    { "username": "user2", "password": "secret2",
      "permissions": {
	"publish": ["hello.*"],
	"subscribe": ["hello.world"]
      }
    }
  ],
  "default_permissions": {
    "publish": ["SANDBOX.*"],
    "subscribe": ["PUBLIC.>"]
  }
}

Example of creating a secret to set the permissions:

kubectl create secret generic nats-clients-auth --from-file=clients-auth.json

Now when creating a NATS cluster it is possible to set the permissions as in the following example:

apiVersion: "nats.io/v1alpha2"
kind: "NatsCluster"
pmetadata:
  name: "example-nats-auth"
spec:
  size: 3
  version: "1.1.0"

  auth:
    # Definition in JSON of the users permissions
    clientsAuthSecret: "nats-clients-auth"

    # How long to wait for authentication
    clientsAuthTimeout: 5

Configuration Reload

On Kubernetes +v1.10 clusters that have been started with support for sharing the process namespace (via --feature-gates=PodShareProcessNamespace=true), it is possible to enable on-the-fly reloading of configuration for the servers that are part of the cluster. This can also be combined with the authorization support, so in case the user permissions change, then the servers will reload and apply the new permissions.

apiVersion: "nats.io/v1alpha2"
kind: "NatsCluster"
metadata:
  name: "example-nats-auth"
spec:
  size: 3
  version: "1.1.0"

  pod:
    # Enable on-the-fly NATS Server config reload
    # Note: only supported in Kubernetes clusters with PID namespace sharing enabled.
    enableConfigReload: true

    # Possible to customize version of reloader image
    reloaderImage: connecteverything/nats-server-config-reloader
    reloaderImageTag: "0.2.2-v1alpha2"
    reloaderImagePullPolicy: "IfNotPresent"
  auth:
    # Definition in JSON of the users permissions
    clientsAuthSecret: "nats-clients-auth"

    # How long to wait for authentication
    clientsAuthTimeout: 5

Development

Building the Docker Image

To build the nats-operator Docker image:

$ docker build -f docker/operator/Dockerfile . <image:tag>

To build the nats-server-config-reloader:

$ docker build -f docker/reloader/Dockerfile . <image:tag>

You'll need Docker 17.06.0-ce or higher.

Updating the NatsCluster type (code generation)

If you are adding a new field to the NatsCluster, then you have to get the deepcopy-gen tools first.

$ go get -u github.com/kubernetes/gengo/examples/deepcopy-gen

Then run the code generation script in order to recreate pkg/spec/zz_generated.deepcopy.go with the required methods to support that field filled in:

$ ./hack/codegen.sh

Running outside the cluster for debugging

For debugging purposes, it is also possible to run the operator outside of the cluster without having to build an image:

MY_POD_NAMESPACE=default MY_POD_NAME=nats-operator go run cmd/operator/main.go --debug-kube-config-path=$HOME/.kube/config

Direct access to the cluster

For debugging and development you might want to access the NATS cluster directly. For example, if you created the cluster with name example-nats-cluster in namespace nats-io you can forward ports of the pod with name example-nats-cluster-1 as follows:

$ kubectl port-forward -n nats-io example-nats-cluster-1 4222:4222