Skip to content

Commit

Permalink
documentation
Browse files Browse the repository at this point in the history 
Signed-off-by: Caleb Lloyd <caleb@synadia.com>
  • Loading branch information
Caleb Lloyd committed Apr 17, 2023
1 parent abcb1ad commit 7dd1bbd
Show file tree
Hide file tree
Showing 4 changed files with 270 additions and 68 deletions.
310 changes: 243 additions & 67 deletions helm/charts/nats/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,28 +1,56 @@
# nats-next
# nats-helm

Experimental composable Helm chart for NATS.
```shell
helm repo add nats https://nats-io.github.io/k8s/helm/charts/
# --devel flag is required while 1.x is in beta
helm upgrade --install nats nats/nats --devel
```

The chart has very few explicit values defined. Everything in the NATS Config or Kubernetes Resources can be overridden by merging or patching.
## Values

- Merges are performed using the Helm `mergeOverwrite` function
- Patches are performed using [JSON Patch](https://jsonpatch.com/)
There are a handful of explicitly defined options which are documented with comments in the [values.yaml](values.yaml) file.

Additionally, anything in `values.yaml` can be templated:
Everything in the NATS Config or Kubernetes Resources can be overridden by `merge` and `patch`, which is supported for the following values:

- maps matching the following syntax will be templated and parsed as YAML:
```yaml
$tplYaml: |
yaml template
```
- maps matching the follow syntax will be templated, parsed as YAML, and spread into the parent map/slice
```yaml
$tplYamlSpread: |
yaml template
```
| key | type | enabled by default |
|----------------------------------|-----------------------------------------------------------------------------------------------------------------------------|-----------------------------------------|
| `config` | [NATS Config](https://docs.nats.io/running-a-nats-service/configuration) | yes |
| `config.cluster` | [NATS Cluster](https://docs.nats.io/running-a-nats-service/configuration/clustering/cluster_config) | no |
| `config.cluster.tls` | [NATS TLS](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls) | no |
| `config.jetstream` | [NATS JetStream](https://docs.nats.io/running-a-nats-service/configuration#jetstream) | no |
| `config.jetstream.fileStore.pvc` | [k8s PVC](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#persistentvolumeclaim-v1-core) | yes, when `config.jetstream` is enabled |
| `config.nats.tls` | [NATS TLS](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls) | no |
| `config.leafnode` | [NATS LeafNode](https://docs.nats.io/running-a-nats-service/configuration/leafnode/leafnode_conf) | no |
| `config.leafnode.tls` | [NATS TLS](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls) | no |
| `config.websocket` | [NATS WebSocket](https://docs.nats.io/running-a-nats-service/configuration/websocket/websocket_conf) | no |
| `config.websocket.tls` | [NATS TLS](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls) | no |
| `config.websocket.ingress` | [k8s Ingress](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#ingress-v1-networking-k8s-io) | no |
| `config.mqtt` | [NATS MQTT](https://docs.nats.io/running-a-nats-service/configuration/mqtt/mqtt_config) | no |
| `config.mqtt.tls` | [NATS TLS](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls) | no |
| `config.gateway` | [NATS Gateway](https://docs.nats.io/running-a-nats-service/configuration/gateways/gateway#gateway-configuration-block) | no |
| `config.gateway.tls` | [NATS TLS](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls) | no |
| `config.resolver` | [NATS Resolver](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/jwt/resolver) | no |
| `config.resolver.pvc` | [k8s PVC](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#persistentvolumeclaim-v1-core) | yes, when `config.resolver` is enabled |
| `container` | nats [k8s Container](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#container-v1-core) | yes |
| `reloader` | config reloader [k8s Container](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#container-v1-core) | yes |
| `promExporter` | prometheus exporter [k8s Container](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#container-v1-core) | no |
| `promExporter.podMonitor` | [prometheus PodMonitor](https://prometheus-operator.dev/docs/operator/api/#monitoring.coreos.com/v1.PodMonitor) | no |
| `service` | [k8s Service](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#service-v1-core) | yes |
| `statefulSet` | [k8s StatefulSet](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#statefulset-v1-apps) | yes |
| `podTemplate` | [k8s PodTemplate](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#pod-v1-core) | yes |
| `headlessService` | [k8s Service](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#service-v1-core) | yes |
| `configMap` | [k8s ConfiegMap](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#configmap-v1-core) | yes |
| `natsBox.context.default` | [NATS Context](https://docs.nats.io/using-nats/nats-tools/nats_cli#nats-contexts) | yes |
| `natsBox.context.[name]` | [NATS Context](https://docs.nats.io/using-nats/nats-tools/nats_cli#nats-contexts) | no |
| `natsBox.container` | nats-box [k8s Container](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#container-v1-core) | yes |
| `natsBox.deployment` | [k8s Deployment](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#deployment-v1-apps) | yes |
| `natsBox.podTemplate` | [k8s PodTemplate](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#pod-v1-core) | yes |
| `natsBox.contextsSecret` | [k8s Secret](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#secret-v1-core) | yes |
| `natsBox.contentsSecret` | [k8s Secret](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#secret-v1-core) | yes |

## NATS Config
### Merge

**Merge** - add accounts
Merging is performed using the Helm `merge` function. Example - add NATS accounts and container resources:

```yaml
config:
Expand All @@ -31,110 +59,258 @@ config:
A:
users:
- {user: a, password: a}
B:
B:
users:
- {user: b, password: b}
natsBox:
contexts:
a:
merge: {user: a, password: a}
b:
merge: {user: b, password: b}
defaultContextName: a
```

**Patch** - remove http monitoring
## Patch

Patching is performed using [JSON Patch](https://jsonpatch.com/). Example - add additional route to end of route list:

```yaml
config:
patch:
- op: remove
path: /http
cluster:
enabled: true
patch:
- op: add
path: /routes/-
value: nats://demo.nats.io:6222
```

## Common Configurations

### JetStream Cluster on 3 separate hosts

```yaml
config:
cluster:
enabled: true
replicas: 3
jetstream:
enabled: true
fileStore:
pvc:
size: 10Gi

podTemplate:
topologySpreadConstraints:
kubernetes.io/hostname:
maxSkew: 1
```

**Template** - add cluster authorization
### NATS Container Resources

```yaml
replicas: 3
container:
env:
# different from k8s units, suffix must be B, KiB, MiB, GiB, or TiB
# should be ~90% of memory limit
GOMEMLIMIT: 7GiB
merge:
# recommended limit is at least 2 CPU cores and 8Gi Memory for production JetStream clusters
resources:
requests:
cpu: "2"
memory: 8Gi
limits:
cpu: "2"
memory: 8Gi
```

### Specify Image Version

```yaml
container:
nats:
image:
tag: x.y.z-alpine
```

### Operator Mode with NATS Resolver

Run `nsc generate config --nats-resolver` and replace the `OPERATOR_JWT`, `SYS_ACCOUNT_ID`, and `SYS_ACCOUNT_JWT` with your values.
Make sure that you do not include the trailing `,` in the `SYS_ACCOUNT_JWT`.

```
config:
resolver:
enabled: true
merge:
type: full
interval: 2m
timeout: 1.9s
merge:
operator: OPERATOR_JWT
system_account: SYS_ACCOUNT_ID
resolver_preload:
SYS_ACCOUNT_ID: SYS_ACCOUNT_JWT
```

## Advanced Features

### Templating Values

Anything in `values.yaml` can be templated:

- maps matching the following syntax will be templated and parsed as YAML:
```yaml
$tplYaml: |
yaml template
```
- maps matching the follow syntax will be templated, parsed as YAML, and spread into the parent map/slice
```yaml
$tplYamlSpread: |
yaml template
```

Example - add cluster authorization:

```yaml
config:
cluster:
enabled: true
merge:
authorization:
user: foo
password:
$tplYaml: >
{{ printf "bar" | bcrypt }}
{{ bcrypt "bar" }}
routes:
- $tplYamlSpread: |
{{- range $i, $_ := until ($.Values.replicas | int) }}
{{- range $i, $_ := until (int $.Values.config.cluster.replicas) }}
- {{ printf "nats://foo:bar@%s-%d.%s:6222" $.Values.statefulSet.name $i $.Values.headlessService.name }}
{{- end }}
```

templates to the `nats.conf`:

```json
```
{
"cluster": {
"authorization": {
"password": "$2a$10$hC7z.u7LyEeBVcBsuZjmDege8Cf448JaNWHQGbpgrHv8WOSksQ8qy",
"password": "$2a$10$iPs.JbHVKFlFnE.NAN.jF.I1PNi72UycEE83TzyUd1rZsXfFQteQ6",
"user": "foo"
},
"routes": [
"nats://foo:bar@nats-0.nats-headless:6222",
"nats://foo:bar@nats-1.nats-headless:6222",
"nats://foo:bar@nats-2.nats-headless:6222"
]
}
},
"port": 4222,
...
}
```

## NATS Container
### NATS Config Units and Variables

**Merge** - increase resources
NATS configuration extends JSON, and can represent Units and Variables. They must be wrapped in `<< >>` in order to template correctly. Example:

```yaml
container:
config:
merge:
resources:
requests:
memory: 8Gi
cpu: "2"
limits:
memory: 16Gi
cpu: "4"
authorization:
# variable
token: << $TOKEN >>
# units
max_payload: << 2MB >>
```

**Patch** - add a wss port
templates to the `nats.conf`:

```yaml
container:
patch:
- op: add
path: /ports/-
value:
containerPort: 443
name: wss
```
{
"authorization": {
"token": $TOKEN
},
"max_payload": 2MB,
"port": 4222,
...
}
```

## PodTemplate
### NATS Config Includes

**Merge** - add an annotation and a security context
Any NATS Config key ending in `$include` will be replaced with an include directive. Included files should be in paths relative to `/etc/nats-config`. Example:

```yaml
podTemplate:
config:
merge:
$include: auth.conf
zzz$include: params.conf
configMap:
merge:
metadata:
annotations:
nats/is: awesome
spec:
securityContext:
runAsUser: 1000
data:
auth.conf: |
accounts: {
A: {
users: [
{user: a, password: a}
]
},
B: {
users: [
{user: b, password: b}
]
},
}
params.conf: |
max_payload: 2MB
```

templates to the `nats.conf`:

```
include auth.conf;
"port": 4222,
...
include params.conf;
```

**Patch** - add a volume
### Extra Resources

Enables adding additional arbitrary resources. Example - expose WebSocket via VirtualService in Istio:

```yaml
podTemplate:
patch:
- op: add
path: /spec/volumes/-
value:
name: tls
secret:
secretName: my-tls-cert
config:
websocket:
enabled: true
extraResources:
- apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name:
$tplYaml: >
{{ include "nats.fullname" $ | quote }}
labels:
$tplYaml: |
{{ include "nats.labels" $ }}
spec:
hosts:
- demo.nats.io
gateways:
- my-gateway
http:
- name: default
match:
- name: root
uri:
exact: /
route:
- destination:
host:
$tplYaml: >
{{ .Values.service.name | quote }}
port:
number:
$tplYaml: >
{{ .Values.config.websocket.port }}
```
7 changes: 7 additions & 0 deletions helm/charts/nats/files/stateful-set/pod-template.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,3 +49,10 @@ spec:
secretName: {{ $configProtocol.tls.secretName | quote }}
{{- end }}
{{- end }}

{{- with .Values.podTemplate.topologySpreadConstraints }}
topologySpreadConstraints:
{{- range $k, $v := . }}
- {{ merge (dict "topologyKey" $k "labelSelector" (dict "matchLabels" (include "nats.selectorLabels" $ | fromYaml))) $v | toYaml | nindent 4 }}
{{- end }}
{{- end}}
Loading

0 comments on commit 7dd1bbd

Please sign in to comment.