Add Standalone gateway and separate brokers

[salaboy]

Like here:
https://github.com/camundacon2019/zeebe-ft-demo/blob/master/docker-compose.yml

[Zelldon]

For our deployments/benchmark we will need this as well.

Here is a configuration we currently use to deploy a standalone gateway:

Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gateway
spec:
  replicas: 1
  template:
    spec:
      volumes:
        - name: configuration
          configMap:
            name: gateway
            defaultMode: 0744
      containers:
        - name: gateway
          image: camunda/zeebe:latest
          ports:
            - containerPort: 26500
              name: gateway
            - containerPort: 26502
              name: cluster
            - containerPort: 9600
              name: monitoring
          env:
            - name: ZEEBE_STANDALONE_GATEWAY
              value: "true"
            - name: ZEEBE_GATEWAY_CLUSTER_MEMBER_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          volumeMounts:
            - name: configuration
              mountPath: /usr/local/zeebe/conf/log4j2.xml
              subPath: log4j2.xml
          securityContext:
            capabilities:
                add: ["SYS_ADMIN"]
          readinessProbe:
            tcpSocket:
              port: gateway
            initialDelaySeconds: 20
            periodSeconds: 5

Service:

apiVersion: v1
kind: Service
metadata:
  name: gateway
spec:
  ports:
    - port: 26500
      name: grpc
    - port: 9600
      name: monitoring

Might help to implement this.

[salaboy]

@Zelldon @npepinpe @saig0 can you guys point me out how does the standalone gateway connects with the brokers? where is that configuration?

[salaboy]

I am currently getting
2020-03-10 09:23:23.303 [] [atomix-partition-group-membership-service-0] WARN io.atomix.primitive.partition.impl.DefaultPartitionGroupMembershipService - Failed to locate management group via bootstrap nodes. Please ensure partition groups are configured either locally or remotely and the node is able to reach partition group members. in the gateway

[Zelldon]

Hey @salaboy this is currently expected, we have no fix for that but it should still work

[salaboy]

let me try to deploy some processes
and I will send an initial PR to review

[salaboy]

Still not finding the brokers.. there is something that I'm missing.. in the previous deployment descriptor there is a configMap, which seems to only contain the logging config files.. but is there something else require for the gateway to locate the brokers?
Currently I am running

salaboy> zbctl status --insecure                                                               
Cluster size: 0
Partitions count: 0
Replication factor: 0
Brokers:

Where no brokers are found, even when the brokers are up and running right besides the gateway.

[salaboy]

does this make sense: ->
ZEEBE_GATEWAY_CLUSTER_MEMBER_ID: zeebe-zeebe-gateway-7f565445fd-5jm2h (v1:metadata.name)

[salaboy]

that is basically the name of the pod hosting the gateway.

[Zelldon]

@npepinpe i think you setup the last time the standalone gateway any idea? I think in the gateway config the contactPoint needs to be set correctly.

Probably something like:

[cluster]
contactPoint = "NODE_ONE_HOST:INTERNAL_API"

With the ports i'm always confused we need to document this better

[npepinpe]

You need to point the contactPoint to at least one of the brokers - this isn't optimal in terms of discoverability obviously, but that's how it is for now. I don't remember having to do anything in particular tbh.

Having the pod name as the node ID is totally fine, I think that's also what I was doing.

[npepinpe]

I additionally configured the following environment vars:

            - name: ZEEBE_GATEWAY_CONTACT_POINT
              value: "$(BROKER_SERVICE_NAME)"
            - name: ZEEBE_GATEWAY_HOST
              value: 0.0.0.0

where $(BROKER_SERVICE_NAME) was taken from the downstream API as

  - name: BROKER_SERVICE_NAME
    objref:
      kind: Service
      name: broker
      apiVersion: v1
    fieldref:
      fieldpath: metadata.name

This was for the kustomize setup, but should be easy enough to adapt.

[Zelldon]

Hey guys I think the current setup doesn't work, at least for me.

I had to disable the standalone and enable the embedded again to get my benchmarks working

gatewayMetrics: true
gateway.replicas: 0

env:
  - name: ZEEBE_BROKER_GATEWAY_ENABLE
    value: "true"

[npepinpe]

Is this fixed by now?

[ezolnbl-dpc]

anybody experienced a double env var for ZEEBE_BROKER_GATEWAY_ENABLE when defining the helm chart(zeebe 0.0.122) like this:

zeebe-cluster:
  global:
    zeebe: "zeebe-zeebe"
    elasticsearch:
      host: "zeebe-elasticsearch"
  clusterSize: "1"
  partitionCount: "1"
  replicationFactor: "1"
  cpuThreadCount: "2"
  ioThreadCount: "2"
  pvcSize: "10Gi"
  pvcAccessModes: [ "ReadWriteOnce" ]

  resources:
    requests:
      cpu: 100m

  env:
    - name: ZEEBE_BROKER_GATEWAY_ENABLE
      value: "true"

  gateway:
    replicas: 0
    logLevel: warn
    env:
      - name: ZEEBE_GATEWAY_THREADS_MANAGEMENTTHREADS
        value: "4"

and in kubectl desribe:

 Environment:
   ZEEBE_BROKER_CLUSTER_CLUSTERNAME:                zeebe-zeebe
   ZEEBE_LOG_LEVEL:                                 debug
   ZEEBE_BROKER_CLUSTER_PARTITIONSCOUNT:            1
   ZEEBE_BROKER_CLUSTER_CLUSTERSIZE:                1
   ZEEBE_BROKER_CLUSTER_REPLICATIONFACTOR:          1
   ZEEBE_BROKER_THREADS_CPUTHREADCOUNT:             2
   ZEEBE_BROKER_THREADS_IOTHREADCOUNT:              2
   ZEEBE_BROKER_GATEWAY_ENABLE:                     false
   ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_CLASSNAME:  io.zeebe.exporter.ElasticsearchExporter
   ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_URL:   http://zeebe-elasticsearch:9200
   ZEEBE_BROKER_NETWORK_COMMANDAPI_PORT:            26501
   ZEEBE_BROKER_NETWORK_INTERNALAPI_PORT:           26502
   ZEEBE_BROKER_NETWORK_MONITORINGAPI_PORT:         9600
   K8S_POD_NAME:                                    zeebe-zeebe-0 (v1:metadata.name)
   JAVA_TOOL_OPTIONS:                               -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:MaxRAMPercentage=25.0 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+PrintFlagsFinal -Xmx4g -Xms4g -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/zeebe/da
-XX:ErrorFile=/usr/local/zeebe/data/zeebe_error%p.log
   ZEEBE_BROKER_GATEWAY_ENABLE:                     true

and embedded gateway is not started because of this

[salaboy]

@ezolnbl hmm that is interesting.. and totally possible with the current setup.. that is because, half of the ENV variables are defined in the templates and the other half can be injected using the env key.
Instead of overriding with env you should try to set it with the values themselves. But in order for that to happen, we need to make this line a variable:

https://github.com/zeebe-io/zeebe-cluster-helm/blob/master/zeebe-cluster/templates/statefulset.yaml#L54

Would you mind sending a PR for that?

[ezolnbl-dpc]

to which repository? zeebe or zeebe-helm?

[salaboy]

@ezolnbl this repository.. you need to change the file that I've linked in my previous message.

[salaboy]

@npepinpe I will close this issue as this is fixed..
@ezolnbl would you mind creating another issue for adding a variable to the gateway enable in the broker?

[ssxv]

Still not finding the brokers.. there is something that I'm missing.. in the previous deployment descriptor there is a configMap, which seems to only contain the logging config files.. but is there something else require for the gateway to locate the brokers?
Currently I am running

salaboy> zbctl status --insecure                                                               
Cluster size: 0
Partitions count: 0
Replication factor: 0
Brokers:

Where no brokers are found, even when the brokers are up and running right besides the gateway.

version: "2"

networks:
  zeebe_network:
    driver: bridge

services:
  gateway:
    restart: always
    container_name: gateway
    image: camunda/zeebe:0.26.0
    environment:
      - ZEEBE_LOG_LEVEL=debug
      - ZEEBE_STANDALONE_GATEWAY=true
      - ZEEBE_GATEWAY_NETWORK_HOST=0.0.0.0
      - ZEEBE_GATEWAY_NETWORK_PORT=26500
      - ZEEBE_GATEWAY_CLUSTER_CONTACTPOINT=node0:26502
      - ZEEBE_GATEWAY_CLUSTER_PORT=26502
      - ZEEBE_GATEWAY_CLUSTER_HOST=gateway
    ports:
      - "26500:26500"
    networks:
      - zeebe_network
  
  node0:
    container_name: zeebe_broker_1
    image: camunda/zeebe:0.26.0
    environment:
      - ZEEBE_LOG_LEVEL=debug
      - ZEEBE_BROKER_CLUSTER_NODEID=0
      - ZEEBE_BROKER_CLUSTER_PARTITIONSCOUNT=2
      - ZEEBE_BROKER_CLUSTER_REPLICATIONFACTOR=3
      - ZEEBE_BROKER_CLUSTER_CLUSTERSIZE=3
      - ZEEBE_BROKER_CLUSTER_INITIALCONTACTPOINTS=node1:26502,node2:26502
    networks:
      - zeebe_network
  
  node1:
    container_name: zeebe_broker_2
    image: camunda/zeebe:0.26.0
    environment:
      - ZEEBE_LOG_LEVEL=debug
      - ZEEBE_BROKER_CLUSTER_NODEID=1
      - ZEEBE_BROKER_CLUSTER_PARTITIONSCOUNT=2
      - ZEEBE_BROKER_CLUSTER_REPLICATIONFACTOR=3
      - ZEEBE_BROKER_CLUSTER_CLUSTERSIZE=3
      - ZEEBE_BROKER_CLUSTER_INITIALCONTACTPOINTS=node0:26502,node2:26502
    networks:
      - zeebe_network
    depends_on:
      - node0
  
  node2:
    container_name: zeebe_broker_3
    image: camunda/zeebe:0.26.0
    environment:
      - ZEEBE_LOG_LEVEL=debug
      - ZEEBE_BROKER_CLUSTER_NODEID=2
      - ZEEBE_BROKER_CLUSTER_PARTITIONSCOUNT=2
      - ZEEBE_BROKER_CLUSTER_REPLICATIONFACTOR=3
      - ZEEBE_BROKER_CLUSTER_CLUSTERSIZE=3
      - ZEEBE_BROKER_CLUSTER_INITIALCONTACTPOINTS=node0:26502,node1:26502
    networks:
      - zeebe_network
    depends_on:
      - node1

this worked for me

ZEEBE_GATEWAY_CLUSTER_CONTACTPOINT=node0:26502