helm stable/redis does not work with istio sidecar

[kminehart]

Here's the yaml i generated from helm install stable/redis.

# Source: redis/templates/redis-master-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: rune-redis-master
  labels:
    app: redis
    chart: redis-3.2.0
    release: "rune-redis"
    heritage: "Tiller"
  annotations:
spec:
  type: ClusterIP
  ports:
  - name: redis
    port: 6379
    targetPort: redis
  selector:
    app: redis
    release: "rune-redis"
    role: master
---
# Source: redis/templates/redis-slave-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: rune-redis-slave
  labels:
    app: redis
    chart: redis-3.2.0
    release: "rune-redis"
    heritage: "Tiller"
  annotations:
spec:
  type: ClusterIP
  ports:
  - name: redis
    port: 6379
    targetPort: redis
  selector:
    app: redis
    release: "rune-redis"
    role: slave
---
# Source: redis/templates/redis-slave-deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: rune-redis-slave
  labels:
    app: redis
    chart: redis-3.2.0
    release: "rune-redis"
    heritage: "Tiller"
spec:
  replicas: 3
  template:
    metadata:
      labels:
        release: "rune-redis"
        role: slave
        app: redis
      annotations:
        sidecar.istio.io/inject: "false"
        
    spec:      
      securityContext:
        fsGroup: 1001
        runAsUser: 1001
      containers:
      - name: rune-redis
        image: docker.io/bitnami/redis:4.0.9
        imagePullPolicy: "Always"
        env:
        - name: REDIS_REPLICATION_MODE
          value: slave
        - name: REDIS_MASTER_HOST
          value: rune-redis-master
        - name: REDIS_PORT
          value: "6379"
        - name: REDIS_MASTER_PORT_NUMBER
          value: "6379"
        - name: ALLOW_EMPTY_PASSWORD
          value: "yes"
        - name: REDIS_DISABLE_COMMANDS
          value: 
        ports:
        - name: redis
          containerPort: 6379        
        livenessProbe:
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
          exec:
            command:
            - redis-cli
            - ping        
        readinessProbe:
          initialDelaySeconds: 5
          periodSeconds: 10
          timeoutSeconds: 1
          successThreshold: 1
          failureThreshold: 5
          exec:
            command:
            - redis-cli
            - ping
        resources:
          null
---
# Source: redis/templates/redis-master-statefulset.yaml
apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
  name: rune-redis-master
  labels:
    app: redis
    chart: redis-3.2.0
    release: "rune-redis"
    heritage: "Tiller"
spec:
  selector:
    matchLabels:
      release: "rune-redis"
      role: master
      app: redis
  serviceName: "redis-master"  
  template:
    metadata:
      labels:
        release: "rune-redis"
        role: master
        app: redis
      annotations:
        sidecar.istio.io/inject: "false"
        
    spec:
      securityContext:
        fsGroup: 1001
        runAsUser: 1001
      containers:
      - name: rune-redis
        image: "docker.io/bitnami/redis:4.0.9"
        imagePullPolicy: "Always"
        env:
        - name: REDIS_REPLICATION_MODE
          value: master
        - name: ALLOW_EMPTY_PASSWORD
          value: "yes"
        - name: REDIS_PORT
          value: "6379"
        - name: REDIS_DISABLE_COMMANDS
          value: 
        ports:
        - name: redis
          containerPort: 6379
        livenessProbe:
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
          exec:
            command:
            - redis-cli
            - ping
        readinessProbe:
          initialDelaySeconds: 5
          periodSeconds: 10
          timeoutSeconds: 1
          successThreshold: 1
          failureThreshold: 5
          exec:
            command:
            - redis-cli
            - ping
        resources:
          null
          
        volumeMounts:
        - name: redis-data
          mountPath: /bitnami/redis/data
          subPath: 
  volumeClaimTemplates:
    - metadata:
        name: redis-data
        labels:
          app: "redis"
          chart: redis-3.2.0
          component: "master"
          release: "rune-redis"
          heritage: "Tiller"
      spec:
        accessModes:
          - "ReadWriteOnce"
        resources:
          requests:
            storage: "8Gi"

Most of it is pretty irrelevant but what you'd expect to be there is there:

1. The pods do not have the istio sidecar on them.
2. The service ports are named:

  ports:
  - name: redis
    port: 6379
    targetPort: redis

Istio version: 7.0.1

I thought this was fixed in 5.0.0?

Istio auto-injector version: 7.0.1

[kminehart]

Some more info:

kubectl run --namespace default redis-client --rm --tty -i --image docker.io/bitnami/redis:4.0.9 -- bash

I have no name!@redis-client-9f45fc688-dxs5c:/$ redis-cli -h rune-redis-master
Could not connect to Redis at rune-redis-master:6379: Connection refused

It doesn't matter if I don't name the service's port or if it's named redis.

However, there is an older version of this chart on the cluster and it does did work from a sidecarred service. It no longer does though after today... How can I find out what the difference is to istio, exactly?

[kminehart]

Whoops, meant to put this in istio/issues

[proddam]

I have the same issue. Did you find a solution? I think istio sidecars don't fit k8s statefulsets.

[kminehart]

They do though.

It really looked like Istio wasn't able to handle the "redis" protocol traffic. I had it working for a while before I started running into this issue; make sure the port is named "redis".

I ditched istio whenever i couldn't figure it out, unfortunately.

[swistaczek]

@proddam did you managed to run helm/stable redis chart on istio cluster?

[emmekappa]

I'm having the same issue.

[kminehart]

I think the issue might be that StatefulSets usually have 2 services mapped to the same pod, which Istio does not support. (Or did not support at the time). Did an update fix this or anything?

[iroller]

If the issue that you see is istio sidecar not starting then check istio sidecar logs and look for envoy missing listener for inbound application port. If that's the case then creating a dummy service for your redis pods will help to get it working.

See https://github.com/istio/istio/blob/master/pilot/cmd/pilot-agent/status/ready/probe.go#L41-L65 and spotahome/redis-operator#93 (comment).

[amyroh]

Commenting out securityContext from deployment.yaml works for me.

    spec:
#      securityContext:
#        runAsUser: 1001
#        fsGroup: 1001

[jaygorrell]

@iroller I'm getting the message you mentioned (envoy missing listener for inbound application port) but I do have a service. The only thing of note here is that the container port and the service are both UDP... is it required to have a TCP-based port on the service?

[iroller]

@jaygorrell you probably have rfs-app-redis service for sentinel, right? Try creating rfr-...-redis service for redis pods. I provided an example below.

apiVersion: v1
kind: Service
metadata:
  name: rfr-app-redis-dummy
spec:
  ports:
    - name: redis-dummy
      port: 16379
      protocol: TCP
      targetPort: 6379
  selector:
    app: redis-failover
    component: redis
    creator: redisfailover
    redisfailover: app-redis

At least that's how services are named in redis-operator. You might want to tweak the selector for redis helm chart but the idea is create a service for redis pods, not only for sentinel.

[jaygorrell]

Sorry @iroller, I didn't give enough info. This isn't actually a Redis service but your comment was the closest I've seen to the problem (missing listener for inbound application port).

It's a service with only a UDP port exposed. I have a service for that port with the protocol specified, but Istio fails to become healthy with that error message.

[grantbachman]

I ran into a similar issue, where my redis pod would CrashLoop after enabling the istio sidecar proxy. The root cause was actually in the Istio InitContainer, which needed to be run as root. Setting the following in the redis helm chart fixed it for me:

master:
  securityContext:
    runAsUser: 0

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 90 days. It will be closed in the next 30 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had activity in the last month and a half. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.

[philrud]

Confirming what previously has been mentioned: this not a sidecar/traffic management issue. The problem is that the securityContext.runAsUser = 1001 directive is on a Pod level hence affecting all containers in the Pod including istio-init. istio-init has to be run as root in order to configure iptables.

Commenting out runAsUser field (or moving it to the container level) resolves the issue and Redis Cluster works fine with sidecars enabled. Replicas are able to establish connections to master just fine and I'm able to execute commands via redis-cli on both master and replicas.

I suspect many charts may not work because of similar issue, one thing we could do is explicitly adding securityContext.runAsUser = 0 to the istio-init container (by default, with an option to disable that via a Helm option) so that this setting takes precedence over the pod-level one. istio-init currently requires running as root anyways.

[XBeg9]

@philrud that's totally right, we should disable runAsUser if want to inject istio's sidecars.

I am curious, do you use mTLS? Have you got a chance to run redis with mTLS? Thanks

[philrud]

@XBeg9 yes, I was running that setup with mTLS=on

[XBeg9]

@philrud namespace-policy or global one?

[howardjohn]

@philrud is this fixed now?

[philrud]

yes, the fix will go live with 1.1.3, closing