Skipper container runs out of memory

[MarounMaroun]

Description:

I'm trying to build the following streams:

STREAM_2=:messages > filter --expression="#jsonPath(payload, '$.id')==1" | rabbit --queues=id_1 --host=rabbitmq --routing-key=id_1 --exchange=ex_1 --own-connection=true
STREAM_3=:messages > filter --expression="#jsonPath(payload, '$.id')==2" | rabbit --queues=id_2 --host=rabbitmq --routing-key=id_2 --exchange=ex_1
STREAM_4=:messages > filter --expression="#jsonPath(payload, '$.id')==3" | rabbit --queues=id_3 --host=rabbitmq --routing-key=id_3 --exchange=ex_1
STREAM_1=rabbit --queues=hello_queue --host=rabbitmq > :messages

When I deploy, it takes a couple of minutes until the skipper container reaches the max memory usage, and finally fails on random streams (and sometimes kills the container).

docker stats:

CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
9a0e3ff0beb8        dataflow-server     0.18%               1.293GiB / 11.71GiB   11.04%              573kB / 183kB       92.1MB / 279kB      49
2a448b3583a3        scdf_kafka_1        7.00%               291.6MiB / 11.71GiB   2.43%               4.65MB / 3.64MB     40.4MB / 36.9kB     73
eb9a70ce2a0e        scdf_rabbitmq_1     2.15%               94.21MiB / 11.71GiB   0.79%               172kB / 92.5kB      41.7MB / 139kB      128
06dd2d6a1501        scdf_zookeeper_1    0.16%               81.72MiB / 11.71GiB   0.68%               77.8kB / 99.2kB     36.7MB / 45.1kB     25
1f1b782ad66d        skipper             8.64%               6.55GiB / 11.71GiB    55.93%              3.63MB / 4.73MB     213MB / 0B          324

The workaround, for now, is increasing the Docker memory. I'm wondering if it's normal for the Skipper container to consume this amount of memory.

Release versions:

DATAFLOW_VERSION=2.0.1.RELEASE
SKIPPER_VERSION=2.0.0.RELEASE

Steps to reproduce:
Run the stream provided in the description.

Link for SO question: https://stackoverflow.com/questions/55371129/streams-deploying-stuck-and-then-fails-with-no-errors-in-the-logs

[markpollack]

That is definitely not normal. It shot up to 6.5GB just from deploying this stream once, or was it deployed/undeployed over and over. Trying to understand how to best reproduce to investigate.

[MarounMaroun]

@markpollack Only once. Here's my YAML in case it helps:

version: '3'

services:
  rabbitmq:
    image: "rabbitmq:3-management"
    ports:
      - "5672:5672"
      - "15672:15672"
    expose:
      - "5672"
  dataflow-server:
    image: springcloud/spring-cloud-dataflow-server:${DATAFLOW_VERSION:?DATAFLOW_VERSION is not set!}
    container_name: dataflow-server
    ports:
      - "9393:9393"
    environment:
      - spring.cloud.dataflow.applicationProperties.stream.spring.rabbitmq.host=rabbitmq
        #- spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.kafka.binder.brokers=kafka:9092
        #- spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.kafka.binder.zkNodes=zookeeper:2181
      - spring.cloud.skipper.client.serverUri=http://skipper-server:7577/api
        # - spring.cloud.dataflow.applicationProperties.stream.management.metrics.export.influx.enabled=true
        # - spring.cloud.dataflow.applicationProperties.stream.management.metrics.export.influx.db=myinfluxdb
        # - spring.cloud.dataflow.applicationProperties.stream.management.metrics.export.influx.uri=http://influxdb:8086
        # - spring.cloud.dataflow.grafana-info.url=http://localhost:3000
    depends_on:
      - rabbitmq
  app-import:
    image: springcloud/openjdk:latest
    depends_on:
      - dataflow-server
    command: >
      /bin/sh -c "
        while ! nc -z dataflow-server 9393;
        do
          sleep 1;
        done;
        wget -qO- 'http://dataflow-server:9393/apps' --post-data='uri=http://bit.ly/Einstein-SR2-stream-applications-rabbit-maven';
        echo 'Stream apps imported'
        wget -qO- 'http://dataflow-server:9393/apps' --post-data='uri=http://bit.ly/Dearborn-SR1-task-applications-maven&force=true';
        echo 'Task apps imported'"
  skipper-server:
    image: springcloud/spring-cloud-skipper-server:${SKIPPER_VERSION:?SKIPPER_VERSION is not set!}
    container_name: skipper
    ports:
    - "7577:7577"
    - "9000-9010:9000-9010"

[markpollack]

sorry, my bad, I thought it was skipper itself (the jvm process) that was running out of memory, but as @chrisjs mentioned to me out of band - all the apps deployed by skipper are in that container...so eventually you are going to blow up that container's memory because 7 JVM apps are spawned in there.....

[MarounMaroun]

@markpollack Can you please elaborate more? What apps are spawned in the Skipper container? I'm new to SCDF and I'm not sure I'm following the best practices. Is it a problem with my streams' definition, with the setup, or with anything else?

Thanks for your fast response!

[chrisjs]

@MarounMaroun stream components (parts of the pipe) are deployed as applications. those applications are deployed into the skipper container (as well as the skipper application itself) since skipper deploys streams. the more applications that get deployed (parts of the pipe, streams, etc) the more memory is used.

so for example, when deploying your streams, when inspecting the processes running on the skipper container, you can see individual java processes that are spawned, ie:

root         1 32.2 11.6 6233292 1193148 ?     Ssl  16:44   1:42 java -Djava.security.egd=file:/dev/./urandom -jar /spring-cloud-skipper-server.jar
root        80 93.2  5.1 5169184 525160 ?      Sl   16:46   0:36 /opt/openjdk/bin/java -jar /root/.m2/repository/org/springframework/cloud/stream/app/rabbit-sink-rabbit/2.1.0.RELEASE/rabbit-sink-rabbit-2.1.0.RELEASE.jar
root       103 79.2  4.2 5160880 429620 ?      Sl   16:46   0:21 /opt/openjdk/bin/java -jar /root/.m2/repository/org/springframework/cloud/stream/app/filter-processor-rabbit/2.1.0.RELEASE/filter-processor-rabbit-2.1.0.RELEASE.jar
root       128 56.1  2.6 5154588 266216 ?      Sl   16:47   0:10 /opt/openjdk/bin/java -jar /root/.m2/repository/org/springframework/cloud/stream/app/rabbit-sink-rabbit/2.1.0.RELEASE/rabbit-sink-rabbit-2.1.0.RELEASE.jar
root       135 59.7  2.5 5156636 258956 ?      Sl   16:47   0:10 /opt/openjdk/bin/java -jar /root/.m2/repository/org/springframework/cloud/stream/app/filter-processor-rabbit/2.1.0.RELEASE/filter-processor-rabbit-2.1.0.RELEASE.jar
root       162 48.3  1.7 5146280 176060 ?      Sl   16:47   0:06 /opt/openjdk/bin/java -jar /root/.m2/repository/org/springframework/cloud/stream/app/rabbit-sink-rabbit/2.1.0.RELEASE/rabbit-sink-rabbit-2.1.0.RELEASE.jar
root       169 50.0  1.4 5026368 149188 ?      Sl   16:47   0:06 /opt/openjdk/bin/java -jar /root/.m2/repository/org/springframework/cloud/stream/app/filter-processor-rabbit/2.1.0.RELEASE/filter-processor-rabbit-2.1.0.RELEASE.jar

and so on. so as you pointed out in your SO post - increasing the available memory is what you would need to do here.

[MarounMaroun]

@chrisjs Thank you for the response!

[MarounMaroun]

@chrisjs Is there a way to reduce the number of streams created? For now, if I have a simple log sink, it'll create a process for it. We have 3 logs, 3 HTTP posts, and 3 RMQ listeners. It spawns many processes, reaching almost 12GiB.