webp-server uses to much memory

[bhzhu203]

At the first time , it uses 1GB memory . But after 7 hours , it grows to 6.7GB and continue to grow slowly.

[n0vad3v]

🤔 Thanks for reporting, we need to dig more on to this issue, are you using version 0.7.0 on this?

Before that you can try running it with docker-compose with limited resource allocated to it for mitigating this problem, more info can be seen on https://docs.webp.sh/usage/docker/ and #75.

[bhzhu203]

So , how to continue improve the memory usage ?

[n0vad3v]

Could you please have a try adding MALLOC_ARENA_MAX=1 env before running the program?
Ref: https://github.com/davidbyttow/govips#memory-usage-note

In my testing case the RAM it used after benching using method from #200, has reduced to less than 100MiB after testing:

eb5eff3296d1   webp-server-webp-1   0.04%     95.56MiB / 45.89GiB   0.20%     54.6kB / 11.9MB   98.3kB / 28.6MB   20

My docker-compose.yml example:

version: '3'

services:
  webp:
    image: webpsh/webp-server-go
    environment:
      - MALLOC_ARENA_MAX=1
    volumes:
      - ./pics:/opt/pics
      - ./exhaust:/opt/exhaust
    ports:
      -  127.0.0.1:3333:3333
    command: '/usr/bin/webp-server --config /etc/config.json -v'

[Ingiboy]

Could you please have a try adding MALLOC_ARENA_MAX=1 env before running the program?

Yes, but it eat 1-2-3Gb.
I use docker exec webp-server-go sh -c 'kill -HUP 1' every 15 min

[n0vad3v]

@Ingiboy OK, that seems better.

BTW, you can use docker-compose.yml described in https://docs.webp.sh/usage/docker/ with resource limit so it will automatically restart if RAM has reached your limitation, instead of restarting it manually, example as below.

version: '3'

services:
  webp:
    image: webpsh/webp-server-go
    # image: ghcr.io/webp-sh/webp_server_go
    restart: always
    volumes:
      - ./path/to/pics:/opt/pics
      - ./path/to/exhaust:/opt/exhaust
    ports:
      -  127.0.0.1:3333:3333
    deploy:
      resources:
        limits:
          memory: 200M

Use with docker-compose --compatibility up -d to start.

[Ingiboy]

When i set deploy section with mem limit=512M, LA grows up to 30-40, CPU in wait state 60-80%. And using swap up to 2.5G.

Without 'deploy' LA less 1, no swap.

[BennyThink]

that is because docker engine will try to kill and then restart the container quiet often, so high load average.

@n0vad3v can we consider adding back MaxCacheMem and MaxCacheSize? We can read environment variables so it's totally up to end users.

[Ingiboy]

that is because docker engine will try to kill and then restart the container quiet often, so high load average.

No, i see 'docker ps' - container uptime is more than 6h. I think memlimit is only limits RAM, and container eats swap)

[n0vad3v]

@Ingiboy
I've dig some more on https://docs.docker.com/config/containers/resource_constraints/#--memory-swap-details and found that:

If --memory-swap is unset, and --memory is set, the container can use as much swap as the --memory setting, if the host container has swap memory configured. For instance, if --memory="300m" and --memory-swap is not set, the container can use 600m in total of memory and swap.

So using the docker-compose.yml as above:

version: '3'

services:
  webp:
    image: webpsh/webp-server-go
    # image: ghcr.io/webp-sh/webp_server_go
    restart: always
    volumes:
      - ./path/to/pics:/opt/pics
      - ./path/to/exhaust:/opt/exhaust
    ports:
      -  127.0.0.1:3333:3333
    deploy:
      resources:
        limits:
          memory: 200M

The container can use 200M of RAM and 200M of swap, can be confirmed by docker inspect

"Memory": 209715200,
...
"MemorySwap": 419430400,

In order to limit the swap usage, here is an working example(Note I've added MALLOC_ARENA_MAX in env):

version: '3'

services:
  webp:
    image: webpsh/webp-server-go
    restart: always
    environment:
      - MALLOC_ARENA_MAX=1
    volumes:
      - ./path/to/pics:/opt/pics
      - ./path/to/exhaust:/opt/exhaust
    ports:
      -  127.0.0.1:3333:3333

    deploy:
      resources:
        limits:
          memory: 200M
    memswap_limit: 200M

In this case, the container can use no swap and only 200M of RAM, and will restart the container if ram runs above 200M.

Example inspired by https://stackoverflow.com/questions/70095537/how-to-use-swap-memory-in-docker-compose, have to say the structure is wierd somehow, not sure why it is designed like that.

Please have a try on this, BTW, a higher ram limit than 512MB is suggested.

[Ingiboy]

@n0vad3v, o, thanks! i try it

[Ingiboy]

Hmm, don't helps.

version: "3.3"
services:
  webp-http:
    container_name: webp-http
    image: webpsh/webp-server-go:latest
    restart: unless-stopped
    user: 1001:1001
    cap_drop:
      - ALL
    environment:
      - MALLOC_ARENA_MAX=1
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./webp/config.json:/etc/config.json:ro
      - /var/tmp/webp-http/exhaust:/opt/exhaust
      - /var/tmp/webp-http/remote-raw:/opt/remote-raw
    ports:
     - 127.0.0.1:3335:3333
    deploy:
      resources:
        limits:
          memory: 196M
    memswap_limit: 196M

after some minuts LA grows up to 30-40, CPU in wa state about 80%. webp utilizes CPU more than 100%
Temporarily reverted to no-limit scheme and kill -HUP 1 every 15 minutes.

[n0vad3v]

@Ingiboy

Hmm, don't helps.

Seems weird, in the compose file content provided above, in my understanding with memswap_limit set it should no write to swap, but you've described "CPU in wa state about 80%", are your images large or your machine's disk is quite slow, or if using VPS, might that be caused by CPU overselling?

May I have some more info on your case:

• What type of disk are you using? (HDD, SSD, NVMe SSD?) The disk type can have an impact on the overall performance, especially when dealing with I/O operations.
• When "LA grows up to 30-40, CPU in wa state about 80%. webp utilizes CPU more than 100%", it's important to understand the underlying reasons. Could you provide more details about the number of requests being sent to the service and the sizes of the images involved? This information will help in identifying potential bottlenecks.
• From your previous reply "Yes, but it eat 1-2-3Gb.", I'm assuming your machine has more than 3GB's RAM, if you raise memory and memswap_limit to, say 1024MB, what will happen after running?

[Ingiboy]

@n0vad3v

1. I use VPS, no info about disk, i think it's not a hdd;
2. For example, 12000+ requests per hour:
   main.findSmallestFiles() - 4382,
   main.proxyHandler() - 5489,
   main.convertLog() - 2344,
   Average size of image - 600kb;
3. Yes, it have 8Gb RAM. I try to set 1Gb limit, but i saw the peak - 1,6Gb

[Ingiboy]

All not good) I tried to put memory=1G and memswap_limit=1G || memswap_limit=0. If the memory usage is about gigabyte - LA about 40, CPU wa about 80% etc(