roundcubemail container is constantly infected with kdevtmpfsi and kinsing

[manuviens]

In htop on the host where is running my containers, I noticed that a process (kdevtmpfsi) was launched twice from /tmp. Each of them was using 50% of my CPU. While searching on the web, I learned that kdevtmpfsi was often related to another file, kinsing, and that it is probably a cryptocurrency miner.
Not finding these files on the host and noticing that the processes disappeared from htop if I turned off the containers, I searched inside the containers and found /tmp/kinsing in the roundcubemail container.
Deleting the container and restarting it solves the problem for a while only, within 24 hours both kdevtmpfsi processes are hogging my CPU again.

Have others experienced this problem, or is it my configuration that is to blame?

Here is a copy of my docker-compose.yml :

services:
  mailserver:
    image: ghcr.io/docker-mailserver/docker-mailserver:latest
    container_name: mailserver
    hostname: {{ mailserver_hostname }}
    env_file: mailserver.env
    ports:
      - "25:25"    # SMTP  (explicit TLS => STARTTLS)
      - "143:143"  # IMAP4 (explicit TLS => STARTTLS)
      - "465:465"  # ESMTP (implicit TLS)
      - "587:587"  # ESMTP (explicit TLS => STARTTLS)
      - "993:993"  # IMAP4 (implicit TLS)
      - "110:110"  # POP3
      - "995:995"  # POP3 (with TLS)
    volumes:
      - ./data/dms/mail-data/:/var/mail/
      - ./data/dms/mail-state/:/var/mail-state/
      - ./data/dms/mail-logs/:/var/log/mail/
      - ./data/dms/config/:/tmp/docker-mailserver/
      - /etc/localtime:/etc/localtime:ro
      - ./data/certbot/certs/:/etc/letsencrypt/:ro
    restart: always
    stop_grace_period: 1m
    cap_add:
      - NET_ADMIN
    healthcheck:
      test: "ss --listening --tcp | grep -P 'LISTEN.+:smtp' || exit 1"
      timeout: 3s
      retries: 0

  roundcubemail:
    image: roundcube/roundcubemail:latest-fpm-alpine
    container_name: roundcubemail
    restart: unless-stopped
    healthcheck:
      test: ["CMD-SHELL", "php -r '$$c = @fsockopen(\"localhost\", 9000); if (is_resource($$c)) { fwrite(STDOUT, \"OK\"); fclose($$c); exit(0); } else { fwrite(STDERR, \"FAIL\"); exit(1); }'"]
      interval: 5s
      timeout: 3s
      retries: 30
      start_period: 10s
    depends_on:
      roundcubedb:
        condition: service_healthy
    links:
      - roundcubedb
    ports:
      - 9000:9000
    volumes:
      - ./data/rcm/www:/var/www/html
      - /etc/localtime:/etc/localtime:ro
    environment:
      - ROUNDCUBEMAIL_DB_TYPE=pgsql
      - ROUNDCUBEMAIL_DB_HOST=roundcubedb
      - ROUNDCUBEMAIL_DB_NAME=roundcube
      - ROUNDCUBEMAIL_DB_USER=roundcube
      - ROUNDCUBEMAIL_DB_PASSWORD={{ roundcubemail_db_password }}
      - ROUNDCUBEMAIL_SKIN={{ roundcubemail_skin }}
      - ROUNDCUBEMAIL_DEFAULT_HOST={{ roundcubemail_default_host }}
      - ROUNDCUBEMAIL_SMTP_SERVER={{ roundcubemail_smtp_server }}

  roundcubedb:
    image: postgres:alpine
    container_name: roundcubedb
    restart: unless-stopped
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U roundcube"]
      interval: 5s
      timeout: 3s
      retries: 30
      start_period: 10s
    depends_on:
      mailserver:
        condition: service_healthy
    ports:
      - 5432:5432
    volumes:
      - ./data/rcm/pgsql:/var/lib/postgresql/data
      - /etc/localtime:/etc/localtime:ro
    environment:
      - POSTGRES_DB=roundcube
      - POSTGRES_USER=roundcube
      - POSTGRES_PASSWORD={{ roundcubemail_db_password }}

  roundcubenginx:
    image: nginx:alpine
    container_name: roundcubenginx
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost/?ping=ping"]
      interval: 5s
      timeout: 3s
      retries: 30
      start_period: 10s
    ports:
      - '80:80'
      - '443:443'
    depends_on:
      roundcubemail:
        condition: service_healthy
    links:
      - roundcubemail
    volumes:
      - ./data/rcm/www:/var/www/html
      - ./rcm/fpm-alpine/templates:/etc/nginx/templates
      - ./nginx-default.conf:/etc/nginx/conf.d/default.conf
      - ./data/certbot/certs/:/etc/letsencrypt/
      - /etc/localtime:/etc/localtime:ro
    environment:
      - NGINX_HOST={{ roundcubemail_nginx_host }}
      - NGINX_PHP_CGI=roundcubemail:9000

  watchtower:
    image: containrrr/watchtower
    container_name: watchtower
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /etc/localtime:/etc/localtime:ro
    command: --schedule "10 10 4 * * *" --cleanup

Maybe the breach is in my nginx configuration?

server {
    listen 80 default_server;
    server_tokens off;
    autoindex off;
    location / { return 301 https://{{ website_hostname }}; }
}

server {
    listen 443 ssl http2 default_server;
    server_tokens off;
    autoindex off;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!MEDIUM:!LOW:!aNULL:!NULL:!SHA;
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:10m;
    ssl_session_tickets off;
    ssl_certificate     /etc/letsencrypt/live/{{ mailserver_fqdn }}/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/{{ mailserver_fqdn }}/privkey.pem;
    location / { return 301 https://{{ website_hostname }}; }
}

server {
    listen 80;
    server_name {{ roundcubemail_nginx_host }};
    location / { return 301 https://$host$request_uri; }
}

server {
    listen 443 ssl http2;
    server_name {{ roundcubemail_nginx_host }};
  
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!MEDIUM:!LOW:!aNULL:!NULL:!SHA;
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:10m;
    ssl_session_tickets off;
    ssl_certificate     /etc/letsencrypt/live/{{ roundcubemail_nginx_host }}/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/{{ roundcubemail_nginx_host }}/privkey.pem;

    root /var/www/html;

    location / {
        index index.php;
    }

    location ~ \.php$ {
        fastcgi_pass roundcubemail:9000;
        # regex to split $uri to $fastcgi_script_name and $fastcgi_path
        fastcgi_split_path_info ^(.+\.php)(/.+)$;

        # Check that the PHP script exists before passing it
        try_files $fastcgi_script_name =404;

        # Bypass the fact that try_files resets $fastcgi_path_info
        # see: https://trac.nginx.org/nginx/ticket/321
        set $path_info $fastcgi_path_info;
        fastcgi_param PATH_INFO $path_info;

        fastcgi_index index.php;
        include fastcgi.conf;
    }
}

Other than that, everything works fine.

P.S. I'm using this VPS for 1 week only, what you see in docker-compose.yml is the only things running on this server.
I'm using Ansible to manage this server, I can provide you how I install docker, etc. if it can help.

[manuviens]

Ok, I just deleted my roundcubemail container, deleted the contents of data/rcm/www and data/rcm/pgsql then restarted and looked at the logs.

Here is what I just saw:

roundcubemail   | chattr: can't stat '/etc/ld.so.preload': No such file or directory
roundcubemail   | chattr: setting flags on /var/spool/cron: Permission denied
roundcubemail   | chattr: can't open '/var/spool/cron/crontabs': Symbolic link loop
roundcubemail   | chattr: can't stat '/etc/crontab': No such file or directory
roundcubemail   | bash: line 9: ufw: command not found
roundcubemail   | bash: line 10: iptables: command not found
roundcubemail   | bash: line 11: /proc/sys/kernel/nmi_watchdog: Read-only file system
roundcubemail   | bash: line 12: /etc/sysctl.conf: Permission denied
roundcubemail   | netstat: showing only processes with your user ID
roundcubemail   | netstat: showing only processes with your user ID
roundcubemail   | netstat: showing only processes with your user ID
roundcubemail   | netstat: showing only processes with your user ID
roundcubemail   | netstat: showing only processes with your user ID
roundcubemail   | cat: /tmp/.X11-unix/01: No such file or directory
roundcubemail   | cat: /tmp/.X11-unix/11: No such file or directory
roundcubemail   | cat: /tmp/.X11-unix/22: No such file or directory
roundcubemail   | cat: /tmp/.pg_stat.0: No such file or directory
roundcubemail   | cat: /tmp/.pg_stat.1: No such file or directory
roundcubemail   | cat: /home/www-data/data/./oka.pid: No such file or directory
roundcubemail   | ps: unrecognized option: w
roundcubemail   | BusyBox v1.35.0 (2022-11-19 10:13:10 UTC) multi-call binary.
roundcubemail   | Usage: ps [-o COL1,COL2=HEADER] [-T]
roundcubemail   | Show list of processes
roundcubemail   | 	-o COL1,COL2=HEADER	Select columns for display
roundcubemail   | 	-T			Show threads
roundcubemail   | ps: unrecognized option: w
roundcubemail   | BusyBox v1.35.0 (2022-11-19 10:13:10 UTC) multi-call binary.
roundcubemail   | Usage: ps [-o COL1,COL2=HEADER] [-T]
roundcubemail   | Show list of processes
roundcubemail   | 	-o COL1,COL2=HEADER	Select columns for display
roundcubemail   | 	-T			Show threads
roundcubemail   | md5sum: /tmp/kinsing: No such file or directory
roundcubemail   | /tmp/kinsing is not 2c44b4e4706b8bd95d1866d7867efa0e, actual 
roundcubemail   | chmod: cannot access '/tmp/kinsing': No such file or directory
roundcubemail   |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
roundcubemail   |                                  Dload  Upload   Total   Spent    Left  Speed
roundcubenginx  | 127.0.0.1 - - [04/May/2023:14:46:23 -0400] "GET /?ping=ping HTTP/1.1" 301 162 "-" "curl/7.88.1" "-"
100 6168k  100 6168k    0     0  1565k      0  0:00:03  0:00:03 --:--:-- 1565k
roundcubemail   | /tmp/kinsing is 2c44b4e4706b8bd95d1866d7867efa0e
roundcubemail   | md5sum: /tmp/libsystem.so: No such file or directory
roundcubemail   | /tmp/libsystem.so is not ccef46c7edf9131ccffc47bd69eb743b, actual 
roundcubemail   | chmod: cannot access '/tmp/libsystem.so': No such file or directory
roundcubemail   |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
roundcubemail   |                                  Dload  Upload   Total   Spent    Left  Speed
100 26800  100 26800    0     0  68787      0 --:--:-- --:--:-- --:--:-- 68894
roundcubemail   | /tmp/libsystem.so is ccef46c7edf9131ccffc47bd69eb743b
roundcubemail   |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
roundcubemail   |                                  Dload  Upload   Total   Spent    Left  Speed
100 26800  100 26800    0     0  75147      0 --:--:-- --:--:-- --:--:-- 75280
roundcubemail   | /tmp/libsystem.so is ccef46c7edf9131ccffc47bd69eb743b
roundcubemail   | main: line 276: /etc/ld.so.preload: Permission denied
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | crontab: must be suid to work properly
roundcubemail   | bash: line 390: echo: write error: Broken pipe

Then... :

# docker exec -it roundcubemail /bin/sh
/var/www/html # find / -name kdevtmpfsi
/var/www/html # find / -name kinsing
/tmp/kinsing

But kdevtmpfsi it is not visible for now in htop on the host.

[manuviens]

I deleted everything, except the data from my mailserver.

rm -rf data/rcm/*
docker system prune -af --volumes

then I tried again, but this time without the alpine versions of the containers, that is :
roundcube/roundcubemail:latest-fpm
postgres:latest
nginx:latest

I also replaced my nginx config with this:

server {
    listen 80 default_server;
    server_tokens off;
    autoindex off;
    location / { return 301 https://{{ website_hostname }}; }
}

server {
    listen 443 ssl http2 default_server;
    server_tokens off;
    autoindex off;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!MEDIUM:!LOW:!aNULL:!NULL:!SHA;
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:10m;
    ssl_session_tickets off;
    ssl_certificate     /etc/letsencrypt/live/{{ mailserver_fqdn }}/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/{{ mailserver_fqdn }}/privkey.pem;
    location / { return 301 https://{{ website_hostname }}; }
}

server {
    listen 80;
    server_name {{ roundcubemail_nginx_host }};
    location / { return 301 https://$host$request_uri; }
}

server {
    listen 443 ssl http2;
    server_name {{ roundcubemail_nginx_host }};
  
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!MEDIUM:!LOW:!aNULL:!NULL:!SHA;
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:10m;
    ssl_session_tickets off;
    ssl_certificate     /etc/letsencrypt/live/{{ roundcubemail_nginx_host }}/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/{{ roundcubemail_nginx_host }}/privkey.pem;

    root /var/www/html;

    location / {
        index index.php;
    }

    location ~ \.php$ {
        try_files $uri =404;
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_pass roundcubemail:9000;
        fastcgi_index index.php;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param PATH_INFO $fastcgi_path_info;
    }
}

The malware was back when I woke up this morning.

I just found that I'm not the only one with this problem : "I also have been pawned by this malware"

[erlangparasu]

https://techcommunity.microsoft.com/t5/microsoft-defender-for-cloud/initial-access-techniques-in-kubernetes-environments-used-by/ba-p/3697975

[thomascube]

According to @erlangparasu 's comment, the weak spot might be postgres. Running a plain Roundcube container does not show such files and thus there's little we can do about. But I'm leaving this issue open to collect more information and possibly solutions for those who are affected.

[navossoc]

@manuviens
You shouldn't leave your php-fpm port open to the world.
You are allowing anyone to run some kind of PHP script on your server.

Docker and firewalls (on the host) don't play nicely, so you probably are not blocking that port at all.
Try to check if the port is open outside your network with an external site, something like this: https://www.yougetsignal.com/tools/open-ports/

If you host everything on the same host, there is no need to expose the port 9000.

[erlangparasu]

i dont open fpm port. @navossoc 's comment is spam or may be scam.

[navossoc]

@erlangparasu You maybe not, but he did.

I just tagged the wrong person.
#129