[BUG]I'm passing the script through salt base64 to the target machine and running it, but salt-master EventPublisher CPU 100%,Whether this thread supports extension

[764276020]

Description
I'm passing the script through salt base64 to the target machine and running it, but salt-master EventPublisher CPU 100%,Whether this thread supports extension?

We use saltstack as the mission system, salt-minion number 8000+

Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Please be as specific as possible and give set-up details.

• on-prem machine
• [ √] VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior
(Include debug logs if possible and relevant)

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

PASTE HERE

Additional context
Add any other context about the problem here.

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[764276020]

[root@saltmaster-m192-49 tmp]# salt --versions-report
Salt Version:
Salt: 3005.1

Dependency Versions:
cffi: Not Installed
cherrypy: unknown
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.8.1
libgit2: Not Installed
M2Crypto: 0.33.0
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: Not Installed
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 18.0.1
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.1.4

System Versions:
dist: centos 7 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-1127.19.1.el7.x86_64
system: Linux
version: CentOS Linux 7 Core

[764276020]

@garethgreenaway Could you answer that, please!

[OrangeDog]

8000 is a lot of minions. It's very easy to overload a single master if you're not careful.

I don't understand what "passing the script through salt base64" means.
Please provide actual reproduction of the issue, including relevant config, sls files, and the command(s) you are running.

[764276020]

8000 is a lot of minions. It's very easy to overload a single master if you're not careful.

I don't understand what "passing the script through salt base64" means. Please provide actual reproduction of the issue, including relevant config, sls files, and the command(s) you are running.

I suspect eventpublisher cpu 100% is shell script in hashutil.base64_decodefile

I used two salt-syndic nodes， But only the top master has problems

[764276020]

8000 is a lot of minions. It's very easy to overload a single master if you're not careful.
I don't understand what "passing the script through salt base64" means. Please provide actual reproduction of the issue, including relevant config, sls files, and the command(s) you are running.

I suspect eventpublisher cpu 100% is shell script in hashutil.base64_decodefile

I used two salt-syndic nodes， But only the top master has problems

like this

Currently don't know how to expand

[OrangeDog]

Please provide actual reproduction of the issue, including relevant config, sls files, and the command(s) you are running.

[764276020]

请提供问题的实际重现，包括相关配置、sls 文件和您正在运行的命令。

i am executing salt '*' test.ping

[OrangeDog]

How is that "passing the script through salt base64 to the target machine and running it"?

What are you actually doing? What script? What config? How are you passing it and running it?

[764276020]

How is that "passing the script through salt base64 to the target machine and running it"?

What are you actually doing? What script? What config? How are you passing it and running it?

I'm sorry，i mean saltstack minion too many, salt top master event pubulisher single process(thread) performance is not enough, I want to extend it, or is there any other way to solve it

[764276020]

How is that "passing the script through salt base64 to the target machine and running it"?

What are you actually doing? What script? What config? How are you passing it and running it?

even running salt '*' test.ping causes salt top master eventpublisher thread cpu 100%.
I want to ask is there a solution?

[OrangeDog]

Possible duplicate: #61845

[764276020]

Possible duplicate: #61845

no, i see salt top master eventpublisher cpu 100%, I want to extend it, or is there any other way to solve it

[764276020]

I've tried extending salt-top-master-eventpublisher-processing to multiple processes, but it doesn't seem to work

[764276020]

Possible duplicate: #61845

no, i see salt top master eventpublisher cpu 100%, I want to extend it, or is there any other way to solve it

may be salt minion too many lead to insufficient performance，Is salt top master eventpublisher possible to make it run in multiple processes

[764276020]

Excuse me, is there any progress?

[dwoz]

Is zmq_filtering enabled?

[764276020]

Is zmq_filtering enabled?

not enable

configuration:
interface: 192.168.192.49
worker_threads: 8
auto_accept: False
keysize: 2048
master_id: saltmaster-m192-49
keep_jobs: 24
job_cache: True
cachedir: /home/data/salt/cache/master
sqlite_queue_dir: /home/data/salt/cache/master/queues
sock_pool_size: 10240
tcp_master_workers: 10240
ipc_mode: ipc
default_include: master.d/*.yml
file_roots:
base:
- /home/data/salt/data/states
- /home/data/salt/data/services
pillar_roots:
base:
- /home/data/salt/data/pillar

order_masters: True

rest_cherrypy:
port: 8888
disable_ssl: 1
host: 192.168.192.49
thread_pool: 2048
socket_queue_size: 1024
debug: True
log_error_file: /home/data/salt/var/log/api/error.log
log_access_file: /home/data/salt/var/log/api/access.log
expire_responses: False

mysql_auth: ....

gather_job_timeout: 120
timeout: 60
show_timeout: True
show_jid: True

log_level: warn