Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Celery 僵尸进程 #2869

Open
an9wer opened this issue Jul 2, 2019 · 3 comments

Comments

Projects
None yet
4 participants
@an9wer
Copy link

commented Jul 2, 2019

运行环境:

  • OS: CentOS 7
  • Database: Sqlite3
  • Jumpserver: 1.4.10

现象:

  • Jumpserver 启动一段时间后,会出现僵尸进程,观察后发现是 celery 的某些进程。

问题:

  1. 为什么会产生僵尸进程?
  2. 这些 celery 进程的主要作用是什么?
  3. 在 jms 启动时只选择 gunicorn,不启动 celery 和 beat,会有哪方面的影响?

僵尸进程:

$ ps aux | grep Z
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      4577  0.0  0.0      0     0 ?        Z    7月01   0:00 [python36] <defunct>
root      4841  0.0  0.0      0     0 ?        Z    7月01   0:00 [python36] <defunct>
root      5645  0.0  0.0      0     0 ?        Z    02:42   0:00 [python36] <defunct>
root      5646  0.0  0.0      0     0 ?        Z    02:42   0:00 [python36] <defunct>
root      5647  0.0  0.0      0     0 ?        Z    02:42   0:00 [python36] <defunct>
root      5907  0.0  0.0      0     0 ?        Z    03:42   0:00 [python36] <defunct>

僵尸进程的父进程:

$ pstree -p -s 4577
systemd(1)───python36(3673)───python36(3694)───python36(4577)

$ ps -p '3673' '3694'
  PID TTY      STAT   TIME COMMAND
 3673 ?        S      1:39 /opt/jumpserver-venv/bin/python36 -m celery worker -A ops -l INFO --autoscale 20,4 --logfile=/opt/jumpserver/logs/celery.log --pidfile=/opt/
 3694 ?        S      0:01 /opt/jumpserver-venv/bin/python36 -m celery worker -A ops -l INFO --autoscale 20,4 --logfile=/opt/jumpserver/logs/celery.log --pidfile=/opt/
@lzj3278

This comment has been minimized.

Copy link

commented Jul 3, 2019

前天从1.4.3升级到1.5.0,同样出现celery僵尸进程。
[root@s0 bin]# ps -A -ostat,ppid,pid,cmd | grep -e '^[Zz]'
Z 120282 157873 [celery]
Z 120284 170027 [celery]

[root@s0 bin]# ps -p '120282' '120284'
PID TTY STAT TIME COMMAND
120282 ? S 0:05 /home/jumpserver/py3/bin/python3 /home/jumpserver/py3/bin/celery worker -A ops -l INFO --pidfile /home/jumpserver/jumpserver/tmp/celery.pid --autoscale 20,4
120284 ? S 0:02 /home/jumpserver/py3/bin/python3 /home/jumpserver/py3/bin/celery worker -A ops -l INFO --pidfile /home/jumpserver/jumpserver/tmp/celery.pid --autoscale 20,4

[root@s0 bin]# systemctl status jumpserver
● jumpserver.service - jumpserver
Loaded: loaded (/etc/systemd/system/jumpserver.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2019-07-03 10:13:34 CST; 4h 56min ago
Main PID: 120053 (python3)
Memory: 889.1M
CGroup: /system.slice/jumpserver.service
├─120053 python3 /home/jumpserver/jumpserver/jms start all
├─120212 /home/jumpserver/py3/bin/python3 /home/jumpserver/py3/bin/gunicorn jumpserver.wsgi -b 0.0.0.0:8088 -k gthread --threads 10 -w 4 --max-requests 4096 --access-logformat %(h)s %(t)s "%(r)s" %(s)s %(b)s -p /home/j...
├─120213 /home/jumpserver/py3/bin/python3 /home/jumpserver/py3/bin/celery worker -A ops -l INFO --pidfile /home/jumpserver/jumpserver/tmp/celery.pid --autoscale 20,4
├─120214 /home/jumpserver/py3/bin/python3 /home/jumpserver/py3/bin/celery beat -A ops --pidfile /home/jumpserver/jumpserver/tmp/beat.pid -l DEBUG --scheduler django_celery_beat.schedulers:DatabaseScheduler --max-interva...
├─120224 /home/jumpserver/py3/bin/python3 /home/jumpserver/py3/bin/gunicorn jumpserver.wsgi -b 0.0.0.0:8088 -k gthread --threads 10 -w 4 --max-requests 4096 --access-logformat %(h)s %(t)s "%(r)s" %(s)s %(b)s -p /home/j...
├─120226 /home/jumpserver/py3/bin/python3 /home/jumpserver/py3/bin/gunicorn jumpserver.wsgi -b 0.0.0.0:8088 -k gthread --threads 10 -w 4 --max-requests 4096 --access-logformat %(h)s %(t)s "%(r)s" %(s)s %(b)s -p /home/j...
├─120228 /home/jumpserver/py3/bin/python3 /home/jumpserver/py3/bin/gunicorn jumpserver.wsgi -b 0.0.0.0:8088 -k gthread --threads 10 -w 4 --max-requests 4096 --access-logformat %(h)s %(t)s "%(r)s" %(s)s %(b)s -p /home/j...
├─120230 /home/jumpserver/py3/bin/python3 /home/jumpserver/py3/bin/gunicorn jumpserver.wsgi -b 0.0.0.0:8088 -k gthread --threads 10 -w 4 --max-requests 4096 --access-logformat %(h)s %(t)s "%(r)s" %(s)s %(b)s -p /home/j...
├─120282 /home/jumpserver/py3/bin/python3 /home/jumpserver/py3/bin/celery worker -A ops -l INFO --pidfile /home/jumpserver/jumpserver/tmp/celery.pid --autoscale 20,4
├─120284 /home/jumpserver/py3/bin/python3 /home/jumpserver/py3/bin/celery worker -A ops -l INFO --pidfile /home/jumpserver/jumpserver/tmp/celery.pid --autoscale 20,4
├─120285 /home/jumpserver/py3/bin/python3 /home/jumpserver/py3/bin/celery worker -A ops -l INFO --pidfile /home/jumpserver/jumpserver/tmp/celery.pid --autoscale 20,4
└─120287 /home/jumpserver/py3/bin/python3 /home/jumpserver/py3/bin/celery worker -A ops -l INFO --pidfile /home/jumpserver/jumpserver/tmp/celery.pid --autoscale 20,4


jumpserver的systemd

[Unit]
Description=jumpserver
After=network.target
[Service]
Type=forking
User=root
Environment="PATH=/home/jumpserver/py3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin"
#ExecStart=/bin/bash -c 'PATH=/home/jumpserver/py3/bin:$PATH exec /home/jumpserver/jumpserver/jms start all'
ExecStart=/home/jumpserver/jumpserver/jms start all -d
ExecReload=
ExecStop=/home/jumpserver/jumpserver/jms stop

[Install]
WantedBy=multi-user.target


不使用systemd启动 使用jms start all -d 启动也会出现僵尸进程

@zhanghailu

This comment has been minimized.

Copy link

commented Jul 18, 2019

这个我也遇见 了 这个类似的情况,有什么解决办法吗

@ibuler

This comment has been minimized.

Copy link
Member

commented Jul 22, 2019

不影响,重启后僵尸进程就没了,应该是celery管理worker有关系,https://stackoverflow.com/questions/43131145/avoid-zombie-processes-by-regularly-calling-join

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.