Set Process title for Worker when using LocalExecutor
#16623
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This has annoyed me for a long time. When using ``LocalExecutor``, it was difficult to see which process is a worker as it just showed up as below -- which had same title as parent scheduler process. This PR/commit adds a title for idle workers and when a task is running it has the "command" that is running in the title, similar to our supervising process Before: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 1449 0.8 2.6 988356 326312 pts/1 Sl+ Jun23 0:16 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 1584 0.0 0.4 121068 56864 pts/1 S+ Jun23 0:01 | \_ gunicorn: master [airflow-webserver] root 1587 0.6 2.5 986144 318712 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1588 0.6 2.5 984776 317672 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1589 0.6 2.5 985688 318148 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1590 0.6 2.5 985200 317776 pts/1 Sl+ Jun23 0:11 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 13933 31.0 0.9 466596 117656 pts/2 S+ 00:22 0:01 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13941 0.0 0.7 466340 97988 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13942 3.2 0.8 1392072 100136 pts/2 Sl+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13950 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13952 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13955 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13958 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13962 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13966 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13969 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13975 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13979 6.5 0.8 466596 99956 pts/2 S 00:22 0:00 \_ airflow scheduler -- DagFileProcessorManager ``` After (with no running tasks - idle workers): ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 1449 0.8 2.6 988356 326312 pts/1 Sl+ Jun23 0:16 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 1584 0.0 0.4 121068 56864 pts/1 S+ Jun23 0:01 | \_ gunicorn: master [airflow-webserver] root 1587 0.6 2.5 985752 318184 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1588 0.6 2.5 984776 317672 pts/1 Sl+ Jun23 0:11 | \_ [ready] gunicorn: worker [airflow-webserver] root 1589 0.6 2.5 985688 318148 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1590 0.6 2.5 985200 317776 pts/1 Sl+ Jun23 0:11 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 13237 25.7 0.9 466596 117692 pts/2 S+ 00:20 0:02 \_ airflow worker -- LocalExecutor root 13245 0.1 0.7 466340 97804 pts/2 S+ 00:20 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13246 2.1 0.8 1318340 100104 pts/2 Sl+ 00:20 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13254 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13256 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13259 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13263 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13267 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13271 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13274 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13276 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13282 4.1 0.8 466596 99952 pts/2 S 00:20 0:00 \_ airflow scheduler -- DagFileProcessorManager ``` After (with running tasks): ``` root@a7c8aa590704:/opt/airflow# ps auxf USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 6434 0.0 0.0 6652 4584 pts/3 Ss 00:01 0:00 /bin/bash root 19250 0.0 0.0 9556 3064 pts/3 R+ 00:39 0:00 \_ ps auxf root 1 0.0 0.0 2148 720 ? Ss Jun23 0:00 /usr/bin/dumb-init -- /entrypoint root 7 0.0 0.0 6656 4400 pts/0 Ss Jun23 0:00 /bin/bash root 121 0.0 0.0 8220 3228 pts/0 S+ Jun23 0:00 \_ tmux root 101 0.0 0.0 15856 4272 ? Ss Jun23 0:00 /usr/sbin/sshd root 123 0.0 0.0 10176 5148 ? Ss Jun23 0:00 tmux root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 1449 0.6 2.6 988356 326312 pts/1 Sl+ Jun23 0:20 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 1584 0.0 0.4 121068 56864 pts/1 S+ Jun23 0:01 | \_ gunicorn: master [airflow-webserver] root 1587 0.4 2.5 986144 318712 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1588 0.4 2.5 984776 317672 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1589 0.4 2.5 985848 318600 pts/1 Sl+ Jun23 0:13 | \_ [ready] gunicorn: worker [airflow-webserver] root 1590 0.4 2.5 985628 318424 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 19030 17.9 0.9 467108 118628 pts/2 S+ 00:38 0:02 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 19038 0.0 0.7 466084 97776 pts/2 S+ 00:38 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 19039 1.4 0.8 1318084 99804 pts/2 Sl+ 00:38 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 19047 0.0 0.8 466084 98692 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_2', '2021-06-24T00:39:06.539715+00:00', '--local', ' root 19240 25.3 0.8 470820 104400 pts/2 S+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_2', '2021-06-24T00:39:06.539715+00:00', '--local', '--po root 19246 0.0 0.8 470956 103980 pts/2 S 00:39 0:00 | \_ airflow task runner: example_bash_operator runme_2 2021-06-24T00:39:06.539715+00:00 91 root 19049 0.1 0.8 466084 98696 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_1', '2021-06-24T00:39:06.539715+00:00', '--local', ' root 19241 26.0 0.8 470824 104408 pts/2 S+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_1', '2021-06-24T00:39:06.539715+00:00', '--local', '--po root 19248 0.0 0.8 470824 103720 pts/2 S 00:39 0:00 | \_ airflow task runner: example_bash_operator runme_1 2021-06-24T00:39:06.539715+00:00 93 root 19052 0.1 0.8 466084 98760 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_0', '2021-06-24T00:39:06.539715+00:00', '--local', ' root 19244 26.0 0.8 470824 104404 pts/2 S+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_0', '2021-06-24T00:39:06.539715+00:00', '--local', '--po root 19245 0.0 0.8 471212 104032 pts/2 S 00:39 0:00 | \_ airflow task runner: example_bash_operator runme_0 2021-06-24T00:39:06.539715+00:00 90 root 19056 0.1 0.8 466084 98760 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'this_will_skip', '2021-06-24T00:39:06.539715+00:00', '--lo root 19243 24.6 0.8 470824 104400 pts/2 S+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'this_will_skip', '2021-06-24T00:39:06.539715+00:00', '--local' root 19247 0.0 0.8 470956 103712 pts/2 S 00:39 0:00 | \_ airflow task runner: example_bash_operator this_will_skip 2021-06-24T00:39:06.539715+00:00 92 root 19057 0.1 0.8 466084 98760 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'also_run_this', '2021-06-24T00:39:06.539715+00:00', '--loc root 19242 26.6 0.8 470824 104404 pts/2 R+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'also_run_this', '2021-06-24T00:39:06.539715+00:00', '--local', root 19249 0.0 0.8 470824 101976 pts/2 S 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'also_run_this', '2021-06-24T00:39:06.539715+00:00', '--loc root 19062 0.0 0.8 466084 98300 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor root 19066 0.0 0.8 466084 98300 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor root 19069 0.0 0.8 466084 98300 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor root 19075 2.7 0.8 466596 100144 pts/2 S 00:38 0:00 \_ airflow scheduler -- DagFileProcessorManager ``` Once the worker is done executing a task, the worker is renamed back to `airflow worker -- LocalExecutor`
potiuk
approved these changes
Jun 24, 2021
|
Nice! |
|
@kaxil I've set this to 2.1.2 -- 2.1.1. is in feature freeze now. |
Jorricks
pushed a commit
to Jorricks/airflow
that referenced
this pull request
Jun 24, 2021
This has annoyed me for a long time. When using ``LocalExecutor``, it was difficult to see which process is a worker as it just showed up as below -- which had same title as parent scheduler process. This PR/commit adds a title for idle workers and when a task is running it has the "command" that is running in the title, similar to our supervising process Before: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 1449 0.8 2.6 988356 326312 pts/1 Sl+ Jun23 0:16 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 1584 0.0 0.4 121068 56864 pts/1 S+ Jun23 0:01 | \_ gunicorn: master [airflow-webserver] root 1587 0.6 2.5 986144 318712 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1588 0.6 2.5 984776 317672 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1589 0.6 2.5 985688 318148 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1590 0.6 2.5 985200 317776 pts/1 Sl+ Jun23 0:11 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 13933 31.0 0.9 466596 117656 pts/2 S+ 00:22 0:01 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13941 0.0 0.7 466340 97988 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13942 3.2 0.8 1392072 100136 pts/2 Sl+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13950 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13952 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13955 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13958 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13962 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13966 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13969 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13975 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13979 6.5 0.8 466596 99956 pts/2 S 00:22 0:00 \_ airflow scheduler -- DagFileProcessorManager ``` After (with no running tasks - idle workers): ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 1449 0.8 2.6 988356 326312 pts/1 Sl+ Jun23 0:16 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 1584 0.0 0.4 121068 56864 pts/1 S+ Jun23 0:01 | \_ gunicorn: master [airflow-webserver] root 1587 0.6 2.5 985752 318184 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1588 0.6 2.5 984776 317672 pts/1 Sl+ Jun23 0:11 | \_ [ready] gunicorn: worker [airflow-webserver] root 1589 0.6 2.5 985688 318148 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1590 0.6 2.5 985200 317776 pts/1 Sl+ Jun23 0:11 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 13237 25.7 0.9 466596 117692 pts/2 S+ 00:20 0:02 \_ airflow worker -- LocalExecutor root 13245 0.1 0.7 466340 97804 pts/2 S+ 00:20 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13246 2.1 0.8 1318340 100104 pts/2 Sl+ 00:20 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13254 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13256 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13259 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13263 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13267 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13271 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13274 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13276 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13282 4.1 0.8 466596 99952 pts/2 S 00:20 0:00 \_ airflow scheduler -- DagFileProcessorManager ``` After (with running tasks): ``` root@a7c8aa590704:/opt/airflow# ps auxf USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 6434 0.0 0.0 6652 4584 pts/3 Ss 00:01 0:00 /bin/bash root 19250 0.0 0.0 9556 3064 pts/3 R+ 00:39 0:00 \_ ps auxf root 1 0.0 0.0 2148 720 ? Ss Jun23 0:00 /usr/bin/dumb-init -- /entrypoint root 7 0.0 0.0 6656 4400 pts/0 Ss Jun23 0:00 /bin/bash root 121 0.0 0.0 8220 3228 pts/0 S+ Jun23 0:00 \_ tmux root 101 0.0 0.0 15856 4272 ? Ss Jun23 0:00 /usr/sbin/sshd root 123 0.0 0.0 10176 5148 ? Ss Jun23 0:00 tmux root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 1449 0.6 2.6 988356 326312 pts/1 Sl+ Jun23 0:20 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 1584 0.0 0.4 121068 56864 pts/1 S+ Jun23 0:01 | \_ gunicorn: master [airflow-webserver] root 1587 0.4 2.5 986144 318712 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1588 0.4 2.5 984776 317672 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1589 0.4 2.5 985848 318600 pts/1 Sl+ Jun23 0:13 | \_ [ready] gunicorn: worker [airflow-webserver] root 1590 0.4 2.5 985628 318424 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 19030 17.9 0.9 467108 118628 pts/2 S+ 00:38 0:02 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 19038 0.0 0.7 466084 97776 pts/2 S+ 00:38 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 19039 1.4 0.8 1318084 99804 pts/2 Sl+ 00:38 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 19047 0.0 0.8 466084 98692 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_2', '2021-06-24T00:39:06.539715+00:00', '--local', ' root 19240 25.3 0.8 470820 104400 pts/2 S+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_2', '2021-06-24T00:39:06.539715+00:00', '--local', '--po root 19246 0.0 0.8 470956 103980 pts/2 S 00:39 0:00 | \_ airflow task runner: example_bash_operator runme_2 2021-06-24T00:39:06.539715+00:00 91 root 19049 0.1 0.8 466084 98696 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_1', '2021-06-24T00:39:06.539715+00:00', '--local', ' root 19241 26.0 0.8 470824 104408 pts/2 S+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_1', '2021-06-24T00:39:06.539715+00:00', '--local', '--po root 19248 0.0 0.8 470824 103720 pts/2 S 00:39 0:00 | \_ airflow task runner: example_bash_operator runme_1 2021-06-24T00:39:06.539715+00:00 93 root 19052 0.1 0.8 466084 98760 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_0', '2021-06-24T00:39:06.539715+00:00', '--local', ' root 19244 26.0 0.8 470824 104404 pts/2 S+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_0', '2021-06-24T00:39:06.539715+00:00', '--local', '--po root 19245 0.0 0.8 471212 104032 pts/2 S 00:39 0:00 | \_ airflow task runner: example_bash_operator runme_0 2021-06-24T00:39:06.539715+00:00 90 root 19056 0.1 0.8 466084 98760 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'this_will_skip', '2021-06-24T00:39:06.539715+00:00', '--lo root 19243 24.6 0.8 470824 104400 pts/2 S+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'this_will_skip', '2021-06-24T00:39:06.539715+00:00', '--local' root 19247 0.0 0.8 470956 103712 pts/2 S 00:39 0:00 | \_ airflow task runner: example_bash_operator this_will_skip 2021-06-24T00:39:06.539715+00:00 92 root 19057 0.1 0.8 466084 98760 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'also_run_this', '2021-06-24T00:39:06.539715+00:00', '--loc root 19242 26.6 0.8 470824 104404 pts/2 R+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'also_run_this', '2021-06-24T00:39:06.539715+00:00', '--local', root 19249 0.0 0.8 470824 101976 pts/2 S 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'also_run_this', '2021-06-24T00:39:06.539715+00:00', '--loc root 19062 0.0 0.8 466084 98300 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor root 19066 0.0 0.8 466084 98300 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor root 19069 0.0 0.8 466084 98300 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor root 19075 2.7 0.8 466596 100144 pts/2 S 00:38 0:00 \_ airflow scheduler -- DagFileProcessorManager ``` Once the worker is done executing a task, the worker is renamed back to `airflow worker -- LocalExecutor`
kaxil
added a commit
to astronomer/airflow
that referenced
this pull request
Jun 25, 2021
Follow up of apache#16623. This PR/commits adds title to serve-logs command and multiprocessing manager for LocalExecutor. The serve-logs process is on celery worker when using CeleryExecutor but for LocalExecutor, it is a separate process in Scheduler. **Before**: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 25299 25.3 2.6 988372 326344 pts/1 Sl+ 01:30 0:09 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 25510 3.6 0.4 121068 57152 pts/1 S+ 01:31 0:00 | \_ gunicorn: master [airflow-webserver] root 25555 35.7 2.5 983584 316564 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25556 35.7 2.5 983840 316684 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25557 35.5 2.5 983840 316548 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25558 37.2 2.5 984920 317700 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 25090 5.8 0.9 467508 118808 pts/2 S+ 01:30 0:03 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25098 0.0 0.7 466080 97800 pts/2 S+ 01:30 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25099 0.4 0.8 1391812 99788 pts/2 Sl+ 01:30 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25107 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25109 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25114 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25117 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25120 0.0 0.8 466080 98552 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25125 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--lo root 26139 0.0 0.8 468988 102204 pts/2 S+ 01:31 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--local' root 25128 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25132 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25137 1.1 0.8 466592 100016 pts/2 S 01:30 0:00 \_ airflow scheduler -- DagFileProcessorManager root@a7c8aa590704:/opt/airflow# ps aux ``` **After**: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 25299 25.3 2.6 988372 326344 pts/1 Sl+ 01:30 0:09 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 25510 3.6 0.4 121068 57152 pts/1 S+ 01:31 0:00 | \_ gunicorn: master [airflow-webserver] root 25555 35.7 2.5 983584 316564 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25556 35.7 2.5 983840 316684 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25557 35.5 2.5 983840 316548 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25558 37.2 2.5 984920 317700 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 25090 5.8 0.9 467508 118808 pts/2 S+ 01:30 0:03 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25098 0.0 0.7 466080 97800 pts/2 S+ 01:30 0:00 \_ airflow serve-logs root 25099 0.4 0.8 1391812 99788 pts/2 Sl+ 01:30 0:00 \_ airflow executor -- LocalExecutor root 25107 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25109 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25114 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25117 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25120 0.0 0.8 466080 98552 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25125 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--lo root 26139 0.0 0.8 468988 102204 pts/2 S+ 01:31 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--local' root 25128 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25132 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25137 1.1 0.8 466592 100016 pts/2 S 01:30 0:00 \_ airflow scheduler -- DagFileProcessorManager root@a7c8aa590704:/opt/airflow# ps aux ```
kaxil
added a commit
that referenced
this pull request
Jun 25, 2021
Follow up of #16623. This PR/commits adds title to serve-logs command and multiprocessing manager for LocalExecutor. The serve-logs process is on celery worker when using CeleryExecutor but for LocalExecutor, it is a separate process in Scheduler. **Before**: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 25299 25.3 2.6 988372 326344 pts/1 Sl+ 01:30 0:09 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 25510 3.6 0.4 121068 57152 pts/1 S+ 01:31 0:00 | \_ gunicorn: master [airflow-webserver] root 25555 35.7 2.5 983584 316564 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25556 35.7 2.5 983840 316684 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25557 35.5 2.5 983840 316548 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25558 37.2 2.5 984920 317700 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 25090 5.8 0.9 467508 118808 pts/2 S+ 01:30 0:03 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25098 0.0 0.7 466080 97800 pts/2 S+ 01:30 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25099 0.4 0.8 1391812 99788 pts/2 Sl+ 01:30 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25107 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25109 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25114 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25117 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25120 0.0 0.8 466080 98552 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25125 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--lo root 26139 0.0 0.8 468988 102204 pts/2 S+ 01:31 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--local' root 25128 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25132 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25137 1.1 0.8 466592 100016 pts/2 S 01:30 0:00 \_ airflow scheduler -- DagFileProcessorManager root@a7c8aa590704:/opt/airflow# ps aux ``` **After**: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 25299 25.3 2.6 988372 326344 pts/1 Sl+ 01:30 0:09 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 25510 3.6 0.4 121068 57152 pts/1 S+ 01:31 0:00 | \_ gunicorn: master [airflow-webserver] root 25555 35.7 2.5 983584 316564 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25556 35.7 2.5 983840 316684 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25557 35.5 2.5 983840 316548 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25558 37.2 2.5 984920 317700 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 25090 5.8 0.9 467508 118808 pts/2 S+ 01:30 0:03 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25098 0.0 0.7 466080 97800 pts/2 S+ 01:30 0:00 \_ airflow serve-logs root 25099 0.4 0.8 1391812 99788 pts/2 Sl+ 01:30 0:00 \_ airflow executor -- LocalExecutor root 25107 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25109 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25114 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25117 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25120 0.0 0.8 466080 98552 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25125 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--lo root 26139 0.0 0.8 468988 102204 pts/2 S+ 01:31 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--local' root 25128 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25132 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25137 1.1 0.8 466592 100016 pts/2 S 01:30 0:00 \_ airflow scheduler -- DagFileProcessorManager root@a7c8aa590704:/opt/airflow# ps aux ```
kaxil
added a commit
to astronomer/airflow
that referenced
this pull request
Jul 6, 2021
) Follow up of apache#16623. This PR/commits adds title to serve-logs command and multiprocessing manager for LocalExecutor. The serve-logs process is on celery worker when using CeleryExecutor but for LocalExecutor, it is a separate process in Scheduler. **Before**: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 25299 25.3 2.6 988372 326344 pts/1 Sl+ 01:30 0:09 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 25510 3.6 0.4 121068 57152 pts/1 S+ 01:31 0:00 | \_ gunicorn: master [airflow-webserver] root 25555 35.7 2.5 983584 316564 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25556 35.7 2.5 983840 316684 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25557 35.5 2.5 983840 316548 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25558 37.2 2.5 984920 317700 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 25090 5.8 0.9 467508 118808 pts/2 S+ 01:30 0:03 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25098 0.0 0.7 466080 97800 pts/2 S+ 01:30 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25099 0.4 0.8 1391812 99788 pts/2 Sl+ 01:30 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25107 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25109 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25114 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25117 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25120 0.0 0.8 466080 98552 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25125 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--lo root 26139 0.0 0.8 468988 102204 pts/2 S+ 01:31 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--local' root 25128 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25132 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25137 1.1 0.8 466592 100016 pts/2 S 01:30 0:00 \_ airflow scheduler -- DagFileProcessorManager root@a7c8aa590704:/opt/airflow# ps aux ``` **After**: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 25299 25.3 2.6 988372 326344 pts/1 Sl+ 01:30 0:09 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 25510 3.6 0.4 121068 57152 pts/1 S+ 01:31 0:00 | \_ gunicorn: master [airflow-webserver] root 25555 35.7 2.5 983584 316564 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25556 35.7 2.5 983840 316684 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25557 35.5 2.5 983840 316548 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25558 37.2 2.5 984920 317700 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 25090 5.8 0.9 467508 118808 pts/2 S+ 01:30 0:03 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25098 0.0 0.7 466080 97800 pts/2 S+ 01:30 0:00 \_ airflow serve-logs root 25099 0.4 0.8 1391812 99788 pts/2 Sl+ 01:30 0:00 \_ airflow executor -- LocalExecutor root 25107 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25109 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25114 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25117 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25120 0.0 0.8 466080 98552 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25125 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--lo root 26139 0.0 0.8 468988 102204 pts/2 S+ 01:31 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--local' root 25128 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25132 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25137 1.1 0.8 466592 100016 pts/2 S 01:30 0:00 \_ airflow scheduler -- DagFileProcessorManager root@a7c8aa590704:/opt/airflow# ps aux ``` (cherry picked from commit 1b2535e)
jhtimmins
pushed a commit
that referenced
this pull request
Aug 9, 2021
This has annoyed me for a long time. When using ``LocalExecutor``, it was difficult to see which process is a worker as it just showed up as below -- which had same title as parent scheduler process. This PR/commit adds a title for idle workers and when a task is running it has the "command" that is running in the title, similar to our supervising process Before: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 1449 0.8 2.6 988356 326312 pts/1 Sl+ Jun23 0:16 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 1584 0.0 0.4 121068 56864 pts/1 S+ Jun23 0:01 | \_ gunicorn: master [airflow-webserver] root 1587 0.6 2.5 986144 318712 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1588 0.6 2.5 984776 317672 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1589 0.6 2.5 985688 318148 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1590 0.6 2.5 985200 317776 pts/1 Sl+ Jun23 0:11 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 13933 31.0 0.9 466596 117656 pts/2 S+ 00:22 0:01 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13941 0.0 0.7 466340 97988 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13942 3.2 0.8 1392072 100136 pts/2 Sl+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13950 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13952 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13955 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13958 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13962 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13966 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13969 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13975 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13979 6.5 0.8 466596 99956 pts/2 S 00:22 0:00 \_ airflow scheduler -- DagFileProcessorManager ``` After (with no running tasks - idle workers): ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 1449 0.8 2.6 988356 326312 pts/1 Sl+ Jun23 0:16 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 1584 0.0 0.4 121068 56864 pts/1 S+ Jun23 0:01 | \_ gunicorn: master [airflow-webserver] root 1587 0.6 2.5 985752 318184 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1588 0.6 2.5 984776 317672 pts/1 Sl+ Jun23 0:11 | \_ [ready] gunicorn: worker [airflow-webserver] root 1589 0.6 2.5 985688 318148 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1590 0.6 2.5 985200 317776 pts/1 Sl+ Jun23 0:11 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 13237 25.7 0.9 466596 117692 pts/2 S+ 00:20 0:02 \_ airflow worker -- LocalExecutor root 13245 0.1 0.7 466340 97804 pts/2 S+ 00:20 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13246 2.1 0.8 1318340 100104 pts/2 Sl+ 00:20 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13254 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13256 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13259 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13263 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13267 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13271 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13274 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13276 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13282 4.1 0.8 466596 99952 pts/2 S 00:20 0:00 \_ airflow scheduler -- DagFileProcessorManager ``` After (with running tasks): ``` root@a7c8aa590704:/opt/airflow# ps auxf USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 6434 0.0 0.0 6652 4584 pts/3 Ss 00:01 0:00 /bin/bash root 19250 0.0 0.0 9556 3064 pts/3 R+ 00:39 0:00 \_ ps auxf root 1 0.0 0.0 2148 720 ? Ss Jun23 0:00 /usr/bin/dumb-init -- /entrypoint root 7 0.0 0.0 6656 4400 pts/0 Ss Jun23 0:00 /bin/bash root 121 0.0 0.0 8220 3228 pts/0 S+ Jun23 0:00 \_ tmux root 101 0.0 0.0 15856 4272 ? Ss Jun23 0:00 /usr/sbin/sshd root 123 0.0 0.0 10176 5148 ? Ss Jun23 0:00 tmux root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 1449 0.6 2.6 988356 326312 pts/1 Sl+ Jun23 0:20 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 1584 0.0 0.4 121068 56864 pts/1 S+ Jun23 0:01 | \_ gunicorn: master [airflow-webserver] root 1587 0.4 2.5 986144 318712 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1588 0.4 2.5 984776 317672 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1589 0.4 2.5 985848 318600 pts/1 Sl+ Jun23 0:13 | \_ [ready] gunicorn: worker [airflow-webserver] root 1590 0.4 2.5 985628 318424 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 19030 17.9 0.9 467108 118628 pts/2 S+ 00:38 0:02 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 19038 0.0 0.7 466084 97776 pts/2 S+ 00:38 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 19039 1.4 0.8 1318084 99804 pts/2 Sl+ 00:38 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 19047 0.0 0.8 466084 98692 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_2', '2021-06-24T00:39:06.539715+00:00', '--local', ' root 19240 25.3 0.8 470820 104400 pts/2 S+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_2', '2021-06-24T00:39:06.539715+00:00', '--local', '--po root 19246 0.0 0.8 470956 103980 pts/2 S 00:39 0:00 | \_ airflow task runner: example_bash_operator runme_2 2021-06-24T00:39:06.539715+00:00 91 root 19049 0.1 0.8 466084 98696 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_1', '2021-06-24T00:39:06.539715+00:00', '--local', ' root 19241 26.0 0.8 470824 104408 pts/2 S+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_1', '2021-06-24T00:39:06.539715+00:00', '--local', '--po root 19248 0.0 0.8 470824 103720 pts/2 S 00:39 0:00 | \_ airflow task runner: example_bash_operator runme_1 2021-06-24T00:39:06.539715+00:00 93 root 19052 0.1 0.8 466084 98760 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_0', '2021-06-24T00:39:06.539715+00:00', '--local', ' root 19244 26.0 0.8 470824 104404 pts/2 S+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_0', '2021-06-24T00:39:06.539715+00:00', '--local', '--po root 19245 0.0 0.8 471212 104032 pts/2 S 00:39 0:00 | \_ airflow task runner: example_bash_operator runme_0 2021-06-24T00:39:06.539715+00:00 90 root 19056 0.1 0.8 466084 98760 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'this_will_skip', '2021-06-24T00:39:06.539715+00:00', '--lo root 19243 24.6 0.8 470824 104400 pts/2 S+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'this_will_skip', '2021-06-24T00:39:06.539715+00:00', '--local' root 19247 0.0 0.8 470956 103712 pts/2 S 00:39 0:00 | \_ airflow task runner: example_bash_operator this_will_skip 2021-06-24T00:39:06.539715+00:00 92 root 19057 0.1 0.8 466084 98760 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'also_run_this', '2021-06-24T00:39:06.539715+00:00', '--loc root 19242 26.6 0.8 470824 104404 pts/2 R+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'also_run_this', '2021-06-24T00:39:06.539715+00:00', '--local', root 19249 0.0 0.8 470824 101976 pts/2 S 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'also_run_this', '2021-06-24T00:39:06.539715+00:00', '--loc root 19062 0.0 0.8 466084 98300 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor root 19066 0.0 0.8 466084 98300 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor root 19069 0.0 0.8 466084 98300 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor root 19075 2.7 0.8 466596 100144 pts/2 S 00:38 0:00 \_ airflow scheduler -- DagFileProcessorManager ``` Once the worker is done executing a task, the worker is renamed back to `airflow worker -- LocalExecutor` (cherry picked from commit c8a628a)
jhtimmins
pushed a commit
that referenced
this pull request
Aug 13, 2021
This has annoyed me for a long time. When using ``LocalExecutor``, it was difficult to see which process is a worker as it just showed up as below -- which had same title as parent scheduler process. This PR/commit adds a title for idle workers and when a task is running it has the "command" that is running in the title, similar to our supervising process Before: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 1449 0.8 2.6 988356 326312 pts/1 Sl+ Jun23 0:16 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 1584 0.0 0.4 121068 56864 pts/1 S+ Jun23 0:01 | \_ gunicorn: master [airflow-webserver] root 1587 0.6 2.5 986144 318712 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1588 0.6 2.5 984776 317672 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1589 0.6 2.5 985688 318148 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1590 0.6 2.5 985200 317776 pts/1 Sl+ Jun23 0:11 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 13933 31.0 0.9 466596 117656 pts/2 S+ 00:22 0:01 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13941 0.0 0.7 466340 97988 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13942 3.2 0.8 1392072 100136 pts/2 Sl+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13950 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13952 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13955 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13958 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13962 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13966 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13969 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13975 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13979 6.5 0.8 466596 99956 pts/2 S 00:22 0:00 \_ airflow scheduler -- DagFileProcessorManager ``` After (with no running tasks - idle workers): ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 1449 0.8 2.6 988356 326312 pts/1 Sl+ Jun23 0:16 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 1584 0.0 0.4 121068 56864 pts/1 S+ Jun23 0:01 | \_ gunicorn: master [airflow-webserver] root 1587 0.6 2.5 985752 318184 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1588 0.6 2.5 984776 317672 pts/1 Sl+ Jun23 0:11 | \_ [ready] gunicorn: worker [airflow-webserver] root 1589 0.6 2.5 985688 318148 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1590 0.6 2.5 985200 317776 pts/1 Sl+ Jun23 0:11 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 13237 25.7 0.9 466596 117692 pts/2 S+ 00:20 0:02 \_ airflow worker -- LocalExecutor root 13245 0.1 0.7 466340 97804 pts/2 S+ 00:20 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13246 2.1 0.8 1318340 100104 pts/2 Sl+ 00:20 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13254 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13256 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13259 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13263 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13267 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13271 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13274 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13276 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13282 4.1 0.8 466596 99952 pts/2 S 00:20 0:00 \_ airflow scheduler -- DagFileProcessorManager ``` After (with running tasks): ``` root@a7c8aa590704:/opt/airflow# ps auxf USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 6434 0.0 0.0 6652 4584 pts/3 Ss 00:01 0:00 /bin/bash root 19250 0.0 0.0 9556 3064 pts/3 R+ 00:39 0:00 \_ ps auxf root 1 0.0 0.0 2148 720 ? Ss Jun23 0:00 /usr/bin/dumb-init -- /entrypoint root 7 0.0 0.0 6656 4400 pts/0 Ss Jun23 0:00 /bin/bash root 121 0.0 0.0 8220 3228 pts/0 S+ Jun23 0:00 \_ tmux root 101 0.0 0.0 15856 4272 ? Ss Jun23 0:00 /usr/sbin/sshd root 123 0.0 0.0 10176 5148 ? Ss Jun23 0:00 tmux root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 1449 0.6 2.6 988356 326312 pts/1 Sl+ Jun23 0:20 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 1584 0.0 0.4 121068 56864 pts/1 S+ Jun23 0:01 | \_ gunicorn: master [airflow-webserver] root 1587 0.4 2.5 986144 318712 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1588 0.4 2.5 984776 317672 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1589 0.4 2.5 985848 318600 pts/1 Sl+ Jun23 0:13 | \_ [ready] gunicorn: worker [airflow-webserver] root 1590 0.4 2.5 985628 318424 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 19030 17.9 0.9 467108 118628 pts/2 S+ 00:38 0:02 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 19038 0.0 0.7 466084 97776 pts/2 S+ 00:38 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 19039 1.4 0.8 1318084 99804 pts/2 Sl+ 00:38 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 19047 0.0 0.8 466084 98692 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_2', '2021-06-24T00:39:06.539715+00:00', '--local', ' root 19240 25.3 0.8 470820 104400 pts/2 S+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_2', '2021-06-24T00:39:06.539715+00:00', '--local', '--po root 19246 0.0 0.8 470956 103980 pts/2 S 00:39 0:00 | \_ airflow task runner: example_bash_operator runme_2 2021-06-24T00:39:06.539715+00:00 91 root 19049 0.1 0.8 466084 98696 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_1', '2021-06-24T00:39:06.539715+00:00', '--local', ' root 19241 26.0 0.8 470824 104408 pts/2 S+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_1', '2021-06-24T00:39:06.539715+00:00', '--local', '--po root 19248 0.0 0.8 470824 103720 pts/2 S 00:39 0:00 | \_ airflow task runner: example_bash_operator runme_1 2021-06-24T00:39:06.539715+00:00 93 root 19052 0.1 0.8 466084 98760 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_0', '2021-06-24T00:39:06.539715+00:00', '--local', ' root 19244 26.0 0.8 470824 104404 pts/2 S+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_0', '2021-06-24T00:39:06.539715+00:00', '--local', '--po root 19245 0.0 0.8 471212 104032 pts/2 S 00:39 0:00 | \_ airflow task runner: example_bash_operator runme_0 2021-06-24T00:39:06.539715+00:00 90 root 19056 0.1 0.8 466084 98760 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'this_will_skip', '2021-06-24T00:39:06.539715+00:00', '--lo root 19243 24.6 0.8 470824 104400 pts/2 S+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'this_will_skip', '2021-06-24T00:39:06.539715+00:00', '--local' root 19247 0.0 0.8 470956 103712 pts/2 S 00:39 0:00 | \_ airflow task runner: example_bash_operator this_will_skip 2021-06-24T00:39:06.539715+00:00 92 root 19057 0.1 0.8 466084 98760 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'also_run_this', '2021-06-24T00:39:06.539715+00:00', '--loc root 19242 26.6 0.8 470824 104404 pts/2 R+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'also_run_this', '2021-06-24T00:39:06.539715+00:00', '--local', root 19249 0.0 0.8 470824 101976 pts/2 S 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'also_run_this', '2021-06-24T00:39:06.539715+00:00', '--loc root 19062 0.0 0.8 466084 98300 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor root 19066 0.0 0.8 466084 98300 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor root 19069 0.0 0.8 466084 98300 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor root 19075 2.7 0.8 466596 100144 pts/2 S 00:38 0:00 \_ airflow scheduler -- DagFileProcessorManager ``` Once the worker is done executing a task, the worker is renamed back to `airflow worker -- LocalExecutor` (cherry picked from commit c8a628a)
kaxil
added a commit
that referenced
this pull request
Aug 17, 2021
This has annoyed me for a long time. When using ``LocalExecutor``, it was difficult to see which process is a worker as it just showed up as below -- which had same title as parent scheduler process. This PR/commit adds a title for idle workers and when a task is running it has the "command" that is running in the title, similar to our supervising process Before: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 1449 0.8 2.6 988356 326312 pts/1 Sl+ Jun23 0:16 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 1584 0.0 0.4 121068 56864 pts/1 S+ Jun23 0:01 | \_ gunicorn: master [airflow-webserver] root 1587 0.6 2.5 986144 318712 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1588 0.6 2.5 984776 317672 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1589 0.6 2.5 985688 318148 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1590 0.6 2.5 985200 317776 pts/1 Sl+ Jun23 0:11 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 13933 31.0 0.9 466596 117656 pts/2 S+ 00:22 0:01 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13941 0.0 0.7 466340 97988 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13942 3.2 0.8 1392072 100136 pts/2 Sl+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13950 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13952 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13955 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13958 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13962 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13966 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13969 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13975 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13979 6.5 0.8 466596 99956 pts/2 S 00:22 0:00 \_ airflow scheduler -- DagFileProcessorManager ``` After (with no running tasks - idle workers): ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 1449 0.8 2.6 988356 326312 pts/1 Sl+ Jun23 0:16 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 1584 0.0 0.4 121068 56864 pts/1 S+ Jun23 0:01 | \_ gunicorn: master [airflow-webserver] root 1587 0.6 2.5 985752 318184 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1588 0.6 2.5 984776 317672 pts/1 Sl+ Jun23 0:11 | \_ [ready] gunicorn: worker [airflow-webserver] root 1589 0.6 2.5 985688 318148 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1590 0.6 2.5 985200 317776 pts/1 Sl+ Jun23 0:11 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 13237 25.7 0.9 466596 117692 pts/2 S+ 00:20 0:02 \_ airflow worker -- LocalExecutor root 13245 0.1 0.7 466340 97804 pts/2 S+ 00:20 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13246 2.1 0.8 1318340 100104 pts/2 Sl+ 00:20 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13254 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13256 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13259 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13263 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13267 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13271 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13274 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13276 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13282 4.1 0.8 466596 99952 pts/2 S 00:20 0:00 \_ airflow scheduler -- DagFileProcessorManager ``` After (with running tasks): ``` root@a7c8aa590704:/opt/airflow# ps auxf USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 6434 0.0 0.0 6652 4584 pts/3 Ss 00:01 0:00 /bin/bash root 19250 0.0 0.0 9556 3064 pts/3 R+ 00:39 0:00 \_ ps auxf root 1 0.0 0.0 2148 720 ? Ss Jun23 0:00 /usr/bin/dumb-init -- /entrypoint root 7 0.0 0.0 6656 4400 pts/0 Ss Jun23 0:00 /bin/bash root 121 0.0 0.0 8220 3228 pts/0 S+ Jun23 0:00 \_ tmux root 101 0.0 0.0 15856 4272 ? Ss Jun23 0:00 /usr/sbin/sshd root 123 0.0 0.0 10176 5148 ? Ss Jun23 0:00 tmux root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 1449 0.6 2.6 988356 326312 pts/1 Sl+ Jun23 0:20 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 1584 0.0 0.4 121068 56864 pts/1 S+ Jun23 0:01 | \_ gunicorn: master [airflow-webserver] root 1587 0.4 2.5 986144 318712 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1588 0.4 2.5 984776 317672 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1589 0.4 2.5 985848 318600 pts/1 Sl+ Jun23 0:13 | \_ [ready] gunicorn: worker [airflow-webserver] root 1590 0.4 2.5 985628 318424 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 19030 17.9 0.9 467108 118628 pts/2 S+ 00:38 0:02 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 19038 0.0 0.7 466084 97776 pts/2 S+ 00:38 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 19039 1.4 0.8 1318084 99804 pts/2 Sl+ 00:38 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 19047 0.0 0.8 466084 98692 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_2', '2021-06-24T00:39:06.539715+00:00', '--local', ' root 19240 25.3 0.8 470820 104400 pts/2 S+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_2', '2021-06-24T00:39:06.539715+00:00', '--local', '--po root 19246 0.0 0.8 470956 103980 pts/2 S 00:39 0:00 | \_ airflow task runner: example_bash_operator runme_2 2021-06-24T00:39:06.539715+00:00 91 root 19049 0.1 0.8 466084 98696 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_1', '2021-06-24T00:39:06.539715+00:00', '--local', ' root 19241 26.0 0.8 470824 104408 pts/2 S+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_1', '2021-06-24T00:39:06.539715+00:00', '--local', '--po root 19248 0.0 0.8 470824 103720 pts/2 S 00:39 0:00 | \_ airflow task runner: example_bash_operator runme_1 2021-06-24T00:39:06.539715+00:00 93 root 19052 0.1 0.8 466084 98760 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_0', '2021-06-24T00:39:06.539715+00:00', '--local', ' root 19244 26.0 0.8 470824 104404 pts/2 S+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_0', '2021-06-24T00:39:06.539715+00:00', '--local', '--po root 19245 0.0 0.8 471212 104032 pts/2 S 00:39 0:00 | \_ airflow task runner: example_bash_operator runme_0 2021-06-24T00:39:06.539715+00:00 90 root 19056 0.1 0.8 466084 98760 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'this_will_skip', '2021-06-24T00:39:06.539715+00:00', '--lo root 19243 24.6 0.8 470824 104400 pts/2 S+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'this_will_skip', '2021-06-24T00:39:06.539715+00:00', '--local' root 19247 0.0 0.8 470956 103712 pts/2 S 00:39 0:00 | \_ airflow task runner: example_bash_operator this_will_skip 2021-06-24T00:39:06.539715+00:00 92 root 19057 0.1 0.8 466084 98760 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'also_run_this', '2021-06-24T00:39:06.539715+00:00', '--loc root 19242 26.6 0.8 470824 104404 pts/2 R+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'also_run_this', '2021-06-24T00:39:06.539715+00:00', '--local', root 19249 0.0 0.8 470824 101976 pts/2 S 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'also_run_this', '2021-06-24T00:39:06.539715+00:00', '--loc root 19062 0.0 0.8 466084 98300 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor root 19066 0.0 0.8 466084 98300 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor root 19069 0.0 0.8 466084 98300 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor root 19075 2.7 0.8 466596 100144 pts/2 S 00:38 0:00 \_ airflow scheduler -- DagFileProcessorManager ``` Once the worker is done executing a task, the worker is renamed back to `airflow worker -- LocalExecutor` (cherry picked from commit c8a628a)
jhtimmins
pushed a commit
that referenced
this pull request
Aug 17, 2021
This has annoyed me for a long time. When using ``LocalExecutor``, it was difficult to see which process is a worker as it just showed up as below -- which had same title as parent scheduler process. This PR/commit adds a title for idle workers and when a task is running it has the "command" that is running in the title, similar to our supervising process Before: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 1449 0.8 2.6 988356 326312 pts/1 Sl+ Jun23 0:16 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 1584 0.0 0.4 121068 56864 pts/1 S+ Jun23 0:01 | \_ gunicorn: master [airflow-webserver] root 1587 0.6 2.5 986144 318712 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1588 0.6 2.5 984776 317672 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1589 0.6 2.5 985688 318148 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1590 0.6 2.5 985200 317776 pts/1 Sl+ Jun23 0:11 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 13933 31.0 0.9 466596 117656 pts/2 S+ 00:22 0:01 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13941 0.0 0.7 466340 97988 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13942 3.2 0.8 1392072 100136 pts/2 Sl+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13950 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13952 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13955 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13958 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13962 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13966 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13969 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13975 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13979 6.5 0.8 466596 99956 pts/2 S 00:22 0:00 \_ airflow scheduler -- DagFileProcessorManager ``` After (with no running tasks - idle workers): ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 1449 0.8 2.6 988356 326312 pts/1 Sl+ Jun23 0:16 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 1584 0.0 0.4 121068 56864 pts/1 S+ Jun23 0:01 | \_ gunicorn: master [airflow-webserver] root 1587 0.6 2.5 985752 318184 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1588 0.6 2.5 984776 317672 pts/1 Sl+ Jun23 0:11 | \_ [ready] gunicorn: worker [airflow-webserver] root 1589 0.6 2.5 985688 318148 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1590 0.6 2.5 985200 317776 pts/1 Sl+ Jun23 0:11 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 13237 25.7 0.9 466596 117692 pts/2 S+ 00:20 0:02 \_ airflow worker -- LocalExecutor root 13245 0.1 0.7 466340 97804 pts/2 S+ 00:20 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13246 2.1 0.8 1318340 100104 pts/2 Sl+ 00:20 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 13254 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13256 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13259 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13263 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13267 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13271 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13274 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13276 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor root 13282 4.1 0.8 466596 99952 pts/2 S 00:20 0:00 \_ airflow scheduler -- DagFileProcessorManager ``` After (with running tasks): ``` root@a7c8aa590704:/opt/airflow# ps auxf USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 6434 0.0 0.0 6652 4584 pts/3 Ss 00:01 0:00 /bin/bash root 19250 0.0 0.0 9556 3064 pts/3 R+ 00:39 0:00 \_ ps auxf root 1 0.0 0.0 2148 720 ? Ss Jun23 0:00 /usr/bin/dumb-init -- /entrypoint root 7 0.0 0.0 6656 4400 pts/0 Ss Jun23 0:00 /bin/bash root 121 0.0 0.0 8220 3228 pts/0 S+ Jun23 0:00 \_ tmux root 101 0.0 0.0 15856 4272 ? Ss Jun23 0:00 /usr/sbin/sshd root 123 0.0 0.0 10176 5148 ? Ss Jun23 0:00 tmux root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 1449 0.6 2.6 988356 326312 pts/1 Sl+ Jun23 0:20 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 1584 0.0 0.4 121068 56864 pts/1 S+ Jun23 0:01 | \_ gunicorn: master [airflow-webserver] root 1587 0.4 2.5 986144 318712 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1588 0.4 2.5 984776 317672 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 1589 0.4 2.5 985848 318600 pts/1 Sl+ Jun23 0:13 | \_ [ready] gunicorn: worker [airflow-webserver] root 1590 0.4 2.5 985628 318424 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 19030 17.9 0.9 467108 118628 pts/2 S+ 00:38 0:02 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 19038 0.0 0.7 466084 97776 pts/2 S+ 00:38 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 19039 1.4 0.8 1318084 99804 pts/2 Sl+ 00:38 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 19047 0.0 0.8 466084 98692 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_2', '2021-06-24T00:39:06.539715+00:00', '--local', ' root 19240 25.3 0.8 470820 104400 pts/2 S+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_2', '2021-06-24T00:39:06.539715+00:00', '--local', '--po root 19246 0.0 0.8 470956 103980 pts/2 S 00:39 0:00 | \_ airflow task runner: example_bash_operator runme_2 2021-06-24T00:39:06.539715+00:00 91 root 19049 0.1 0.8 466084 98696 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_1', '2021-06-24T00:39:06.539715+00:00', '--local', ' root 19241 26.0 0.8 470824 104408 pts/2 S+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_1', '2021-06-24T00:39:06.539715+00:00', '--local', '--po root 19248 0.0 0.8 470824 103720 pts/2 S 00:39 0:00 | \_ airflow task runner: example_bash_operator runme_1 2021-06-24T00:39:06.539715+00:00 93 root 19052 0.1 0.8 466084 98760 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_0', '2021-06-24T00:39:06.539715+00:00', '--local', ' root 19244 26.0 0.8 470824 104404 pts/2 S+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_0', '2021-06-24T00:39:06.539715+00:00', '--local', '--po root 19245 0.0 0.8 471212 104032 pts/2 S 00:39 0:00 | \_ airflow task runner: example_bash_operator runme_0 2021-06-24T00:39:06.539715+00:00 90 root 19056 0.1 0.8 466084 98760 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'this_will_skip', '2021-06-24T00:39:06.539715+00:00', '--lo root 19243 24.6 0.8 470824 104400 pts/2 S+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'this_will_skip', '2021-06-24T00:39:06.539715+00:00', '--local' root 19247 0.0 0.8 470956 103712 pts/2 S 00:39 0:00 | \_ airflow task runner: example_bash_operator this_will_skip 2021-06-24T00:39:06.539715+00:00 92 root 19057 0.1 0.8 466084 98760 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'also_run_this', '2021-06-24T00:39:06.539715+00:00', '--loc root 19242 26.6 0.8 470824 104404 pts/2 R+ 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'also_run_this', '2021-06-24T00:39:06.539715+00:00', '--local', root 19249 0.0 0.8 470824 101976 pts/2 S 00:39 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'also_run_this', '2021-06-24T00:39:06.539715+00:00', '--loc root 19062 0.0 0.8 466084 98300 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor root 19066 0.0 0.8 466084 98300 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor root 19069 0.0 0.8 466084 98300 pts/2 S+ 00:38 0:00 \_ airflow worker -- LocalExecutor root 19075 2.7 0.8 466596 100144 pts/2 S 00:38 0:00 \_ airflow scheduler -- DagFileProcessorManager ``` Once the worker is done executing a task, the worker is renamed back to `airflow worker -- LocalExecutor` (cherry picked from commit c8a628a)
leahecole
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Mar 10, 2022
Follow up of apache/airflow#16623. This PR/commits adds title to serve-logs command and multiprocessing manager for LocalExecutor. The serve-logs process is on celery worker when using CeleryExecutor but for LocalExecutor, it is a separate process in Scheduler. **Before**: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 25299 25.3 2.6 988372 326344 pts/1 Sl+ 01:30 0:09 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 25510 3.6 0.4 121068 57152 pts/1 S+ 01:31 0:00 | \_ gunicorn: master [airflow-webserver] root 25555 35.7 2.5 983584 316564 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25556 35.7 2.5 983840 316684 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25557 35.5 2.5 983840 316548 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25558 37.2 2.5 984920 317700 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 25090 5.8 0.9 467508 118808 pts/2 S+ 01:30 0:03 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25098 0.0 0.7 466080 97800 pts/2 S+ 01:30 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25099 0.4 0.8 1391812 99788 pts/2 Sl+ 01:30 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25107 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25109 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25114 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25117 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25120 0.0 0.8 466080 98552 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25125 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--lo root 26139 0.0 0.8 468988 102204 pts/2 S+ 01:31 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--local' root 25128 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25132 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25137 1.1 0.8 466592 100016 pts/2 S 01:30 0:00 \_ airflow scheduler -- DagFileProcessorManager root@a7c8aa590704:/opt/airflow# ps aux ``` **After**: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 25299 25.3 2.6 988372 326344 pts/1 Sl+ 01:30 0:09 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 25510 3.6 0.4 121068 57152 pts/1 S+ 01:31 0:00 | \_ gunicorn: master [airflow-webserver] root 25555 35.7 2.5 983584 316564 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25556 35.7 2.5 983840 316684 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25557 35.5 2.5 983840 316548 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25558 37.2 2.5 984920 317700 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 25090 5.8 0.9 467508 118808 pts/2 S+ 01:30 0:03 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25098 0.0 0.7 466080 97800 pts/2 S+ 01:30 0:00 \_ airflow serve-logs root 25099 0.4 0.8 1391812 99788 pts/2 Sl+ 01:30 0:00 \_ airflow executor -- LocalExecutor root 25107 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25109 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25114 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25117 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25120 0.0 0.8 466080 98552 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25125 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--lo root 26139 0.0 0.8 468988 102204 pts/2 S+ 01:31 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--local' root 25128 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25132 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25137 1.1 0.8 466592 100016 pts/2 S 01:30 0:00 \_ airflow scheduler -- DagFileProcessorManager root@a7c8aa590704:/opt/airflow# ps aux ``` GitOrigin-RevId: 1b2535e442599d4607eb6497864af26ad5afa807
leahecole
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Jun 4, 2022
Follow up of apache/airflow#16623. This PR/commits adds title to serve-logs command and multiprocessing manager for LocalExecutor. The serve-logs process is on celery worker when using CeleryExecutor but for LocalExecutor, it is a separate process in Scheduler. **Before**: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 25299 25.3 2.6 988372 326344 pts/1 Sl+ 01:30 0:09 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 25510 3.6 0.4 121068 57152 pts/1 S+ 01:31 0:00 | \_ gunicorn: master [airflow-webserver] root 25555 35.7 2.5 983584 316564 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25556 35.7 2.5 983840 316684 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25557 35.5 2.5 983840 316548 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25558 37.2 2.5 984920 317700 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 25090 5.8 0.9 467508 118808 pts/2 S+ 01:30 0:03 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25098 0.0 0.7 466080 97800 pts/2 S+ 01:30 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25099 0.4 0.8 1391812 99788 pts/2 Sl+ 01:30 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25107 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25109 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25114 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25117 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25120 0.0 0.8 466080 98552 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25125 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--lo root 26139 0.0 0.8 468988 102204 pts/2 S+ 01:31 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--local' root 25128 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25132 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25137 1.1 0.8 466592 100016 pts/2 S 01:30 0:00 \_ airflow scheduler -- DagFileProcessorManager root@a7c8aa590704:/opt/airflow# ps aux ``` **After**: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 25299 25.3 2.6 988372 326344 pts/1 Sl+ 01:30 0:09 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 25510 3.6 0.4 121068 57152 pts/1 S+ 01:31 0:00 | \_ gunicorn: master [airflow-webserver] root 25555 35.7 2.5 983584 316564 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25556 35.7 2.5 983840 316684 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25557 35.5 2.5 983840 316548 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25558 37.2 2.5 984920 317700 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 25090 5.8 0.9 467508 118808 pts/2 S+ 01:30 0:03 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25098 0.0 0.7 466080 97800 pts/2 S+ 01:30 0:00 \_ airflow serve-logs root 25099 0.4 0.8 1391812 99788 pts/2 Sl+ 01:30 0:00 \_ airflow executor -- LocalExecutor root 25107 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25109 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25114 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25117 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25120 0.0 0.8 466080 98552 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25125 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--lo root 26139 0.0 0.8 468988 102204 pts/2 S+ 01:31 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--local' root 25128 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25132 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25137 1.1 0.8 466592 100016 pts/2 S 01:30 0:00 \_ airflow scheduler -- DagFileProcessorManager root@a7c8aa590704:/opt/airflow# ps aux ``` GitOrigin-RevId: 1b2535e442599d4607eb6497864af26ad5afa807
kosteev
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Jul 10, 2022
Follow up of apache/airflow#16623. This PR/commits adds title to serve-logs command and multiprocessing manager for LocalExecutor. The serve-logs process is on celery worker when using CeleryExecutor but for LocalExecutor, it is a separate process in Scheduler. **Before**: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 25299 25.3 2.6 988372 326344 pts/1 Sl+ 01:30 0:09 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 25510 3.6 0.4 121068 57152 pts/1 S+ 01:31 0:00 | \_ gunicorn: master [airflow-webserver] root 25555 35.7 2.5 983584 316564 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25556 35.7 2.5 983840 316684 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25557 35.5 2.5 983840 316548 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25558 37.2 2.5 984920 317700 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 25090 5.8 0.9 467508 118808 pts/2 S+ 01:30 0:03 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25098 0.0 0.7 466080 97800 pts/2 S+ 01:30 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25099 0.4 0.8 1391812 99788 pts/2 Sl+ 01:30 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25107 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25109 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25114 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25117 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25120 0.0 0.8 466080 98552 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25125 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--lo root 26139 0.0 0.8 468988 102204 pts/2 S+ 01:31 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--local' root 25128 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25132 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25137 1.1 0.8 466592 100016 pts/2 S 01:30 0:00 \_ airflow scheduler -- DagFileProcessorManager root@a7c8aa590704:/opt/airflow# ps aux ``` **After**: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 25299 25.3 2.6 988372 326344 pts/1 Sl+ 01:30 0:09 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 25510 3.6 0.4 121068 57152 pts/1 S+ 01:31 0:00 | \_ gunicorn: master [airflow-webserver] root 25555 35.7 2.5 983584 316564 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25556 35.7 2.5 983840 316684 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25557 35.5 2.5 983840 316548 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25558 37.2 2.5 984920 317700 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 25090 5.8 0.9 467508 118808 pts/2 S+ 01:30 0:03 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25098 0.0 0.7 466080 97800 pts/2 S+ 01:30 0:00 \_ airflow serve-logs root 25099 0.4 0.8 1391812 99788 pts/2 Sl+ 01:30 0:00 \_ airflow executor -- LocalExecutor root 25107 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25109 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25114 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25117 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25120 0.0 0.8 466080 98552 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25125 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--lo root 26139 0.0 0.8 468988 102204 pts/2 S+ 01:31 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--local' root 25128 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25132 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25137 1.1 0.8 466592 100016 pts/2 S 01:30 0:00 \_ airflow scheduler -- DagFileProcessorManager root@a7c8aa590704:/opt/airflow# ps aux ``` GitOrigin-RevId: 1b2535e442599d4607eb6497864af26ad5afa807
leahecole
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Aug 27, 2022
Follow up of apache/airflow#16623. This PR/commits adds title to serve-logs command and multiprocessing manager for LocalExecutor. The serve-logs process is on celery worker when using CeleryExecutor but for LocalExecutor, it is a separate process in Scheduler. **Before**: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 25299 25.3 2.6 988372 326344 pts/1 Sl+ 01:30 0:09 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 25510 3.6 0.4 121068 57152 pts/1 S+ 01:31 0:00 | \_ gunicorn: master [airflow-webserver] root 25555 35.7 2.5 983584 316564 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25556 35.7 2.5 983840 316684 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25557 35.5 2.5 983840 316548 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25558 37.2 2.5 984920 317700 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 25090 5.8 0.9 467508 118808 pts/2 S+ 01:30 0:03 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25098 0.0 0.7 466080 97800 pts/2 S+ 01:30 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25099 0.4 0.8 1391812 99788 pts/2 Sl+ 01:30 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25107 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25109 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25114 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25117 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25120 0.0 0.8 466080 98552 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25125 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--lo root 26139 0.0 0.8 468988 102204 pts/2 S+ 01:31 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--local' root 25128 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25132 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25137 1.1 0.8 466592 100016 pts/2 S 01:30 0:00 \_ airflow scheduler -- DagFileProcessorManager root@a7c8aa590704:/opt/airflow# ps aux ``` **After**: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 25299 25.3 2.6 988372 326344 pts/1 Sl+ 01:30 0:09 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 25510 3.6 0.4 121068 57152 pts/1 S+ 01:31 0:00 | \_ gunicorn: master [airflow-webserver] root 25555 35.7 2.5 983584 316564 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25556 35.7 2.5 983840 316684 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25557 35.5 2.5 983840 316548 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25558 37.2 2.5 984920 317700 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 25090 5.8 0.9 467508 118808 pts/2 S+ 01:30 0:03 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25098 0.0 0.7 466080 97800 pts/2 S+ 01:30 0:00 \_ airflow serve-logs root 25099 0.4 0.8 1391812 99788 pts/2 Sl+ 01:30 0:00 \_ airflow executor -- LocalExecutor root 25107 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25109 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25114 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25117 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25120 0.0 0.8 466080 98552 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25125 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--lo root 26139 0.0 0.8 468988 102204 pts/2 S+ 01:31 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--local' root 25128 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25132 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25137 1.1 0.8 466592 100016 pts/2 S 01:30 0:00 \_ airflow scheduler -- DagFileProcessorManager root@a7c8aa590704:/opt/airflow# ps aux ``` GitOrigin-RevId: 1b2535e442599d4607eb6497864af26ad5afa807
leahecole
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Oct 4, 2022
Follow up of apache/airflow#16623. This PR/commits adds title to serve-logs command and multiprocessing manager for LocalExecutor. The serve-logs process is on celery worker when using CeleryExecutor but for LocalExecutor, it is a separate process in Scheduler. **Before**: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 25299 25.3 2.6 988372 326344 pts/1 Sl+ 01:30 0:09 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 25510 3.6 0.4 121068 57152 pts/1 S+ 01:31 0:00 | \_ gunicorn: master [airflow-webserver] root 25555 35.7 2.5 983584 316564 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25556 35.7 2.5 983840 316684 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25557 35.5 2.5 983840 316548 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25558 37.2 2.5 984920 317700 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 25090 5.8 0.9 467508 118808 pts/2 S+ 01:30 0:03 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25098 0.0 0.7 466080 97800 pts/2 S+ 01:30 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25099 0.4 0.8 1391812 99788 pts/2 Sl+ 01:30 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25107 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25109 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25114 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25117 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25120 0.0 0.8 466080 98552 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25125 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--lo root 26139 0.0 0.8 468988 102204 pts/2 S+ 01:31 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--local' root 25128 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25132 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25137 1.1 0.8 466592 100016 pts/2 S 01:30 0:00 \_ airflow scheduler -- DagFileProcessorManager root@a7c8aa590704:/opt/airflow# ps aux ``` **After**: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 25299 25.3 2.6 988372 326344 pts/1 Sl+ 01:30 0:09 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 25510 3.6 0.4 121068 57152 pts/1 S+ 01:31 0:00 | \_ gunicorn: master [airflow-webserver] root 25555 35.7 2.5 983584 316564 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25556 35.7 2.5 983840 316684 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25557 35.5 2.5 983840 316548 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25558 37.2 2.5 984920 317700 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 25090 5.8 0.9 467508 118808 pts/2 S+ 01:30 0:03 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25098 0.0 0.7 466080 97800 pts/2 S+ 01:30 0:00 \_ airflow serve-logs root 25099 0.4 0.8 1391812 99788 pts/2 Sl+ 01:30 0:00 \_ airflow executor -- LocalExecutor root 25107 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25109 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25114 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25117 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25120 0.0 0.8 466080 98552 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25125 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--lo root 26139 0.0 0.8 468988 102204 pts/2 S+ 01:31 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--local' root 25128 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25132 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25137 1.1 0.8 466592 100016 pts/2 S 01:30 0:00 \_ airflow scheduler -- DagFileProcessorManager root@a7c8aa590704:/opt/airflow# ps aux ``` GitOrigin-RevId: 1b2535e442599d4607eb6497864af26ad5afa807
aglipska
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Oct 7, 2022
Follow up of apache/airflow#16623. This PR/commits adds title to serve-logs command and multiprocessing manager for LocalExecutor. The serve-logs process is on celery worker when using CeleryExecutor but for LocalExecutor, it is a separate process in Scheduler. **Before**: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 25299 25.3 2.6 988372 326344 pts/1 Sl+ 01:30 0:09 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 25510 3.6 0.4 121068 57152 pts/1 S+ 01:31 0:00 | \_ gunicorn: master [airflow-webserver] root 25555 35.7 2.5 983584 316564 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25556 35.7 2.5 983840 316684 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25557 35.5 2.5 983840 316548 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25558 37.2 2.5 984920 317700 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 25090 5.8 0.9 467508 118808 pts/2 S+ 01:30 0:03 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25098 0.0 0.7 466080 97800 pts/2 S+ 01:30 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25099 0.4 0.8 1391812 99788 pts/2 Sl+ 01:30 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25107 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25109 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25114 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25117 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25120 0.0 0.8 466080 98552 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25125 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--lo root 26139 0.0 0.8 468988 102204 pts/2 S+ 01:31 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--local' root 25128 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25132 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25137 1.1 0.8 466592 100016 pts/2 S 01:30 0:00 \_ airflow scheduler -- DagFileProcessorManager root@a7c8aa590704:/opt/airflow# ps aux ``` **After**: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 25299 25.3 2.6 988372 326344 pts/1 Sl+ 01:30 0:09 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 25510 3.6 0.4 121068 57152 pts/1 S+ 01:31 0:00 | \_ gunicorn: master [airflow-webserver] root 25555 35.7 2.5 983584 316564 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25556 35.7 2.5 983840 316684 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25557 35.5 2.5 983840 316548 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25558 37.2 2.5 984920 317700 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 25090 5.8 0.9 467508 118808 pts/2 S+ 01:30 0:03 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25098 0.0 0.7 466080 97800 pts/2 S+ 01:30 0:00 \_ airflow serve-logs root 25099 0.4 0.8 1391812 99788 pts/2 Sl+ 01:30 0:00 \_ airflow executor -- LocalExecutor root 25107 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25109 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25114 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25117 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25120 0.0 0.8 466080 98552 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25125 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--lo root 26139 0.0 0.8 468988 102204 pts/2 S+ 01:31 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--local' root 25128 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25132 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25137 1.1 0.8 466592 100016 pts/2 S 01:30 0:00 \_ airflow scheduler -- DagFileProcessorManager root@a7c8aa590704:/opt/airflow# ps aux ``` GitOrigin-RevId: 1b2535e442599d4607eb6497864af26ad5afa807
leahecole
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Dec 7, 2022
Follow up of apache/airflow#16623. This PR/commits adds title to serve-logs command and multiprocessing manager for LocalExecutor. The serve-logs process is on celery worker when using CeleryExecutor but for LocalExecutor, it is a separate process in Scheduler. **Before**: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 25299 25.3 2.6 988372 326344 pts/1 Sl+ 01:30 0:09 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 25510 3.6 0.4 121068 57152 pts/1 S+ 01:31 0:00 | \_ gunicorn: master [airflow-webserver] root 25555 35.7 2.5 983584 316564 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25556 35.7 2.5 983840 316684 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25557 35.5 2.5 983840 316548 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25558 37.2 2.5 984920 317700 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 25090 5.8 0.9 467508 118808 pts/2 S+ 01:30 0:03 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25098 0.0 0.7 466080 97800 pts/2 S+ 01:30 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25099 0.4 0.8 1391812 99788 pts/2 Sl+ 01:30 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25107 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25109 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25114 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25117 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25120 0.0 0.8 466080 98552 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25125 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--lo root 26139 0.0 0.8 468988 102204 pts/2 S+ 01:31 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--local' root 25128 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25132 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25137 1.1 0.8 466592 100016 pts/2 S 01:30 0:00 \_ airflow scheduler -- DagFileProcessorManager root@a7c8aa590704:/opt/airflow# ps aux ``` **After**: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 25299 25.3 2.6 988372 326344 pts/1 Sl+ 01:30 0:09 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 25510 3.6 0.4 121068 57152 pts/1 S+ 01:31 0:00 | \_ gunicorn: master [airflow-webserver] root 25555 35.7 2.5 983584 316564 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25556 35.7 2.5 983840 316684 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25557 35.5 2.5 983840 316548 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25558 37.2 2.5 984920 317700 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 25090 5.8 0.9 467508 118808 pts/2 S+ 01:30 0:03 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25098 0.0 0.7 466080 97800 pts/2 S+ 01:30 0:00 \_ airflow serve-logs root 25099 0.4 0.8 1391812 99788 pts/2 Sl+ 01:30 0:00 \_ airflow executor -- LocalExecutor root 25107 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25109 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25114 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25117 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25120 0.0 0.8 466080 98552 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25125 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--lo root 26139 0.0 0.8 468988 102204 pts/2 S+ 01:31 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--local' root 25128 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25132 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25137 1.1 0.8 466592 100016 pts/2 S 01:30 0:00 \_ airflow scheduler -- DagFileProcessorManager root@a7c8aa590704:/opt/airflow# ps aux ``` GitOrigin-RevId: 1b2535e442599d4607eb6497864af26ad5afa807
leahecole
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Jan 27, 2023
Follow up of apache/airflow#16623. This PR/commits adds title to serve-logs command and multiprocessing manager for LocalExecutor. The serve-logs process is on celery worker when using CeleryExecutor but for LocalExecutor, it is a separate process in Scheduler. **Before**: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 25299 25.3 2.6 988372 326344 pts/1 Sl+ 01:30 0:09 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 25510 3.6 0.4 121068 57152 pts/1 S+ 01:31 0:00 | \_ gunicorn: master [airflow-webserver] root 25555 35.7 2.5 983584 316564 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25556 35.7 2.5 983840 316684 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25557 35.5 2.5 983840 316548 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25558 37.2 2.5 984920 317700 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 25090 5.8 0.9 467508 118808 pts/2 S+ 01:30 0:03 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25098 0.0 0.7 466080 97800 pts/2 S+ 01:30 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25099 0.4 0.8 1391812 99788 pts/2 Sl+ 01:30 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25107 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25109 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25114 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25117 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25120 0.0 0.8 466080 98552 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25125 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--lo root 26139 0.0 0.8 468988 102204 pts/2 S+ 01:31 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--local' root 25128 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25132 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25137 1.1 0.8 466592 100016 pts/2 S 01:30 0:00 \_ airflow scheduler -- DagFileProcessorManager root@a7c8aa590704:/opt/airflow# ps aux ``` **After**: ``` root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash root 25299 25.3 2.6 988372 326344 pts/1 Sl+ 01:30 0:09 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver root 25510 3.6 0.4 121068 57152 pts/1 S+ 01:31 0:00 | \_ gunicorn: master [airflow-webserver] root 25555 35.7 2.5 983584 316564 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25556 35.7 2.5 983840 316684 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25557 35.5 2.5 983840 316548 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 25558 37.2 2.5 984920 317700 pts/1 Sl+ 01:31 0:08 | \_ [ready] gunicorn: worker [airflow-webserver] root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash root 25090 5.8 0.9 467508 118808 pts/2 S+ 01:30 0:03 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler root 25098 0.0 0.7 466080 97800 pts/2 S+ 01:30 0:00 \_ airflow serve-logs root 25099 0.4 0.8 1391812 99788 pts/2 Sl+ 01:30 0:00 \_ airflow executor -- LocalExecutor root 25107 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25109 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25114 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25117 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25120 0.0 0.8 466080 98552 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25125 0.0 0.8 466080 98548 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--lo root 26139 0.0 0.8 468988 102204 pts/2 S+ 01:31 0:00 | \_ airflow task supervisor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T01:31:30.507415+00:00', '--local' root 25128 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25132 0.0 0.7 466080 98076 pts/2 S+ 01:30 0:00 \_ airflow worker -- LocalExecutor root 25137 1.1 0.8 466592 100016 pts/2 S 01:30 0:00 \_ airflow scheduler -- DagFileProcessorManager root@a7c8aa590704:/opt/airflow# ps aux ``` GitOrigin-RevId: 1b2535e442599d4607eb6497864af26ad5afa807
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This has annoyed me for a long time. When using
LocalExecutor, it was difficult to see which process is a worker as it just showed up as below -- which had same title as parent scheduler process. This PR/commit adds a title for idle workers and when a task is running it has the "command" that is running in the title, similar to our supervising processBefore:
After (with no running tasks - idle workers):
After (with running tasks):
Once the worker is done executing a task, the worker is renamed back to
airflow worker -- LocalExecutor^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.