[AIRFLOW-2733] Reconcile psutil and subprocess in webserver cli

[gwax]

Make sure you have checked all steps below.

JIRA

• My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
  • https://issues.apache.org/jira/browse/AIRFLOW-2733

Description

• Here are some details about my PR, including screenshots of any UI changes:
  • use psutil.Popen to reconcile psutil and subprocess usage

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

• My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.

Code Quality

• Passes git diff upstream/master -u -- "*.py" | flake8 --diff

[codecov-io]

Codecov Report

Merging #3586 into master will increase coverage by 0.88%.
The diff coverage is 50%.

@@            Coverage Diff             @@
##           master    #3586      +/-   ##
==========================================
+ Coverage   76.67%   77.56%   +0.88%     
==========================================
  Files         199      204       +5     
  Lines       16186    15767     -419     
==========================================
- Hits        12410    12229     -181     
+ Misses       3776     3538     -238

Impacted Files	Coverage Δ
airflow/bin/cli.py	64.43% <50%> (-0.4%)	⬇️
airflow/operators/slack_operator.py	0% <0%> (-97.37%)	⬇️
airflow/sensors/s3_key_sensor.py	31.03% <0%> (-68.97%)	⬇️
airflow/sensors/s3_prefix_sensor.py	41.17% <0%> (-58.83%)	⬇️
airflow/example_dags/example_python_operator.py	78.94% <0%> (-15.79%)	⬇️
airflow/utils/helpers.py	71.34% <0%> (-13.04%)	⬇️
airflow/hooks/mysql_hook.py	78% <0%> (-12%)	⬇️
airflow/sensors/sql_sensor.py	90.47% <0%> (-9.53%)	⬇️
airflow/utils/sqlalchemy.py	73.91% <0%> (-7.52%)	⬇️
airflow/configuration.py	83.95% <0%> (-5.47%)	⬇️
... and 94 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e703d6b...e7e5a68. Read the comment docs.

[Fokko]

I would suggest to move all the subprocess calls to psutil :-)

Beside the gunicorn.wait, is there a difference in behaviour?

[gwax]

In the usage of both of these objects, poll is the only method that is not shared between psutil.Process, psutil.Popen, and subprocess.Popen. I cannot find a common function for this purpose between psutil.Process and subprocess.Popen.

psutil.Popen is a convenience wrapper around subprocess.Popen that adds methods provided by psutil.Process.

As an alternative, in this case:

gunicorn_master_proc = psutil.Popen(run_args, close_fds=True)

could have been written as:

gunicorn_master_subproc = psutil.Popen(run_args, close_fds=True)
gunicorn_master_proc = psutil.Process(gunicorn_master_subproc.pid)

but that seems needlessly verbose.

I support the idea of migrating subprocess.Popen calls to psutil.Popen but feel that would be a larger undertaking and would prefer to do that as more of a search/replace separate from functional changes like this poll -> wait change.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.