Deadlock scenario in process._remote_check due to use of sub-process stdout/stderr pipes and read methods

[cfsnyder]

The use of read methods on subprocess stdin/stderr pipes can cause deadlocks. The following warning is noted in the python docs:

Warning Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.

This can happen in process.py._remote_check.

In this particular case, a deadlock occurs when the sub-process fills the OS stderr pipe buffer and is blocked while attempting to continue writing to stderr. In that case, stdout never reaches EOF and _remote_check is blocked trying to read from stdout.

This issue was discovered because it was causing a deadlock while attempting to upgrade a Ceph cluster from 15.2.10 -> 16.2.3. This deadlock brought our whole upgrade process to a halt.