Python 3.7 sh hangs when using in threads, forking and logging

[mzbuild]

BPO	37740
Nosy	@gpshead, @ned-deily, @ambv, @cagney

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-08-01.18:16:42.271>
created_at = <Date 2019-08-01.17:58:33.403>
labels = ['3.7', 'invalid', 'type-bug', 'library']
title = 'Python 3.7 sh hangs when using in threads, forking and logging'
updated_at = <Date 2019-08-01.18:16:42.267>
user = 'https://bugs.python.org/mzbuild'

bugs.python.org fields:

activity = <Date 2019-08-01.18:16:42.267>
actor = 'gregory.p.smith'
assignee = 'none'
closed = True
closed_date = <Date 2019-08-01.18:16:42.271>
closer = 'gregory.p.smith'
components = ['Library (Lib)']
creation = <Date 2019-08-01.17:58:33.403>
creator = 'mzbuild'
dependencies = []
files = []
hgrepos = []
issue_num = 37740
keywords = []
message_count = 2.0
messages = ['348869', '348870']
nosy_count = 7.0
nosy_names = ['gregory.p.smith', 'ned.deily', 'lukasz.langa', 'cagney', 'hugh', 'ebizzinfotech', 'mzbuild']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue37740'
versions = ['Python 3.7']

[mzbuild]

Hi,
We use the sh module in a threaded context to execute shell commands. When we were migrating to python 3 from python 2 we encountered some commands hanging. We created a test script that recreates the issue we are seeing.
`import sh
import logging
import concurrent.futures.thread

def execmd(exe):
print(sh.ls())
execution_pool = concurrent.futures.thread.ThreadPoolExecutor(20)
i = 0
thread_results = []
while i<500:
i+= 1
thread_results.append(execution_pool.map(execmd, ['ls']))

execution_pool.shutdown()

When running this in python 3.7 it hangs but in python 3.6 it works fine. We think it is releated to this issue https://bugs.python.org/issue36533. Installing the latest 3.7.4 didn't fix the issue. The sh module uses logging and forking and the top script uses threading so we think there is a locking issue with 3.7. Any help would be great.

[gpshead]

os.fork() cannot be safely used in an application that uses threads in any manner. This is not something Python can fix. This is a POSIX limitation.

The "sh" module on PyPI is incompatible with threaded applications on POSIX platforms due to its use of os.fork().

https://github.com/amoffat/sh/blob/master/sh.py#L1861

Raise this issue with them.