HDFS access within multiprocessing.Process can lead to deadlocks

[simleo]

The following code can hang indefinitely:

from multiprocessing import Process
from pydoop.hdfs.core import init
hdfs = init()

def connect():
    fs = None
    try:
        fs = hdfs.CoreHdfsFs("default", 0)
    finally:
        if fs:
            fs.close()

connect()
p = Process(target=connect)
p.start()
p.join()

The call to connect in the main program completes successfully, but the one spawned as a Process hangs indefinitely in:

hdfsBuilderConnect
  ==> invokeMethod(... "getDefaultUri" ...)
    ==> (*env)->CallStaticObjectMethodV

Note that spawning multiple processes does NOT lead to problems if HDFS is not accessed in the main program, i.e., the following is OK:

procs = [Process(target=connect) for _ in range(3)]
for p in procs:
    p.start()
for p in procs:
    p.join()

The issue also disappears if multiprocessing is replaced with threading:

from threading import Thread
[...]
connect()
t = Thread(target=connect)
t.start()
t.join()

[ben-davidson-6]

changing the way you start the process fixes this problem. See here dask/hdfs3#100. In short if you add

import multiprocessing as mp

...

if __name__ == '__main__':
    mp.set_start_method('forkserver')
    ...

it works.

[simleo]

Thanks for the tip @ben-davidson-6! It's been a while, but I could not even reproduce the issue (it's working for me with all start methods). Anyway, closing this since there's a workaround.