[Python] Hdfs client isn't fork-safe

[asfimport]

Given the following script:

 

import multiprocessing as mp
import pyarrow as pa


def ls(h):
    print("calling ls")
    return h.ls("/tmp")


if __name__ == '__main__':
    h = pa.hdfs.connect()
    print("Using 'spawn'")
    pool = mp.get_context('spawn').Pool(2)
    results = pool.map(ls, [h, h])
    sol = h.ls("/tmp")
    for r in results:
        assert r == sol
    print("'spawn' succeeded\n")

    print("Using 'fork'")
    pool = mp.get_context('fork').Pool(2)
    results = pool.map(ls, [h, h])
    sol = h.ls("/tmp")
    for r in results:
        assert r == sol
    print("'fork' succeeded")

 

Results in the following output:

 

$ python test.py
Using 'spawn'
calling ls
calling ls
'spawn' succeeded

Using 'fork

 

The process then hangs, and I have to kill -9 the forked worker processes.

 

I'm unable to get the libhdfs3 driver to work, so I'm unsure if this is a problem with libhdfs or just arrow's use of it (a quick google search didn't turn up anything useful).

Reporter: Jim Crist / @jcrist

Related issues:

• [Python] pyarrow.hdfs.connect crashes when executed asynchronously in processes (supercedes)

_{Note: This issue was originally created as ARROW-2081. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
I think this has to do with the general policy around forking with an embedded JVM. It may not be supported, but I didn't turn up any immediate references

[asfimport]

Wes McKinney / @wesm:
Is there a way we can detect the fork in the child process(es) and at least avoid a hang or segfault?

[asfimport]

Antoine Pitrou / @pitrou:
For the record, if you want decent multiprocessing performance together with fork safety, I would suggest using the "forkserver" method, not "spawn".

(Note the C libhdfs3 library isn't fork-safe, so no need to try it out IMHO :-))

[asfimport]

Antoine Pitrou / @pitrou:
Closing as won't fix for several reasons:

1. we don't have any HDFS expertise among the developer team (AFAIK)
2. it is likely that the fork-safety issue resides in the underlying C library (libhdfs)
3. no interested party has been willing to investigate and propose a fix