Avoid forking HDFS connection with multiprocessing

[belldandyxtq]

Problem statement

After forking with prior HDFS calls in the main process, the program freezes at any future HDFS calls.
For example:

import chainerio
import multiprocessing


def func(hdfs_handler):
    # freeze here (2)
    print(list(hdfs_handler.list()))


hdfs_handler = chainerio.create_handler("hdfs")

# create HDFS connection interally (1)
print(list(hdfs_handler.list()))

p = multiprocessing.Process(target=func, args=(hdfs_handler, ))
p.start()
p.join()

Cause
Chainerio uses the pyarrow module to access the HDFS internally, and the pyarrow uses the HDFS Java module. The HDFS connection is pooled inside, and If the connection is first created (implicitly though calls to HDFS, like (1)) and then forked, it breaks the pooling and cause future HDFS calls to freeze.

Solution
Fork before the creation of HDFS connection, i.e. fork before any calls to the HDFS, e.g. (1).

[kuenishi]

Closing in favor of v2 API, which detects fork automatically with _check_fork() call, which raises exception.