Parallel build fails non-deterministically.

[asfimport]

The following script fails non-deterministically (most of the time) on my machine.

With make -j1 it seems to work (though I only tried a few times). With make -j8 it fails most of the time (though I have seen it succeed).

git clone https://github.com/apache/arrow.git
cd arrow/cpp
git checkout 8f386374eca26d0eebe562beac52fc75459f352c
mkdir release
cd release
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_FLAGS="-g" -DCMAKE_CXX_FLAGS="-g" -DARROW_BUILD_TESTS=OFF ..
make VERBOSE=1 -j8
cd ../../..

The output from both a successful run (arrow_build_make_j1.txt) and an unsuccessful run (arrow_build_make_j8.txt) are attached, but the error may be the following.

install: mkdir /Users/rkn/Workspace/testingarrow/arrow/cpp/release/jemalloc_ep-prefix/src/jemalloc_ep/dist: File exists
make[3]: *** [install_include] Error 71
make[3]: *** Waiting for unfinished jobs....

Environment: OS X 10.12.1
Reporter: Robert Nishihara / @robertnishihara
Assignee: Uwe Korn / @xhochy

Original Issue Attachments:

• arrow_build_make_j1.txt
• arrow_build_make_j8.txt

_{Note: This issue was originally created as ARROW-739. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
That's very odd. It seems that race conditions with external projects are possible. I see:

  if (JEMALLOC_VENDORED)
    add_dependencies(jemalloc_shared jemalloc_ep)
    add_dependencies(jemalloc_static jemalloc_ep)
  endif()

What happens if you remove the dependency for jemalloc_static?

[asfimport]

Wes McKinney / @wesm:
Workarounds are to pass -DARROW_JEMALLOC=off for now, or to install jemalloc on your system (brew install jemalloc)

[asfimport]

Robert Nishihara / @robertnishihara:
Thanks! Removing the line add_dependencies(jemalloc_static jemalloc_ep) worked the few times I tried it, but I haven't tried enough to say for sure.

[asfimport]

Wes McKinney / @wesm:
I have a better fix, I'm about to put up a patch

[asfimport]

Wes McKinney / @wesm:
PR: #453, let me know if that also works

[asfimport]

Uwe Korn / @xhochy:
PR #2: #456

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 456
#456