ARROW-4873: [C++] Clarify documentation about how to use external ARROW_PACKAGE_PREFIX while also using CONDA dependency resolution #3901

wesm · 2019-03-14T15:41:11Z

This clarifies the "hybrid" developer case where the library toolchain is maintained externally, but the Python bits are managed by an active conda environment.

wesm · 2019-03-14T19:43:51Z

Hm this seems to break the Windows builds. I'll take a closer look...

https://ci.appveyor.com/project/wesm/arrow/builds/23076536

bkietz · 2019-03-14T20:03:41Z

@wesm I'm seeing a similar error to this, it seems that some of the conda dependencies don't have a Find<dep>.cmake in $CONDA_PREFIX/share/cmake-3.13/Modules, so setting <dep>_ROOT isn't getting picked up by anything. This was (hackily) fixed by passing -D<dep>_DIR=$CONDA_PREFIX/<path to directory containing depConfig.cmake>

bkietz · 2019-03-14T20:24:45Z

In your case, it looks like FindDoubleConversion.cmake is looking for DoubleConversion_ROOT but ThirdPartyToolchain.cmake is setting double-conversion_ROOT

bkietz · 2019-03-14T20:25:47Z

@xhochy am I reading this wrong?

wesm · 2019-03-14T21:25:44Z

@xhochy This removes the logic that overrides the AUTO default dependency source when $CONDA_PREFIX is set. I think that we should make -DARROW_DEPENDENCY_SOURCE=CONDA explicit

wesm · 2019-03-14T21:27:08Z

@bkietz yeah that is probably the issue. Is there a JIRA about the double-conversion thing?

bkietz · 2019-03-14T23:03:16Z

@wesm no jira for double-conversion, I created 4879 for Flatbuffers (since that's what's happening to me)

wesm · 2019-03-14T23:57:24Z

The modules should be in $CONDA_PREFIX/lib/cmake or $CONDA_PREFIX/lib/pkgconfig not share/cmake-3.13. For me it seems to be building Flatbuffers from source. I'll comment in ARROW-4879

wesm · 2019-03-15T00:46:25Z

Hmm looks like the macOS build is finding the system boost =/

-- Boost version: 1.66.0
-- Found the following Boost libraries:
--   regex
--   system
--   filesystem
-- Boost include dir: /usr/local/include
-- Boost libraries: /usr/local/lib/libboost_regex-mt.dylib/usr/local/lib/libboost_system-mt.dylib/usr/local/lib/libboost_filesystem-mt.dylib
-- Added shared library dependency boost_system_shared: /usr/local/lib/libboost_system-mt.dylib
-- Added shared library dependency boost_filesystem_shared: /usr/local/lib/libboost_filesystem-mt.dylib
-- Added shared library dependency boost_regex_shared: /usr/local/lib/libboost_regex-mt.dylib

wesm · 2019-03-15T01:50:42Z

Here's an in-progress Appveyor build on my fork: https://ci.appveyor.com/project/wesm/arrow/builds/23089752

wesm · 2019-03-15T02:49:12Z

Needed to rebase to get the zstd fix. Will take this up in the morning

xhochy · 2019-03-15T09:27:26Z

cpp/CMakeLists.txt

  # * BREW: Use SYSTEM but search for select packages with brew.
-  if(NOT "$ENV{CONDA_PREFIX}" STREQUAL "")


I would actually like to keep this as the main strategy. Having conda packages and the build toolchain split is something that I would only advise to expert users since the compiler migration. With the new toolchain using system compilers, binutils, … is going to be a pain unless one is on a brand new system. I rather have CMake error out on missing conda packages than weird compilers as the binutils/compiler/ABI is not matching.

Alright, I can write -DARROW_DEPENDENCY_SOURCE=SYSTEM -DARROW_PACKAGE_PREFIX=$TOOLCHAIN in the meantime

xhochy · 2019-03-15T09:28:05Z

cpp/cmake_modules/ThirdpartyToolchain.cmake

 endif()

+message(STATUS "Using ${ARROW_ACTUAL_DEPENDENCY_SOURCE} approach to find dependencies")


Moving this message here, we should also add a messge when the CONDA_PREFIX was applied.

wesm · 2019-03-15T13:34:22Z

@xhochy done. I updated the PR title and JIRA to make this essentially a documentation issue

wesm · 2019-03-15T13:34:44Z

BTW I'm going to do a pass today on our developer docs and will incorporate this into them

bkietz · 2019-03-15T13:40:23Z

cpp/CMakeLists.txt

-  #   If your system packages are in general in a non-default location, you can
-  #   set a general default for the *_ROOT variables using ARROW_PACKAGE_PREFIX.
-  # * CONDA: Same as system but set all *_ROOT variables to ENV{CONDA_PREFIX}.
+  #   from the outside using the *_ROOT variables.  If your system packages are


nit:

Suggested change

# from the outside using the *_ROOT variables. If your system packages are

# using the environment by setting *_ROOT variables. If your system packages are

Is this right? What the documentation currently says is to do:

cmake .. -DGTest_ROOT=$GTEST_PATH

https://github.com/apache/arrow/blob/master/cpp/cmake_modules/ThirdpartyToolchain.cmake#L86

Actually both are working in CMake 3.11+, before that only -DGTest_ROOT=$GTEST_PATH works.

wesm · 2019-03-15T14:34:25Z

Looks like I have to rebase again. Is this OK to merge once the build runs?

xhochy

+1, LGTM, merge on green

wesm · 2019-03-15T15:14:15Z

Appveyor build in my fork: https://ci.appveyor.com/project/wesm/arrow/builds/23104032. I'll merge once this makes it past the 3rd and 4th builds

xhochy · 2019-03-15T17:09:46Z

@wesm This fails to find Boost in the toolchain build :(

wesm · 2019-03-15T17:29:45Z

Ugh, ok, having a look

…nment variables can be used with CMake >= 3.11 Change-Id: I6658d89c5ea24f1334161764f60bf43d5b30203f

wesm · 2019-03-15T17:33:52Z

Fixed, I had accidentally undone the \Library thing when I rebased

wesm · 2019-03-15T18:21:16Z

Appveyor build just started on my fork: https://ci.appveyor.com/project/wesm/arrow/builds/23109006

wesm · 2019-03-15T20:20:16Z

All systems go. Merging

wesm force-pushed the toolchain-option branch from b2c9a08 to a75990a Compare March 15, 2019 02:48

xhochy requested changes Mar 15, 2019

View reviewed changes

wesm force-pushed the toolchain-option branch from a75990a to d3a5ca5 Compare March 15, 2019 13:31

wesm changed the title ~~ARROW-4873: [C++] Do not override user-supplied ARROW_PACKAGE_PREFIX with ENV{CONDA_PREFIX} by default~~ ARROW-4873: [C++] Clarify documentation about how to use external ARROW_PACKAGE_PREFIX while also using CONDA dependency resolution Mar 15, 2019

bkietz reviewed Mar 15, 2019

View reviewed changes

wesm force-pushed the toolchain-option branch from f10abac to f37126b Compare March 15, 2019 14:40

xhochy approved these changes Mar 15, 2019

View reviewed changes

wesm added 3 commits March 15, 2019 12:30

Clarify how to use external package toolchain and conda together

cf4c3f4

Add additional clarification for dependency sources, note that enviro…

889f846

…nment variables can be used with CMake >= 3.11 Change-Id: I6658d89c5ea24f1334161764f60bf43d5b30203f

Fix rebase artifact to restore \Library on MSVC

9cff629

wesm force-pushed the toolchain-option branch from f37126b to 9cff629 Compare March 15, 2019 17:33

wesm closed this in 359ba42 Mar 15, 2019

wesm deleted the toolchain-option branch March 15, 2019 20:21

asfimport mentioned this pull request Mar 15, 2019

[C++] Clarify documentation about how to use external ARROW_PACKAGE_PREFIX while also using CONDA dependency resolution #21383

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARROW-4873: [C++] Clarify documentation about how to use external ARROW_PACKAGE_PREFIX while also using CONDA dependency resolution #3901

ARROW-4873: [C++] Clarify documentation about how to use external ARROW_PACKAGE_PREFIX while also using CONDA dependency resolution #3901

wesm commented Mar 14, 2019 •

edited

wesm commented Mar 14, 2019

bkietz commented Mar 14, 2019 •

edited

bkietz commented Mar 14, 2019

bkietz commented Mar 14, 2019

wesm commented Mar 14, 2019

wesm commented Mar 14, 2019

bkietz commented Mar 14, 2019

wesm commented Mar 14, 2019

wesm commented Mar 15, 2019

wesm commented Mar 15, 2019

wesm commented Mar 15, 2019

xhochy Mar 15, 2019

wesm Mar 15, 2019

xhochy Mar 15, 2019

wesm commented Mar 15, 2019

wesm commented Mar 15, 2019

bkietz Mar 15, 2019

wesm Mar 15, 2019

xhochy Mar 15, 2019

wesm commented Mar 15, 2019

xhochy left a comment

wesm commented Mar 15, 2019

xhochy commented Mar 15, 2019

wesm commented Mar 15, 2019

wesm commented Mar 15, 2019

wesm commented Mar 15, 2019

wesm commented Mar 15, 2019

		# * BREW: Use SYSTEM but search for select packages with brew.
		if(NOT "$ENV{CONDA_PREFIX}" STREQUAL "")

		endif()

		message(STATUS "Using ${ARROW_ACTUAL_DEPENDENCY_SOURCE} approach to find dependencies")

	# from the outside using the *_ROOT variables. If your system packages are
	# using the environment by setting *_ROOT variables. If your system packages are

ARROW-4873: [C++] Clarify documentation about how to use external ARROW_PACKAGE_PREFIX while also using CONDA dependency resolution #3901

ARROW-4873: [C++] Clarify documentation about how to use external ARROW_PACKAGE_PREFIX while also using CONDA dependency resolution #3901

Conversation

wesm commented Mar 14, 2019 • edited

wesm commented Mar 14, 2019

bkietz commented Mar 14, 2019 • edited

bkietz commented Mar 14, 2019

bkietz commented Mar 14, 2019

wesm commented Mar 14, 2019

wesm commented Mar 14, 2019

bkietz commented Mar 14, 2019

wesm commented Mar 14, 2019

wesm commented Mar 15, 2019

wesm commented Mar 15, 2019

wesm commented Mar 15, 2019

xhochy Mar 15, 2019

Choose a reason for hiding this comment

wesm Mar 15, 2019

Choose a reason for hiding this comment

xhochy Mar 15, 2019

Choose a reason for hiding this comment

wesm commented Mar 15, 2019

wesm commented Mar 15, 2019

bkietz Mar 15, 2019

Choose a reason for hiding this comment

wesm Mar 15, 2019

Choose a reason for hiding this comment

xhochy Mar 15, 2019

Choose a reason for hiding this comment

wesm commented Mar 15, 2019

xhochy left a comment

Choose a reason for hiding this comment

wesm commented Mar 15, 2019

xhochy commented Mar 15, 2019

wesm commented Mar 15, 2019

wesm commented Mar 15, 2019

wesm commented Mar 15, 2019

wesm commented Mar 15, 2019

wesm commented Mar 14, 2019 •

edited

bkietz commented Mar 14, 2019 •

edited