Fix velox dylib loading paths for conda build on macos #355

bearzx · 2022-06-02T01:00:02Z

In this PR I'm attempting to introduce a temporary workaround to fix the velox dylib issue, via a post-build hack. This is to make conda install torcharrow work out of the box.

Currently, you will encounter an error of dylibs not found when velox is dlopen-ing them. This is because the location of those dylibs were determined at compile time. For example: /usr/local/opt/gflags/lib/libgflags.2.2.dylib was installed to a default location by brew install.

We can specify the dylib dependencies in the conda recipe, but they will be installed to the conda environment locations, rather than the system ones (/user/local/...). As a result, the velox dylib (_torcharrow.*.so) won't be able to find those libraries.

On macos there is a tool called install_name_tool that can change the dylib paths, my plan is to modify those dylib paths from absolute ones to @loader_path/... so that the velox .so can find the dylibs installed by conda.

wenleix · 2022-06-02T21:06:42Z

packaging/torcharrow/meta.yaml

    - libsodium
-    {{ environ.get('CONDA_PYTORCH_CONSTRAINT') }}
+    - pytorch
+    # {{ environ.get('CONDA_PYTORCH_CONSTRAINT') }}


shall we just remove this?

yeah I'm experimenting with this, sometimes having the pytorch nightly build as a dependency can make conda confused when installing. Previously I thought it was solved by only leaving pytorch in the "run" section, but it actually didn't solve the problem. Just having pytorch here seems to be simpler.

wenleix · 2022-06-02T21:07:06Z

packaging/torcharrow/meta.yaml

    - BUILD_VERSION
    - CPU_TARGET

-#GitHub Actions usually provide Mac without AVX2 support, for now just skip the test until Velox can be run without AVX2


so the test is already enabled now, right?

yep, that's why I'm removing the stale comment here.

You might need to add command pytest --no-header -v torcharrow/test back the test

yeah here I'm only running an import test for now. For some reason the last time I ran the tests, there were some ones failing on some path comparison issues (don't remember the details, but something like comparing './abc' to 'abc failed). This will go into another PR to investigate.

wenleix · 2022-06-02T21:07:47Z

packaging/torcharrow/meta.yaml

    - glog==0.4.0
    - libsodium
-    {{ environ.get('CONDA_PYTORCH_CONSTRAINT') }}
+    - pytorch


curious: why do we need to build with pytorch? (since we don't build text UDFs by default)

this is in the run section, which specifies the dependencies needed to use torcharrow. I think torcharrow still imports pytorch right?

bearzx · 2022-06-06T23:38:22Z

Ok after a million times of trial-and-error, I finally found a way to adjust the recipe to

Run the conda build as usual.
Use install_name_tool to change the dylib paths for _torcharrow.so post build.
Specify the right versions of runtime dependencies that velox is happy with (notably glog==0.6.0 and arrow-cpp==8.0.0).

The most tricky part is perhaps in step 2, because conda build doesn't provide a good entry point to do a final change before it packages everything into a .tar.bz2 file. Adding stuff after python setup.py in build.sh sort of worked, but there are some extra steps to specify the compatible dylibs versions after, so it will still end up creating broken build (I still don't know all the details here, just making the observations based on my experiments).

Since the conda package is just a .tar.bz2 file, the approach I ended up taking was to unpack it, install_name_tool, pack it again using the tar command, before uploading happens. One more detail here: we'll need to run tar within the target folder so that the file list doesn't have a ./ prefix, this is critical because anaconda upload specifically hard-coded this to read the metadata. It sounds and works pretty hacky, but might be fine as a temp fix before we have static link option available on velox.

Now you should be able to run conda install -c conda-forge -c pytorch-nightly torcharrow to install TA on macos and have it works out of the box.

bearzx · 2022-06-06T23:40:03Z

Test CI run: https://github.com/facebookresearch/torcharrow/runs/6764330823?check_suite_focus=true

facebook-github-bot · 2022-06-06T23:42:01Z

@bearzx has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2022-06-06T23:46:40Z

@bearzx has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2022-06-06T23:48:23Z

@bearzx has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

bearzx · 2022-06-06T23:51:55Z

packaging/torcharrow/meta.yaml

    - typing
    - tabulate
    - pyarrow
-    - arrow


btw arrow on conda is a completely different lib ...

right , or pyarrow i guess? https://arrow.apache.org/docs/python/install.html

and arrow-cpp

wenleix · 2022-06-07T00:18:16Z

packaging/torcharrow/meta.yaml

+    - arrow-cpp==8.0.0
    - cffi
-    - glog==0.4.0
+    - glog==0.6.0


anything changed on Velox side?

I think velox always uses the glog installed from brew, which was recently updated from 0.5.0 to 0.6.0. Building with glog0.6.0 and running with glog0.4.0 will not work (some symbol not found error when running). However, the conda default channel hasn't updated to 0.6.0 yet, it only exists on conda-forge, hence this change.

This is also why we need to specify -c conda-forge when installing.

wenleix

LGTM.

wenleix · 2022-06-07T00:18:55Z

packaging/fix_conda_dylib_paths.sh

@@ -0,0 +1,33 @@
+function fix_velox_dylib_paths() {
+  velox_so=$1


nit: can we check there is exactly one parameter ?

can you elaborate?

wenleix · 2022-06-07T00:21:36Z

.github/workflows/nightly-conda.yml

+          rm ./${pkg_name}.tar.bz2
+          cd ./${pkg_name}
+
+          source ../../../packaging/fix_conda_dylib_paths.sh


can we just use ${GITHUB_WORKSPACE}/packging/...?

Summary: In this PR I'm attempting to introduce a temporary workaround to fix the velox dylib issue, via a post-build hack. This is to make `conda install torcharrow` work out of the box. Currently, you will encounter an error of dylibs not found when velox is dlopen-ing them. This is because the location of those dylibs were determined at compile time. For example: `/usr/local/opt/gflags/lib/libgflags.2.2.dylib` was installed to a default location by `brew install`. We can specify the dylib dependencies in the conda recipe, but they will be installed to the conda environment locations, rather than the system ones (`/user/local/...`). As a result, the velox dylib (`_torcharrow.*.so`) won't be able to find those libraries. On macos there is a tool called `install_name_tool` that can change the dylib paths, my plan is to modify those dylib paths from absolute ones to `loader_path/...` so that the velox .so can find the dylibs installed by conda. Pull Request resolved: #355 Reviewed By: wenleix Differential Revision: D36956804 Pulled By: bearzx fbshipit-source-id: 2551910eaec0c52b59b769de67c52aae633bbbc6

facebook-github-bot · 2022-06-07T00:27:44Z

This pull request was exported from Phabricator. Differential Revision: D36956804

Summary: In this PR I'm attempting to introduce a temporary workaround to fix the velox dylib issue, via a post-build hack. This is to make `conda install torcharrow` work out of the box. Currently, you will encounter an error of dylibs not found when velox is dlopen-ing them. This is because the location of those dylibs were determined at compile time. For example: `/usr/local/opt/gflags/lib/libgflags.2.2.dylib` was installed to a default location by `brew install`. We can specify the dylib dependencies in the conda recipe, but they will be installed to the conda environment locations, rather than the system ones (`/user/local/...`). As a result, the velox dylib (`_torcharrow.*.so`) won't be able to find those libraries. On macos there is a tool called `install_name_tool` that can change the dylib paths, my plan is to modify those dylib paths from absolute ones to `loader_path/...` so that the velox .so can find the dylibs installed by conda. Pull Request resolved: #355 Reviewed By: wenleix Differential Revision: D36956804 Pulled By: bearzx fbshipit-source-id: 0bf133a8cd9b63466706e355c7ac18004b807609

facebook-github-bot · 2022-06-07T00:42:47Z

This pull request was exported from Phabricator. Differential Revision: D36956804

Summary: In this PR I'm attempting to introduce a temporary workaround to fix the velox dylib issue, via a post-build hack. This is to make `conda install torcharrow` work out of the box. Currently, you will encounter an error of dylibs not found when velox is dlopen-ing them. This is because the location of those dylibs were determined at compile time. For example: `/usr/local/opt/gflags/lib/libgflags.2.2.dylib` was installed to a default location by `brew install`. We can specify the dylib dependencies in the conda recipe, but they will be installed to the conda environment locations, rather than the system ones (`/user/local/...`). As a result, the velox dylib (`_torcharrow.*.so`) won't be able to find those libraries. On macos there is a tool called `install_name_tool` that can change the dylib paths, my plan is to modify those dylib paths from absolute ones to `loader_path/...` so that the velox .so can find the dylibs installed by conda. X-link: #355 Reviewed By: wenleix Differential Revision: D36956804 Pulled By: bearzx fbshipit-source-id: 720ce2fcc7815fdb0211f8b5d9d72f9d817f6080

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jun 2, 2022

wenleix reviewed Jun 2, 2022

View reviewed changes

bearzx requested review from ejguan and kgpai June 6, 2022 23:40

bearzx commented Jun 6, 2022

View reviewed changes

wenleix reviewed Jun 7, 2022

View reviewed changes

wenleix approved these changes Jun 7, 2022

View reviewed changes

bearzx force-pushed the conda-build-1 branch from 7ac6e7c to 07b35a0 Compare June 7, 2022 00:27

bearzx force-pushed the conda-build-1 branch from 07b35a0 to 94d4bf6 Compare June 7, 2022 00:42

facebook-github-bot closed this in 0f3491f Jun 7, 2022

		@@ -0,0 +1,33 @@
		function fix_velox_dylib_paths() {
		velox_so=$1

Fix velox dylib loading paths for conda build on macos #355

Fix velox dylib loading paths for conda build on macos #355

Uh oh!

Conversation

bearzx commented Jun 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bearzx Jun 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bearzx commented Jun 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bearzx commented Jun 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented Jun 6, 2022

Uh oh!

facebook-github-bot commented Jun 6, 2022

Uh oh!

facebook-github-bot commented Jun 6, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bearzx Jun 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bearzx Jun 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wenleix left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Jun 7, 2022

Uh oh!

facebook-github-bot commented Jun 7, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

bearzx commented Jun 2, 2022 •

edited

Loading

bearzx Jun 2, 2022 •

edited

Loading

bearzx commented Jun 6, 2022 •

edited

Loading

bearzx commented Jun 6, 2022 •

edited

Loading

bearzx Jun 7, 2022 •

edited

Loading

bearzx Jun 7, 2022 •

edited

Loading