Skip to content
This repository has been archived by the owner on Nov 21, 2020. It is now read-only.

Build wheels properly for macOS (with dynamic libs included) #51

Closed
wants to merge 8 commits into from

Conversation

McSinyx
Copy link
Owner

@McSinyx McSinyx commented Mar 16, 2020

OK so WTF is the problem with delocate?

alure2_include_dirs=/usr/local/opt/openal-soft/include/AL;/usr/local/include;/usr/local/include/AL
alure2_extra_objects=/usr/local/opt/openal-soft/lib/libopenal.dylib;/usr/local/lib/libvorbisfile.dylib;/usr/local/lib/libvorbis.dylib;/usr/local/lib/libopusfile.dylib;/usr/local/lib/libopus.dylib;/usr/local/lib/libogg.dylib;/usr/local/lib/libsndfile.dylib;/usr/local/lib/libalure2.dylib

@McSinyx
Copy link
Owner Author

McSinyx commented Mar 17, 2020

@TrungNguyen1909, could please give me a hand on this? The issue is basically:

delocate on macOS cannot seems to add dynamic libraries:

UserWarning: Couldn't find @rpath/libalure2.dylib on paths:
  warnings.warn(
/usr/local/Cellar/libogg/1.3.4/lib/libogg.0.dylib
/usr/local/Cellar/libsndfile/1.0.28/lib/libsndfile.1.dylib
/usr/local/Cellar/libvorbis/1.3.6/lib/libvorbis.0.dylib
/usr/local/Cellar/libvorbis/1.3.6/lib/libvorbisfile.3.dylib
/usr/local/Cellar/openal-soft/1.20.1/lib/libopenal.1.20.1.dylib
/usr/local/Cellar/opus/1.3.1/lib/libopus.0.dylib
/usr/local/Cellar/opusfile/0.11_1/lib/libopusfile.0.dylib
@rpath/libalure2.dylib

This might be fixable by tweaking the alure cmake target passing in setup.py and CMakeLists.txt.

Originally posted by @McSinyx in #1 (comment)

With matthew-brett/delocate#26 I thought 2018721 would work but it didn't 🙁 The steps to reproduce would be

  1. Install alure2 linking against openal-soft
  2. Build the wheel: git clone https://github.com/McSinyx/palace && pip wheel palace
  3. Use delocate to add DYnamic LIBraries to the wheel

Then after unzipping the wheel there should be a .dylib folder with libalure2, libopus, etc.

@TrungNguyen1909
Copy link

.dylib is not a folder. it is basically something like an shared object(.so)

@McSinyx
Copy link
Owner Author

McSinyx commented Mar 17, 2020

Thank you genius, I mean when you unpack the wheel there should be a folder named like that with files of that extension inside, i.e. .dylib/*.dylib, which we currently don't have, so it's the problem. Is it my English or people just never slow down to read carefully?

Anyway, any advice/finding on this?

@TrungNguyen1909
Copy link

why couldn't you install it with homebrew? I dont think it will statically linked that

@McSinyx
Copy link
Owner Author

McSinyx commented Mar 17, 2020

I asked you a simple favor to just find a way to make manually compiled lib delocated, but if you are familiar with brew enough to package alure2, please do. The instruction at the project page is enough I hope, though I can't do it myself due to the lack of facilities for local testing (aight TBH I know hackintosh is an option, but dedicating time and effort to install something I'll never use is not, ehm, my kinda kink).

@TrungNguyen1909
Copy link

Urgh alure build failed:

/usr/local/include/dumb.h:41:27: error: invalid suffix on literal; C++11
      requires a space between literal and identifier
      [-Wreserved-user-defined-literal]
#define DUMB_NAME "DUMB v"DUMB_VERSION_STR
                          ^

/usr/local/include/dumb.h:53:28: error: invalid suffix on literal; C++11
      requires a space between literal and identifier
      [-Wreserved-user-defined-literal]
#define DUMB_MONTH_STR2 "0"DUMB_MONTH_STR1
                           ^

/usr/local/include/dumb.h:59:26: error: invalid suffix on literal; C++11
      requires a space between literal and identifier
      [-Wreserved-user-defined-literal]
#define DUMB_DAY_STR2 "0"DUMB_DAY_STR1
                         ^

/usr/local/include/dumb.h:71:39: error: invalid suffix on literal; C++11
      requires a space between literal and identifier
      [-Wreserved-user-defined-literal]
#define DUMB_DATE_STR DUMB_DAY_STR1"."DUMB_MONTH_STR1"."DUMB_YEAR_STR4
                                      ^

/usr/local/include/dumb.h:71:57: error: invalid suffix on literal; C++11
      requires a space between literal and identifier
      [-Wreserved-user-defined-literal]
#define DUMB_DATE_STR DUMB_DAY_STR1"."DUMB_MONTH_STR1"."DUMB_YEAR_STR4```

@McSinyx
Copy link
Owner Author

McSinyx commented Mar 17, 2020

And how to reproduce that? BTW dumb is an optional dependency only used for examples, so you can drop it if it doesn't work though.

@McSinyx
Copy link
Owner Author

McSinyx commented Mar 20, 2020

Gentle ping @TrungNguyen1909! Also thank you for offering help on this!

@TrungNguyen1909
Copy link

@McSinyx here is the build output

# trung @ Trungs-MacBook-Pro in ~/Projects/palace on git:master o [12:15:08] C:130
$ bin/delocate-listdeps products/palace-0.0.12-cp37-cp37m-macosx_10_14_x86_64.whl
/Users/trung/Projects/palace/lib/python3.7/site-packages/delocate/libsana.py:110: UserWarning: Couldn't find @rpath/libalure2.dylib on paths:

  '\n\t'.join(realpath(path) for path in rpaths),
/usr/local/Cellar/libogg/1.3.4/lib/libogg.0.dylib
/usr/local/Cellar/libsndfile/1.0.28/lib/libsndfile.1.dylib
/usr/local/Cellar/libvorbis/1.3.6/lib/libvorbis.0.dylib
/usr/local/Cellar/libvorbis/1.3.6/lib/libvorbisfile.3.dylib
/usr/local/Cellar/openal-soft/1.20.1/lib/libopenal.1.20.1.dylib
/usr/local/Cellar/opus/1.3.1/lib/libopus.0.dylib
/usr/local/Cellar/opusfile/0.11_1/lib/libopusfile.0.dylib
@rpath/libalure2.dylib

@TrungNguyen1909
Copy link

Path = palace-0.0.12-cp37-cp37m-macosx_10_14_x86_64.whl
Type = zip
Physical Size = 193832

   Date      Time    Attr         Size   Compressed  Name
------------------- ----- ------------ ------------  ------------------------
2020-03-20 05:13:46 .....       601744       188160  palace.cpython-37m-darwin.so
2020-03-20 05:13:46 .....         7652         2618  palace-0.0.12.dist-info/LICENSE
2020-03-20 05:13:46 .....         3684         1775  palace-0.0.12.dist-info/METADATA
2020-03-20 05:13:46 .....          111          108  palace-0.0.12.dist-info/WHEEL
2020-03-20 05:13:46 .....            7            9  palace-0.0.12.dist-info/top_level.txt
2020-03-20 05:13:46 .....          473          310  palace-0.0.12.dist-info/RECORD
------------------- ----- ------------ ------------  ------------------------
2020-03-20 05:13:46             613671       192980  6 files

What do you even expect?

@TrungNguyen1909
Copy link

Oh, I see,

$ otool -L palace.cpython-37m-darwin.so
palace.cpython-37m-darwin.so:
	/usr/local/opt/openal-soft/lib/libopenal.1.dylib (compatibility version 1.0.0, current version 1.20.1)
	/usr/local/opt/libvorbis/lib/libvorbisfile.3.dylib (compatibility version 7.0.0, current version 7.7.0)
	/usr/local/opt/libvorbis/lib/libvorbis.0.dylib (compatibility version 5.0.0, current version 5.8.0)
	/usr/local/opt/opusfile/lib/libopusfile.0.dylib (compatibility version 5.0.0, current version 5.4.0)
	/usr/local/opt/opus/lib/libopus.0.dylib (compatibility version 9.0.0, current version 9.0.0)
	/usr/local/opt/libogg/lib/libogg.0.dylib (compatibility version 9.0.0, current version 9.4.0)
	/usr/local/opt/libsndfile/lib/libsndfile.1.dylib (compatibility version 2.0.0, current version 2.28.0)
	@rpath/libalure2.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.0.0)

@TrungNguyen1909
Copy link

install_name_tool should work? I'm not familiar with cmake so I probably can't help you, @McSinyx

@McSinyx
Copy link
Owner Author

McSinyx commented Mar 20, 2020

Thanks for the output, it'd take me an hour of stress to wait for those on Travis otherwise. What is the output of otool after install_name_tool -id libalure2.dylib libalure2.dylib? if I may ask?

@McSinyx
Copy link
Owner Author

McSinyx commented Mar 20, 2020

Path = palace-0.0.12-cp37-cp37m-macosx_10_14_x86_64.whl
Type = zip
Physical Size = 193832

   Date      Time    Attr         Size   Compressed  Name
------------------- ----- ------------ ------------  ------------------------
2020-03-20 05:13:46 .....       601744       188160  palace.cpython-37m-darwin.so
2020-03-20 05:13:46 .....         7652         2618  palace-0.0.12.dist-info/LICENSE
2020-03-20 05:13:46 .....         3684         1775  palace-0.0.12.dist-info/METADATA
2020-03-20 05:13:46 .....          111          108  palace-0.0.12.dist-info/WHEEL
2020-03-20 05:13:46 .....            7            9  palace-0.0.12.dist-info/top_level.txt
2020-03-20 05:13:46 .....          473          310  palace-0.0.12.dist-info/RECORD
------------------- ----- ------------ ------------  ------------------------
2020-03-20 05:13:46             613671       192980  6 files

What do you even expect?

This is that's inside of the manylinux wheel after auditwheel (a tool similar to delocate on macOS, but it also check for libc version for cross-distro compatibility):

Archive:  palace-0.0.11-cp37-cp37m-manylinux2014_x86_64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
  2984328  2020-03-01 09:45   palace.cpython-37m-x86_64-linux-gnu.so
        7  2020-03-01 09:45   palace-0.0.11.dist-info/top_level.txt
     1576  2020-03-01 09:45   palace-0.0.11.dist-info/RECORD
     3684  2020-03-01 09:45   palace-0.0.11.dist-info/METADATA
     7652  2020-03-01 09:45   palace-0.0.11.dist-info/LICENSE
      112  2020-03-01 09:45   palace-0.0.11.dist-info/WHEEL
   276936  2020-03-01 09:45   .libspalace/libopus-abf1ee53.so.0.3.0
 16036816  2020-03-01 09:45   .libspalace/libopenal-d73bb64a.so.1.20.1
    50880  2020-03-01 09:45   .libspalace/libvorbisfile-c5d289a9.so.3.3.5
   195328  2020-03-01 09:45   .libspalace/libvorbis-02341991.so.0.4.6
   315984  2020-03-01 09:45   .libspalace/libFLAC-b0904aee.so.8.3.0
    50496  2020-03-01 09:45   .libspalace/libgsm-ed01cacd.so.1.0.12
   408408  2020-03-01 09:45   .libspalace/libsndfile-2ee11aa9.so.1.0.25
    63112  2020-03-01 09:45   .libspalace/libopusfile-a90fc0fd.so.0.3.0
  3471616  2020-03-01 09:45   .libspalace/libalure2-53a453ce.so
  2957816  2020-03-01 09:45   .libspalace/libvorbisenc-e32e3ab5.so.2.0.9
    30856  2020-03-01 09:45   .libspalace/libogg-24eaa32b.so.0.8.0
---------                     -------
 26855607                     17 files

@TrungNguyen1909
Copy link

# trung @ Trungs-MacBook-Pro in ~/Projects/palace on git:master x [16:22:37]
$ sudo install_name_tool -id /usr/local/lib/libalure2.dylib /usr/local/lib/libalure2.dylib
(palace)
# trung @ Trungs-MacBook-Pro in ~/Projects/palace on git:master x [16:23:14]
$ pip wheel . -w products --no-deps
Processing /Users/trung/Projects/palace
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: palace
  Building wheel for palace (PEP 517) ... done
  Created wheel for palace: filename=palace-0.0.12-cp37-cp37m-macosx_10_14_x86_64.whl size=193827 sha256=ec55b31d1d393959ec6dfc6a459dfa0ac9962552f99b9c4bc9622a8b7905c7de
  Stored in directory: /private/var/folders/33/c04gr2z16_v4wq__56sgv27c0000gn/T/pip-ephem-wheel-cache-9x09wvt6/wheels/a0/b5/16/5dfe32be6c8234f82c038adb95e1e810bb705160fe12f8d5dd
Successfully built palace
(palace)
# trung @ Trungs-MacBook-Pro in ~/Projects/palace on git:master x [16:23:43]
$ bin/delocate-listdeps products/palace-0.0.12-cp37-cp37m-macosx_10_14_x86_64.whl
/usr/local/Cellar/libogg/1.3.4/lib/libogg.0.dylib
/usr/local/Cellar/libsndfile/1.0.28/lib/libsndfile.1.dylib
/usr/local/Cellar/libvorbis/1.3.6/lib/libvorbis.0.dylib
/usr/local/Cellar/libvorbis/1.3.6/lib/libvorbisfile.3.dylib
/usr/local/Cellar/openal-soft/1.20.1/lib/libopenal.1.20.1.dylib
/usr/local/Cellar/opus/1.3.1/lib/libopus.0.dylib
/usr/local/Cellar/opusfile/0.11_1/lib/libopusfile.0.dylib
/usr/local/lib/libalure2.dylib

@TrungNguyen1909
Copy link

TrungNguyen1909 commented Mar 20, 2020

But why does it matter? Normal build produced a shared object without any problems.
You don't even need install_name_tool and everything still load anyway.
dlopen/CDLL works without any problems.

#include <stdio.h>
#include <dlfcn.h>
int main(int argc, char *argv[]) {
	printf("%p\n",dlopen("/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/Python",0));
	printf("%p\n",dlopen("/Users/trung/Projects/palace/products/palace.cpython-37m-darwin.so",0));
	printf("%s\n",dlerror());
}

.travis.yml Outdated Show resolved Hide resolved
.travis.yml Outdated Show resolved Hide resolved
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants