New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python bindings not importing on OSX #19

Open
asluk opened this Issue Aug 13, 2016 · 31 comments

Comments

Projects
None yet
9 participants
@asluk
Contributor

asluk commented Aug 13, 2016

I know that the OSX build is currently experimental, but just want to verify that I'm seeing the currently expected behavior wrto Python bindings:

from pxr import Tf
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python/pxr/Tf/init.py", line 85, in
from . import _tf
ImportError: cannot import name _tf

Likely due to http://stackoverflow.com/questions/2488016/how-to-make-python-load-dylib-on-osx , but I haven't found the right combination of xcode build settings to build _tf as a bundle with an .so extension yet, if that's the expected fix.

Cheers!
-a.

@meshula

This comment has been minimized.

meshula commented Aug 13, 2016

I wrote a little script to symlink the dylibs to the name so. That works. You need to symlink, because of the embedded rpath stuff. There's probably a way to get cmake to generate .so instead of .dylib ... ?

@asluk

This comment has been minimized.

Contributor

asluk commented Aug 14, 2016

Thanks Nick; that gets me a bit further-- when importing _tf, the Py_IsInitialized() axiom fails, and then the interpreter hangs. The boost::python examples at https://github.com/TNG/boost-python-examples seem to build and run ok.

I am on Python 2.7.12; will try 2.7.5 and see if it makes a difference.

@meshula

This comment has been minimized.

meshula commented Aug 14, 2016

You might try running python -v, and reading the dynamic loading spew (thanks to sunyab for this tip). Crashes will occur if same-named dylibs hiding in other search paths are taking precedence over the ones you linked against. Maybe there's already boost_python elsewhere in your system...

@asluk

This comment has been minimized.

Contributor

asluk commented Aug 14, 2016

Cool, that tip got me in the right direction-- running with DYLD_PRINT_LIBRARIES=1 showed me that Tf was linked against the wrong Python library. After specifying the desired python locations to cmake, I'm able to import _tf.

I'll leave this open to track build support for .so's on OSX, so that we don't have to make the symlinks. Thanks again!

@amooty

This comment has been minimized.

amooty commented Aug 16, 2016

Filed as internal issue #135883.

@meshula

This comment has been minimized.

meshula commented Sep 2, 2016

So the solution is that the Python lib generation script should specify the ".so" suffix.

diff --git a/cmake/macros/Public.cmake b/cmake/macros/Public.cmake
index a04c374..cad924f 100644
--- a/cmake/macros/Public.cmake
+++ b/cmake/macros/Public.cmake
@@ -173,6 +173,7 @@ function(pxr_shared_library LIBRARY_NAME)
set_target_properties(${LIBRARY_NAME}
PROPERTIES
PREFIX ""

  •            SUFFIX ".so"
             FOLDER "${PXR_PREFIX}/_python"
             INSTALL_RPATH ${rpath}
     )
    
@zhoub

This comment has been minimized.

zhoub commented Sep 30, 2016

python2.7 run.py 
dyld: loaded: /usr/local/bin/python2.7
dyld: loaded: /usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/Python
dyld: loaded: /usr/lib/libSystem.B.dylib
dyld: loaded: /usr/lib/system/libcache.dylib
dyld: loaded: /usr/lib/system/libcommonCrypto.dylib
dyld: loaded: /usr/lib/system/libcompiler_rt.dylib
dyld: loaded: /usr/lib/system/libcopyfile.dylib
dyld: loaded: /usr/lib/system/libcorecrypto.dylib
dyld: loaded: /usr/lib/system/libdispatch.dylib
dyld: loaded: /usr/lib/system/libdyld.dylib
dyld: loaded: /usr/lib/system/libkeymgr.dylib
dyld: loaded: /usr/lib/system/liblaunch.dylib
dyld: loaded: /usr/lib/system/libmacho.dylib
dyld: loaded: /usr/lib/system/libquarantine.dylib
dyld: loaded: /usr/lib/system/libremovefile.dylib
dyld: loaded: /usr/lib/system/libsystem_asl.dylib
dyld: loaded: /usr/lib/system/libsystem_blocks.dylib
dyld: loaded: /usr/lib/system/libsystem_c.dylib
dyld: loaded: /usr/lib/system/libsystem_configuration.dylib
dyld: loaded: /usr/lib/system/libsystem_coreservices.dylib
dyld: loaded: /usr/lib/system/libsystem_coretls.dylib
dyld: loaded: /usr/lib/system/libsystem_dnssd.dylib
dyld: loaded: /usr/lib/system/libsystem_info.dylib
dyld: loaded: /usr/lib/system/libsystem_kernel.dylib
dyld: loaded: /usr/lib/system/libsystem_m.dylib
dyld: loaded: /usr/lib/system/libsystem_malloc.dylib
dyld: loaded: /usr/lib/system/libsystem_network.dylib
dyld: loaded: /usr/lib/system/libsystem_networkextension.dylib
dyld: loaded: /usr/lib/system/libsystem_notify.dylib
dyld: loaded: /usr/lib/system/libsystem_platform.dylib
dyld: loaded: /usr/lib/system/libsystem_pthread.dylib
dyld: loaded: /usr/lib/system/libsystem_sandbox.dylib
dyld: loaded: /usr/lib/system/libsystem_secinit.dylib
dyld: loaded: /usr/lib/system/libsystem_trace.dylib
dyld: loaded: /usr/lib/system/libunc.dylib
dyld: loaded: /usr/lib/system/libunwind.dylib
dyld: loaded: /usr/lib/system/libxpc.dylib
dyld: loaded: /usr/lib/libobjc.A.dylib
dyld: loaded: /usr/lib/libauto.dylib
dyld: loaded: /usr/lib/libc++abi.dylib
dyld: loaded: /usr/lib/libc++.1.dylib
dyld: loaded: /usr/lib/libDiagnosticMessagesClient.dylib
dyld: loaded: /usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
dyld: loaded: /usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/Python
dyld: loaded: /usr/lib/libSystem.B.dylib
dyld: loaded: /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
dyld: loaded: /usr/lib/system/libcache.dylib
dyld: loaded: /usr/lib/system/libcommonCrypto.dylib
dyld: loaded: /usr/lib/system/libcompiler_rt.dylib
dyld: loaded: /usr/lib/system/libcopyfile.dylib
dyld: loaded: /usr/lib/system/libcorecrypto.dylib
dyld: loaded: /usr/lib/system/libdispatch.dylib
dyld: loaded: /usr/lib/system/libdyld.dylib
dyld: loaded: /usr/lib/system/libkeymgr.dylib
dyld: loaded: /usr/lib/system/liblaunch.dylib
dyld: loaded: /usr/lib/system/libmacho.dylib
dyld: loaded: /usr/lib/system/libquarantine.dylib
dyld: loaded: /usr/lib/system/libremovefile.dylib
dyld: loaded: /usr/lib/system/libsystem_asl.dylib
dyld: loaded: /usr/lib/system/libsystem_blocks.dylib
dyld: loaded: /usr/lib/system/libsystem_c.dylib
dyld: loaded: /usr/lib/system/libsystem_configuration.dylib
dyld: loaded: /usr/lib/system/libsystem_coreservices.dylib
dyld: loaded: /usr/lib/system/libsystem_coretls.dylib
dyld: loaded: /usr/lib/system/libsystem_dnssd.dylib
dyld: loaded: /usr/lib/system/libsystem_info.dylib
dyld: loaded: /usr/lib/system/libsystem_kernel.dylib
dyld: loaded: /usr/lib/system/libsystem_m.dylib
dyld: loaded: /usr/lib/system/libsystem_malloc.dylib
dyld: loaded: /usr/lib/system/libsystem_network.dylib
dyld: loaded: /usr/lib/system/libsystem_networkextension.dylib
dyld: loaded: /usr/lib/system/libsystem_notify.dylib
dyld: loaded: /usr/lib/system/libsystem_platform.dylib
dyld: loaded: /usr/lib/system/libsystem_pthread.dylib
dyld: loaded: /usr/lib/system/libsystem_sandbox.dylib
dyld: loaded: /usr/lib/system/libsystem_secinit.dylib
dyld: loaded: /usr/lib/system/libsystem_trace.dylib
dyld: loaded: /usr/lib/system/libunc.dylib
dyld: loaded: /usr/lib/system/libunwind.dylib
dyld: loaded: /usr/lib/system/libxpc.dylib
dyld: loaded: /usr/lib/libobjc.A.dylib
dyld: loaded: /usr/lib/libauto.dylib
dyld: loaded: /usr/lib/libc++abi.dylib
dyld: loaded: /usr/lib/libc++.1.dylib
dyld: loaded: /usr/lib/libDiagnosticMessagesClient.dylib
dyld: loaded: /usr/lib/libicucore.A.dylib
dyld: loaded: /usr/lib/libz.1.dylib
dyld: loaded: /usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_locale.so
dyld: loaded: /Users/Bo/Downloads/USD-Clang-libcpp/dist/lib/python/pxr/Tf/_tf.so
dyld: loaded: /Users/Bo/Downloads/USD-Clang-libcpp/dist/lib/libtf.dylib
dyld: loaded: /Users/Bo/Downloads/USD-Clang-libcpp/dist/lib/libarch.dylib
dyld: loaded: /Applications/Autodesk/maya2017/Maya.app/Contents/MacOS/libtbb.dylib
dyld: loaded: /Applications/Autodesk/maya2017/Maya.app/Contents/MacOS/libtbbmalloc.dylib

Hi, I also got this problem, I'm using static boost.python with the Python 2.7.12 from brew, but the TBB came from Maya 2017. The workaround switch for TBB "-Wl,-Bsymbolic-functions" didn't work on my Clang 7.3. The simple python script always hangs at the libtbbmalloc. Is there any ideas about this case ? Thanks very much.

@meshula

This comment has been minimized.

meshula commented Sep 30, 2016

When you configured cmake the first time, I assume you pointed to the tbb in Maya, and that's how it was discovered? Also I assume you are attempting to be able to use Maya's Python bindings, and USD python bindings simultaneously?

If yes to both, I think you should open a new issue: Using libtbbmalloc on macOS causes hang. I'm able to use the USD Python bindings because I don't have libtbbmalloc in my build at all. Also, I don't have to interoperate with Maya's bindings, so I got a get out of jail free card.

@asluk

This comment has been minimized.

Contributor

asluk commented Sep 30, 2016

As a datapoint, I have libtbbmalloc on OSX, but no Maya, and am able to use
the USD Python bindings as well. My tbb is built and installed from source.

On Fri, Sep 30, 2016 at 6:43 PM, Nick Porcino notifications@github.com
wrote:

When you configured cmake the first time, I assume you pointed to the tbb
in Maya, and that's how it was discovered? Also I assume you are attempting
to be able to use Maya's Python bindings, and USD python bindings
simultaneously?

If yes to both, I think you should open a new issue: Using libtbbmalloc on
macOS causes hang. I'm able to use the USD Python bindings because I
don't have libtbbmalloc in my build at all. Also, I don't have to
interoperate with Maya's bindings, so I got a get out of jail free card.


You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#19 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB_7ZpPY0vMdDMI5EedgJOGlYRbd9NKsks5qvTwWgaJpZM4JjrGI
.

@zhoub

This comment has been minimized.

zhoub commented Oct 1, 2016

@meshula Yes, that's what my setup, use the single tbb from Maya and also the external bindings. But my problem seems a bit more complicated, I'm using the 10.11 but still use the OSX 10.9 SDK for the all dependencies, but the Python comes from Brew was compiled with 10.11 the latest SDK, I'm not sure if that's the issue, the problem in Python. I will try again. Thank you very much guys.

@meshula

This comment has been minimized.

meshula commented Oct 1, 2016

As you suspect, you'll need to conform your build to Maya completely and not mix-and-match SDKs... By conform it, I mean, compile vs. the same SDKs as Maya. FWIW, I'm personally building against system Python, the one in /System/Library/Frameworks, and that works, Maya seems to like it okay.

@zhoub

This comment has been minimized.

zhoub commented Oct 1, 2016

I fixed this at my local in a weird way.

 TF_REGISTRY_FUNCTION(TfDebug) {

    TfType::Define< TfDebugSymbolsChangedNotice,
        TfType::Bases<TfNotice> >();
    TfType::Define< TfDebugSymbolEnableChangedNotice,
        TfType::Bases<TfNotice> >();

I debugged the whole routine for initialization of Tf, it always reported the TfDebugSymbolsChangedNotice was not registered, so I move the code from debugNotice.cpp to debugCodes.cpp before the lines of TF_DEBUG_ENVIRONMENT_SYMBOL, then it worked. Not sure this is the correct fix but there is the proper order to be handled during the initialization. At least my local Python binding is working now. Thanks very much.

@meshula

This comment has been minimized.

meshula commented Oct 2, 2016

Wow, congratulations on working out a very tricky dependency ordering problem, I haven't seen that problem before. I think that is worth tracking as an issue of its own.

@drwave

This comment has been minimized.

drwave commented Oct 5, 2016

Hey @zhoub - I'm now seeing this same issue, and I really appreciate that you got it working, but I'm having trouble understanding what your fix was...

So you removed the above code from debugNotice.cpp, but then I'm not sure what the reconfigured code in debugCodes.cpp looks like. Can you paste what that looks like? Sorry to be so dense...

@zhoub

This comment has been minimized.

zhoub commented Oct 6, 2016

Hi @drwave

This is m commit https://github.com/zhoub/USD/commit/67200e58a7a1df276bc4caca00c04fabcc63ceb5

debugNotice.cpp keep it unchanged, then copy the lines to the debugCodecs.cpp

Then just recompile it. When you're trying to load module it might report the warning about duplicated importing but it should work. To be honest this is absolutely not a good fix, ideally this problem should not even exist, also the logic of PyUSD is so complicated, maybe in future they will get this fixed.

If you still have problems please let me know.

@drwave

This comment has been minimized.

drwave commented Oct 6, 2016

Thanks - that’s exactly what I needed.

Sorry, I’m still at the beginning stages of learning how to navigate github…

I’m hoping someone on the core USD team can give me some insight into what’s going on.

On Oct 5, 2016, at 6:32 PM, Bo Zhou notifications@github.com wrote:

Hi @drwave https://github.com/drwave
This is m commit zhoub@67200e5 https://github.com/zhoub/USD/commit/67200e58a7a1df276bc4caca00c04fabcc63ceb5
debugNotice.cpp keep it unchanged, then copy the lines to the debugCodecs.cpp

Then just recompile it. When you're trying to load module it might report the warning about duplicated importing but it should work. To be honest this is absolutely not a good fix, ideally this problem should not even exist, also the logic of PyUSD is so complicated, maybe in future they will get this fixed.

If you still have problems please let me know.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub #19 (comment), or mute the thread https://github.com/notifications/unsubscribe-auth/ABmgOvexFORl1jxaZbjd6X187hmksO6xks5qxE_FgaJpZM4JjrGI.

@robpieke

This comment has been minimized.

robpieke commented Oct 11, 2016

Huge thanks to all for this. Combining:

  • .dylib -> .so
  • linking against homebrew libpython
  • DebugSymbols patch

I'm up and running (at least to run the HelloWorld tutorial) on my MacBook!

@robpieke

This comment has been minimized.

robpieke commented Oct 13, 2016

Sadly I can't get through the endToEnd example though. Crashes when dealing with variants. Logged Issue #78

@meshula

This comment has been minimized.

meshula commented Oct 13, 2016

@robpieke FWIW, the patches for DebugSymbols and DebugNotice are patched into dev.

@robpieke

This comment has been minimized.

robpieke commented Oct 13, 2016

@meshula sorry, didn't quite get that. are you saying the patches have now been merged into dev? i'm not seeing this after a "git pull"

@meshula

This comment has been minimized.

meshula commented Oct 15, 2016

My mistake. Only #67 got merged.

@sunyab

This comment has been minimized.

Contributor

sunyab commented Dec 10, 2016

Hi, github auto-closed this since there was a commit that fixed the original issue, but I've broken out the TfDebugSymbolsChangedNotice problem that @zhoub and @robpieke noticed into issue #117. Thanks!

@TimvanScherpenzeel

This comment has been minimized.

TimvanScherpenzeel commented Jun 19, 2018

Hi,

After some trial and error I was able to compile USD on MacOS 10.13.4 (XCode 10 Beta) by following the instructions mentioned in the README with minor adjustments (update OpenImageIO/oiio to 1.8.12 and installing PySide2 as mentioned in #60 ). I did not use XCode directly but instead ran the recommended build_scripts/build_usd.py script.

Unfortunately running usdcat (or any other binary) yields the following error:

➜  pixar usdcat -o ar-quicklook-gallery/cupandsaucer/CupAndSaucer.usda ar-quicklook-gallery/cupandsaucer/CupAndSaucer.usdc

------------------------ 'Python' is dying ------------------------
Python crashed. FATAL ERROR: Failed axiom: ' Py_IsInitialized() '
in operator() at line 148 of /Users/timvanscherpenzeel/Projects/pixar/USD/pxr/base/lib/tf/pyTracing.cpp

The stack can be found in Tims-MacBook-Pro.local:/var/folders/s9/rnmbn61120s2hwww26_gk3k40000gn/T//st_Python.99125
done.
------------------------------------------------------------------
[1]    99125 abort      usdcat -o ar-quicklook-gallery/cupandsaucer/CupAndSaucer.usda

I suspect that something similar might be going on as mentioned by OP. I am quite new to build systems so I don't fully understand the various solutions mentioned. Would anyone be able to help me out?

As mentioned in the build log of USD I've added the following to my environment:

    The following in your PYTHONPATH environment variable:
    /Users/timvanscherpenzeel/Projects/pixar/BUILD/lib/python

    The following in your PATH environment variable:
    /Users/timvanscherpenzeel/Projects/pixar/BUILD/bin

My goal is to convert the AR QuickLook Gallery examples to USDA so I can inspect their file-structure.

Many thanks in advance,

Tim

@robpieke

This comment has been minimized.

robpieke commented Jun 22, 2018

Annoyingly, I'm in the same boat and can't remember how I solved this before. It looks like lib/python/pxr/Usd/_usd.so ends up linking to /System/Library/Frameworks/Python.framework/Versions/2.7/Python instead of /usr/lib/...

@meshula

This comment has been minimized.

meshula commented Jun 22, 2018

I guess you did

otool -L _usd.so | grep ython

and see

/System/Library/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.10)
	@rpath/libboost_python.dylib (compatibility version 0.0.0, current version 0.0.0)

That's normal, if you intend everything to purely system python.

If you are running into PyInitialize problems, it could be due to having both a system python 2.7, and also a homebrew, or Maya version of python in the path and/or cross wiring site-packages for different python installations via the environment variable.

Pretty sure that if you want to build against a non-system python, you need to invoke the python executable that corresponds to way you want to run things. So if you want to build in a Maya compatible way, I think you need to invoke the script from Maya's bundled python executable, not the system python...

@robpieke

This comment has been minimized.

robpieke commented Jun 22, 2018

Hey Nick, yes I used otool exactly as you indicated. I'm trying to build using the homebrew python. At least part of the complication seems to be that /usr/local/bin/python doesn't exist (only /usr/local/bin/python2.7), and some part(s) of the USD build process explicitly (re)invoke "python" regardless what you actually used as the interpreter when launching the build script. I've fired off one more test with python->python2.7 and python-config->python-config2.7 symlinks manually added just before leaving the office, so will need to check on Monday if that did the trick. Fingers-crossed.

@meshula

This comment has been minimized.

meshula commented Jun 23, 2018

I think that will work. It sounds like the build_usd.py script needs to detect the hosting python and dispatch to the appropriate python using a full path.

@robpieke

This comment has been minimized.

robpieke commented Jun 25, 2018

Nope, another failure. Google suggests Homebrew+FindPythonLibs=worldOfPain. Supposedly it's better in CMake 3.12 with a more generic FindPython.
In the interim, I did manage to get things working, by hacking the build script:

diff --git a/build_scripts/build_usd.py b/build_scripts/build_usd.py
index 37fb2a5e..047324de 100644
--- a/build_scripts/build_usd.py
+++ b/build_scripts/build_usd.py
@@ -855,6 +855,12 @@ def InstallUSD(context):
 
         if context.buildPython:
             extraArgs.append('-DPXR_ENABLE_PYTHON_SUPPORT=ON')
+            if MacOS():
+              import distutils.sysconfig
+              pyLibPath = distutils.sysconfig.get_config_var('LIBDIR')
+              pyIncPath = distutils.sysconfig.get_config_var('INCLUDEPY')
+              extraArgs.append('-DPYTHON_LIBRARY=' + pyLibPath + '/libpython2.7.dylib')
+              extraArgs.append('-DPYTHON_INCLUDE_DIR=' + pyIncPath)
         else:
             extraArgs.append('-DPXR_ENABLE_PYTHON_SUPPORT=OFF')
@meshula

This comment has been minimized.

meshula commented Jun 25, 2018

(Runs off to read release notes.) The new find scripts do look promising. The namespaced imported targets are an excellent step forward.

https://cmake.org/cmake/help/git-stage/module/FindPython2.html#module:FindPython2

@c64kernal

This comment has been minimized.

Contributor

c64kernal commented Aug 1, 2018

Reopening to track @meshula excellent suggestion of at least documenting this better!

@TimvanScherpenzeel

This comment has been minimized.

TimvanScherpenzeel commented Aug 2, 2018

Using the solution mentioned by @robpieke I was able to correctly build and run USD! Thanks 👍

I've documented my installation here: https://github.com/timvanscherpenzeel/gltf-to-usdz#development as part of ongoing research.

I thought I would post it here as it perhaps would be useful if further documentation changes are done in the future.

aloysbaillet pushed a commit to AnimalLogic/USD that referenced this issue Aug 31, 2018

@sirpalee sirpalee referenced this issue Oct 4, 2018

Closed

Support MacOS #6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment