vtk 5.10 & add pyqt+sip #12807

2bits · 2012-06-13T13:02:17Z

Upgrade vtk to version 5.10.
Add deps on sip and pyqt if the user options --pyqt
Add cmake args to enable SIP/PyQt if --pyqt
Add cmake args for -DSIP_PYQT_DIR as HOMEBREW_PREFIX/share/sip.
Add skip_clean :all to help vtkpython find the symbol _environ
Change the location of the python .so modules so they and the Egg
get installed into lib/which_python/'site-packages'.
Add code so site-packages is created and prepended to the
PYTHONPATH to stop build errors.
Add -DPYTHON_INCLUDE_DIR so the right headers are located.
Remove the workaround code that was previously added to support
Python getting RPATHs, as RPATHS are fixed in Homebrew Core.
Add code that removes three duplicate files in site-packages
that cause the linking step to fail, files that already
exist in HB site-packages when using brewed Python.
Add code to mv the python module directory vtk out of the
VTK-5.10-py2.7.egg and into site-packages. Without that,
you can't import vtk into Python.

Tested on Lion using clang and llvm from XCode-4.3.3 with the CLT
and without the CLT against system Python, HB framework Python,
HB non-framework Python, Qt-4.8.2, sip, pyqt, pyside and shiboken.

2bits · 2012-06-13T13:11:18Z

I know @samueljohn and @youngtaekoh have been working on this formula independently of each other and myself, but I wanted to submit this pull request as one possible solution that I think is valid. cf #12612.

I took @youngtaekoh's sip+pyqt changes and applied them.

I found that by removing the workarounds that the python .so modules contain RPATHS, but I also found that they are wrong because they refer to /tmp. I think it's better to remove the workaround like I did and to rely on the HB installer scripts to finalize the install by running install_name_tool on the all the libraries. Currently it does recurse through the subdirectories like site-packages but I think only works on .dylib files. My idea is that it should adjust .so files as well. If that were the case, then this formula would work, but that would need to be tested thoroughly because there are too many paths to manually adjust with install_name_tool. So I'm waiting for it to be automatic. Thoughts?

2bits · 2012-06-13T13:42:37Z

Had to fix a typo in 32e4916

samueljohn · 2012-06-13T14:14:28Z

@2bits I am working on the fix_xcode thing right now.
Perhaps my works on vtk 5.10 with python bindings do help you?

2bits · 2012-06-14T07:48:41Z

That helps to see your take on it. Vtk is strange software, and it's not fun how it takes 15min to compile each time I want to try something different. I doubt I'll work on it unless 12810 happens, especially because your bug report to them got no replies.

samueljohn · 2012-06-15T14:05:07Z

Have you found a cure for this bug/work-around reported upstream ?

2bits · 2012-06-15T19:12:59Z

Let's see. My solution is this pull request, but it doesn't work because HB during keg_fix_install_names does not operate on the vtk python .so files. So I get a dlopen error. I took the approach of reducing the build to just

cmake *args
make install

and see what happens. When it almost worked except for the keg_fix_install_names, I guessed that vtk-5.10 was better written and didn't need the RPATH workaround. Because it didn't need the RPATH workaround, their developers never answered you. That was my guess. I'm testing it now to see if I can import the module if I shell out before the install exits, leaving all the files it needs in /tmp still there and brew link vtk.

If instead I use all the RPATH workarounds, I think it was crashing when I try to import the module. When I looked at your branch and saw you setting -DPYTHON_INCLUDE_DIR I was going to test that, but I got put off by the 15 min. compiles. Does your new vtk formula work?

Because I spent a whole day compiling this dozens of times, the different errors I had are all running together, and I can't really trust I'm remembering the problems anymore, sigh.

jacknagel · 2012-06-15T19:21:57Z

I'll try and take a look at this tonight.

2bits · 2012-06-15T19:32:11Z

Well I just tested my formula by shelling out of the install, linking vtk, and it will import the vtk module into system python. So that's good. And it writes the module into Installed /usr/local/Cellar/vtk/5.10.0/lib/python2.7/site-packages/VTK-5.10.0-py2.7.egg. So that's good too.

2bits · 2012-06-15T22:09:33Z

@samueljohn From my testing, this formula works with both system python and HB python doing brew install --python vtk if I shell out before the install finishes and brew link vtk then open a new terminal tab and run python and import vtk. So I'm happy about that. So it doesn't have the problem you bug reported last month.

Testing still needs to be done to prove to myself that the X11, tcl, and Qt4 bindings work. I'll get to that in a bit.

samueljohn · 2012-06-16T09:56:27Z

I'll test it on my systems. Yesterday, the build was not successful on my system but you pushed a new version since then. But I don't have much time till monday.

2bits · 2012-06-16T10:55:20Z

Sorry about that. I fixed an error. No rush here because Jack decided to close my request to modify keg_fix_install_names.rb or mach.rb.

samueljohn · 2012-06-16T14:07:52Z

@2bits, I have updated my vtk branch. I can now successfully build (and import vtk) against a brewed python with --framework. The other settings (python without framework) still have to be tested.

The problem I had (on my xcode-only system) was that I had to add
-DVTK_USE_TK:BOOL=OFF in order to have a successful build. I don't ever use TK.

It's all work in progress, though.

samueljohn · 2012-06-16T15:56:28Z

@2bits just a question: I have seen you removed the ":BOOL" and ":STRING" parts of the args given to cmake. So they are not needed?

samueljohn · 2012-06-16T16:01:48Z

Library/Formula/vtk.rb

+
+    # Remove duplicate files that will cause the final link phase to fail.
+    %w(easy-install.pth site.py site.pyc).each do |f|
+      rm pydir/f


If the --python option was not given, will this cause an error because the files are not there?

That's one of the things I fixed which got lost in a commit blizzard.

2bits · 2012-06-16T17:34:33Z

I just pushed a new commit to fix the missing if ARGV.include? '--python' for line 92.

I have seen you removed the ":BOOL" and ":STRING" parts of the args given to cmake. So they are not needed?

It turns out that those type modifiers are only needed if you preload the cache in a text file like we talked about on your XCode thread. When you supply cmake variables from the command line, they can be left out. Personally I find those obscure the code.

2bits · 2012-06-17T12:15:23Z

Those typedefs are also required if the variable doesn't exist in CMakeCache.txt, which happens when the variable is never used in CMakeLists.txt. So in the case of vtk, these two need the typedef:

      -DCMAKE_INSTALL_RPATH:STRING='#{lib}/vtk-5.10'
      -DCMAKE_INSTALL_NAME_DIR:STRING='#{lib}/vtk-5.10'

Currently I found my formula doesn't work with HB non-framework python entirely. Though I can run python and import vtk, I get an error running vtkpython that is patched for in fcgi.rb, namely

$ vtkpython
dyld: Symbol not found: _environ
  Referenced from: /usr/local/lib/libpython2.7.dylib
  Expected in: dynamic lookup

From reading this, https://github.com/Lokaltog/vim-powerline/issues/13#issuecomment-3997965 it turns out that exporting DYLD_LIBRARY_PATH=/usr/lib makes it work. Ugly. So blah and a general blerg.

It looks like Jack is going to pull a solution for #12810, though, which is cool.

2bits · 2012-06-17T12:21:11Z

In c898792 I added back in the typedefs for those two vars and changed to non-interpolated strings.

samueljohn · 2012-06-17T13:58:11Z

I have never used vtkpython. I always import vtk.

samueljohn · 2012-06-17T16:13:37Z

@2bits you have to add skip_clean :all in the vtk formula to avoid vtkpython fail. Look #11705.

samueljohn · 2012-06-17T16:20:40Z

On my system, I still need the -DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=ON flag and the ugly hack around that in order to get the linking right.
Further, -DVTK_USE_TK:BOOL=OFF to build successfully when --python is enabled, too.
Then it builds with homebrew python in both cases (with and without framework)

2bits · 2012-06-17T17:07:42Z

Thanks for the info about skip clean. I will try that.

I still need the -DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=ON flag and the ugly hack

This formula won't need the RPATH workaround nor the -DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=ON because Jack is going to pull jacknagel/homebrew@737a2f30ed
which patches for those problems by running install_name_tool on bundles as well as dylibs. In this case, the python .so files are Mach-O bundles.

Further, -DVTK_USE_TK:BOOL=OFF to build successfully when --python is enabled, too.

On that issue, I'd like to see a gist of your build/CMakeCache.txt. Thanks.
For reference, this is a gist of my CMakeCache from my version of vtk.rb when brew install -vd --python vtk.

2bits · 2012-06-17T17:30:52Z

SIde note: I tried adding a dep on ffmpeg and enabling that CMake variable, but got build errors due to API changes. I didn't try the dep on R or adding deps on libtiff, jpeg, etc.

samueljohn · 2012-06-17T17:58:11Z

I did not apply @jacknagel's patch. I'll test your formula with Jack's patch tomorrow.
I am glad if you could have a look at my build/CMakeCahce.txt (Note, I run the Xcode-only setup here already).

2bits · 2012-06-17T19:02:23Z

Sounds good. Make a gist of your CMakeCache.txt when you have some time.
The last commit, 1219ddf, adds skip_clean :all which solved my last problem.
Everything seems to work.

samueljohn · 2012-06-18T09:37:47Z

@2bits the difference between your and my CMakeCache.txt is due to my Xcode-only setup and cmake.
Sometimes cmake finds the libs and includes located in Xcode. Sometimes it does not.
Frameworks in /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk/Frameworks are not automatically found. The Frameworks in /System/Library/Frameworks do not contain the includes needed to build software (only the libs to run software).

Perhaps there is a cmake env var, I could set so cmake picks up the Frameworks from inside of Xcode.app?

//Path to a file.
TK_INCLUDE_PATH:PATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk/usr/include

this is fine, since the headers are symlinked from the framework to that dir.

//The path to the Tk internal headers (tkMacOSXDefault.h).
TK_INTERNAL_PATH:PATH=/tmp/homebrew-vtk-5.10.0-Rfpt/VTK/Utilities/TclTk/internals/tk8.5

this is bullshit. The private headers are not symlinked to Xcode.app/.../usr/include :-(

//Path to a library.
TK_LIBRARY:FILEPATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk/usr/lib/libtk.dylib

The same is true for opengl. The libs are found in /System/Library but not the freamework from inside Xcode.app.
Perhaps you have a hint for me, @2bits ?

//Include for OpenGL on OSX
OPENGL_INCLUDE_DIR:PATH=OPENGL_INCLUDE_DIR-NOTFOUND

//OpenGL lib for OSX
OPENGL_gl_LIBRARY:FILEPATH=/System/Library/Frameworks/OpenGL.framework

//AGL lib for OSX
OPENGL_glu_LIBRARY:FILEPATH=/System/Library/Frameworks/AGL.framework

samueljohn · 2012-06-18T10:45:27Z

Okay, I fixed the OPENGL and AGL stuff by adding
append 'CMAKE_FRAMEWORK_PATH', "#{sdk}/System/Library/Frameworks", ':' (I'll have to fix this in #10510 !)

However, since the std_cmake_parameters include CMAKE_FIND_FRAMEWORK=LAST, I think cmake mixes some includes from apple with some includes (the private ones) from the tk headers bundled with vtk. Investigating...

2bits · 2012-06-18T11:07:39Z

It sounds like you've gotten into some gory CMake issues. What I can offer is limited, but I'll mention that CMake does a good job of making separate Find modules that you should read when it's not finding what you want. They are located in

/usr/local/Cellar/cmake/2.8.8/share/cmake/Modules

From reading FindTCL.cmake, it looks like there are some variables you can preload. I don't know at this point how to tune the search directories globally, but you did come across the one, CMAKE_FIND_FRAMEWORK. So are likely more, but we'll have to dig for those.

samueljohn · 2012-07-16T07:50:37Z

Library/Formula/vtk.rb

+        else
+          args << "-DPYTHON_INCLUDE_DIR='#{python_prefix}/Headers'"
+        end
+


this line is empty.

samueljohn · 2012-07-16T09:16:16Z

Besides the caveats, I am super-satisfied now.
Please leave a small note in the caveats where the Examples are to find. (They are not yet copied into the cellar. If you like we can do this only if --examples is given as an option.)

Tested on my machines. Like charm. It runs!

2bits · 2012-07-16T16:14:02Z

Examples are small so I will just install them. They are worth it to someone who is crafting something complex with vtk. The caveats seem good, but they seem to imply that PyQt would work with vtk even without the --pyqt option. I'll try to reword that or explain why there is a --pyqt option. Any thoughts?

I didn't otool the --x11 files yet, but what I did was analyze the brew list --verbose vtk when I used no options versus when I used --x11 and I found the exact same number of files in the exact same directory structure. So it may be just a linking thing as you surmised. Thanks, I'll look into that.

samueljohn · 2012-07-16T16:36:27Z

The stuff in vtk.qt4 works even if built without --pyqt. It
imports pyqt on the fly if available (or PySide). However, if you
integrate the vtk widgets into an existing qt application you probably
need to build with --pyqt.

2bits · 2012-07-16T18:15:34Z

Right on. Then my long description for the --pyqt option needs work. What would you suggest is a better one? When looking at what that option does, it creates 1file for each of the 14 libraries in egg/vtk, e.g.

/usr/local/Cellar/vtk/5.10.0/lib/python2.7/site-packages/VTK-5.10.0-py2.7.egg/vtk/vtkChartsPython.so
/usr/local/Cellar/vtk/5.10.0/lib/python2.7/site-packages/VTK-5.10.0-py2.7.egg/vtk/vtkChartsPythonSIP.so
/usr/local/Cellar/vtk/5.10.0/lib/python2.7/site-packages/VTK-5.10.0-py2.7.egg/vtk/vtkCommonPython.so
/usr/local/Cellar/vtk/5.10.0/lib/python2.7/site-packages/VTK-5.10.0-py2.7.egg/vtk/vtkCommonPythonSIP.so
/usr/local/Cellar/vtk/5.10.0/lib/python2.7/site-packages/VTK-5.10.0-py2.7.egg/vtk/vtkFilteringPython.so
/usr/local/Cellar/vtk/5.10.0/lib/python2.7/site-packages/VTK-5.10.0-py2.7.egg/vtk/vtkFilteringPythonSIP.so

That listing is from before it moves vtk.

samueljohn · 2012-07-16T19:37:34Z

@2bits I read this as

the python bindings are built if --python is provided. And I tested that they work.
If --pyqt is given additionally, all the vtk (c++) classes are wrapped with SIP, too. This helps you, if you want to code a big PyQt application with interaction of your widgets with the widgets from vtk.

To the long description for our --pyqt option, I'd go with the description given by ccmake. And I think their description is vague as well. I actually think the current description is ok except that I would remove the "at runtime.".

Btw.: Concerning the examples. I just tested to build them, too. It works without problems, so basically we could add an --examples option that switches the examples ON and copies them to the cellar and prints the caveat.

samueljohn · 2012-07-18T16:04:26Z

Library/Formula/vtk.rb

+      system 'make'
+      system 'make install' # Finally move libs in their places.
+      # And move the examples into the Cellar, 2.5 MB, which are very useful.
+      system 'exit 1'


Sollbruchstelle? (This is a German word) :-D

samueljohn · 2012-07-18T17:10:52Z

@2bits no problem. Ping me for a final test-run.
Btw. if you are at it and have the time, perhaps adding the option --examples would be nice. It could switch on building the Examples -DBUILD_EXAMPLES=ON.

2bits · 2012-07-18T17:34:12Z

I decided that 4MB of examples without an option is more than justified considering any project using vtk is bound to be highly complex and would find this data handy.

samueljohn · 2012-07-18T17:36:44Z

That is true and I agree. But some may prefer to have the C/C++ and other examples to be compiled (which is harder to do it afterwars oneself). That takes extra time and therefore an option might be nice.

samueljohn · 2012-07-18T17:39:50Z

I am building your formula right now. Looks good ...

2bits · 2012-07-18T19:17:47Z

oic, the BUILD_EXAMPLES cmake option as a formula option. Sure why not. After 20 min. of compiling, I don't want to increase that. So an option sounds good along with code to only install the Examples directory if that option is thrown.
Thanks for pointing it out.

samueljohn · 2012-07-18T19:34:49Z

Perhaps we should bottle vtk. But not in this pull request.

You did a fantastic job on this formula!

2bits · 2012-07-18T19:54:42Z

Thanks. It was a team effort. Did we learn anything, though? Is there something we should be reporting upstream?

samueljohn · 2012-07-19T07:31:27Z

Well, I'd like to refactor all the python specific code and place it somewhere central. Not only which_python but the other support functions, too. Besides that the vtk formula looks quite clean, now.

The VTK team is working on a new, modular VTK 6 version. Let's hope, it's not too different/difficult to adapt :-/

jacknagel · 2012-07-19T08:08:07Z

Is this ready for final review, then? I should have some time tomorrow.

samueljohn · 2012-07-19T09:17:28Z

@jacknagel almost. @2bits wants to add the --examples option and add little caveat to tell where the examples are located. But the core of this formula works and I have tested the different python modules and GUIs, too.

On Xcode-only with the new XQuartz, there seems to be a problem though, but I'd not hold this formula any longer because of that.

Upgrade vtk to version 5.10. Add deps on `sip` and `pyqt` if the user options `--pyqt` Add cmake args to enable SIP/PyQt if `--pyqt` Add cmake args for `-DSIP_PYQT_DIR` as `HOMEBREW_PREFIX/share/sip`. Add `skip_clean :all` to help vtkpython find the symbol `_environ` Change the location of the python `.so` modules so they and the Egg get installed into `lib/which_python/'site-packages'`. Add code so `site-packages` is created and prepended to the PYTHONPATH to stop build errors. Add `-DPYTHON_INCLUDE_DIR` so the right headers are located. Remove the workaround code that was previously added to support Python getting RPATHs, as RPATHS are fixed in Homebrew Core. Add code that removes three duplicate files in `site-packages` that cause the linking step to fail, files that already exist in HB `site-packages` when using brewed Python. Add code to `mv` the python module directory `vtk` out of the `VTK-5.10-py2.7.egg` and into `site-packages`. Without that, you can't import vtk into Python. Add code to install the `Examples` Tested on Lion using clang and llvm from XCode-4.3.3 with the CLT and without the CLT against system Python, HB framework Python, HB non-framework Python, Qt-4.8.2, sip, pyqt, pyside and shiboken.

2bits · 2012-07-20T16:53:36Z

Amended to add --examples and a caveat.

samueljohn · 2012-07-23T11:47:02Z

@jacknagel this is ready for final review. @2bits has put huge amounts of time into it.

I'd pull it :-)

jacknagel · 2012-08-03T02:26:56Z

Pulled, sorry for the delay.

Upgrade vtk to version 5.10. Add deps on `sip` and `pyqt` if the user options `--pyqt` Add cmake args to enable SIP/PyQt if `--pyqt` Add cmake args for `-DSIP_PYQT_DIR` as `HOMEBREW_PREFIX/share/sip`. Add `skip_clean :all` to help vtkpython find the symbol `_environ` Change the location of the python `.so` modules so they and the Egg get installed into `lib/which_python/'site-packages'`. Add code so `site-packages` is created and prepended to the PYTHONPATH to stop build errors. Add `-DPYTHON_INCLUDE_DIR` so the right headers are located. Remove the workaround code that was previously added to support Python getting RPATHs, as RPATHS are fixed in Homebrew Core. Add code that removes three duplicate files in `site-packages` that cause the linking step to fail, files that already exist in HB `site-packages` when using brewed Python. Add code to `mv` the python module directory `vtk` out of the `VTK-5.10-py2.7.egg` and into `site-packages`. Without that, you can't import vtk into Python. Add code to install the `Examples` Tested on Lion using clang and llvm from XCode-4.3.3 with the CLT and without the CLT against system Python, HB framework Python, HB non-framework Python, Qt-4.8.2, sip, pyqt, pyside and shiboken. Closes Homebrew#12807. Signed-off-by: Jack Nagel <jacknagel@gmail.com>

2bits mentioned this pull request Jun 13, 2012

fix_install_names - should it fix .so files also? #12810

Closed

samueljohn reviewed Jun 16, 2012
View reviewed changes

samueljohn reviewed Jul 16, 2012
View reviewed changes

Library/Formula/vtk.rb

else

args << "-DPYTHON_INCLUDE_DIR='#{python_prefix}/Headers'"

end

Copy link

Contributor

samueljohn Jul 16, 2012

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this line is empty.

samueljohn reviewed Jul 18, 2012
View reviewed changes

jacknagel closed this in 2cc2739 Aug 3, 2012

Homebrew locked and limited conversation to collaborators Feb 16, 2016

vtk 5.10 & add pyqt+sip #12807

vtk 5.10 & add pyqt+sip #12807

Conversation

2bits commented Jun 13, 2012

2bits commented Jun 13, 2012

2bits commented Jun 13, 2012

samueljohn commented Jun 13, 2012

2bits commented Jun 14, 2012

samueljohn commented Jun 15, 2012

2bits commented Jun 15, 2012

jacknagel commented Jun 15, 2012

2bits commented Jun 15, 2012

2bits commented Jun 15, 2012

samueljohn commented Jun 16, 2012

2bits commented Jun 16, 2012

samueljohn commented Jun 16, 2012

samueljohn commented Jun 16, 2012

samueljohn Jun 16, 2012

Choose a reason for hiding this comment

2bits Jun 16, 2012

Choose a reason for hiding this comment

2bits commented Jun 16, 2012

2bits commented Jun 17, 2012

2bits commented Jun 17, 2012

samueljohn commented Jun 17, 2012

samueljohn commented Jun 17, 2012

samueljohn commented Jun 17, 2012

2bits commented Jun 17, 2012

2bits commented Jun 17, 2012

samueljohn commented Jun 17, 2012

2bits commented Jun 17, 2012

samueljohn commented Jun 18, 2012

samueljohn commented Jun 18, 2012

2bits commented Jun 18, 2012

samueljohn Jul 16, 2012

Choose a reason for hiding this comment

samueljohn commented Jul 16, 2012

2bits commented Jul 16, 2012

samueljohn commented Jul 16, 2012

2bits commented Jul 16, 2012

samueljohn commented Jul 16, 2012

samueljohn Jul 18, 2012

Choose a reason for hiding this comment

samueljohn commented Jul 18, 2012

2bits commented Jul 18, 2012

samueljohn commented Jul 18, 2012

samueljohn commented Jul 18, 2012

2bits commented Jul 18, 2012

samueljohn commented Jul 18, 2012

2bits commented Jul 18, 2012

samueljohn commented Jul 19, 2012

jacknagel commented Jul 19, 2012

samueljohn commented Jul 19, 2012

2bits commented Jul 20, 2012

samueljohn commented Jul 23, 2012

jacknagel commented Aug 3, 2012