migrate osx build to using jhbuild and modulesets

[totaam]

Issue migrated from trac ticket # 533

component: packaging | priority: blocker | resolution: fixed | keywords: osx focus

2014-03-14 13:06:01: totaam created the issue

[totaam]

2014-03-14 22:32:46: smo commented

---

Started work on this here is the first start. Hoping these can be added to svn somewhere.

Still needs work for dependencies however they all seem to build nicely.
Only one patch was needed for "nasm" to support make install DESTDIR= or jhbuild complains.

I'm attaching the first files to this ticket.

[totaam]

2014-03-14 22:33:31: smo uploaded file xpra.modules (11.4 KiB)

Xpra Moduleset

[totaam]

2014-03-14 22:33:38: smo uploaded file nasm-makefile-destdir.patch (0.9 KiB)

[totaam]

2014-03-17 19:33:24: smo commented

---

Added these in r5829

After doing

jhbuild bootstrap
jhbuild build
jhbuild buildone gstreamer
jhbuild buildone gst-plugins-base
jhbuild buildone gst-plugins-good

You should be able to now cd to trunk/osx/jhbuild

Run jhbuild -m ./xpra.modules build meta-osx-xpra-deps

This should build and install everything else necessary to build xpra..

Currently it is missing (py)gtkglext which is the next step to be worked on.

Optional if you want subversion updated and installed you can try this as well

jhbuild -m ./xpra.modules build meta-xpra-subversion

[totaam]

2014-03-21 11:14:27: totaam commented

---

Very good stuff, what a timesaver overall! I've updated the winswitch osx build page to refer to this moduleset.

For reference here's a link to the [/browser/xpra/trunk/osx/jhbuild jhbuild directory]

[[BR]]

Only found a few minor things that we want to tweak / note:

• I've bumped the version of Pillow (2.3.1) and lz4 (0.6.1) in r5882
• we should use https for the download links and/or add checksums for the downloads
• GTK and pygobject need patching (probably patch the module before building):

• include this patch to fix some warnings, at least until upstream finally merges it (if ever?)
• include this patch for fullscreen support (see support multi-head configurations better - fix fullscreen? #496)

• gstreamer update: we should either get gtk-osx upstream to bump the gstreamer version to the latest 0.10, carry a patch against the moduleset, or duplicate the gstreamer modules - no biggie
• libpng, libtiff and libjpeg: see update libpng and libjpeg in binary builds #544
• PyOpenGL should be bumped to 3.1 beta (see upgrade osx to PyOpenGL 3.1 beta #531), using the same download or build arguments as setuptools if needed (in the setup.cfg?), I've just recorded it as I did the upgrade:

Downloading https://pypi.python.org/packages/source/P/PyOpenGL/PyOpenGL-3.1.0b1.zip#md5=98cf868fac68e57d1712bc8b52bc8a4b
Processing PyOpenGL-3.1.0b1.zip
Writing /tmp/easy_install-18IUtE/PyOpenGL-3.1.0b1/setup.cfg
Running PyOpenGL-3.1.0b1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-18IUtE/PyOpenGL-3.1.0b1/egg-dist-tmp-L7SbV6
(...)
Downloading https://pypi.python.org/packages/source/P/PyOpenGL-accelerate/PyOpenGL-accelerate-3.1.0b1.zip#md5=8db7b69ba0553ca5b927fe2ddbf2424e
Processing PyOpenGL-accelerate-3.1.0b1.zip
Writing /tmp/easy_install-33_Ewg/PyOpenGL-accelerate-3.1.0b1/setup.cfg
Running PyOpenGL-accelerate-3.1.0b1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-33_Ewg/PyOpenGL-accelerate-3.1.0b1/egg-dist-tmp-7MQaJK

• important: I had problems resolving pypi.douban.com from here (today?) and even downloading from my UK servers it was pretty slow too at times (5KB/s at some point), which made building the python bits really tedious. Not sure if there's a better mirror we could/should be using? (easy_install must be using a different download domain, because it didn't have any problems)
• for sure, we should not be using pypi.douban.com for things like Cython (and probably others too) which have known good upstream download locations
• easy_install will automagically grab the latest release for us, whereas the moduleset will require updating everytime one of the dependencies is updated. I'm OK with that, it will make the upgrades more obvious (we'll have a commit reference). The old macosx build page has links to most of the dependencies to make it easier to check for new releases, we should probably do the same for the python dependencies, maybe here: [old macosx build page](http: winswitch.org dev macosx-manual.html) has links to most of the dependencies to make it easier to check for new releases, we should probably do the same for the python dependencies, maybe here: [Building](old macosx build page](http:--winswitch.org-dev-macosx-manual.html) has links to most of the dependencies to make it easier to check for new releases, we should probably do the same for the python dependencies, maybe here: [../wiki/Building) or in a new wiki page
• the ffmpeg build is the winswitch one, it should be the xpra minimal one instead, ideally we could have both in the module somehow (unfortunately, I don't think that jhbuild does "virtual packages" ala portage)
• nice to have: add gdb (not sure how to handle the chgrp and chmod at the end..)
• it would be nice not to have to use the --insecure download flag at all to bootstrap the process, but I can't think of an easy way..
• I seem to have a broken (py)gtkglext (opengl rendering on osx #226) again now... can't wait to see that streamlined!

[totaam]

2014-04-09 00:42:58: smo commented

---

• added libpng, libjpeg-turbo, libtiff in r6068
• added meta module named meta-osx-image-libs for easier building of the 3 libs

I've had mixed results with not using --insecure or -k with curl in the end I ended up having to add the flag to the tarball.py within jhbuild to get by some issues.

[totaam]

2014-04-14 19:28:57: smo commented

---

• PyOpenGL and PyOpenGL version bump in r6090 built both no problem still needs testing

[totaam]

2014-04-15 07:07:07: totaam commented

---

New updates:

• Pillow 2.4.0
• ffmpeg 2.2.1

For ffmpeg, I am now using a better, more limited configure line (also on win32):

./configure --prefix=${JHBUILD_PREFIX} \
    --cpu=i686 --enable-runtime-cpudetect --enable-pic --enable-memalign-hack \
    --enable-static --enable-shared --enable-gpl \
    --disable-everything \
    --enable-swscale --enable-libx264 --enable-decoder=h264 \
    --enable-libvpx --enable-decoder=vp8 --enable-decoder=vp9 --enable-decoder=hevc \
    --disable-protocol=tcp --disable-protocol=rtp \
    --disable-filter=aformat --disable-filter=crop --disable-filter=setpts \
    --disable-filter=anull --disable-filter=format --disable-filter=trim \
    --disable-filter=atrim --disable-filter=null \
    --disable-programs --disable-avfilter

It took a lot of fiddling and trial and error:

• --disable-all sounds like what we want... but it doesn't work: nothing gets installed
• we now get problems with assembler optimizations... maybe we should move to OSX Lion as a build platform and target 64-bit only in the future?

[totaam]

2014-04-15 10:28:07: smo commented

---

Added patch for gtkglext in r6096

Can use it by doing something like this

git clone git://git.gnome.org/gtkglext
cd gtkglext
curl -O http://xpra.org/svn/Xpra/trunk/osx/jhbuild/xpra-gtkglext.patch
patch -p1 <xpra-gtkglext.patch 
autoreconf -fiv
./configure --prefix=${JHBUILD_PREFIX}
make && make install

This isn't ideal but jhbuild doesn't allow me to apply a patch when pulling source from git only when using tarballs

Next is pygtkglext.

[totaam]

2014-04-15 20:50:59: smo commented

---

Added

python-macholib
python-modulegraph
python-altgraph 

in r6098

As I was unable to build without these perhaps easy_install pulls these in for us normally?

After compiling and installing these I was able to build and package xpra.

Installing pygtkglext is still quite a hack as the installing portion usually fails and requires you to install them manually into site-packages/

[totaam]

2014-04-18 10:33:45: totaam commented

---

New ones:

• ORC 0.4.19 is out and is worth upgrading to as it fixes memory leaks. download link
• libjpeg-turbo 1.3.1 is out. download link
• numpy 1.8.1 is out

These are going to keep accumulating until this ticket is closed..

As for the question above: macholib, modulegraph and altgraph are normally pulled in automatically when installing py2app via easy_install.

[totaam]

2014-05-01 21:51:56: smo commented

---

• ORC version bump in r6292
• numpy version bump in r6293

[totaam]

2014-05-09 05:40:21: totaam uploaded file pygtkglext-git.patch (17.7 KiB)

patch for building git version of pygtkglext

[totaam]

2014-05-09 05:45:53: totaam commented

---

The patch above builds pygtkglext for me using:

PYTHONPATH=$JHBUILD_PREFIX/share/pygobject/2.0/:$PYTHONPATH python2.7 \
    ./setup.py build

We cannot use setup.py for installing (see #226 for details), so we install it with rsync (ugly):

rsync -rpvlgto build/lib.macosx*/gtk/* $JHBUILD_PREFIX/lib/python2.7/site-packages/gtk-2.0/gtk/

Note: be aware of #563#comment:14 when choosing which version to use (git tree vs 1.1 snapshots)

[totaam]

2014-05-09 06:13:10: antoine uploaded file gtkglext-1.2.0.osx-hack.tar.gz (1340.3 KiB)

1.2 snapshot for osx found here: http://lists.gnu.org/archive/html/bug-gnubg/2011-04/msg00006.html

[totaam]

2014-05-14 11:20:09: totaam commented

---

Newer versions should be updated in the moduleset:

• nasm 2.11.04
• setuptools 3.6

Note to smo: work is going to keep piling on until you close this ticket...

[totaam]

2014-05-15 19:20:19: smo commented

---

updated versions in r6479

Compiling this whole moduleset to make sure its still working. I will use the patch that I made against gtkglext this time but should I be using this new hack that you attached? Have you tried this?

[totaam]

2014-05-16 05:26:57: totaam commented

---

The 1.2.0 patched version I posted is a newer version with some osx patches already applied, in particular the gtk>=2.20 compatibility patches. I haven't found any regressions, so I think we should use this rather than patching an older version ourselves.

[totaam]

2014-05-27 09:07:33: totaam commented

---

New:

• PyOpenGL 3.1b3 is now available.

[totaam]

2014-06-03 04:36:00: totaam commented

---

And more updates will continue to accumulate until this ticket is closed..

Today:

• Python 2.7.7

[totaam]

2014-06-10 06:15:11: smo commented

---

PyOpenGL version update in r6691
Python 2.7.7 I also changed the url in .jhbuildrc-custom and recompiled everything with no issues.

I have some modifications that work well for this as well that should be updated in the winswitch osx dev page.

Can we host gtkglext and the patch for pygtkglext somewhere on xpra.org so that I can put those into the moduleset as well?

Was your pygtkglext patch for the tarball?

[totaam]

2014-06-10 06:21:40: totaam commented

---

Can we host gtkglext and the patch for pygtkglext somewhere on xpra.org so that I can put those into the moduleset as well?
[[BR]]
Sure, point me to them and I'll upload them.
[[BR]]
Was your pygtkglext patch for the tarball?
[[BR]]
Which one?

[totaam]

2014-06-10 08:08:52: smo commented

---

They were both attachments to this ticket that you posted before however as I see now the pygtkglext patch is for git which won't work with jhbuild.

We can however put the hacked copy of gtkglext in the moduleset and that removes one more thing.

[totaam]

2014-06-10 08:55:52: totaam commented

---

Done: [http://xpra.org/src/gtkglext-1.2.0.osx-hack.tar.gz]

[totaam]

2014-06-11 07:38:53: smo commented

---

Added gtkglext-1.2.0.osx-hack.tar.gz in r6720

had to configure with --enable-gtk-doc or make install failed with missing html doc file

[totaam]

2014-06-11 16:26:23: totaam commented

---

I've had to pull the 0.12.4 OSX build because pygtkglext is now completely broken (not that I could easily have noticed since I don't have GL in my macosx vms...), rebuilding it against the latest gtk+ update does not seem sufficient to make it run, see #593.
You can easily get a crash just by trying to import gtkgl:

python -c "from gtk.gtkgl import *"

The error seems to be the one mentioned here Fatal Python error: can't initialize module gtk.gtkgl but I can't see anything wrong, this same source code used to build and run fine...

[totaam]

2014-06-12 05:55:40: totaam commented

---

I've just reviewed the GTK+ changes since 2.24.16, and we want quite a few of those (many are also relevant to win32 rebuilds discussed in #263#comment:10):

• Quartz: Cursor doesn't reset when the mouse leaves a toplevel window (2.24.21)
• Text input is too slow and some keys are broken after changing input source (2.24.20)
• OS X: Various input handling fixes: (2.24.21)

• Avoid a crash on first key press (I have seen this one before)
• Fall back from native input method to GtkIMContextSimple
• Avoid beeping when using native input method
• Reset preedit state when needed

• null pointer dereferenced due to multiple bugs in gtkclipboard-quartz.c (2.24.23)

[totaam]

2014-07-15 00:25:50: smo commented

---

r6897 updates many libraries

I had issues trying to rebuild after this as gst-plugins-ugly links against x264

jhbuild -m ./xpra.modules uninstall gst-plugins-ugly-xpra 
jhbuild -m ./xpra.modules -f buildone gst-plugins-ugly-xpra 

Should be enough to get it building properly again.

[totaam]

2014-07-24 09:47:13: totaam changed priority from minor to critical

[totaam]

2014-07-24 09:47:13: totaam commented

---

It's taken a while but I think we will soon be able to close this ticket!

Please update as discussed:

• Python 2.7.8
• PyOpenGL 3.1 and PyOpenGL_accelerate 3.1: had to do the latter by hand as easy_update failed to build it for some reason (not investigated)
• Pillow 2.5.1 (and check webp is enabled)
• ffmpeg 2.3
• x264 stable latest
• py2app 0.8.1
• gdb 7.8 which as Betters Python Scripting
• gtk+ 2.24.24
• gdk-pixbuf 2.30.8
• pango 1.36.5
• glib 2.40
• gmp 6.0.0 - important: please specify a build host type to prevent building cpu optimizations which may not be available at runtime
• setuptools 5.4.1
• libpng 1.6.12
• pyobjc 3.0.1 (worth a try - first update in almost 2 years, should be a drop in replacement, more lightweight too)

Finally, you can find three patches which allow us to build (py)gtkglext from git which I have added to svn in r6940 + r6941:

• gtkglext: xpra-gtkglext.patch and gtkglext-osx-quartztagfix.diff

git clone git://git.gnome.org/gtkglext
cd gtkglext
curl -O "http://xpra.org/trac/browser/xpra/trunk/osx/jhbuild/xpra-gtkglext.patch?rev=6940"
patch -p1 < xpra-gtkglext.patch
autoreconf -fiv
curl -O "http://xpra.org/trac/browser/xpra/trunk/osx/jhbuild/gtkglext-osx-quartztagfix.diff?rev=6940"
patch -p1 < gtkglext-osx-quartztagfix.patch
./configure --prefix=${JHBUILD_PREFIX} --with-gdktarget=quartz
make && make install

• pygtkglext: pygtkglext-osx-v4.patch

git clone git://git.gnome.org/pygtkglext
cd pygtkglext
curl -O "http://xpra.org/trac/browser/xpra/trunk/osx/jhbuild/pygtkglext-osx-v4.patch?rev=6940"
patch -p1 < pygtkglext-osx-v4.patch
./autogen.sh --prefix ${JHBUILD_PREFIX}
make && make install

If needed for the moduleset, we can host the patches in a more canonical location, or even make a patched source archive with the patches already applied.

Note: the pygtkglext patch is not so different from the previous version (which may still work?), and I've seen weird things happen as I rebuilt the libraries. I believe jhbuild leaves some crud behind and we must forcibly remove the build directories before trying to rebuild a library. Something to keep in mind.. since I don't see a way of forcing jhbuild to completely remove a build directory.

It may be worthwhile adding a switch somewhere so we can get debug builds for all those libraries. Not sure exactly how to add -g to the CFLAGS however.

Once that's all done and tested, please re-assign to me so I can close this.

[totaam]

2015-04-24 12:41:02: antoine commented

---

(just setting milestone)

[totaam]

2015-04-30 09:08:10: antoine commented

---

Tested with a completely clean user account on my 10.5.8 VM and got this error:

configure: error: Oops, mp_limb_t is 32 bits, but the assembler code
in this configuration expects 64 bits.
You appear to have set $CFLAGS, perhaps you also need to tell GMP the
intended ABI, see "ABI and ISA" in the manual.
*** Error during phase configure of gmplib: ########## \
    Error running ./configure --prefix /Users/osx/gtk/inst --libdir '/Users/osx/gtk/inst/lib'    *** [71/97]

Fixed by re-adding --build=i386-darwin. That's odd, is anything else building for 64 bit??
@smo: we may have to re-add this one and workaround it for 64-bit.

Another really annoying problem is that the apr downloads seem to disappear with new releases (see [http://www.eu.apache.org/dist/apr/]), which is a really bad idea for those building from source like us, maybe we should mirror those? Otherwise we run the risk of not being able to build things in the future.
I've used apr 1.5.2 for building and updated the moduleset, also updated subversion to 1.8.13

[totaam]

2015-04-30 15:56:26: antoine commented

---

I have just uploaded an osx beta build (tagged with r9198) made from this moduleset environment.
@afarr: testing it wouldn't hurt, making sure there are no regressions from previous "non moduleset" builds.

PS: unless problems are reported against these builds, from now on all the 0.15 builds (and later) will be made from this "moduleset" environment.

[totaam]

2015-05-01 04:02:08: antoine commented

---

Just noticed an important update missing: gtk 2.24.27, see:

• [http://ftp.gnome.org/pub/gnome/sources/gtk+/2.24/gtk+-2.24.25.news]
• [http://ftp.gnome.org/pub/gnome/sources/gtk+/2.24/gtk+-2.24.26.news]
• [http://ftp.gnome.org/pub/gnome/sources/gtk+/2.24/gtk+-2.24.27.news]

There are some crashes fixed in these releases.

[totaam]

2015-05-07 21:53:20: smo changed status from new to closed

[totaam]

2015-05-07 21:53:20: smo set resolution to fixed

[totaam]

2015-05-07 21:53:20: smo commented

---

r9279 updates gtk to 2.24.27 builds fine

I'm going to close this now while I can and move on to 64bit builds.

[totaam]

2015-07-31 18:20:32: antoine commented

---

@smo: FYI r10091 bumped x264 to a newer stable snapshot. This caused hard to decipher problems at package time because some dylibs still referred to the old one. (libx264.142.dylib vs libx264.146.dylib)

Fixed by rebuilding gstreamer with:

jhbuild buildone -f 

I wished jhbuild had the equivalent of revdep-rebuild under Gentoo. There might be a way of cooking up a check script using otool to verify all the dylibs found in gtk/inst/lib can be loaded.
Until then, we'll fix things when they break and hope the breakage manifests itself at package time rather than at runtime..

Similar dependency problem in #971.

Follow up for 64-bit in #840.

[totaam]

2016-01-10 13:03:09: antoine commented

---

@smo: FYI lots of updates in r11631.

[totaam]

2017-02-16 17:16:46: antoine commented

---

Note: the pygtkglext patch should be updated to support "get_depth" which we commented out: #1443