building all the dependencies from source on win32

[totaam]

Issue migrated from trac ticket # 678

component: platforms | priority: blocker | resolution: fixed | keywords: win32

2014-09-14 07:21:34: totaam created the issue

---

Split from #90 (see also #640). Similar to #533 for OSX.

Note: cross compiling is not an option because of the Cython extensions which seem to confuse mingw..

Some links:

• Compiling the GTK+ (and Clutter) stack using Visual C++ 2008 and later
• python3 and gtk3 support #90 has a lot of information
• Some links in this answer to Where can I download precompiled GTK+ 3 binaries or windows installer?
• [http://www.tarnyko.net/repo/gtk3_build_system/]
• [http://www.gtk.org/download/win32_tutorial.php]
• [http://opensourcepack.blogspot.com/p/pygobject-pygi-aio.html]
• [https://wiki.gnome.org/action/show/Projects/PyGObject]
• msys2
• [https://github.com/Alexpux/MINGW-packages]
• [http://tesla.duckdns.org/jewelkit-1-0-released/]

[totaam]

2014-09-16 14:15:41: totaam commented

---

Note: this is required for #263#comment:12.

[totaam]

(added new links found on the mailing list)

[totaam]

2015-07-03 02:52:10: antoine commented

---

GTK+3 build environment for Win32/Win64 - temporary fork for upstreaming

[totaam]

2015-08-14 12:22:39: antoine commented

---

GTK3 is not looking good.

[totaam]

2015-11-09 10:10:47: antoine commented

---

All GTK3 things will now be tracked in #640, we still want to do this for GTK2.

[totaam]

2015-11-27 17:32:04: antoine commented

---

A solution based on hexchat's [https://github.com/codekiddy2/Visual-Studio-gtkmm/wiki].
Also: [https://github.com/fanc999/gtk-msvc-projects] - MSVC Projects to build GTK+ from Stock Installation of MSVC 2008-2013

See this thread: [https://www.mail-archive.com/gtkmm-list@gnome.org/msg19515.html].

[totaam]

2016-03-26 06:33:46: antoine commented

---

Becoming more urgent every day (missing out on fixes and new features added since 0.24.10! that's lots of them), ie: #1131#comment:5

[totaam]

2016-03-27 18:52:35: antoine commented

---

The latest hexchat scripts can be found here: [https://github.com/hexchat/gtk-win32], for GTK3 see #640#comment:36

[totaam]

2016-04-14 19:16:51: smo changed owner from smo to antoine

[totaam]

2016-04-14 19:16:51: smo commented

---

I've gone ahead with getting a machine ready to compile what we can with the hexchat scripts.

Not done yet but maybe we can get a headstart by using the precompiled binaries from hexchat for testing?

https://hexchat.readthedocs.org/en/latest/building.html

https://dl.hexchat.net/gtk-win32/vc14/x86/gtk-Win32.7z
https://dl.hexchat.net/gtk-win32/vc14/x64/gtk-x64.7z

[totaam]

2016-04-15 04:02:34: antoine changed owner from antoine to smo

[totaam]

2016-04-15 04:02:34: antoine commented

---

Building GTK itself is documented here: [https://github.com/hexchat/gtk-win32/blob/master/README.md].
It does include all the latest versions, which is great.
(re-scheduling)

[totaam]

2016-04-28 07:00:05: antoine commented

---

Build GStreamer on Windows: A Guide

[totaam]

2016-05-14 08:05:13: antoine commented

---

There are 3 tickets that should be consolidated into one: #917, #678 and #300

[totaam]

2016-06-22 05:23:03: antoine commented

---

We'll probably have to also make 64-bit builds in order to support newer CUDA and NVENC versions for shadowing (#558) as the latest sdks don't support 32-bit..

Preliminary steps:

• hexchat build setup instructions, but install python in the more standard location (root of C drive) and choose Python 2.7 and 3.4 (3.5 is not supported yet by cx_freeze or the gi bindings)
• install Microsoft Visual C++ Compiler for Python 2.7
• install tortoisesvn for accessing the xpra source code (optional)
• install pip for python 2.7
• using pip, install most of the python Dependencies for both python 2.7 and 3.4 (the exact names to use may vary slightly in the pypi registry), some special cases:

• py2exe 0.6.9 installed from exe for python 2.7, not needed with python 3.x
• pywin32 installed from exe

• problematic dependencies: lz4, lzo, pillow, cryptography
• build x264, libyuv, vpx and ffmpeg and place them in XPRA_WIN32_BUILD_LIB_PREFIX

[totaam]

2016-06-23 19:22:39: smo commented

---

From the hexchat build setup instructions I installed Visual Studio 2015 Community and didn't end up with what was needed for building.

Should install the 2nd one instead Visual C++ Build Tools 2015

It also installs less stuff and installs much faster.

[totaam]

2016-06-23 19:32:27: smo commented

---

Also since it only uses msys2 for tools like patch etc.. it is not necessary to have the 64bit version

[totaam]

2016-06-23 23:56:57: smo commented

---

Following instructions for hexchat I can now build both win32 and x64 builds of gtk and the other libraries needed for Hexchat.

Hexchat's build system does this by using vcproj files to facilitate building from the command line.

We will have to do the same for some of the libraries that you listed as problematic since we'd like to build these all with msvc as well.

[totaam]

2016-07-07 08:30:12: antoine changed priority from critical to blocker

[totaam]

2016-07-07 08:30:12: antoine commented

---

Blocker for #1072 and #1230

[totaam]

2016-07-12 17:52:23: antoine commented

---

Milestone renamed

[totaam]

2016-09-27 10:07:01: antoine commented

---

We'll have to make sure everything runs obviously, so this should cover #628.

[totaam]

2016-12-28 15:19:08: antoine commented

---

Edited a brain fart: building the GTK bundles from source

Then I noticed that there are also some files that are supposed to build pygtk2 for win32: [https://github.com/Alexpux/MINGW-packages/tree/master/mingw-w64-python2-pygtk]
And jhbuild was used at least once for cross compiling: Cross Compiling Jhbuild.
And there's even a jhbuild for windows: [http://afuera.me.uk/jhbuild-windows/]

Another thing worth considering: do we want to continue to support 32-bit windows? Maybe just leave that for the 1.0.x branch and focus on 64-bit? (so we can use more modern things like AVX acceleration in openssl, NVENC, etc)

[totaam]

2016-12-30 11:25:20: antoine commented

---

Closed #300 and #917 as duplicates of this ticket.

I now have two machines with everything installed as per hexchat's building the GTK bundles from source. One 32-bit VM ("Windows 7 Pro x86") for regular builds and one 64-bit dedicated system ("Windows 7 Pro x64") with an Nvidia card for NVENC and hardware testing.

Note: I had to install "Visual Studio 2015 Community" and not "Visual C++ Build Tools 2015". The latter caused build failures, it seems that the directory layout does not match the expectations of the powershell build script.

Next we need to build a number of extra packages, some of those may be available via MSYS2 or from here: [https://github.com/Alexpux/MINGW-packages/tree/master/], others may be found here: [https://github.com/codekiddy2/Visual-Studio-gtkmm/wiki/Packages], OTOH:

• gtkglext + pygtkgl
• yasm / nasm
• openssl
• gmp / mpfr?
• libjpeg-turbo?
• gstreamer, its plugins and their dependencies - we can make progress without this first, and we may be able to get by using the binaries if needed - this may help: Build GStreamer on Windows: A Guide
• x264 / ffmpeg / libvpx and their dependencies
• python (probably easier to just call setuptools / pip): cython, rencode, netifaces, python-cryptography, pillow, numpy, lz4, lzo, pyopengl, pycuda, pynvml, gi, py2exe / cx_freeze
• python win32: pywin32, comtypes, wmi
• opencv?

How do we update this build script? fork it? (PITA) or maintain an overlay?
ie: we want a newer libpng.

The packaging scripts will also need to be modified to find (py)GTK in its new location.

[totaam]

2016-12-31 09:24:16: antoine commented

---

For setting up a pure mingw development environment (here for 32-bit):

pacman -S base-devel msys2-devel mingw-w64-i686-toolchain

Since the packages exist in MSYS2, why can't we just use that? (and those are also built from source if needed - or we can tweak the PKGBUILD file).

Here's a list of versions found as of today here: [https://github.com/Alexpux/MINGW-packages], looks like things get updated fairly regularly too:

• pygtk2 2.24.0 and all its dependencies - this is already using the latest versions too:

• gtk 2.24.31
• glib 2.50.2
• gdk-pixbuf 2.36.0
• pygoject2 2.28.6
• libjpeg-turbo 1.5.1
• libpng 1.6.26 (needs updating)

• winpthreads (useful for ffmpeg and others)
• orc 0.4.26
• yasm 1.2.0
• x264 [r2721](../commit/1abd158360c5b905f56d0b0aef9fa12264e0007f).72d53ab
• ffmpeg 3.2: will need tweaking to reduce the dependencies as per x265 encoding #445#comment:4
• opus 1.1.3, flac 1.3.1, wavpack 4.80.0, gmp 6.1.1, lame 3.99.5, libmad 0.15.1b, libmatroska 1.4.5: all present
• gstreamer 1.8.3: we can trim the plugins during the packaging phase
• gobject-introspection bindings 1.50.0
• lz4 1.7.1
• lzo2 2.09
• xxhash 0.6.1
• openssl 1.0.2j: no 1.1 yet, no biggie
• python2 2.7.12: 2.7.13 should land soon I expect
• python3 3.5.2
  And it even includes the python packages:
• numpy 1.11.1
• pillow 3.3.0
• pywin32 219
• setuptools 28.6.0

extras we may want to use:

• opencv 3.1.0
• xpdf (tools only) 3.04
• evince 3.22.1
• ghostscript 9.20
• switch to libssh / libssh2 instead of executing putty?

Good tip: MSVC and MinGW DLLs, or how we can generate lib / def files.

[totaam]

2016-12-31 12:27:18: antoine commented

---

With the pure mingw approach:

• looks like we would need the enum vs gflags patch (shouldn't this be upstreamed?): python-gobject has wrong types
• pywin32 will have to be replaced: 751: mingw-w64-python-pywin32 is on git but not in the repos, or install using a wheel package? or using unoffical windows binaries
  We use it for:
• win32con everywhere..
• win32api.SetConsoleTitle
• win32console in wait for input
• win32api.EnumDisplayMonitors, win32api.MonitorFromWindow, win32api.GetMonitorInfo: gui screen detection
• win32com.propsys (win32_propsys_set_group_leader): window hooks
• win32api.GetSystemMetrics, gdi32.GetDeviceCaps, win32gui.GetDoubleClickTime: settings
• win32api.GetWindowLong / win32gui.SetWindowLong: gui
• win32api.ClipCursor: grabs
• win32gui.FindWindow, win32api.SendMessage in show_desktop
• win32api.MapVirtualKey, win32api.GetAsyncKeyState, win32api.GetKeyState, win32api.GetKeyboardLayoutList, win32api.GetKeyboardLayout, win32gui.SystemParametersInfo: keyboard
• win32com.shell.SHGetFolderPath, win32api.RegOpenKey, win32api.RegQueryValueEx: paths, etc
• win32print.EnumPrinters: printing
• win32process.GetWindowThreadProcessId, win32gui.GetWindowText, win32gui.GetWindowRect, win32gui.EnumWindows, win32gui.GetDesktopWindow, win32gui.GetWindowDC, win32ui.CreateDCFromHandle, etc. for shadow
• winxpgui and win32gui, win32gui.Shell_NotifyIcon, win32api.GetCursorPos: tray
• win32ts.WTSRegisterSessionNotification / win32ts.WTSUnRegisterSessionNotification: events
• comtypes webcam
  It is a lot, but managing all the builds provided by MSYS2 ourselves is probably worse..
  Most of these calls can be changed to ctypes relatively easily.
  It may help to write unit tests for these functions to make sure we re-implement them correctly.

Easily installed with pacman:

• yasm, lz4, lzo2, xxhash, openssl
• python modules: python2-numpy, python2-pillow, cython2 (only at 0.24.1)
  Then with setuptools, these python modules:
• rencode, python-lz4, xxhash, cryptography, nvidia-ml-py, netifaces
• comtypes, wmi (will these actually work without pywin32?)
  Python modules that need fixing:
• lzo (needs pointer to the C source code)
• py2exe or cx_freeze
• gtkglext + pygtkglext: the big ones, if that's too hard then we can look at GTK3 again (we have the GTK source so we can build cython extension if we need the speed)
• pycuda (needs CUDA installed first)

[totaam]

2017-01-01 09:59:17: antoine commented

---

Added:

• subversion, cmake
• python2-cx_Freeze
• libvpx, x264, ffmpeg
• gst-plugins-good, gst-plugins-bad, gst-plugins-ugly
  Added using their own installer:
• ghostscript
• gsview
  Added with easy_install:
• PyOpenGL + PyOpenGL_accelerate

Needs sorting out:

• uglifyjs + yuicompressor
• libyuv (see patch and cmake . && make)
• websockify (installed OK via easy_install, but needs our patches for win32)

With those changes and also some minor fixes to the build file: r14673, r14674, r14675, r14676
I can build xpra (as long as I replace win32con with a handmade copy) using:

python ./setup.py build_ext --inplace --without-enc_x265
python ./setup.py install_exe --install=./install-dir

The x265 plugin crashes hard, probably a bug in our code - we should either disable it or remove it altogether.

The resulting executables are almost usable, apart from the pywin32 bits missing.. (and pygtkglext)

[totaam]

2017-01-01 11:43:12: antoine uploaded file libyuv-mjpeg.patch (0.3 KiB)

(incomplete) patch for compiling libyuv with mingw

[totaam]

2017-01-01 14:47:47: antoine commented

---

gtkglext and pygtkglInstall:

• gtk-doc
• gobject-introspection
• gtkglext

Download pygtkglext (the download links on the gnome gtkglext project site are dead):

git clone https://github.com/GNOME/pygtkglext
cd pygtkglext
#remove sanity check which would fail:
sed -i -e 's/if not pkgc.*/if False:/g' setup.py
#tell the build tools where to find codegen:
export PYTHONPATH=${MINGW_PREFIX}/share/pygobject/2.0/
#fix the pygtk dsextras class to support gcc 6:
wget "https://xpra.org/trac/raw-attachment/ticket/678/dsextras-gcc.patch"
patch -p1 -d ${MINGW_PREFIX} < dsextras-gcc.patch
#build pygtkglext:
python ./setup.py build
#install phase fails on "add_template_option", see #147 for details
#so do it by hand:
rsync -rplogtv build/lib.mingw-2.7/gtk/* ${MINGW_PREFIX}/lib/python2.7/site-packages/gtk-2.0/gtk/

With r14677 (detect mingw and use minor build tweaks) + r14678 (honour install directory for pyopengl modules), I get a working pygtkgl installation!

[totaam]

2017-01-01 15:03:39: antoine uploaded file platform-win32.diff (10.0 KiB)

temporary patching for build with mingw, without any pywin32 modules

[totaam]

2017-07-16 12:43:46: antoine commented

---

More regressions caused by the move to ctypes (these method signatures should be generated correctly once and for all): #1545, r16289.
Python3 fix: r16281.

[totaam]

More pywin32 legacy code removed in r17317.

[totaam]

Some recent update broke setuptools (that whole thing is a terrible mess), you get ImportError: No module named 'pkg_resources._vendor.packaging'.
More information on that here: No module named pkg_resources.

The solution I have used to get going again (and hope that MSYS2 fixes the package sooner or later), on 64-bit:

pacman -S mingw-w64-x86_64-python2-pip
rm -fr /mingw64/lib/python2.7/site-packages/setuptools*
pip install setuptools

Edit: see #2033

[totaam]

FWIW pywin32 may get ported to MSYS2: [https://github.com/msys2/MINGW-packages/pull/5615]