How to achieve uniform behaviour across platforms?

[vog]

I noticed that lots of the issues of MXE could be solved if MXE would behave more uniformly across all those different platforms. I wonder if we could fix those issues with a better design. This GitHub issue is meant to collect ideas and opinions on the future design of MXE.

Although MXE is faily portable, there are always (and have been) various issues that appear on one platform but not on another. The means people are accidently breaking the build for others without even noticing. Some past problems were even depending on which other software packages are installed on the build system. And I fear that there are possibly lots of subtle problems that nobody noticed yet, waiting to be found the hard way.

The previous testing phases were mostly about different problems of various distros, instead of about whether the cross-compiled applications actually work. Also, the current master has some hairy problems so that I wonder if we should even bother announcing another testing phase. On the other hand, the current testing branch became quite old (#136), so it becomes a less and less useful alternative to the master branch.

Of course we could approach those issues by having virtual machines (VMs) for any distro, which check that all packages build on all platforms, and which probably even automatically file bug reports in the case of discrepancies. But even then, we'd had to create portability fixes one by one for each package on each platform. But in the end, we have just one target platforms (or two, if you count Win32 and Win64 separately).

So I wonder if we could fix those issues with a better design, rather than a piling stack of portability fixes.

The following proposals come to my mind:

1. Design MXE for a single VM

As an extreme idea, we could design MXE for a single, specific virtual machine. At the risk of starting a distro war, let's assume we're using Debian. It is a well maintained, easily installed binary distro, and I'm familar with it ;-). So we'd provide a disk image that anyone can download and run within VirtualBox, KVM or whatever. It would contain a script that updates the Debian system (if needed), and runs the latest MXE. Maybe we'd even have a bootstrap script that builds a Debian VM from scratch such that it fulfils all MXE requirements. Upside: An identical system for everyone, full reproducibility, almost no chance to introduce a bug that affects other but not yourself. Downside: Only really usable on systems with hardware virtualization, otherwise too slow. Also, triggering an automatic build and moving the resulting binaries from the VM is a bit cumbersome. One would have to use SSH or SSHFS or similar.

2. Design MXE for Chroot, LFS-like

We could also design MXE more LFS-like (http://www.linuxfromscratch.org/). So we'd build a full new toolchain within a new Chroot environment we create, with everything compiled from source, ensuring that all native libraries/tools are in the same version, and built from the same sources, as their cross variants. We wouldn't use any pre-installed native tools except for bootstrapping. To build a package, one would chroot into that environment and use that toolchain, disconnected from the main system. Upside: An identical system for everyone, full reproducibility. Also, copying from and to the chroot environment is trivial, as it is in the same file system. Automatic tasks simply call some scripts through chroot. Downside: Chroot usually requires root privileges, but you could drop those provileges to normal user after entering. Also, it is possible that this chroot environment would work only on Linux systems and not on MacOSX or FreeBSD. Or, maybe some tweaks are necessary for Mac and/or BSD, but then the environment would be different on those platforms, which is what we wanted to avoid in the first place. So in the worst case, Mac and BSD users would have to use a Linux VM to build and use this Chroot environment.

3. Use existing build systems

We could make use of existing infrastructure such as Fedora's MinGW packages, openSUSE's MinGW packages or Debian's dpkg-cross. We'd have to decide on one of those platforms, and reshape our src/.mk files to be valid build files for that platform. *Upside: Using existing infrastructure, such as standardized source package formats and existing build farms. Downside: Lots of overhead that made me create MXE in the first place. :-) Also, people with a different distro would probably still have to use a VM, unless they get their final applications (EXE files) built by those build farms, too.

Please share your opinion, refine those ideas, and/or provide more ideas.

[woncount]

Just briefly regarding the VM idea mentioned in your first idea: While debian is obviously a solid foundation (using it myself!), I would probably suggest using a distro that has explicit "remastering" support - preferably for different deployment scenarios. For example a distro based on "TurnKeyLinux" like mentioned in #129

The point is, by targeting a single distro, we'll also want to make remastering as simple as possible - so using a distro that has explicit remastering support and a remastering communtiy would seem useful.

Finally: TurnKeyLinux is based on debian ;-)

[woncount]

Regarding #3 it's also worth mentioning that the toolchain built by mxe is MUCH more up to date than the mingw stuff available in most distros, which are usually based on gcc 3.x - I was pleasantly surprised that mxe uses a fairly recent version of mxe.

[lm8]

Having a base set of programs that work the same on all systems could help. For instance, when I'm switching between Linux and FreeBSD, I noticed tools like sed and patch don't necessarily work the same across platforms. FreeBSD at least offers the GNU version of make if you need it, but they rename it gmake. If a baseline set of build tools that worked the same on each system was available, I think there would be less porting problems. How to get that baseline is another issue.

[mabrand]

Although MXE is faily portable, there are always (and have been) various issues that appear on one platform but not on another.

I'd like to know more about the scope and impact of the problem.

Which platforms have the problems? My experience is that MXE works well on recent Debian and OpenSuse and I imagine that's true of their close cousins. Are most of the problems on the bsds and mac?

I'm aware of the sed issue which seems to be handled nicely by $(SED).

Also, different autotools versions are sometimes a problem. MXE builds of native autotools have been suggested. Volker has pointed out that native building could actually be thornier for MXE than cross building. Does anyone know if building native autotools would be hard in practice?

What are the other known problematic tools? (Sorry if I'm overlooking a list somewhere.) If someone has a list, could you post in this thread so the scope of the problem is clearer?

[tonytheodore]

Are most of the problems on the bsds and mac?

I'd say so, we can cover most of the usual portability issues (bashisms, GNU userland) quite easily, but there are some harder problems:

• FreeBSD hasn't updated to the latest glib so there's no guidance on getting it to work
• MacPorts has automake 1.13 which fails with deprecation errors
• Apple (and FreeBSD soon) have moved to clang with patchy c++11 support, so we end up with hacks like qtbase.mk
• Other *BSDs use a pre-GPL3 version of gcc (4.2.1) that won't build the latest gcc (last time I checked)

In contrast, Debian GNU/kFreeBSD only needed an updated config.guess for some packages and some existing patches from packages.debian.org.

Does anyone know if building native autotools would be hard in practice?

I'm guessing it should be fairly easy, the last time I had to do this was for Open Solaris and it was a standard ./configure && make && make install.

To avoid conflicts with the normal package system, we could install them under the mxe prefix and use something like tools/mxe-activate to set the appropriate env vars. It's less than a chroot and the general idea works well in practice (inspired by https://bitbucket.org/dhellmann/virtualenvwrapper which I use all the time for a sane python setup these days)

I like the idea of VMs and I think it's fairly safe to assume anyone attempting to build mxe probably has the hardware to support virtualisation. The minimal Debian and Ubuntu VMs start up in seconds and I rarely see more than a 5% increase in build times.

VMs have their own issues though. Currently on Ubuntu, something like pcl.mk will merely fail a parallel build and then retry with a single job. On Debian, the parallel build causes a kernel panic and puts the host in a state that needs a restart to recover - I've never seen anything like it.

[tonytheodore]

We could also design MXE more LFS-like (http://www.linuxfromscratch.org/)

Apart from it's own merits, this would be a nice approach if we ever wanted to support a glibc based target.

[tonytheodore]

Also, triggering an automatic build and moving the resulting binaries from the VM is a bit cumbersome. One would have to use SSH or SSHFS or similar.

We could take some ideas from vagrant [1] and use the shared folder functionality to automount a host directory in the VM.

The VM approach also means windows users could use mxe - I've never been able to get it working with either msys or Windows Services For Unix.

[1] http://docs.vagrantup.com/v1/docs/getting-started/ssh.html

[woncount]

Regarding the whole VM-based approach, I've had plenty of experience building MXE inside VMs during the last couple of weeks, using VirtualBox and OpenVZ containers mostly - most portability issues have gone meanwhile, so I agree with Tony here.

So, I'd probably suggest to really consider the VM-based approach, and it is my understanding that TKL (TurnKeyLinux like mentioned in #129 ) handles lots of ugly details here, so that "remastered" TKL distros can be easily deployed in various formats, for different virtualization techniques.

And after all, it's TKL is still based on Debian 6, too. The major difference being that it's a distro designed to be remastered and deployed to different VM solutions.

Also, I have to agree with previous comments regarding availability of hardware, resources - and mxe becoming available on other platforms that way, too.

I would also like to point out that it will be much easier that way to also provide other features, such as integration with IDEs or even build farms through the use of ccache and/or distcc. All of this can be nicely managed via SSH, with preconfigured credentials - and build artefacts could be retrieved from the VM through SCP/SFTP, too.

Finally, there are professional IDEs that have native "Remote Development" support (like Eclipse or NetBeans), where logging into a machine through SSH (be it a VM or not!) is supported "out of the box" by the IDE, so that the VM's development tools can be used to configure/build software and even download the binaries, do remote debugging through gdb, profiling and even leak-checking.

In addition, using virtualization as part of the mxe concept could even mean that unit-tests could actually be run, too - i.e. through qemu, so that console programs that should build properly, can even be tested in a qemu machine. Overall, I'm really convinced that using virtualization means more opportunities and potential for mxe, and it seems the entry barrier is fairly low -and for new mxe users it will even be lower than before, simply because TurnKeyLinux (#129) is all about providing so called "virtual appliances" that are completely preconfigured and which can be directly downloaded, booted/installed and used.
Speaking as a fairly new mxe user, I find that that most straightforward option,

Finally, note that these remote development details can be easily further abstracted by installing a dedicated build server inside the same mxe VM, i.e. something like Jenkins for example - which obviously requires a JRE, but is incredibly powerful, and basically works extremely well with remote development IDEs like Eclipse or NetBeans.

Regarding "remote development" support in Eclipse/NetBeans, I'd suggest to check out these links:
http://netbeans.org/kb/docs/cnd/remotedev-tutorial.html
http://help.eclipse.org/juno/index.jsp?topic=%2Forg.eclipse.ptp.rdt.doc.user%2Fhtml%2Ftoc.html

All in all, these can be pretty compelling arguments, especially to people who are primarily interested in developing software for windows on their non-Windows OS.

[lm8]

FreeBSD hasn't updated to the latest glib so there's no guidance on getting it to work

I'm using an older version of FreeBSD, but I thought later versions of FreeBSD had switched to using llvm. Another option on FreeBSD, there's an emulation mode that lets you run Linux applications similar to native ones. If I remember correctly, you just have to turn on the emulation mode, copy all the files you need (executables and libraries) and brandelf the Linux files. Some people are running Linux printer drivers that way on FreeBSD since FreeBSD doesn't have the drivers for certain printers and Linux does.

Does anyone know if building native autotools would be hard in practice?
I'm guessing it should be fairly easy, the last time I had to do this was for Open Solaris and it was a standard ./configure && make && make install.

I'd be very curious about that. Tried building some of the autotools on Windows and wasn't having much luck. (Maybe it's easier on Linux.) If anyone runs across more information on this, please send me some pointers to it.

To avoid conflicts with the normal package system, we could install them under the mxe prefix and use >something like tools/mxe-activate to set the appropriate env vars. It's less than a chroot and the general idea >works well in practice

Don't know if it's the best solution for MXE, but I like the idea of setting environment variables over chroot.

Here are some issues I hit when building cross-platform applications directly on FreeBSD. If MXE uses the same tools, it could cause similar issues. I'm not sure which version of sed is installed on my FreeBSD machine, but I gave up using it, because it was behaving too differently and ended up building and using minised instead. That doesn't exactly work the same as sed either, but at least I can get a baseline version of sed that works the same across FreeBSD, Linux and Windows. I've seen some Slackware scripts use Perl instead of sed. That results in more differences in syntax. For instance, parenthesis have delimeters if they're literal in one case and if they're grouping marks in the other case. Had some issues with patch on FreeBSD. Remember it couldn't successfully patch a file with a non-standard internationalized character. The character wasn't in utf-8 format and the FreeBSD patch program couldn't handle it. Ended up having to use sed in place of patch for that file.

Only drawback I see to a VM is that you'd need a computer with enough resources to run it. That lets out my FreeBSD machine at the moment. (I have an older, low resource test machine I use for running FreeBSD and testing cross-platform applications.) However, it sounds like it certainly would cut down on many issues related to building for multiple environments. I'll be very interested to see which way MXE goes. There are definitely pros and cons to any approach chosen.

[tonytheodore]

Another option on FreeBSD, there's an emulation mode that lets you run Linux applications similar to native ones.

From memory, Linux binary compatibility is very complete and easy to use (I've used it for Oracle and the Flash plugin in the past). There's still the issue of generating and distributing those Linux binaries - I can see that would be more problematic than bootstrapping a minimal GNU userland for our purposes.

Don't know if it's the best solution for MXE, but I like the idea of setting environment variables over chroot.

I find setting env vars is handy for development and experimentation, but the chroot opens many possibilities like a glibc toolchain for exactly the same versions across all targets, and even a third stage bootstrapping of an msys-like environment for windows.

If MXE uses the same tools, it could cause similar issues. I'm not sure which version of sed is installed on my FreeBSD machine, but I gave up using it, because it was behaving too differently and ended up building and using minised instead

We've standardised on GNU make/patch/sed, check for them in https://github.com/mxe/mxe/blob/master/Makefile#L13, and add them to the pkg_add instructions. This works well in practice, you should give it a try.

Had some issues with patch on FreeBSD

The version in the base system is very old, install it from ports/packages and adjust your path so /usr/local/bin is before /usr/bin (see http://mxe.cc/#requirements-freebsd). Up till the last release, I'd say these types of issues were a solved problem.

Only drawback I see to a VM is that you'd need a computer with enough resources to run it.

That's a tricky one, it's hard to know what the minimum requirements would be and what the average developer runs these days - I tend to think that mxe's user base would be in the upper quartile of hardware specs.

On the other hand, VMs also make it easy to run on other infrastructure so local hardware requirements become less of an issue. Since we added the pkg mirror/cdn, it's costing a coffee or two a month. I know CPU time is more expensive, but I've got some idle hardware that could be put to use if it was trivial to set up.

[tonytheodore]

There's still the issue of generating and distributing those Linux binaries

To take @woncount 's ideas a step further, we could ship a VM that already has a fully built mxe environment or retrieve the build artefacts remotely. This would remove a lot of the resource requirements since the VM only has to share a filesystem.

[woncount]

I tend to think that mxe's user base would be in the upper quartile of hardware specs.

I have to agree here, developers usually tend to have better-than-average hardware, which often even competes with top-notch "gaming" hardware - often, this is even a problem from a development point of view, because software is developed and tested on "state-of-the-art" hardware, instead of consumer/end-user hardware.

In this particular use-case and scenario I would argue that this is an advantage, because mxe's "end users" actually ARE developers, who are likely to have corresponding hardware resources available. Besides, it's not that the hardware requirements are ridiculously high at all - I just tested building mxe in a single-core 1ghz, 1024 MB OpenVZ container, and it just worked (only tested the gcc build).

At work, we even have build farms and clusters where we can run stuff on,not just SMP/multicore boxes, but literally clusters in the sense of HP blade servers with dozens of computers acting as a single environment.

we could ship a VM that already has a fully built mxe environment or retrieve the build artefacts remotely. This would remove a lot of the resource requirements since the VM only has to share a filesystem.

Yes, that was the idea in #129 : not just providing the mxe build script, but a completely pre-built and pre-configured mxe environment. Setting this up such that the file system is accessible via NFS/SSHFS or SMBFS (Samba for Windows), would be a huge usability improvement in my opinion.

Basically, new users would download a debian 6.x based TurnKeyLinux image, install it in VMWare/VirtualBox/QEmu or OpenVZ and could directly use mxe without having to build it from scratch, and the build artefacts would automagically show up via a remote file system such as NFS or SMB.

[woncount]

From a usability perspective, TurnKeyLinux being based on Debian, we could even have our own mxe-specific debian repository to serve updated mxe packages via apt, too - which would make it pretty straightforward to deploy updated packages to mxe end users, who would not necessarily need to have the git repository checked out then or rebuild things from scratch.

And just briefly, regarding Jenkins support, here's a list of plugins related to making build artifacts available via various means, including network shares, rsync copying, ssh, sftp, ftp etc: https://wiki.jenkins-ci.org/display/JENKINS/Plugins#Plugins-Artifactuploaders

While the Jenkins idea may seem a littl efar-fetched to most of you probably, that's what all mxe users ultimately care about: building their binaries via mxe, getting the build artifacts and testing/deploying them.

[woncount]

Only drawback I see to a VM is that you'd need a computer with enough resources to run it. That lets out my FreeBSD machine at the moment. (I have an older, low resource test machine I use for running FreeBSD and testing cross-platform applications.)

If that should turn out to be a serious issue, some of us with idle cores could volunteer to run an MXE VM and enable a distcc daemon via SSH inside it to "donate" horsepower to those folks with less CPU power. Using something like DynDNS that should work fairly well.

http://en.wikipedia.org/wiki/Distcc
https://code.google.com/p/distcc/

[tonytheodore]

@woncount distcc looks very interesting and is somewhat orthogonal to this issue - we could start doing something with it now. I always thought it wouldn't work for cross-compiling, it's mentioned on the homepage but I couldn't find much information about it. The closest was this http://www.gentoo.org/doc/en/cross-compiling-distcc.xml but I didn't get very far.

Do you have any setup/config notes? If so, could you raise a new issue to discuss using distcc with mxe? Thanks.

[woncount]

I don't know how typical a use-case distributed compilation via distcc really is for mxe users ? I only mentioned it because of the previously raised concerns regarding lack of horsepower when running VM-based MXE installations on old FreeBSD hardware. Sorry, didn't mean to hijack this discussion.

[tonytheodore]

@mabrand wrote:

Does anyone know if building native autotools would be hard in practice?

@tonytheodore wrote:

I'm guessing it should be fairly easy

This is true for autotools, but I've been fiddling with guile2 for for the past few days (needs it's own current native version to cross-compile itself). My idea of setting env vars sort of works, but I think it would be fragile at build time across systems - it's probably better suited to interpreted languages like make and python.

It looks like @vog has been there before #136:

In other words: We have a single, well defined toolchain for cross compiling,but not for native compiling.
[...]
a) build all native tools from source, possibly even the compiler

As far as I can tell, guile2 can't be built (in an easily repeatable fashion) with the standard compilers on FreeBSD and OSX. We'd have to build a new compiler to build the native build; the LFS-like route. For these two systems, I don't think we've "dropped the ball" in terms of support, we're pretty much at parity with the package systems - but they are going in different directions and have some hard problems to solve.

Are most of the problems on the bsds and mac?

I am sad to say this, but I'd drop the level of support and consideration we give these systems to "nice to have". The most we can do is develop an exclusion mechanism to provide partial support (which will be a large - if not complete - subset of actual use).

[tonytheodore]

I stumbled across the mxe-octave project which builds (among other interesting things) native versions of autotools, cmake, libtool, and pkg-config:
http://hg.octave.org/mxe-octave

[woncount]

that looks interesting, the question is if we want to (and can) reuse useful stuff there, or maybe even team up with mxe-octave to provide a single mxe distribution that satisfies all shared goals ?

[vog]

@tonytheodore: Thanks for bringing this to my attention. I don't think we'll be able to merge with this project anytime soon, at least not before we support shared builds in addition to static builds. However, I think that mxe-octave it worth mentioning at our website in the "Related projects": http://mxe.cc/#see-also

[tonytheodore]

My current thinking is that VMs are more of a deployment/implementation detail - they don't really play a part in the design of a system. I'm leaning towards an LFS-like system and my next set of experiments will be to see how a native build (and possibly a Canadian Cross) fits into a multi-target approach - likely extended to a multi-host/target.

Distribution of build artefacts is possibly more an extension of #134; if we're going to separate build stages, we might as well install to a DESTDIR, create a manifest (for later uninstall) and *.mxz file, then install from the *.mxz file (and optionally make it available for distribution). The *.mxz file is a yet-to-be-specified mxe package archive.

[woncount]

had some time to play around with TurnKeyLinux and mxe, after the initial setup, everything is really simple to deploy/replicate now - in fact, I am considering now to upload a TKL appliance with mxe support, and maybe even provide debian packages for amd-x86_64 just to reduce the installation overhead.

Most of the time, having access to just a VM that cross-compiles things would suffice for me ... obviously, the next cool thing would be jenkins/CI integration (i.e. or through eclipse/netbeans), so that the VM would be hooked up to a build server farm or the IDE, so that projects can be easily cross-compiled just by providing SSH credentials.

Bottom line being, TurnKeyLinux is all about deployment (for different virualization solutions), but at the end of the day it is simply debian 6.0 - so even just providing deb packages (and an apt repo) would drastrically reduce the setup overhead. The question is if the mxe project could handle maintaining such an apt repo ?

I think we only need to be offering 64bit packages, because a 32bit system is unlikely to run mxe,right ?

[tonytheodore]

had some time to play around with TurnKeyLinux and mxe, after the initial setup, everything is really simple to deploy/replicate now

Thanks for the update, good to hear you're having some success with it.

The question is if the mxe project could handle maintaining such an apt repo ?

What's involved in setting this up? It doesn't sound like a trivial exercise.

I think we only need to be offering 64bit packages, because a 32bit system is unlikely to run mxe,right ?

There are open issues with ocaml on 32-bit systems and lower memory systems can't take full advantage of parallel options - if you have a choice, target 64-bit build systems only.

[TheJosh]

Is there a reason why the current system couldn't simply be used as-is, but a pre-built VM option provided (as well as a script in the repository for creating said VMs), and then users can continue to run locally, and if they have issues switch to the VM?

[tonytheodore]

The only problem with that is one of perception from the user. I imagine I'd be quite annoyed if I'd been running a build for hours, it fails, and the solution is to install a VM??

If we're going to provide VMs, we should be clear that those are the recommended way to use MXE and only people wanting to work on MXE itself should try building it.

[lbguilherme]

One proposal could be to use a docker container for building MXE. This is somewhat equivalent to using a VM, but without the overhead and slowness on hardware without virtualization. The only requirement here would be to run a Linux kernel version 3.10 (Jun 2013) or newer (I believe FreeBSD and some others are supported too, not sure). For systems with other operating systems or older Linuxes, a VM could be provided to simply run docker (there are many ready images for that, boot2docker for instance).

The pros here are a fully controlled and reproducible environment, with easy methods of exchanging files with the host. Things could run as simply as:

docker run mxe i686-w64-mingw32.shared-gcc main.c -o main.exe

I don't really see a downside for this.

[vog]

I agree that Docker has become an option. It is a nice alternative to a hand-written chroot or LXC / jails environment. However, I'm still a bit suspicious about relying on Docker, as it is some kind of container standard with just one implementation. But I guess that's is just a matter of personal taste.

The exact technology is up to whoever will sit down and implement that thing. :-)

Does anyone volunteer to implement that thing in Docker? (or any other technology?)

[lbguilherme]

I agree that Docker is not perfect, but right now is the most mature option, the only viable one IMHO. Anyway, a implementation using it would be convertible to any other container alternative, they don't differ too much.

I'll be working on a prototype of this.

[TimothyGu]

I big +1 from me. I would prefer if the base system is Debian since that is what I'm most used to. Docker is open-source and on the rising edge, so I wouldn't worry about compatibility for now :)

[vog]

+1 for Debian/Stable (currently Jessie) from my side, too.

[lbguilherme]

@TimothyGu @vog Debian was indeed my choice too.

Here is the initial prototype: https://github.com/lbguilherme/dmxe

The idea is to provide a command line tool for two tasks: Creating images and running them,
where image has a single MXE_TARGET and a set of mxe libraries. Updating anything involves recreating the image from scratch (slower, but will always be very clean). When running the image as a container, it will mount the current working directory as a volume and open a shell directly on it. So running commands there will affects the files outside. I've also added symbolic links to all binaries on mxe/usr/bin removing the target part. So gcc points to i686-w64-mingw32.static-gcc. Note: Currently there is no way of debugging an error, all logs are simply lost.

I didn't try compiling anything non-trivial yet, I'm mostly looking at the design of it for now. What do you think?

[vog]

This looks very promising. Nice work! However, I have two minor objections to the current state, though:

1. Do we really need 3 executables here? (dmxe, dmxe-build and dmxe-run). I'd prefer a single executable, named dmxe. But maybe there are important reasons to split it?
2. I'm afraid the prefixless symlinks will to more harm than good. With that in place, the application's build system may believe it is in a native build rather than a cross build. Also, many applications want access to both, the native and the cross compiler. (e.g. to run a code generator written in C/C++, or if you have a compiler project that needs to bootstrap itself)

If those issues are fixed, I propose to include it into MXE's tools/ directory, and to put the contents of your README.md into a new section of MXE's documentation (index.html). Finally, announce it at the mailing list and encourage people to try tools/dmxe for compiling their application(s).

[tonytheodore]

Here is the initial prototype: https://github.com/lbguilherme/dmxe

Great start! Works as expected on OSX using the new docker toolbox (which also runs on windows).

[lbguilherme]

@vog:

1. Not really, it was more for organization and it is indeed better to merge them. Done.
2. You are right here, I removed the symbolic links and also allowed more than one target on the same image.

---

I added support for pulling the repo and rebuilding just what changed, thus it doesn't need 't need a full rebuilt. I think it is mostly read. I'm now running a full build of all targets (will take a while...) to ensure it all works and there's still documentation to write.

[UniqueFool]

Regarding the docker comments, some of you may find this interesting: https://www.packer.io/intro

[ashkulz]

I just proposed an approach for building packages more regularly in #2021, and if that gets implemented it's trivial to create a docker image: just include the repository in sources.list.d -- no need to push an image to Docker Hub.

For other platforms (e.g. macOS / FreeBSD), you do need a VM-based approach. I think vagrant would be a good choice, like I'm doing with wkhtmltopdf.

[Zirias]

Another possibility could be configurable build options, and for FreeBSD, maybe use jails for building packages.

I just managed to cross-compile my work-in-progress Qt application for windows, fully statically linked, using the latest MXE on FreeBSD. Now that it works, it's awesome. I had the following issues:

• building with clang causes several problems. MXE doesn't find GCC because FreeBSD ports of GCC only install binaries named like gcc8, g++9, etc. I had to create symlinks on my system myself, same thing for python -> python2.7. Some configuration options and a recommendation to install and use a GCC port on FreeBSD would help with this.
• For glib, I had to add building of the kqueue part. This is only enabled for darwin systems, but FreeBSD needs it, too. That's where I think a configurable build option might come in handy -- easier than trying to autodetect what has to be done on any system.
• Another similar thing: I couldn't build cmake at first because it tries to build a bundled curl, which tries to link against the system-wide OpenSSL. I use libressl on my system instead and curl is known to be incompatible with libressl. In my case, adding a --system-curl in the cmake package solved this issue. Again something that one could make configurable.
• I had yet another problem with cmake because some stuff on my host system in /usr/local/lib/cmake broke the build. This is where I think building in a jail would have a huge benefit. Unfortunately, this would be a FreeBSD-only thing and require quite some work to get it running :(

I don't really like the idea to use a vm, as this vm would probably also be needed to use MXE, and one great benefit of MXE for me is that I can do my native and cross builds from the same source tree without even switching terminals. In fact, adding Linux as a cross-compilation target would be awesome ;)

[vog]

I wouldn't mind adding FreeBSD-specific changes to MXE, but would prefer to have them generically.

Example: It should use "g++9" instead of "g++" not because the system is FreeBSD, but because the system has no "g++" but "g++9".

Same for Python, etc.

Another example: --system-curl could be added unconditionally, as I'm pretty sure this will work on all Linux systems as well (but we should test that nevertheless, of course).

So, @Zirias, feel free to create pull requests which fix some or all of the issues you identified.