Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows/OS X cross builds generalization and cleanup. #1928

Merged
merged 47 commits into from
Apr 21, 2014
Merged

Conversation

aszlig
Copy link
Member

@aszlig aszlig commented Mar 12, 2014

This is a merge of my cross-building branches for OS X and Windows (MinGW-w64) and this branch is for working on generalization of both cross-builds and also cleaning up old cruft.

Also, again, I'd welcome any comments/ideas/suggestion on that topic (already asked for that in #1876).

The remaining todos/questions for after the merge for 14.04 are:

  • Add a stdenv.isDarwinTarget, stdenv.isLinuxTarget and stdenv.isWindowsTarget which essentially are something like for example isWindows = if stdenv ? cross then stdenv.cross.libc == "msvcrt" else stdenv.isWindows, so if you're not cross-compiling, the *Target attribute reflects the native architecture.
  • Remove old MinGW(32)-specific stuff. Non-MinGW-w64 builds are broken right now anyway and if noone starts to shed a tear, I'm going to remove it as MinGW-w64 is able to compile to 32 and 64 bit PE.
  • Check whether things such as opensslCrossSystem are still necessary and why they were introduced in the first place.
  • Enable running of tests using runInWindowsVM and runInMacOSXVM.
  • Clean up release-cross.nix.
  • Improve cross-building with cmake.
  • Evaluate use of --with-sysroot for all cross-architectures in gcc-stage-final (according to the GCC
    docs, the options --with-headers and --with-libs are deprecated).
  • Maybe allow to assign/match platform meta attributes with cross targets?

I've added a jobset for native builds and for cross builds on my Hydra. The latter only consists of x86(_64) NixOS machines, so I can't test whether anything breaks something like for example native Darwin builds.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This package wasn't imported by anything in the package tree and also
isn't used for cross-builds. So in order to avoid confusion, it's better
to remove it as it is outdated as well.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
In Windows there is no get(e)uid() call available, so the build fails.

The patch now checks whether OPENSSL_SYS_WINDOWS is defined and only
uses those calls if _not_ on Windows.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
At least for x86_64-w64-mingw32, it doesn't make sense to use the native
strip tool for stripping of symbols. To the contrary it results in
unusable archive files.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Not sure why this even got into the upstream release, as _all_ other
includes use forward slashes.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
D'oh, no wonder the headers are of almost the same size as the main
package.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
The winpthreads library is part of the same source package, so let's
just override the name and the source directory.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
That way we're able to build with the JIT compiler enabled.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This also cleans up the package expression a bit and we no longer
directly override the phases (except patchPhase).

Also for cross-builds we're using the stock mingw target and pass CC, LD
and STRIP to make, because the mingw-w64 targets of the upstream package
are outdated.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This is to make sure that we don't get Xlibs or alsa in cross builds,
because those aren't available on non-Linux/Unix platforms.

Also, until we don't have the DirectX SDK packaged, let's disable it
during cross builds.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This is because autoconf is passing -print-prog-name=ld to the
cross-gcc, which in turn assumes a FHS compliant filesystem hierarchy
and searches ../../../../$crossConfig/bin/ld for the correct ld.

Of course, this won't work on Nix, hence we're explicitly passing the
correct LD program name.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Especially for the 64bit mingw target, because libgcrypt contains
protected mode assembly.

Also, this adds gettext to the crossAttrs of libgpg-error, because it
tries to regenerate the MO files.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Mingw(32) is rather poorly maintaned and has quite a lot of bugs. And
because our Windows cross builds were also poorly maintained and most of
the cross-tests were broken as well, I'm just taking this step and try
to switch to mingw-w64 for everything "cross Windows".

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This is no longer needed, because we've already fixed the dynamic
cross-linker.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This really hase some nice utilities for coping with HFS+ and DMG
format. We're going to use it for extracting the xcode DMG, but it could
be used for more, even generating DMGs.

The reason I'm using the fork instead of the repository from
@planetbeing is thet the fork contains a few fixes, especially for
dealing with symlinks in HFS+ images.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This package provides the SDK and standard library needed for
cross-compiling to Mac US X. We're using xpwn here to extract the DMG.

Also, this version (XCode 5.0.2) only contains the SDKs for version 10.9
and 10.8, so we might need to add requireFile directives for older
versions as well.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This basically is binutils for Mac OS X, but ported to work on
(GNU/)Linux and FreeBSD.

And it's up-to-date as well! I'm mentioning this, because it was quite
hard to find a recent port of it and I just accidentally stumbled on it
while trying to do the port by myself.

So thanks to @tpoechtrager for doing this.

Also, I've added two more patches, which essentially are:

 * ld-rpath-nonfinal:
     This allows -rpath to be used for linking non-final builds, which
     was allowed for earlier versions of cctools and got a check for
     that in more recent versions.

 * ld-ignore-rpath-link:
     Ignores the -rpath-link option, because the cross-wrapper uses it
     in different places. Unfortunately, the cctools linker doesn't
     support it, so we might need to implement this later if it's
     possible (I'm not a Mach-O man^H^H^Hexpert).

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Let's finally hook everything into the existing cross-building
infrastructure. We're using --with-sysroot instead of --with-headers
here, because the XCode SDK contains references to /usr/lib.

I've tried to patch those references, but unfortunately (at least with
install_name_tool) it isn't possible to change those refernces in stub
dylibs.

So after bugging @tpoechtrager with annoying questions (thanks again), I
think my initial approach (patching the SDK itself and/or regenerating
the dylib stubs) was way to complicated so I ended up with this
implementation.

Also, I've added a condition to binutilsCross to use cctools if the libc
is set to libSystem. This might need some cleanups someday, mainly to
figure out how to properly bridge cctools and binutils.

So, as an example on how to cross-compile GNU Hello to Darwin, you can
use something like this:

(import <nixpkgs> {
  crossSystem = {
    config = "x86_64-apple-darwin13";
    arch = "x86_64";
    libc = "libSystem";
    platform = {};
  };
}).hello.crossDrv

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
We're already using tzcode inside tzdata, so let's make it available as
an output here. The reason we need it is in order to compile OpenCFLite.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Cross-compiling stuff against Mac OS X's CoreFoundation won't work
without ObjC support, and we don't want to compile commandline utilities
only, right?

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This toolchain contains Mach-O binaries and might not be useful in the
first place, but there are programs like dsymutil, where Apple didn't
release the source code, so we need a Mach-O loader...

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This provides a port of Mac OS X's CoreFoundation and is needed if we
want to be able to run dsymutil using maloader.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This is the mentioned Mach-O loader that we're yoing to use to execute
Apple's proprietary binaries.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
At the moment, this includes only dyldinfo, dwarfdump and dsymutil, but
we'll see whether we need more of these utilities later.

Tho reason those are wrapped in cctools-port is because it is the
binutils used to cross-compile for Mac OS X.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
The ld from cctools doesn't like the -soname argument, so let's strip it
off for now until we have a binutils <-> cctools bridge.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
The Makefile still contained calls to install with the -s (strip)
option, which we don't want because it uses the native strip utility and
we're going to crossStrip in fixupPhase later anyway.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This is just a minor fix, because when using "darwin64-x86_64-cc" for
config.openssl.system, the OpenSSL build scripts try to compile with
$prefix-cc, which is not available with the gcc-cross-wrapper.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Provide the correct cross ranlib, because native ranlib will fail.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This allows to pass a new attribute osxMinVersion to crossSystem, which
specifies the minimum Mac OS X version you want to be compatible to.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This version just got released two days ago, while we were working for
cross-builds on 5.0.2. From the release notes it shouldn't introduce any
incompatibilities.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Both branches have quite a lot in common, so it's time for a merge and
do the cleanups with respect to both implementations and also generalize
both implementations as much as possible.

This also closes #1876.

Conflicts:
	pkgs/development/interpreters/lua-5/5.2.nix
	pkgs/development/libraries/SDL/default.nix
	pkgs/development/libraries/glew/default.nix
	pkgs/top-level/all-packages.nix
On MinGW, we're passing these programs to the configure script, but this
obviously won't work for non-autoconf-based projects.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This reverts commit c5ab2bf.

As mentioned in the previous commit, this is now no longer needed.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
@shlevy
Copy link
Member

shlevy commented Mar 19, 2014

@aszlig Can you summarize the important changes? There's a lot in here, which is great, but it's a bit hard to review as-is.

@aszlig
Copy link
Member Author

aszlig commented Mar 20, 2014

In case of MinGW it's mostly fixup and cleanup away from building with old MinGW(32) towards MinGW-w64 plus fixes of some packages.

For the Darwin part, it's a new cross-target based on XCode and cctools plus again fixes of some packages.

This PR contains the merge (c7bac81) of both of them, because quite a lot of these fixes and the crossbuild implementation share quite a lot in common, so I'm using this branch to clean them up and generalize the repetitious code.

@offlinehacker
Copy link
Contributor

So what does cross builds for osx mean(i lack knowledge about osx)? Will we be able to build packages for osx on linux?

@aszlig
Copy link
Member Author

aszlig commented Mar 25, 2014

@offlinehacker: Essentially, yes and that's already possible with this branch, although it needs the mentioned cleanup/generalization before being ready to be merged.

@offlinehacker
Copy link
Contributor

Seems reasonable, i reviewed changes as far as i understood them and they sure look good to me :)

@offlinehacker
Copy link
Contributor

Ok i was testing a few cross builds for mingw, and it works well for me. Can you give/add crossSystem config for darwin?

Unfortunately github's *.patch URIs contain version of git generating them,
which changes from time to time. It seems that *.diff doesn't suffer from that.

Conflicts:
	pkgs/development/libraries/libarchive/default.nix
@aszlig
Copy link
Member Author

aszlig commented Mar 30, 2014

Cool, thanks for reviewing/testing, for Darwin I'm currently using:

crossSystem = {
  config = "x86_64-apple-darwin13";
  arch = "x86_64";
  libc = "libSystem";
  platform = {};
  osxMinVersion = "10.7";
  openssl.system = "darwin64-x86_64-cc";
}

@offlinehacker
Copy link
Contributor

Thanks, this works great :)

@aszlig @edolstra: Do you think there's any chance we get cross builds(espeically mingw) with hydra. What i would like to see is building some binaries statically, and have them ready for distribution in different kinds of projects that have needs for having statically built binaries avalible. Now i have to do this with my own hydra instance. One example of this is bunch of projects that use bin-wrapper from nodejs(ex. https://github.com/yeoman/node-optipng-bin) and i've said that i can provide statically built binaries for different platforms. For now i'll be building with my own hydra instance, but it would be great if hydra.nixos.org could build such things :)

@shlevy
Copy link
Member

shlevy commented Apr 5, 2014

This all looks awesome, is there any more testing or work you need help with?

Thanks to @offlinehacker for the fixes.

* 'crossdev' of github.com:offlinehacker/nixpkgs:
  gifsicle: add optional static builds, make gifview optional
  libjpeg_original: add optional static builds
  optipng: use system libpng & zlib, fix cross builds and add ...
  xcode: fix hash
  libarchive: *permanently* fix patch source and hash
@aszlig aszlig added this to the 14.04 milestone Apr 19, 2014
@aszlig aszlig merged commit 2d3376e into master Apr 21, 2014
aszlig added a commit that referenced this pull request Apr 21, 2014
This includes a lot of fixes for cross-building to Windows and Mac OS X
and could possibly fix things even for non-cross-builds, like for
example OpenSSL on Windows.

The main reason for merging this in 14.04 already is that we already
have runInWindowsVM in master and it doesn't work until we actually
cross-build Cygwin's setup binary as the upstream version is a fast
moving target which gets _overwritten_ on every new release.

Conflicts:
	pkgs/top-level/all-packages.nix
@vcunat vcunat deleted the cross-win-osx branch November 30, 2015 10:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants