Toolchain re-org - target dependencies

[tonytheodore]

Following on from #919...

This implements a fairly straightforward two-phase build where cross targets depend on the native target. Visibility is lost about gcc depending on gmp et. al. (#516), but at some point in the toolchain you just have to rely on implied dependencies.

The ordering dependencies like binutils --> pkgconf and <most pkgs> --> gcc remain, these can't be removed until the use cases for phases are fully understood (requires attempts at clang and glibc).

Native builds are enabled by specifying:

$(PKG)_TARGETS  := $(BUILD) [$(MXE_TARGETS)]

but otherwise there's no change in normal usage or *.mk files.

There's no simple way to alter some gcc install directories, so it's mv and rm.

The main impact will be on build-pkg.lua where an ordered build is required. I've added an initial

make print-build-order [MXE_TARGETS=...]

target that is the most reliable way (I know) to determine what make thinks it needs to do. @starius, I'd recommend moving the logic from getPkgs and sortForBuild into the main Makefile (if not the creation of tarballs and manifests altogether).

[starius]

<most pkgs> --> gcc

typo? Should be gcc --> <most pkgs>

[starius]

Package gcc is compiled with the following log message:

[build]     gcc                    i686-w64-mingw32.static

Can you print phase as a part of this log message?

[tonytheodore]

typo? Should be gcc -->

Probably just a matter of context. If talking about dependencies I'd go left to right in the sense "most pkgs depend on gcc", build order would be the reverse of that "gcc is built before most pkgs".

Can you print phase as a part of this log message?

There aren't explicit phases at the moment, just targets. The native phase is x86_64-unknown-linux-gnu, other targets (*-w64-*) depend on that and can be thought of as the second or cross phase.

There's no simple way to alter some gcc install directories, so it's mv and rm.

Only one package (libgsf shared) seems to have a problem with this - I'm looking at it.

[tonytheodore]

Only one package (libgsf shared) seems to have a problem with this - I'm looking at it.

Unrelated sed issue.

[starius]

[build] gcc i686-w64-mingw32.static

,

There aren't explicit phases at the moment, just targets. The native phase is x86_64-unknown-linux-gnu, other targets (-w64-) depend on that and can be thought of as the second or cross phase.

So gcc belongs to cross-phase, doesn't it?

[starius]

Native packages create files usr/<target>/installed/isl as well as usr/x86_64-unknown-linux-gnu/installed/isl.

$ ls usr/*/installed/isl                           
usr/i686-w64-mingw32.static/installed/isl
usr/x86_64-unknown-linux-gnu/installed/isl
usr/x86_64-w64-mingw32.static/installed/isl

Is it a bug or a feature?

[tonytheodore]

So gcc belongs to cross-phase, doesn't it?

Yes, though that can be misleading, think of it as phase two. As I mentioned, phases don't have names yet, just target dependencies.

Is it a bug or a feature?

A feature as the native isl will only be built once instead of multiple times and clobbering the previous build when it was simply under usr/. It also gives us a place to do things like install config.guess and the cmake modules only once in the native mxe-conf.

Running:

$ make isl MXE_TARGETS=`ext/config.guess`
[using autodetected 6 job(s)]
[check requirements]
[build]     mxe-conf               x86_64-apple-darwin14.5.0
[done]      mxe-conf               x86_64-apple-darwin14.5.0                              312 KiB        0m4.637s
[build]     gmp                    x86_64-apple-darwin14.5.0
[done]      gmp                    x86_64-apple-darwin14.5.0                              26840 KiB      0m34.185s
[build]     isl                    x86_64-apple-darwin14.5.0
[done]      isl                    x86_64-apple-darwin14.5.0                              29416 KiB      0m21.104s

and there is no reference to the cross targets. Running make isl will build all of gcc and it's dependents and cross versions of gmp and isl.

In build-pkg.lua, you can remove most of the COMMON_FILES:

diff --git a/tools/build-pkg.lua b/tools/build-pkg.lua
index 7a5df56..2ac4928 100755
--- a/tools/build-pkg.lua
+++ b/tools/build-pkg.lua
@@ -49,37 +49,9 @@ local BLACKLIST = {
 }

 local COMMON_FILES = {
-    ['gcc-isl'] = {
-        '^usr/include/isl/',
-        '^usr/lib/libisl%.',
-        '^usr/lib/pkgconfig/isl.pc$',
-    },
-    ['gcc-mpc'] = {
-        '^usr/include/mpc.h$',
-        '^usr/lib/libmpc%.',
-    },
-    ['gcc-gmp'] = {
-        '^usr/include/gmp.h$',
-        '^usr/lib/libgmp%.',
-    },
-    ['gcc-mpfr'] = {
-        '^usr/include/mpf2mpfr.h$',
-        '^usr/include/mpfr.h$',
-        '^usr/lib/libmpfr%.',
-    },
-    ['gcc'] = {
-        '^usr/lib/libcc1%.',
-    },
-    ['yasm'] = {
-        '^usr/include/libyasm',
-        '^usr/lib/libyasm.a$',
-    },
     ['ncurses'] = {
         '^usr/lib/pkgconfig/',
     },
-    ['pkgconf'] = {
-        '^usr/bin/config.guess$',
-    },
 }

 local ARCH_FOR_COMMON = 'i686-w64-mingw32.static'

Hope that makes sense.

[tonytheodore]

This implements a fairly straightforward two-phase build

Maybe I should elaborate on this. Phase one (the "native" phase) consists of seven packages:

• the gcc deps (gmp, isl, mpc, and mpfr)
• our custom pkgconf (with Cflags.private support)
• yasm (the assembler)
• the implied mxe-conf (for once-off setup)

All these packages currently also exist in phase two:

• gmp, mpc, and mpfr as requirements of other cross-packages
• isl (why not?)
• pkgconf (to create wrapper scripts)
• yasm (to create wrapper scripts for "yasm the assembler", and for cross-libyasm)
• the implied mxe-conf (for target-specific setup)

Arguably, isl and libyasm can be removed from phase two until someone or some package requires it. All other packages behave the same as before.

One package that (seemingly) will exist only in phase one is (the future) cmake. Surely mxe-conf in phase two will configure it correctly?

It does so at the moment, but if we added cmake, I would move the configuration from mxe-conf into a phase two cmake - probably adding a "fake" dependency in binutils (as with pkgconf).

This is far from ideal, but packages should configure themselves. We're not planning on delivering a cmake.exe in the near future - that would require an n-phase build.

A feature as the native isl will only be built once instead of multiple times

In an n-phase build, we could also build gcc only once in a multi-lib configuration. We don't need to build it four times, but the subtleties and complexities of doing that are staggering. I'd like to see how far we can go with a simple two-phase build.

[starius]

Changes for build-pkg required:

• remove most of common packages
• fix getting dependency information. make print-build-order is not sufficient, because it provides only build order, while build-pkg needs a list of dependencies to write them to Debian package. build-pkg generates temporary mk-file to get dependencies. This workaround was written when build-pkg was not a part of MXE. Now the content of this temporary mk-file can be merged into Makefile as new target.

I tried build-pkg with this pull request as is and it works. The only bug I see is that all native packages are squashed into mxe-conf package. It generated 4 shared packages and all of them are empty:

• common-gcc.list
• common-ncurses.list
• common-pkgconf.list
• common-yasm.list

[tonytheodore]

Something like print-deps-for-build-pkg may work in the short term. It lets you refer to packages as gmp~x86_64-unknown-linux-gnu, I've used a tilde but the separator can be anything unlikely to be in a package name.

Update: that won't work, the new print-deps-for-build-pkg seems to be doing the right thing after adding the native target to local TARGETS.

[starius]

To track cross-target dependencies (i686~foo depends on native-bar), target information is required in output of print-deps-for-build-pkg. I like the idea of adding a target and ~. Can you modify print-deps-for-build-pkg to output dependency graph for all targets at once and add targets to names, please? A package could appear multiple times, with different targets. build-pkg.lua would process such full-target dependencies instead of target agnostic dependencies and build all items in one loop.

[tonytheodore]

Okay, now it prints target~pkg, but calling make with pkg~target will also work. It's still based on MXE_TARGETS in case you want a subset but full deps on the native target will always be shown.

[starius]

Patches for build-pkg.lua after toolchain re-org:
https://gist.github.com/starius/deb0902a35bf5ef1a6b2
This patch was applied to tonytheodore:toolchain merged with mxe:master (5c61e16).
I'm building all packages with patched build-pkg. Please review the code.

[tonytheodore]

Nice work!, I added a line to keep the Debian control files:

--- a/tools/build-pkg.lua
+++ b/tools/build-pkg.lua
@@ -435,6 +435,8 @@ local function makeDeb(item, list_path, deps, ver)
         -- make .deb file
         local cmd = 'fakeroot -i deb.fakeroot dpkg-deb -b %s'
         os.execute(cmd:format(dirname))
+    else
+        os.execute(('cp %s %s.deb-control'):format(control_fname, dirname))
     end
     -- cleanup
     os.execute(('rm -fr %s deb.fakeroot'):format(dirname))

and those along with the *list files all look good to me (using MXE_MAX_ITEMS=100). A couple of comments.

https://gist.github.com/starius/deb0902a35bf5ef1a6b2#file-mxe-reorg-build-pkg-patch-L662

return not isInString('unknown', target)

A check against the output of ext/config.guess would probably be better.

https://gist.github.com/starius/deb0902a35bf5ef1a6b2#file-mxe-reorg-build-pkg-patch-L684

because it seems to work faster with lesser MXE_TARGETS.

I've been working on some cleanup and speedups for this in another branch that should be ready in the next few days.

https://gist.github.com/starius/deb0902a35bf5ef1a6b2#file-mxe-reorg-build-pkg-patch-L710

Subject: [PATCH] build-pkg: log cross-target dependencies

Don't log dependencies on native target.

not sure about this last patch, it prints a lot of lines and I'm not sure if they are supposed to be warnings. What is it trying to do?

[tonytheodore]

I'm currently installing the cmake modules in usr/share/cmake but usr/share is blacklisted. Should they be installed somewhere else or change the blacklist?

[starius]

Warning: New toolchain builds usr/i686-w64-mingw32.static/qt5/lib/libQt5Bootstrap.a natively (detected by build-pkg). I'll post full build log when it finishes.

New patches: https://gist.github.com/starius/931118a42fb7793b8b74

I added a line to keep the Debian control files:

You have not pushed this change. I have included it to part 2 of the patch.
I like the idea of keeping Debian control files. Why do you keep them only if not no_debs?

I'm currently installing the cmake modules in usr/share/cmake but usr/share is blacklisted. Should they be installed somewhere else or change the blacklist?

I prefer changing the blacklist. Path usr/share was unlisted and paths usr/share/{doc,gcc*,info,man} were listed.

A check against the output of ext/config.guess would probably be better.

Done

I've been working on some cleanup and speedups for this in another branch that should be ready in the next few days.

make pkg MXE_TARGETS=target keeps working so I prefer this variant because it is more stable.

not sure about this last patch, it prints a lot of lines and I'm not sure if they are supposed to be warnings. What is it trying to do?

It produces a warning when one cross-target (e.g. i686-w64-mingw32.static) depends on another cross-target (e.g. i686-w64-mingw32.shared), which is obviously an error. It doesn't produce a warning when a cross-target depends on a native target (x86_64-unknown-linux-gnu).

My build process is in progress but it has produced no such warnings.

[starius]

The patch 2 was updated (forgot to trim output of shell).
https://gist.github.com/starius/931118a42fb7793b8b74

[tonytheodore]

Warning: New toolchain builds usr/i686-w64-mingw32.static/qt5/lib/libQt5Bootstrap.a natively

I noticed that in the last log also. I imagine it's for bootstrapping the native qmake etc. and can likely be deleted if those tools are static (which they appear to be).

I like the idea of keeping Debian control files. Why do you keep them only if not no_debs?

Mostly because I'm testing on my fast machine (running OSX) without creating debs. I agree, they are very useful and can always be kept - I've changed the commit.

make pkg MXE_TARGETS=target keeps working so I prefer this variant because it is more stable.

Yes, that will always work.

It doesn't produce a warning when a cross-target depends on a native target (x86_64-unknown-linux-gnu).

Fixed now with using config.guess for the native target.

The patch 2 was updated (forgot to trim output of shell).

Thanks! I've pushed those along with some .gitignore and make clean additions.

[tonytheodore]

"make target~pkg" doesn't work if target is not in MXE_TARGETS

I've removed the phony pkg~target and target~pkg targets.

make pkg MXE_TARGETS=target

Is again the only recommended way.

[starius]

Logs:
Wheezy: https://gist.github.com/starius/e9e32a4d206fd268fb0f
Jessie: https://gist.github.com/starius/c242cd70b967ba8f6baf
Wheezy's log doesn't have [downloads] entries because packages had been already downloaded.
No packages are broken.

[tonytheodore]

Thanks! gendef looks like the last one related to this and it's an easy fix.

[starius]

I think speeding up the Makefile is an unrelated change and this pull request is ready for merging.

I've built MXE twice: without this pull request and with it.
Here are changes of the file list made by #925: https://gist.github.com/starius/3b5ae26b804f9820e97b
"<" stands for removed files, ">" stands for added files.
Please analyse this list before merging.

[tonytheodore]

Please analyse this list before merging.

Everything is as intended in that list, just a couple of comments:

• log4cxx is using a hash for filenames that would be expected to change
• libcc1.so has other issues as it is also an *.so on OSX
• vstasm and ytasm are different front-ends for yasm that aren't currently used
• doc exclusion generally needs some focus

I think speeding up the Makefile is an unrelated change

I've removed the speedup commit and rebased, it could do with some re-factoring for readability. Having print-deps-for-build-pkg is a good sanity check.

and this pull request is ready for merging

Thanks for all your help with this! It gives us a lot of flexibility having an additional dependency mechanism and the packaging is all nicely self-contained.