Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Multi-target approach #39

Closed
vog opened this Issue · 16 comments

6 participants

@vog
Owner
vog commented

Improve the main Makefile and directory structure to allow for multiple targets (Windows 32/64-bit static/dynamic, Darwin 32/64-bit static/dynamic, etc.), avoiding code duplication as much as possible, on per-package basis.

A good starting point is Tony's experimental branch: https://github.com/tonytheodore/mxe/commits/multi-target

@vog vog referenced this issue
Closed

makefile improvements #38

@tonytheodore

Hi William,

On 20/05/2012, at 9:51 PM, William wrote:

Hello Tony,

I tried your forked MXE-multi and it seems to work quite nicely.
Here are several comments I could gather.

  • When I updated your upstream repository, merged it with internal packages from mxe, I had an update mess, that could be related >to the gmp.mk/gcc-gmp.mk trick. When I was building gcc, pkg/gmp-5.0.4 was already donwloaded, version was (index.html) 5.0.5, >mxe was not downloading 5.0.5, but then it was complaining 5.0.5 was not there... Solved it don't know how, by forcing download of >5.0.5 somehow.

Not sure what happened there, it's not merging cleanly for me either. I'll have a look at it.

  • I think the build log got pretty ugly with the "[build] gcc i686-static-mingw32" string as replacement of simple "[build] gcc". What >do you think of a log such as : *** target i686-static-mingw32 [build] package1 [done] package1 [build] package2 [done] package2 *** target i686-dynamic-mingw32 [build] package1 [...]

instead of :
[build] package1 i686-static-mingw32
[done] package1 i686-static-mingw32
[build] package2 i686-static-mingw32
[done] package2 i686-static-mingw32
[build] package1 i686-dynamic-mingw32
[...]

?

  • About logs. It gets a bit longer to write the command line "log/gcc_i686-static_mingw32" each time we want to review it (2 more >keys pressed, time to analyse what we want between different targets ...). What about writing logs to for example "log/i686-pc->mingw32-20120520_002035/package" and make a link "log/package -> log/i686-pc-mingw32-20120520_002035/package", instead >of what is done actually "log/20120520_002035/package_i686-pc-mingw32" and a link "log/package_i686-pc-mingw32 -> >log/20120520_002035/package_i686-pc-mingw32" ? This would only show last built file, and it is very likely that only one target is >built at a time, so no big confusions...

I like both these ideas, however, they may not play nicely with inter-package parallel builds (using -jx) which is what I mostly do. I've left logging as it is for the moment, it's largely a matter of preference, and I have no strong opinions on it. It may even be something to override in the settings file.

  • the settings.mk file is perfect to configure the target. I would add a comment in the Makefile, refering to the settings.mk file. And I >would add a line such as this one in the settings.mk file : # TARGET =

Indeed, that is one of it's primary intended uses, but I haven't had a chance to update the multi-target branch yet:

http://lists.nongnu.org/archive/html/mingw-cross-env-list/2012-05/msg00021.html

  • And last but least, why do you wait to integrate it in mxe ?

I'm mostly trying to find the time to do a squashed/rebased version that separates the Makefile, toolchain, and package build rules; and then summarise the remaining areas (logging, scope of options in $(LINK_STYLE), empty build rules vs non-existent rules for failing packages etc.).

Make mxe worth its 'M' !

Ideally, yes :) I'd like to launch with a fair subset of packages (at least qt) supported.

If you are afraid of compatibility problems, why not add a target "i686-pc-mingw32", that makes the same as i686-static-mingw32?
Using the settings.mk capabilities, I would put only the "i686-pc-mingw32" target in the main Makefile, not the other ones by >default, and put in the settings.mk file the commented line :
#TARET = i686-static-mingw32 i686-dynamic-mingw32 x86_64-static-mingw32 x86_64-dynamic-mingw32

  • Finally, I would modify target "i686-static-mingw32" to be used with mingw-w64 32 bits targets (which is the future), and keep >"i686-pc-mingw32" for mingw.org, thus making something compatible for the transition, and with a framework suitable for the future, >without mingw.org ?

Fair points, but there are no current plans to drop mingw.org. There are some subtle points to this (i.e. mingw-w64 exception handling) that need to be fleshed out.

@bruiselee

with the same idea with william.
mingw-64's w32api/wincrypt is up2date, while MXE's wincrypt is so old that gnutls can't be built successfully

As to the multi-target branch, I also build successfully after removing dynamic target in settings.mk, or else libFLAC, libsndfile would fail to make, FYI

@oysteinkrog

I'm using the multitarget branch to build x86_64 and it works well for me (I use it for ffmpeg).
Is there any chance it could be merged back into master or is it still too unstable?

@tonytheodore

I've merged mxe/master back in to my experimental branch (https://github.com/tonytheodore/mxe/commits/multi-target) and made several changes so that it's more pleasant to use. A single target build (the default) looks much the same as it currently does, and it also plays nicely with settings.mk and parallel builds. Haven't done any re-basing yet...

@oysteinkrog
@tonytheodore
Owner

I've rebased the changes in a new branch https://github.com/tonytheodore/mxe/commits/multi-rebase

git clone -b multi-rebase git://github.com/tonytheodore/mxe.git

Target Awareness: tonytheodore@78cd1a0

  • move install markers to $(PREFIX)/$(TARGET)
  • make CMAKE_TOOLCHAIN_FILE & TARGET available as named variables in PKG_BUILD rules (as gmake target-specific-variables[1] instead of global variables)
  • provide fallback to generic build rule if no target rule is specified. We'd previously decided against this [2], however, explicit rules get tired very quickly and you end up just adding all at once then disabling the failing ones. This is no different to implicit rules, other than there's a lot of boilerplate left over with explicit rules.
  • distinguish between a missing rule (no-op) and an empty rule (excluded)

Mingw-W64 Toolchain: tonytheodore@82c6287

  • replace TARGET with MXE_TARGET so it can safely be used on the command line without propagating to package builds
  • add mingw-w64 package and target specific build rules (in the future we could supply packaged crt/headers to bypass the standalone gcc build)
  • enable sjlj exceptions for mingw-w64 toolchain
  • enabled toolchains: i686-pc-mingw32 x86_64-w64-mingw32 i686-w64-mingw32
  • add winpthreads (from mingw-w64 svn), rename pthreads to pthreads-w32, and make pthreads a virtual package that depends on the actual pthreads implementations (packaging inspired by vmime and git-svn)
  • gcc.mk is an example of composable snippets to reduce duplication, PTHREADS_TEST is another

Package Build Rules: tonytheodore@6c83957

Mostly disabling builds, but some examples of handling different targets:

At this stage, targets can be built using make MXE_TARGET=... (or adjusting the Makefile/settings.mk)

Build Multiple Targets: tonytheodore@e1780e9

  • adds target loop and dependencies to build targets serially
  • replace MXE_TARGET with MXE_TARGETS to reflect it's plural nature - still i686-pc-mingw32 by default
  • adjust settings.mk sample to play nicely with above
  • prints target header when building for a visual break and quick reference to source of package list (command line, settings.mk, or src/*)
  • most recent log is accessed via log/pkg, previous logs are log/pkg_target

Now everything can be built with a simple make (provided settings.mk is adjusted to your needs) or run a specific target with command line options make MXE_TARGETS=x86_64-w64-mingw32 etc. The default is still to build i686-pc-mingw32 with the mingw.org toolchain so this can be seamless drop-in for current users and they can adopt the new targets when ready.

This last commit is a very verbose way of achieving much the same as make MXE_TARGET=... && make MXE_TARGET=.... I like the way it works, but it's pushing my make skills and alternate implementations are welcome.

I've left out the static in the vendor part of the target triplet, this is an easy sed command to update, and I think the previous $(LINK_STYLE) should be expanded from (enable|disable)-(static|shared) to an $(MXE_CONFIG_OPTS) that also includes host/prefix/build (also[2]). These touch almost every file in src/, so I'd rather isolate those changes.

[1] http://www.gnu.org/software/make/manual/make.html#Target_002dspecific
[2] http://lists.nongnu.org/archive/html/mingw-cross-env-list/2012-04/msg00050.html

@woncount

not sure how to leave a vote, so +1

@tonytheodore
Owner

It's not so much about voting as getting the design right. The main remaining point of contention for me is implicit vs explicit build rules. I started with implicit, then went to explicit, back to implicit (in this rebase), and am now swinging back to explicit rules.

It's hard to tell until until we actually support multiple targets - it could turn out to be a mute point as most targets will require distinct rules anyway. Implicit rules work well at the moment as it allows merging master with minimal conflicts.

@donbright donbright referenced this issue in openscad/openscad
Closed

64 bit windows version #341

@tonytheodore
Owner

I've left out the static in the vendor part of the target triplet

After attempting to upgrade mingw-w64 to gcc-4.8, it seems the vendor part (-w64-) is actually used to detect and enable functionality. I suspect this means we're left with using a suffix on the triplet.

@tonytheodore

I suspect this means we're left with using a suffix on the triplet.

We can't use a suffix that causes a new segment (i.e. *-*-*-dynamic), as config.sub specifically rejects anything greater than three. The other option is to treat this like a version number (i.e. *-*-mingw32dynamic) but the first package I tried (icu4c) didn't recognise it and I strongly suspect many other package build systems won't like it either.

I think we're going to have to use the triplets as-is and manage the different --prefixes with a new variable ([LIB | LINK]_[STYLE | TYPE]).

I'm also inclined to remove the target loop from the Makefile and mange this with a shell script. Forcing make to serialise the targets is fairly ugly and adding another loop for the link styles will be worse - this could be my lack of skill though.

@tonytheodore

In more annoying prefix/triplet shenanigans, new versions of mingw-w64 toolchains must be installed to a prefix that matches the triplet.

I guess we'll be left with a directory structure that looks like:

usr/
---->dynamic/
--------->i686-pc-mingw32/
--------->i686-w64-mingw32/
--------->x86_64-w64-mingw32/
---->static/
--------->i686-pc-mingw32/
--------->i686-w64-mingw32/
--------->x86_64-w64-mingw32/
etc...
@tonytheodore

Alternately, do we build the static/shared versions in the same pass and have a mix in the same tree? I think this will be okay in lib but I'm reasonably sure pkg-config will be messy and some headers will be generated with specific defines. Does anyone have experience with this? - I'm only guessing.

@tonytheodore

I wonder if we should start referring to hosts instead of targets:

http://gcc.gnu.org/onlinedocs/gccint/Configure-Terms.html

@woncount

given that gcc is the mxe backend, I would agree that terminology should match, in order to avoid unnecessary confusion hopefully.

@TimothyGu
Owner

I guess this issue can be closed as we already have a target-based approach of shared build.

@tonytheodore

Agreed, the next stage is customisable packages, just a couple of post-implementation comments:

provide fallback to generic build rule if no target rule is specified...
it could turn out to be a mute point as most targets will require distinct rules anyway..

Turns out that most rules are fairly generic and this works well in practice - the main use is probably to disable targets. The logic has been consolidated and expanded to find increasingly generic rules for build|deps|file.

distinguish between a missing rule (no-op) and an empty rule (excluded)...
adds target loop and dependencies to build targets serially...
prints target header when building for a visual break...

These are all attempts at micro-managing the build instead of letting make do what it's best at - building what needs to be updated. There's some ugly code that fights make and will be removed soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.