[POLL] Build system change

[andlabs]

I had four reasons for choosing flat makefiles:

1. I wanted libui to not have any extra dependencies (GNU make on Windows was already pushing it, but nmake is just way too primitive).
2. I wanted to ensure that every generated file goes somewhere clean that I can predict away from the actual source code, in this case to just .obj and out directories (obviating the need for Add .gitignore #13 in my opinion, at least)
3. I'm not really a fan of other build systems to begin with, either because of their complexity or their mess (see point 2 above)
4. I wanted to know exactly what was going on in the build process

However it seems that this decision is causing problems for lots of people, mostly on Windows where the command-line Visual Studio tools either behave unpredictably or require special handling depending on the configuration. @kainjow in #15 makes a really good point: most Windows users are likely more familiar with an IDE than with the command-line build system, to the point that command-line systems like cmake are more accessible.

While it's true I originally resisted changing, thinking make would be simpler, the number of build issues keeps growing, especially on other platforms. So now I'm willing ot hear arguments for specific build systems.

I genuinely do not have a preference for any one build system over the other, so please make your case for one you prefer. As part of your case, you should demonstrate what a setup for libui, the test/ folder, and each example would look like, as well as (if possible) a rule to build all examples. Feel free to point out any other advantages or disadvantages.

Here is a page detailing what an ideal build system would be like: https://gist.github.com/andlabs/243d5d817d291fa62722c33a9c58c6ab

Right now the following are being considered:

• cmake
• waf
• scons

I'll make my decision and switch build systems if one appeals to me more than the others.

Thanks for your feedback! I do appreciate it all, and I want to make sure libui works for everyone :)

[kainjow]

I'm maintaining my own CMakeLists.txt file here for ease of development for myself. I plan on adding Windows and GTK support to it when I get a chance.

Currently libui has nearly 20 files that are part of the Makefile. Using more modern versions of CMake (3.1 and later), you can manage multiple targets in a single file very easily.

Some examples of big projects using CMake are Qt and LLVM.

The great thing about CMake is it generates the cross platform projects for you, so one can work in Xcode, Visual Studio, Makefiles, Eclipse, and more.

CMake is fully open source. It can be built from source, or the binaries can be installed via Homebrew on OS X, Chocolatey on Windows, apt-get on Debian/Ubuntu, or downloaded directly for OS X and Windows.

With my example CMakeLists.txt file, you would put that in the root directly, then run these commands (minimum needed), which works on every platform (CMake will choose a default generator based on what it finds):

cmake .
cmake --build .

[andlabs]

For the OS X build, what files does cmake generate, and how?

[kainjow]

On *nix platforms, if you don't specify a generator it makes a Makefile (and other junk/cache files for detecting compiler features), so then you can make as normal. On OS X for example if you pass -G Xcode it makes an Xcode project. cmake --build is an easy built-in way to build whatever CMake generates without having to know the native command (make, msbuild, xcodebuild, etc.).

[andlabs]

And everything is created in .? (So I would do mkdir build; cd build; cmake .. to drop build files elsewhere?)

[kainjow]

Yep, exactly. With CMake it's usually preferred to build out-of-source to keep things clean.

[HalosGhost]

Honestly, I deeply prefer flat makefiles over most build-systems as well. But I have taken a special liking to tup. It builds incredibly fast, has brilliant dependency resolution, has incredibly terse build rule syntax, makes reproducible builds much simpler, is cross-platform and supports Windows, OSX and Linux.

The only things to know are that it is not makefile-generating, so it is a shift away from makefiles (where things like CMake are a layer on top of makefiles) and that it is truly a build-system and nothing more; e.g., it does not have install/uninstall rules.

In my personal projects, I tend to use Tup as my build-system and a flat makefile on top of it to deal with things like install/uninstall rules cleanly.

[moosingin3space]

Agree with @HalosGhost -- tup is nice and simple, and with a Makefile wrapper, you can easily add other targets.

The current system of doing cross-OS builds could be emulated by sticking a Tuprules.tup file in each platform directory, which would assign values to !cc and !ld targets (or something like that), then make $OS could be implemented by copying the correct Tuprules.tup file into the project root and then running tup upd. I'm attempting to do something like this in one of my projects that uses cross-platform C.

[hajimehoshi]

I prefer qo (https://github.com/andlabs/qo). You will like it if you like Go-way to build something.

[SevenBits]

I definitely prefer flat Makefiles. On Windows it might make things more difficult, but in the long term it is probably a good idea to avoid external dependencies; once you add one, then it isn't too long before you concede to adding another, and soon the whole project is overflowing and bloated.

[HalosGhost]

@SevenBits, I agree, though I think it is fair to classify external dependencies in two ways: makedeps and runtime deps; adding makedeps is less of a concern in my opinion as far as bloat.

[andlabs]

The thing about tup that I'm not thrilled about is its use of a database that you have to constantly update yourself (and which I'm not even sure is safe to check into source control); copying build scripts around especially sounds unattractive.

[HalosGhost]

@andlabs, database? When do you need to manually update it?

In addition, you definitely would not need to copy around build-scripts to support multiple targets.

[moosingin3space]

Yeah, I didn't read the Variants section of the tup manual when suggesting that.

[sekrause]

+1 for CMake from me.

I especially like the fact that it simply generates files for the platform's "native" build system, e.g. Makefile on Unix and a Visual Studio project on Windows.

[waddlesplash]

CMake. I hate it, but there's nothing better (yet -- there are a few projects in the works, but they're way behind what libui wants as of now.)

[NattyNarwhal]

I would recommend against CMake - it's way too complex. Makefiles are simple and GNU Make isn't that much of a requirement.

[RotsiserMho]

Makefiles are simple until you want to do even just a little bit more than build the source. For example: running tests. CMake works well for something like that because you don't have to worry about platform-specific path formats, etc.

[aoloe]

on top of what has already being said, there are two reasons for me to prefer cmake:

• it would make very easy to embed libui inside of projects that already use cmake. probably, without hurting the rest of the world.
• most people know about it and you can compile cmake based projects without having to deal with the way a programmer found cool at the time he wrote the library.

i can live with libui's flat makefiles; i would prefer if libui would avoid "hesoteric" (for c++ projects) pre-processors, even if they are superior.
and i could have saved much time if libui were already using cmake...

[kainjow]

@NattyNarwhal what makes CMake complex to you?

[SevenBits]

My thought is that whatever built system that is chosen needs to be relatively simple to setup and use on all of the supported platforms. I've seen build systems (OpenSSL comes to mind) where you can't just simply run make or something similar; you have to engage in this ridiculous and complex dance involving multiple sets of scripts, etc before you can build, or at least you did have to when I last did it.

I love that libui is using flat Makefiles currently; I downloaded the source archive, opened it in my Mac's terminal, and ran make. All done. No hassle. It then produced a single file that I can copy into my project or tie into my existing Makefiles. That should be how this project continues. I'd hate to have to do more than that; I've used enough GUI libraries requiring lots of build time dependencies. So far, libui has been able to avoid that, and I think that that's fantastic.

[jamierocks]

I've only used it once but it seemed good, but gradle's cpp plugin.

[waddlesplash]

I've only used it once but it seemed good, but gradle's cpp plugin.

Gradle is a hulking resource hog of a build system, essentially the complete antithesis of libui. It wouldn't be a good match.

[kainjow]

Seems like building libui as a static library is a popular request. In CMake, when you use add_library to create a library target, it by default builds static. However if BUILD_SHARED_LIBS is defined and true (which the user can easily set on the command line) all libraries instead build as shared (unless explicitly set).

[bkaradzic]

GENie (not build system, rather project generator) https://github.com/bkaradzic/GENie#genie---project-generator-tool

[Vdragon]

I really against any build system that is not space/non-ASCII containing path compatible, GNU Make & it's friends are on my blacklist for this reason.

[jasom]

While I really like tup (mentioned by others) it is really a make replacement, so at least some of the issues you are currently having now with windows will still be around.

However, it is smart about exporting the variables visual studio needs, and it provides a varaible for which host you are on, so "do X if on windows and Y otherwise" is trivial to add.

[andlabs]

So far I've only seen cmake and tup mentioned here. Any advantages to any of the others? Someone else mentioned gyp somewhere else, for instance.

[andlabs]

@pcwalton what build systems are you familiar with that you could backport your static lib visibility changes to? I have a cmake file and am investigating waf; probably others too...

[andlabs]

https://gist.github.com/andlabs/a244eadbf885c5f2b20b3fe4c6dfee19 waf file available for comments

[andlabs]

Actually after writing that I could probably refactor the cmake script a bit more. I could probably also use an intermediary build step to simulate the static library, maybe...

[andlabs]

@billyquith actually, if I had control over whether an add_library() and add_custom_output() gets placed in the build/ directly or kept as an intermediate target, I could recreate the static linking by splitting into three steps: an object library, the change, and another add_library()...

[andlabs]

As for premake and by extension genie:

I'm not sure how to get the OS and toolchain to decide what to do or how, unless I'm not supposed to worry about that? I can't tell. Same for specifying such on the command line.

I don't see a way to have multiple languages in one project (since on Windows we mix the C common files and C++ Windows-specific files).

How would I do the static linking visibility reprocessing tango?

[bkaradzic]

@andlabs

There are multiple ways to do it, for the case of many different compiler/os combinations it's the best to specify it thru option. See example how I do it in my scripts for GCC/Clang/VS:
https://github.com/bkaradzic/bx/blob/master/scripts/toolchain.lua#L11

GENie automatically builds .c and .cpp with their compilers. You could specify options for both C/C++ with buildoptions, or if you need to specify options for C or C++ you could use buildoptions_c or buildoptions_cpp.

[andlabs]

So tomorrow (or when I get a new power adapter for this laptop, IDK) I'm going to also try premake/genie and scons; those four might be the four choices...

@bkaradzic I also see you said you removed "Solaris support"; does this just mean support for suncc (that is, if I build GENie manually on Solaris or Illumos can I use gcc?).

[2a03]

CMake
Not so ideal from the perfectionist developer standpoint (bulky, somewhat overcomplicated and, like many build systems, is «a world of its own», requiring additional learning and reading manuals to be done for accomplishing something meaningful),

But for the average user it's perfect — self-contained, crossplatform and in most cases boils down to «mkdir build && cd build && cmake ../ && make». Also it's well documented, widespread and supported.

I'd also like to firmly advise against using:

• «in–house» or «one man experimental project» systems. As cute, nice and cozy those are for their creators, for most devs it means additional trouble to bring those into project environment. As well as lacking or absent documentation, absence of support and no guarantees for forseeable future.
• any system requiring infrastructure–tied language. This means waf, rake, qo (:D) or anything Python–, Go–, Ruby–, NodeJS– or Java–based. Yes, setting up whole installation of Ruby or Python (and then packet manager, and then packages and packages and packages), just for the sake of building small C/C++ project — it is a major pain in the… well, just the major pain, that is.

[SevenBits]

I would second what @2a03 said: if you must change the build system, move to CMake. Everything else IMO is either too heavy, too complicated, or both.

[billyquith]

@andlabs I'm not exactly sure what you mean by the visibility problem. I assume you only want to export symbols that you've marked as such? I think cmake can do this in a cross-platform way. Info here: http://stackoverflow.com/q/17080869/3233

@SevenBits What about Brainfuck? 😆

[andlabs]

No, I mean removing those hidden symbols from a static library. I know how it works now. I'll work on switching to using configurations, subdirectories, and object libraries; that might make it easiest.

[andlabs]

Let me map this out, reading what @pcwalton did:

A normal static build would do

ar rcs library.a objects

However, this exposes hidden symbols to the caller, which will lead to name clashes which I don't want.

On non-OS X, if we instead do

    ld -r objects -o hugeobject.o
    objcopy --localize-hidden hugeobject.o
    ar rcs library.a hugeobject.o

then we wind up with a library.a with the hidden symbols truly hidden, thanks to how ld -r and objcopy work: ld -r combines multiple .o files into one big .o file and objcopy rewrites that .o file to scramble the hidden symbols.

The equivalent on OS X is

    nm -m objects | sed -E -n 's/^[0-9a-f]* \([A-Z_]+,[a-z_]+\) external //p' > symbols
    ld -exported_symbols_list symbols -r objects -o hugeobject.o
    ar rcs library.a hugeobject.o

The build system I choose would need a transparent way to do all of this, even if I have to key in those commands manually (as long as I can say "these commands produce file.o which is a dependency of libui[-static]")

[andlabs]

That's what we're already doing, and it works for shared libraries, but it isn't enough for static libraries.

[andlabs]

https://gist.github.com/andlabs/243d5d817d291fa62722c33a9c58c6ab I made a page describing what's necessary, to keep everyone on the same page

Right now cmake, waf, and scons are on the table, and maybe premake or genie. cmake is the clear winner here anyway among the people in this discussion :/ I'd need to count github reactions as well.

[andlabs]

So do any of you know how to do that gist in a "cmake"-y way? Do I need to restructure my source tree? What should be configurations? What should be separate CMakeLists.txt files? How can I organize these options?

[billyquith]

@andlabs
For code snippets perhaps Gist is best. For structured information (without discussion) I'd suggest your project wiki. The requirements then act as a kind of documentation for the project going forward. The front page is just a directory.

I already included a link to my branched cmake version, split into sub-folders. I added a pull request for this. I'd suggest commenting on the pull request if anyone has any comments. Note this is just for OSX at the moment. I have other priorities so until you make a decision on the system, it works for me.

I'd suggest the decision is either you want a project file generator, or build system. If you want a project file generator then cmake is the popular choice. If you want a build system then I'd suggest that many of these are overkill for a small project like this, and involve introducing another language as @2a03 mentioned.

Personally I'd restructure the project, e.g. so you don't have header files in the root. Something like Gwork:

libui/
    readme.md - what is it? build instructions. useful links.
    licence.md
    changes.md - change log
    src/
        libui/
            include/
            src/            - common
                windows/
                unix/
                darwin/
        test/
        examples/
    doc/

I also think you need to think about what constitutes a platform and GUI, and the naming. Darwin is the name of the BSD UNIX that OSX (which contains Cocoa, the UI) sits upon. Like, OSX/Darwin, Linux is the base OS, but has the additional complication of multiple GUI libraries (e.g. GTK+ and Qt). I notice there is already a pull request for a Qt version of libui.

So your current platforms are:

• Windows
• OSX
• Linux

And your GUIs are:

• Cocoa
• GTK+
• Win32

This might be important if you need to add platform specific code that can be shared between GUIs. You already have a nice simple driver abstraction for the UI part.

If you added Qt it might become:

Platforms:

• Windows
• OSX
• Linux
• Qt - this is cross-platform

And your GUIs are:

• Cocoa
• Win32
• GTK+
• Qt

I'd suggest starting another issue to discuss this.

[SevenBits]

@andlabs If you do move to cmake, it would be very nice to be able to include and exclude certain features from the build based on our needs. For example, I won't need OpenGL support. This would be very useful, especially since on 64-bit Windows the statically linked sample applications are already about a megabyte in size.

[andlabs]

@billyquith Yours and the other two cmake PRs I've gotten have helped with writing what I have now; thanks :)

@SevenBits I'm not a fan of optional features, sorry. libui itself shouldn't contain many more controls besides what's in here now, grid, table, and the OpenGL area; other spurious controls will be in libui-extra and libui-future.

---

Now I'm stuck with cmake again:

Right now I'm building common/ and darwin/ as object libraries and want to do pcwalton's static linking thing, so I try:

    _add_static(libui
        $<TARGET_OBJECTS:libui-common>
        $<TARGET_OBJECTS:libui-${_OSDIR}>
    )

macro(_add_static _name)
    add_custom_command(
        OUTPUT "libui_static_intermediate.o"
        COMMAND
            nm -m "${ARGN}" | sed -E -n 's/^[0-9a-f]* \([A-Z_]+,[a-z_]+\) external //p' > libui_static_intermediate.lst
        COMMAND
            ld -exported_symbols_list libui_static_intermediate.lst -r "${ARGN}" -o libui_static_intermediate.o
        DEPENDS ${ARGN})
    add_library(libui STATIC "libui_static_intermediate.o")
endmacro()

Unfortunately, this gives me lots of

CMake Error at darwin/CMakeLists.txt:46 (add_custom_command):
  Error evaluating generator expression:

    $<TARGET_OBJECTS:libui-darwin>

  The evaluation of the TARGET_OBJECTS generator expression is only suitable
  for consumption by CMake.  It is not suitable for writing out elsewhere.
Call Stack (most recent call first):
  CMakeLists.txt:140 (_add_static)

followed by

CMake Error: Cannot determine link language for target "libui".
CMake Error: CMake can not determine linker language for target: libui

It seems like the TARGET_OBJECTS generator is only supported in a SOURCES clause due to Xcode bugs. What are my other options?

[kainjow]

IMO, just use what I mentioned above (BUILD_SHARED_LIBS), and leave static/shared up to the user, instead of supporting both. Then in your tests just call the CMake commands twice, one for static, one for shared.

[andlabs]

The problem is shared and static have different requirements: different flags for the library itself (both sources and output), different flags for linking the executables, different build steps.

I've almost finished with cmake anyway. If you have any way to improve what I come up with, feel free to suggest it when I push.

[kainjow]

You can check for if(BUILD_SHARED_LIBS) ... and do what you need to do then (or use NOT). The docs suggest making it an option() so users can set this on the command line (or before it's included in their project).

[andlabs]

Ah, all right. I'll refactor that once I make sure what I have now works.

[kainjow]

Also that message in the CMakeLists file about MinGW and static building... might be better to just use the message() feature for that:

if(${_NOSHARED})
    message(FATAL_ERROR "Sorry, libui for Windows cannot be built as a DLL with MinGW. You will need to either build as a static library or build with MSVC.")
endif()

A FATAL_ERROR will stop CMake execution.

[andlabs]

Yeah, I've already turned it into that :) That comes from a time when static and shared were separate targets instead of separate configurations.

[andlabs]

Okay, I pushed a cmake setup now that works on OS X and Linux. It still doesn't work on Windows. I've filed a bug report at #93 and we can continue there.

Thanks again everyone!

[billyquith]

I'm unclear, does this mean you have chosen cmake as the build solution, or are you still trialling?

[andlabs]

cmake.

[frak0d]

MESON