multiplatform build automation tool?

[jonforums]

Is either of the following build tools viewed as a welcome inclusion to mruby?

waf - Python-based build framework

• toy example: tma
• real-world example: samba

cmake - Makefile/project generator

• real-world example: clang

[matz]

I am not familiar with those tools. Tell me what we get if we accept those tools, especially for embedding purpose.

[jonforums]

Two primary reasons: better target support maintainability, and increased project velocity/robustness.

• Better target support maintainability - don't waste cycles trying to maintain multiple target-specific, cross-compilable Makefiles/project files, maintain a build infrastructure at a higher level of abstraction. As mruby is potentially interesting in a number of embedding scenarios, this is especially important. Projects like luajit do a reasonable job by supporting Makefile for *nix and batch for Windows, but the situation can be made more efficient by not manually maintaining multiple build systems. Also, the situation described in 1.1 and 1.2 of the lua faq can certainly be improved.

• The easier it is to build mruby for multiple embeddable scenarios, the more you're going to be able to attract and retain active contributors that will play with the code on multiple systems. This can lead to a positive "code Darwinism" cycle in which the edge cases/bugs are discovered/fixed much more quickly. For example, say I was interested in embedding Ruby in a custom C app and wanted to build with Windows SDK 7.1 on Win 7. Currently I'm stopped cold before I even begin:

  C:\Users\Jon\Documents\RubyDev\mruby-git>nmake -nologo -f Makefile
  Makefile(9) : fatal error U1034: syntax error : separator missing
  Stop.
  

The great thing about either of the build tools is they enable contributors to spend their precious time in a more efficient way by having the tool do much of the grunt work. Of course, it takes effort to create and maintain the meta-build info, but the leverage and target coverage is greater and easier.

While applicable, I do not consider the autotools infrastructure a desirable or usable option for mruby.

[beoran]

I have some experience with Cmake, and it's better than autotools, and works on more platforms, including Windows.

Another additional advantage (over,, say, waf) is that cmake makes cross-compiling relatively easy. So it becomes easier to make, say, mingw binaries for windows without having to actually run windows. The only downside I can think of to cmake is that it's syntax is a bit different than most other tools. It's not too hard, but it takes a while getting used to. I' think I'll quickly try to make a cmake build system now to see how hard it really could be.

[jonforums]

@beoran I'd cloned mruby and started a cmake branch here if you'd like to have a go at it https://github.com/thecodeshop/mruby/tree/cmake

I paused to see if there was other interest. Let me know if you want commit access to that clone, and there may be others who will want to join in.

[beoran]

Thanks for that but I had already started before I saw your message. I now have a cmake system that works on linux here:
https://github.com/beoran/mruby/tree/cmake
For windows, etc, you'll probably need to modify it as it uses cat, etc... for building the mrblib directory.
Also, the build structure has changed to fit better with how cmake works. It's decribed in the top CMakeLists.txt and I repeat it here so you can read it here as well.

NOTE: The original Makefile build system had a few quirks in them, which I didn't duplicate. In stead the build logic is like this:

1. First libritevm_static.a & libritevm.so are built from all C files in src.
2. Then mrbc is built and linked with libritevm_static.a .
3. Then libmrblib_static.a & lib_mrblib.so are built from the rb files in the mrblib subdirectory
4. Finally, mruby is built and linked with libritevm_static.a and libmrblib_static.a.
   As a result, applications that embed mruby will have to link against libritevm_static.a and libmrblib_static.a in that order. Or use dynamic linking on systems that support it since Cmake compiles shared libaries just as easily as static ones.

TODO: make this work on windows too, support build options to generate mrbconf.h, etc...

I hope this will be useful.

Edit: Another thing I forgot to mention is that in some embedded environments you need to cross-compile a lot from a developer's pc to the architecture of the device that mruby will run on. With cmake, cross compiling will be easier than with a hand-made build system. You need a single toolchain file and then you're good to go, as described here:
http://www.cmake.org/Wiki/CMake_Cross_Compiling

[jonforums]

Fantastic! I just deleted my branch and will look at what you've done.

Best to keep all the focus directed at sending you pull requests so that Matz can quickly see how a CMake solution might add value.

Are you planning to create reusable macros/functions in a top-level cmake dir similar to llvm? For example, I think it's particularly clever how they integrate with their VCS with a CMake module.

[beoran]

Yes, I leave the possibility of a cmake directory open, but only if that is really needed for such a relatively easy to build project as this one. The goal should be to keep the CMakeLists.txt are easier to read and work with that the old Makefiles were.

[beoran]

After testing more I fond a bit of a problem with my cbuild approach. The main problem is that the way it's built now, libritevm_static and libmrblib_static have a cyclical dependency. When you link an application that wants to embed them, you have to do
gcc -o out *.o -lritevm_static -lmrblib_static -lritevm_static
which is annoying to say the very least. I think there should be some modifications in the C sources to remove this cyclical dependency, probably this will need change the API too a bit. Or the build method will have to be changed. Or the parser and code generator should become a separate library?
edit: looks like I'll need this: http://www.cmake.org/Wiki/CMake/Tutorials/Object_Library, as well as the latest version of cmake.

[beoran]

I pushed an update to my cmake branch that solves this, using object libraries, but you need cmake 2.8.8.
Now gcc -o out *.o -lmrubylib_static -lm should be enough for linking programs that use mruby.

[bovi]

Just a small feedback from a Mac to your branch:

~$ uname -a
Darwin boviAir.local 11.3.0 Darwin Kernel Version 11.3.0: Thu Jan 12 18:47:41 PST 2012; root:xnu-1699.24.23~1/RELEASE_X86_64 x86_64

I made a cmake:

cmake$ cmake ../cmake-mruby
-- The C compiler identification is GNU 4.2.1
-- Checking whether C compiler has -isysroot
-- Checking whether C compiler has -isysroot - yes
-- Checking whether C compiler supports OSX deployment target flag
-- Checking whether C compiler supports OSX deployment target flag - yes
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Found PkgConfig: /usr/local/bin/pkg-config (found version "0.25") 
-- Found BISON: /usr/bin/bison (found version "2.3") 
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/daniel/Documents/cmake

And then my build exit with following error:

[ 88%] Building C object src/CMakeFiles/ritevm_object.dir/vm.c.o
[ 88%] Built target ritevm_object
Scanning dependencies of target ritevm_static
Linking C static library ../lib/libritevm_static.a
[ 88%] Built target ritevm_static
Scanning dependencies of target mrbc
[ 90%] Building C object tools/mrbc/CMakeFiles/mrbc.dir/mrbc.c.o
Linking C executable mrbc
Undefined symbols for architecture x86_64:
  "_mrb_parse_file", referenced from:
      _main in mrbc.c.o
  "_parser_dump", referenced from:
      _main in mrbc.c.o
ld: symbol(s) not found for architecture x86_64
collect2: ld returned 1 exit status
make[2]: *** [tools/mrbc/mrbc] Error 1
make[1]: *** [tools/mrbc/CMakeFiles/mrbc.dir/all] Error 2
make: *** [all] Error 2

My cmake version is:

cmake$ cmake --version
cmake version 2.8.8

[beoran]

Thanks for the info, but other than thinking "that is weird", I don't quite know what I could do about this problem... Could you investigate a bit more on your side please? Oh, and you do have bison installed somewhere, right? Otherwise the parser may not get built correctly...

[jonforums]

It appears the cmake prototype needs more refinement before submitting a pull request for Matz to review.

@beoran @bovi as a way to speed up collaboration by enabling a place to refine the cmake mods, I've added you both to the https://github.com/thecodeshop/mruby repo and pulled in @beoran branch under beoran-mods/cmake. The namespaced branch approach will allow multiple people to quickly mix-n-match refine.

While I'm going to focus on Win32 tweaks and out-of-tree builds so as to not clobber existing Makefiles, I strongly feel that the cmake prototype should be able to build *nix, OSX, and Windows before submitting a pull request.

Background on that repo...I specifically setup The Code Shop to enable quick ad-hoc collaboration like this with the goal of getting the mods accepted upstream. I've also set up a moderated mailing list we can use if it makes things easier than trying to collaborate by only replying to this issue. A few of us are working other Ruby mods at TCS, so if you use the ML simply add "[mruby]" to the subject.

I'm not trying to force TCS as the way to collaborate, but rather, I'm offering to open it up if it adds value to anyone wanting tweak on a CMake/mruby build system. I'm most interested in seeing us quickly show a CMake prototype in it's best light for Matz to review.

[beoran]

johnforums, I think all in all that's a good idea, since I'm working on linux, and you seem to work on windows, and bovi on osx. However, I don't like to join mailing lists since they generate a lot of traffic that I find hard to keep up with. I prefer forums or issues like this one. Anyway, let's work on this together.

I already cloned that repo, and I pushed asmall change: a build directory. I think it may be valuable to do out-of-tree builds so the orginal Makefiles can be kept as they are, and the CMake build system then doesn't interfere with the Makefile-based one.

[jonforums]

@beoran agreed re: noisy MLs. Although the ML is relatively low traffic, I've turned on Issues at the repo so we can use either. Just synced with your mod...like the build subdir to keep the source dir and existing Makefiles clean.

[jonforums]

@bovi I've solved the undefined symbols error on both Win7 32bit and Arch 32bit with this commit

Please try my branch and reply to this issue to confirm that the commit fixes the symbols error you saw on Mac.

[bovi]

It's running!

[jonforums]

@bovi great, thanks for testing so quickly!

The focus now switches to:

• cleanup, library naming issues, dynamic lib build support, install tweaks
• add build-time binary archive packaging support (no installers) using cpack
• add support for building with Windows SDK via nmake
• add support for building with MSVC IDE (need a contributor as I don't use MSVC IDE)
• add cross-compiling toolchain samples for building on Arch/Ubuntu/OSX for Windows (need contributor for other cross examples)

Once these have stabilized, I'll squash down the current gaggle-o-dev-commits and submit a pull request for Matz' review. If accepted, further development should happen as part of this repo.

That said, if you're at all interested in this issue, please don't hesitate to test and post issues over at the issues list

ps...currently, it also builds using clang 3.1 (svn) + mingw 4.6.2 on Win7 32bit :)

[jonforums]

It's in good enough shape to ask for review and more testing.

A post to help you quickly get up and going: http://jonforums.github.com/ruby/2012/05/09/cmake-prototype-for-mruby.html

[jonforums]

Thanks to @nkshigeru and 1de8100 the Windows SDK/nmake build using cmake -G "NMake Makefiles" .. now appears to work (lots-o-warnings) on Win7 32bit :)

Build success stories summarized on the CMake-compatible wiki page; please update as you discover more.

[zhzhxtrrk]

IMO, Makefile is OK, just keep the source code and build process clean and simple.
Lua is using Makefile too, many people just delete lua.c and luac.c and merge all code files to their project, and that is all.

[zhzhxtrrk]

By the way, I love cmake. :)

[matz]

Since mruby has libraries written in Ruby, build process is more complex than Lua.
But there might be room to make it simpler.

[jonforums]

Currently, mrbc consumes a concatenated mrblib/*.rb file and generates a C byte array that is added to libmruby.a.

The cmake proto does this with no problems when building natively on OS X/Linux/Windows, but I've not yet coaxed cmake to do the right things for cross compiles. The key issue is to create an mrbc running on the build system that generates the mrblib.ctmp file which then gets cross compiled and embedded into the cross libmruby.a

I think the key is to get cmake to switch build toolchains/environments during the cross build process so it builds both native and cross versions of libmruby.a, mrbc, mruby.

I'd like to hear from someone with deeper cmake experience if/how this is possible and whether there's a cleaner way.

[jonforums]

I'm now able to cross compile for windows from arch or ubuntu 12.04. It needs more testing especially on OS X with a recent mingw-w64 cross-compiler

http://jonforums.github.com/ruby/2012/05/13/cross-compiling-mruby.html

[jonforums]

I've added mrbtest support to the cmake proto as of this commit

It has the following behavior:

• [n]make && [n]make test or [n]make all test - build all and run tests
• [n]make test - build and run only what's needed to run tests

Since the proto currently supports all the major use cases (native builds on Linux/OS X/Windows and cross compiles from Linux/OS X for Windows) I plan to clean up and submit a pull request by next Friday. There are plenty of needed refinements, but nothing currently is a showstopper.

Enhancements should now be done as part of the official mruby repo if Matz accepts the pull request.

As such, I ask that you run it through the test gauntlet in the next few days. Specifically:

• native OS X builds, and cross builds on OS X for Windows 32 and 64bit
• native Linux 64bit builds, and cross builds on Linux for Windows 32 and 64bit
• native Windows 64bit builds with MSYS, NMake, and Visual Studio 10 configurations

As always, submit problems here and list working configs here.

[mitchblank]

I was actually going to look at trying a cmake conversion, happy to see someone else already did one. I've got a lot of cmake experience, so I'll try to look at it as well.

The skinny on cmake: it's a love/hate relationship. Their own language is just barely suitable; doing anything complicated can be really frustrating but it's always possible (I say this as someone who has written cmake macros that are >500 lines long :-) However, once you get it all set up it works great. In my experience it's the best way to maintain code that needs to work cross-platform, since you can get Visual Studio .sln/.vcproj files from the same cmake files that produce your UNIX Makefiles.

tl;dr -- one vote for cmake

[matz]

I don't mind moving to cmake. I am waiting pull-request.

[jonforums]

I'm sculpting commits in preparation for submitting a pull request later today.

Thanks to all who tested and provided feedback. In particular, thanks to @beoran for the initial version, @bovi for watching out for OS X Lion, @luislavena for OS X Snow Leopard testing, and @nkshigeru for ensuring builds work on Visual Studio 10.

[jonforums]

closed via 618697c