Change build system?

[darnuria]

Have you plan to move to an other build system like Cmake or autotools?
This will be in your roadmap #11 ?

Thanks! And sorry to disturb you but this questions seems important.

[techtonik]

I'd be shamelessly promoting SCons both as a build system, test suite and a benchmark.

[martyglaubitz]

agree to @techtonik i'd choose SCons

[Bengt]

For what it'is worth, TravisCI supports autotools, make, cmake and scons as well. Compare #4.

[martyglaubitz]

i already made a fork to do this now...

[martyglaubitz]

@Bengt maybe gyp would be an option ?
it's used by the v8 team at google...
Gyp Manual

[martyglaubitz]

another option: gradle
even though it's primarly designed for java and i used it for android projects, it has a
fairly good support for C/C++ and is, unlike SCons and gyp, well documented

[kmod]

Well, I think Make has been working decently well for our needs, but we've started running into issues with cross-platform support, since for example we don't have a configure script that will help us find libraries or dependencies. I heard CMake is good for this?

[dagar]

dagar@0cba092
Here's an initial (quite rough) cmake config that can build pyston on ubuntu 14.04. This is probably the sanest solution to cross platform development, especially if you ever want pyston on windows.

It would also offers a much cleaner integration with llvm (although I don't have that working yet). See http://llvm.org/docs/CMake.html#embedding-llvm-in-your-project

If you want to go with cmake I could probably replicate most of the current Makefile in a couple days.

[kmod]

Cool :) I tried running it and got this error though:

CMake Error at CMakeLists.txt:49 (find_package):
  Could not find module FindLibLZMA.cmake or a configuration file for package
  LibLZMA.

sorry if this is a simple problem; I haven't used cmake before.

Our Makefile currently has a fair amount of ahem "nontraditional" behavior, which I think would take some time to port to cmake. Would it be possible to incrementally port things to cmake, and have the generated makefile reference our included makefile for the trickier bits? I think a lot of the configuration-type logic would be good to start with, but some of the program-running parts would be both harder and less valuable to port.

[dagar]

FindLibLZMA.cmake is in the cmake-data package in ubuntu.

I think the current Makefile and cmake could coexist during the transition. The only confusion is the default cmake generated unix makefiles are named Makefile which would need to be added to .gitignore. Once cmake is ready to take over compiling, the existing src/Makefile could be renamed to src/pyston.mk and dummy targets added to cmake that invoke make -f src/pyston.mk TARGET for the harder to move parts.

dagar@ce7293a

[kmod]

Hmm I seem to already have cmake-data installed; this is on 11.04, which has been EOL for a while (why did I ever pick a non-LTS release...), but I wonder if, since the point would be to improve cross-platform compatibility, it's possible to avoid relying on platform-provided bits?

Could we have the generated makefile just include what's left of our existing makefile?

[martyglaubitz]

i'd still put my eggs into the gradle basket
i've done some C++ building with it recently and it was pretty nice to use

Plus: you can easily customize build tasks with groovy if you need to do something special
(i was downloading & compiling libSDL + libSDLImage before staticly linking my program against it)

[dagar]

@kmod
https://github.com/dagar/pyston/blob/cmake/CMakeLists.txt
I removed the dependency on the FindLibLZMA cmake module (necessary for using find_package). I think using find_library is sufficient for finding most of the dependencies. The next problem you'll likely run into is that I didn't link -static-libstdc++. I think setting CC, CXX, LD_LIBRARY_PATH, etc should probably be done in your local environment outside of the build system.

I don't have much to contribute to a cmake vs gradle debate. A large part of what I like about cmake is the ability to generate your favorite native makefile (gnu make, nmake, ninja, etc) or ide project (eclipse, xcode, visual studio, etc) and it looks like gradle offers a lot of that.
I think the main difference for pyston will be the ease of integrating a cmake project with llvm (http://llvm.org/docs/CMake.html#embedding-llvm-in-your-project). I'll take a look at how this would simplify the pyston configuration (especially cross platform).

[kmod]

I think we should go full-steam-ahead with the CMake stuff; that's not to rule out any other options, but the decision seems close enough that it will probably be decided by whoever is the most willing to do the corresponding work.

I think you're right about the environment variables, but I do think there's also value in having a predefined "development environment" that combines installation instructions + the appropriate settings for those variables. Right now things are kind of messy since this development environment is embedded into the build system, so yeah we could definitely factor that better; in practice, though, I expect that everyone will use the recommended environment, which is why fixing that has been lower priority than helping people get set up easily.

[dagar]

I now have cmake properly building and linking the stdlib (src/runtime/inline/*.cpp).
dagar/pyston@dropbox:master...a53e195

I discovered Shadow Simulator also uses cmake to emit and link llvm bitcode. I was able to reuse some of their custom LLVMTools cmake module to handle the manual compilation and linking (somewhat awkward in cmake).
See dagar/pyston@dropbox:master...a53e195#diff-2

Next step is proper llvm integration and library searching.

[kmod]

Cool!

It sounds like there will be quite a bit of work getting cmake to do everything our current Makefile does; I would recommend prioritizing getting a minimal cmake setup working 100% (ie relying on the existing Makefile for most of the logic), and then migrating things from the Makefile to cmake over time. I think that will make the process a lot more sane, since we'll be able to work on a few things at a time rather than trying to drop a huge port at once and having to hunt down lots of issues.

Also, right now it crashes for me at CMakeLists.txt:8 with Unknown CMake command "error"...

[dagar]

It's getting there, but still lots of work to do. dagar/pyston@dropbox:master...afc7f84
That error message should be fixed, but I'd expect other problems outside of ubuntu 14.04.

This is how I build

CC=clang CXX=clang++ cmake -DLLVM_DIR=~/pyston_deps/llvm-trunk-build-cmake/share/llvm/cmake ~/pyston
make -j4 

It looks like llvm needs to be built with cmake to generate the required LLVMConfig.cmake in ${LLVM_DIR}/share/llvm/cmake.

Debug/Release mode for make/ninja can be controlled by adding -DCMAKE_BUILD_TYPE=Debug/Release. This can be changed on the fly in IDEs. I think pyston_prof will likely exist as another build type.

Some questions/comments

• How would you like to handle patches long term? Will everything pyston needs eventually be upstream in llvm, clang, and libunwind? I think the cleanest thing to do is completely decouple building llvm from pyston. The packaged llvm-dev or equivalent usually includes the llvm cmake module pyston needs (except ubuntu but there's an open bug), so eventually using the packaged version would be nice.
  For now you could fork llvm-mirror/llvm and include it as a submodule. That would remove patch handling in the makefile, llvm_revision.txt, simplify grabbing dependencies and git bisect.
• LLVM is currently built with/without optimizations, assertions, profiling. Does a debug/release/profile pyston actually need the corresponding llvm or is it just helpful sometimes? Again, I'd like to decouple building llvm from pyston if possible.
• Obviously the inlined stuff needs to be built with clang, but what are the requirements for the rest? Just std=c++11? Cmake can error if it doesn't find the right compiler versions and I'm trying to keep the compiler specific flags separate.

[kmod]

Hi Daniel, I'm impressed that you're up to changing so much :) but I think we should try to keep any workflow changes to a minimum if we're changing the build system at the same time. I think we'll have our hands full for the initial migration with just getting cmake working on the different platforms we're currently on and making sure that everyone can still follow the same workflows. We'll have plenty of time to migrate more over time!

As for your questions

• While yes, for any given patch the goal is to get it upstreamed, as a whole it will probably stay common to have local patches that are in the middle of the upstreaming process.
• We do depend on a trunk version of LLVM, and I don't see that changing any time soon :/ I hope that at some point the JIT stuff will be stable enough that we could run off of a platform-provided release, but I think that's a ways away.
• I think switching the llvm repository to a git submodule could make a lot of sense, and would simplify a lot of the tooling around the revision handling + patching. That should probably be done separately from a cmake migration.
• The pyston_prof target is pretty defunct and the only thing it currently gets used for is to make sure we can get compiled under gcc. I think we do want the debug (full debug symbols), quick (assertions but no debug symbols) and release (no assertions) builds of llvm. We barely use the profile build but it'd be nice to keep that around.
• For the compiler requirement, yeah I think std=c++11 is the main requirement. I haven't tested it, but I would guess that the LLVM requirements are pretty close to ours as well.

[dagar]

Hi Kevin,

Please give this a try when you have a chance. I have a few more questions and details to verify, but I'd like to know what you think and if you can get it to work on older ubuntu before going further.
dagar/pyston@dropbox:master...e9b5ae6

I've included llvm, clang, and libunwind as submodules for simplicity, but this could easily be changed to play nice with the current python_deps and minimize existing workflow disruption.

To build everything

git clone --branch cmake --recursive git@github.com:dagar/pyston.git pyston-cmake
mkdir pyston-build && cd pyston-build
CC=clang CXX=clang++ cmake ../pyston-cmake  # CC, CXX are the llvm host compiler and used to build everything except runtime/inline/*.cpp
make
make test                 

Pyston's cmake adds LLVM as a subdirectory which merges the build systems. This means that when you set the Pyston build mode (Debug or Release), or CC, CXX, flags, etc it's used in Pyston and LLVM. You also have access to all Pyston, LLVM, and Clang targets in the same place with proper dependencies. For example, there are dependencies on opt, llvm-link, and clang++ to compile runtime/inline/*.cpp, so only those binaries are built.

With a few minor changes both cmake and the existing makefile could happily coexist.

[dagar]

Notes on https://github.com/dagar/pyston/tree/cmake

It's currently working in Debug and Release modes across gcc 4.8, clang 3.4, 3.5 with most of the testing. I've still only tried on Ubuntu 14.04 though. That's full Debug or Release mode for Pyston AND LLVM. Do we need a mode that's Pyston debug + LLVM Release+Asserts?

Things I'm looking at adding next

• intel_jit_events <- obsolete?
• pyston_grwl
• profiling

There are some helpers that I'm not sure really belong in cmake.
Things like make run_TESTNAME, dbg_TESTNAME, wdbg_TESTNAME, memcheck_, memcheck_gdb, memleaks_, cachegrind_, etc don't really translate unless I scan the test directory and create a target for each test case at configure time. Would these make more sense as scripts in tools?

Could you clarify the current profiling story, or just tell me which of these you'd like to exist in cmake?

• pyston_oprof
• pyston_pprof
• pyston_prof
• pyston_profile
• perf_report
• oprof_*

Misc things I still need to look at adding, but are probably easy. Please let me know if I'm missing anything important or if any of these are now obsolete.

• demangle
• mcjitcache
• test_opt
• test_asm
• test_cpp_asm
• test_cpp_dwarf
• test_cpp_ll

Currently cmake places the pyston binary at the top level of the output directory, which I think is the best place to put it. If we modify the existing build system (and a few source files) to work with the binary at the top level then cmake could start coexisting with the current makefile.

[dagar]

I was able to get my cmake branch working on Ubuntu 12.04 using gcc 4.8 and clang 3.4 from the ubuntu-toolchain-r ppa and cmake 3 built from source.
sudo add-apt-repository ppa:ubuntu-toolchain-r/test

At this point I believe that all of the core functionality that I'm familiar with is now working in cmake. Missing pieces include profiling, intel_jit_events, scan-build, test running/debugging helpers, and misc tests mentioned above (test_asm, test_opt, etc).

https://github.com/dagar/pyston/tree/cmake

[kmod]

Cool, I was able to build pyston successfully! I had to do a few things to get it to work, which probably should get added to the eventual default instructions -- I had to add
-DGCC_INSTALL_PREFIX=~/pyston_deps/gcc-4.8.2-install -DCMAKE_EXE_LINKER_FLAGS="-Wl,-rpath,~/pyston_deps/gcc-4.8.2-install/lib64"

I set the rpath so that all the tools, which I'm linking with my non-system-wide gcc-4.8.2 libraries, can run without specifying LD_LIBRARY_PATH which personally I think is a bummer to have to do all the time (yes I can set it once but we have to get anyone interested in Pyston to learn to potentially do that too). CMake seems to have built-in support for rpath handling but I already had the linker argument handy.

I would say we need to aim for supporting all the build configurations -- yes some of them are very rare, but they usually do get used from time to time. Also, the Pyston Debug + LLVM Release+Asserts (Unoptimized+Asserts would also be ok) build mode is the main debug mode, since the LLVM debug info weighs in at ~500MB which makes things slow to link and debug when it's included.

I was pretty disappointed that running make on an up-to-date build takes a fair bit of time to run, since it's also checking llvm and clang for anything out of date. Then I tried having cmake generate a ninja build and that's extremely fast, which is pretty cool and makes me feel better about cmake.

How are you envisioning things working for multiple build configurations? I didn't set the mode and I'm not sure what I ended up with, and I'm not sure to generate multiple configurations ex for the unittests. There's a bunch of hackery in the current Makefile regarding the building of the different LLVM configurations including trying to build them in parallel in the same directory -- not saying that that has to be done the same way, but it makes me curious how you're planning on handling it.

I did a tiny bit of research and it looks like cmake really does not play with external makefiles well. Maybe one option for now is to do the reverse: have our existing makefile shell out to the cmake build to do the building, and the existing makefile can take care of the long tail of odd targets? ie something like

.PHONY: pyston_dbg
pyston_dbg:
  make -C ../../pyston-build-dbg
  cp ../pyston-build-dbg/pyston pyston_dbg
weird_run_command: pyston_dbg
   # etc

ugly but could solve the interopability problem for now, and let us start migrating without having to rewrite everything at once.

[dagar]

It's kind of surprising how slow the generated makefile is, but it was useful to see what's happening. I've mostly been using ninja as well.

Currently it defaults to Release mode or you can set Debug (-DCMAKE_BUILD_TYPE=Debug) which gives you symbols, unoptimized, and asserts for both Pyston and LLVM. FYI my full debug Pyston is about 150MB and I had to double the test timeout to consistently pass.

To keep things simple we could stick to just Release or Debug. LLVM debug info could be disabled by default and for release mode you could optionally turn asserts on if you want them (-DLLVM_ENABLE_ASSERTIONS=ON).

You need a build directory for each release type. Then you run the unittests for each release type in that directory (ninja check_pyston).

I think trying to get the existing makefile and cmake to work together might be more trouble than it's worth.

Next steps
There are a few minor changes like moving the pyston binaries up to the root directory and changing the search for codegen/parse_ast.py that would allow cmake to be a secondary build system right now. Then people can start using it and we can iron out the issues.

[dagar]

I'll generate targets for run_TESTNAME and dbg_TESTNAME. If you could list the other must have targets I'll be sure to include them before opening a pull request.

[kmod]

I think in this case "keeping things simple" means to change less, not to implement less. If you want to reduce the work of the migration I've had a few comments, but I definitely don't think that skipping features of the current build system is the way to go.

Are you planning on maintaining the cmake stuff side-by-side with the existing Makefile until we potentially get rid of the Makefile entirely? It seems much less risky to do try to migrate parts at a time.

Random question: how can I specify that I want to build Pyston with the newly-built clang compiler?

[dagar]

I definitely don't want to skip features of the current build system. I thought these modes (with options) would be equivalent to the current build. Have I missed something?

Release (+/- asserts) llvm_quick and llvm_release
Debug (optionally disable llvm symbols) llvm_debug

I'd like to maintain cmake side-by-side. The current makefile will continue to work in source, and cmake can be used out of source. Once a few regular contributors are using cmake it should be easy to knock out any remaining issues. I'd like to be able to migrate parts at a time but embedding llvm forces out of source only builds, so I'm really not sure how they could be used together effectively.

Cmake really needs to know the compiler at configure time. You could try doing this, but it's a bit of a hack and will cause everything to rebuild.

rm -rf CMakeCache.txt CMakeFiles/
CC=llvm/bin/clang CXX=llvm/bin/clang cmake -G"Ninja" ~/pyston

From the cmake wiki
"Once a build tree is created with a given compiler it cannot be changed. There are a variety of implementation reasons for this policy."

The majority of src/runtime/inline/CMakeLists.txt exists simply because it's so difficult to manually reconstruct the compile command in cmake.

[dagar]

I took a second look at your interop suggestion. I think this could work once we're ready to completely commit to cmake for compiling. I don't want to depend on more specific paths, but the old makefile could be responsible for picking the build directory, running cmake, then use the pyston_dbg target as you showed.

[dagar]

Actually I think cmake could just handle it right now. I'll add a POST_BUILD target that copies the pyston binary back to ${CMAKE_SOURCE_DIR} as pyston_{CMAKE_BUILD_TYPE}.

[darnuria]

Hello I viewed this issue still opened in my dashboard. It seems that this #460 PR resolved this bug.
So I think this is time to close it two years later!

A good bug is a resolved and closed bug. :)

[kmod]

Can't believe we (by which I mean @dagar) made it happen!