Add Experimental CMake support

[Ozaq]

Support for building with CMake added for Linux/OSX native builds.

Build output can be controlled by supplying TILE_MODE to the build.
WEB_TILES for building the web version. LOCAL_TILES for native tiles and
NO_TILES for ascii mode.

E.g.

    cmake -DTILE_MODE=WEB_TILES <path to source>

Use of "prebuilt" level compiler sources can be controlled by
-DUSE_PREBUILT_LEVCOMP=ON/OFF

Mergebase can be set with
-DMERGE_BASE=

Arbitray defines can be passed to the build by
-DADDITIONAL_DEFINES=FOO;BAR=1;DEBUG

Build output is generated to 'stage' and crawl can be run diretly from
its build location. All dependent files are copied to stage. Note that
starting the webserver from stage can be done by
./webserver/server.py

Note:
Since there is unfortunately still in-tree generation hapening it is not
advised to have different build types (localtiles/webtiles/notiles)
running side by side w.o. EXPLICITLY cleaning the source folder.

Limitations:

• No special Travis support
• Limited compflags.h (no cflags, no ldflags)
• No crossbuild (linux -> Windows) atm
• No Windows native build atm (not tested)
• Only builds from external deps
• Only support out-of-tree builds
• Still does generate some files in source tree

[alexjurkiewicz]

Wow! Nice work! How does build time compare? What does cmake do better than make? Would this make reproducible builds more possible?

[rawlins]

@alexjurkiewicz see #903 for some background.

Would this make reproducible builds more possible?

Is there a problem with reproducible builds? 4d52153 removed the only issue I'm aware of.

[alexjurkiewicz]

Doesn't seem to work for me: https://gist.github.com/alexjurkiewicz/6a25453c1d1acbf20bb99dc4e23fa656

[Ozaq]

Thanks for testing and providing the build output!

2 unrelated issues I can already see:

• curses and threads are not found.
• you do in in tree build ( make Output dir inside the source tree, '.' In your case. This is currently unsupported but not the cause of the error.

I did some last minute target moving before committing and I might not have tested properly. I'll get back to it after work. CEST here, so it'll be some time :)

[Ozaq]

I reread the output: those are all warnings last two lines say build files have been written correctly. What happens if you do a subsequent make? Can you post the build output as well?

Just in case there is some confusion: cmake is not compiling the sources but creating a makefile/ninja/xcode (untested) setup that then can be used to compile

[alexjurkiewicz]

you do in in tree build ( make Output dir inside the source tree, '.' In your case. This is currently unsupported but not the cause of the error.

About this, I am not sure what you mean. Should I be testing with a different cmake commandline?

[alexjurkiewicz]

Build output:

$ make
Scanning dependencies of target link-settings
Error copying directory from "/Users/alex/personal/crawl/crawl-ref/source/settings" to "settings".
make[2]: *** [CMakeFiles/link-settings] Error 1
make[1]: *** [CMakeFiles/link-settings.dir/all] Error 2
make: *** [all] Error 2

[Ozaq]

you do in in tree build ( make Output dir inside the source tree, '.' In your case. This is currently unsupported but not the cause of the error.

About this, I am not sure what you mean. Should I be testing with a different cmake commandline?

Have a look into README-CMAKE.md

~/crawl        <--source
~/build-crawl  <--build (location of your choice)
cd build-crawl
cmake -DCMAKE_BUILD_TYPE=Debug -DTILE_MODE=LOCAL_TILES ../crawl/cral-ref
make -j8

[Ozaq]

@alexjurkiewicz
After rereading your console output I can see that you selected the wrong root folder for the CMake build and that CMake could not find ncurses on you system.

W.r.t how to run the CMake build I tried to update the documentation to make some things more explicit and clearer. Please have a loot at the README-CMAKE.md your feedback is welcome (and everyone else's as well). It is important to me that the README is correct and actually helpful!

W.r.t ncurses: I am expecting to be able to find ncurses with pgk-conf. I.e. for ncurses installed via homebrew you need to export PKG_CONFIG_PATH="/usr/local/opt/ncurses/lib/pkgconfig" for pkg-conf to be able to find ncurses. I could not make crawl build with MacOS ancient ncurses, if you know how to do that let me know and I will add this to the CMake build.

Since your issue report prompted me to redo all builds an retest them locally I did find 2 more things:

• There was one issue w.r.t to console builds trying to copy png files, that is now fixed.
• Several generated lua files are not yet copied correctly to stage, not yet addressed

And ofc, checkwhite has to be pacified :)

Oh and you asked about builds times:
They are roughly the same, but with the CMake variant incremental rebuilds work. No more recompiling everything.

[rawlins]

W.r.t ncurses: I am expecting to be able to find ncurses with pgk-conf.

This might already be apparent, but the OS X builds currently don't ever use pkg-config, so nothing is tested with pkg-config packages at all on OS X. I think the reason for this is just historical (and maybe it's a bit less reliable overall, I've had problems with pkg-config installed via anaconda), but it does mean that trying to enable this adds an extra layer of potential problems.

As to ncurses, it currently builds with system ncurses, so there shouldn't be anything too complicated about that. On my system, the link libraries are:

contrib/install/x86_64-apple-darwin16.7.0/lib/libz.a -L/usr/lib -lncurses -Lcontrib/install/x86_64-apple-darwin16.7.0/lib contrib/install/x86_64-apple-darwin16.7.0/lib/libz.a contrib/install/x86_64-apple-darwin16.7.0/lib/liblua.a contrib/install/x86_64-apple-darwin16.7.0/lib/libsqlite3.a

In the compile flags there's just -isystem /usr/include/ncurses.

[Ozaq]

@rawlins Thanks for the info, Ill see that I fixup this PR so that it handles MacOS system ncurses

[Ozaq]

I think I have now addressed what was raised so far:

• Checkwhite
• System curses now used on MacOS
• crawl can be directly started in stage folder after build in all configurations (tiles. no tiles, web tiles)

[alexjurkiewicz]

Sorry for the slow response. But nice! It works perfectly on my system too :)

[Ozaq]

Note: If you have checked out this branch be aware that I try to rebase it on top of master once a week to detect breaking changes early.

[perryprog]

What's left for this to be merged? Looking forward to the change!

[rawlins]

Thanks for continuing to work on this! I tried it out and was able to get up to the point of linking where it seemed to fail to link with lua. This is OS X, with lua installed via macports.

-- Found Lua: /opt/local/lib/liblua.dylib (found suitable exact version "5.1.5")

Link errors:

[ 98%] Linking CXX executable ../stage/crawl
Undefined symbols for architecture x86_64:
  "_luaL_loadbuffer", referenced from:
      CLua::loadbuffer(char const*, unsigned long, char const*) in clua.cc.o
      CLua::loadfile(lua_State*, char const*, bool, bool) in clua.cc.o
      _loadline(lua_State*) in luaterp.cc.o
  "_luaL_register", referenced from:
      dluaopen_monsters(lua_State*) in l-mons.cc.o
  "_lua_call", referenced from:
      CLua::init_lua() in clua.cc.o
      _clua_dofile(lua_State*) in clua.cc.o
      crawl_err_trace(lua_State*) in l-crawl.cc.o
  "_lua_insert", referenced from:
      CLua::setuniqregistry() in clua.cc.o
      CLua::setregistry(char const*) in clua.cc.o
      CLua::pushglobal(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) in clua.cc.o
      CLua::callfn(char const*, int, int) in clua.cc.o
      _clua_loadfile(lua_State*) in clua.cc.o
      _clua_guarded_pcall(lua_State*) in clua.cc.o
      lua_datum::lua_datum(CLua&, int, bool) in clua.cc.o
      ...
  "_lua_objlen", referenced from:
      _loadline(lua_State*) in luaterp.cc.o
  "_lua_pcall", referenced from:
      CLua::execfile(char const*, bool, bool, bool) in clua.cc.o
      CLua::execstring(char const*, char const*, int) in clua.cc.o
      CLua::calltopfn(lua_State*, char const*, __va_list_tag*, int, __va_list_tag (*) [1]) in clua.cc.o
      CLua::callfn(char const*, int, int) in clua.cc.o
      _clua_guarded_pcall(lua_State*) in clua.cc.o
      crawl_err_trace(lua_State*) in l-crawl.cc.o
      dgn_with_map_bounds_fn(lua_State*) in l-dgn.cc.o
      ...
  "_lua_remove", referenced from:
      _loadline(lua_State*) in luaterp.cc.o
  "_lua_tonumber", referenced from:
      luaL_safe_tointeger(lua_State*, int) in cluautil.cc.o
      crawl_worley(lua_State*) in l-crawl.cc.o
      crawl_worley_diff(lua_State*) in l-crawl.cc.o
      crawl_split_bytes(lua_State*) in l-crawl.cc.o
      crawl_simplex(lua_State*) in l-crawl.cc.o
      crawl_do_commands(lua_State*) in l-crawl.cc.o
      moninf_get_is(lua_State*) in l-moninf.cc.o
      ...
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [stage/crawl] Error 1
make[1]: *** [source/CMakeFiles/crawl.dir/all] Error 2
make: *** [all] Error 2

Any hints?

[rawlins]

Other possibly superficial comments / questions while taking what is really my first look at this:

• a more natural place for the README contents in our existing tree would be in crawl-ref/doc/develop, or possibly as something like INSTALL-cmake.txt in crawl-ref, where INSTALL.txt is also. (Or part of that file.)
• does switching build arguments always entail running cmake again? (Or having multiple build directories I guess, maybe this would be the more natural cmakeian way?)
• Any particular reason for CMakeLists.txt to be in crawl_ref/?
• How do I build with contribs? On this note I think CMakeLists.txt has some incorrect information -- native builds for mac should probably not use OS packages, because this isn't tested at all. Mac release builds will always use contribs. Actually one of the things I've been hopeful that cmake can give us is some integration with those contribs that do use cmake themselves.
• Can you explain the preference for out-of-source builds? Is this a standard cmake thing?

[perryprog]

Oh, sorry—I'm not the original author, if that's what you thought. I was just wondering what needed to be done still. I can probably answer a few of those questions though, but I'm sure they won't be perfectly accurate.

• I believe cmake will be fairly smart about what it needs to recompile. Some obvious exceptions are things like changes to the compiler args.
• I would guess that @Ozaq chose to put it there as that's sort of the root directory of the source code. It's uncommon to have the CMakeLists file to be in the same directory as the actual source files.
• Out of source builds are usually the norm for cmake projects. I don't know if it has support for in-source builds and I've never seen it done. I personally prefer out-of-source builds.

CMake should be perfectly good for using builds under contribs. It doesn't look like it's currently set up to use contribs programmatically, but you might be able to comment find_package and uncomment the section here. You'll probably have to do a similar change for other deps as well.

Again, take the above with a grain of salt—I'm just answering what I can in case the original author isn't available for a while or something. /shrug

[rawlins]

@perryprog yes, sorry for any confusion, I do realize that you're not the PR author.

[rawlins]

-- Found Lua: /opt/local/lib/liblua.dylib (found suitable exact version "5.1.5")

With a bit more investigation it looks like this step is incorrect: I have both lua 5.1 and lua 5.3 installed via macports and this path is actually to the lua 5.3 lib. Lua 5.3 is in /opt/local/lib and lua 5.1 is in /opt/local/lib/lua-5.1, so what it should have used is /opt/local/lib/lua-5.1/liblua-5.1.dylib. I'm not sure how to debug this further. It possibly interacts with https://gitlab.kitware.com/cmake/cmake/blob/master/Modules/FindLua.cmake, though as far as I can tell my version of that is current except for indentation changes and the like.

Edit: I upgraded cmake (from 3.13.1 to 3.14.5) to no effect, but from some googling I do suspect it may be an issue in one of the cmake FindLua scripts that I'll need to work around (I just don't know how).

[Ozaq]

@rawlins ill look into it, probably not before Friday tho

[Ozaq]

I am on lunch brake so I can only comment to some of the things @rawlins and @perryprog wrote, some of the stuff I have to look into proper.

1.) About Lua link errors, actually no idea. I have tested on MacOS with home-brew. Ill red to investigate this a bit.

2.) About README. I just wanted to have some readme, I will move that where ever you prefer.

3.) Switching build arguments most of the time means you need to re-run cmake. After you called cmake a 'CMakeCache.txt' is added to your build folder (and respectively updated on each cmake invocation). Here you can always check what variables are set to. NOTE: if you re-run cmake it will not clear variables that you did not define on a second invocation, e.g. you call

cmake -DMY_VARIABLE=5 -DMY_OTHER_VARIABLE=6
... hack hack
cmake -DMY_VARIABLE=10

CMakeCache.txt will still list MY_OTHER_VARIABLE=6

Having several build folders is probably the way to go, id say probably because there are quite a few 'nice' things we can do later, like e.g. making WebTiles, LocalTiles, Console top level build targets so that they can be build from a single invocation of make. (But that requires some work on all the code generation tools)

Does that answer your question?

4.) About crawl_ref: @perryprog pretty much nailed it :)

5.) About Contribs: I did start with building from contribs only which you can see from the commented out code. I can change the MacOS build to use contribs only. Will update the PR accordingly. Since this PR is only supposed to get cmake started in crawl I would like to keep it limited for now to in that case then build linux from external deps and macOS from contrib. OK?

6.) About out of source builds: Cmake does support in source builds but for this project I explicitly added a check to disallow them. Out of source builds allow to maintain multiple build trees on a single source folder, i.e. I can quickly switch configs and only rebuild changed parts of my build tree. I think this is very useful for crawls atm since it allows me to quickly compile all three frontends (web, local, console) with a bit of bash, e.g.

cd WebTiles-build && make -j$(nproc) &&  \
cd ../LocalTiles-build && make -j$(nproc) && \
cd ../NoTiles-build && make -j$(nproc) && cd ..

Also one more abstract property of out of source tree builds is that they help to keep dependencies on generated code more explicit since they usually have to be modelled.
Also note that currently the build is not really out of source tree. There are still several tools that generate code into the source tree. This is stuff I would like to address but this would make this PR explode with changes to the existing build system.

EDIT: typos

[Ozaq]

@rawlins & @perryprog I an currently making all contribs available for Linux and MacOS, this will take a bit since testing each change takes a lot of time.

@rawlins In the meantime you could help me with the lua5.3 dynlib issue. If the paths you mentioned above are correct this almost looks like an issue in the FindLua module from cmake. Which would really suck. To understand whats happening I would like to get some trace output from cmake. To do that I would like to ask you to create a new clean build folder and run cmake with --trace-expand, this outputs very statement cmake evaluates with expanded variables. With this and some patience its possible to track down the reason. Can you then post the full output here or on a pastebin? (its going to be a lot of output)

[Ozaq]

So this should address your concerns @rawlins. One thing i noticed today is that PCRE support is yet missing in this PR. It this is immediately required I can add this soon (TM)

[aidanholm]

Thanks for working on this! Unfortunately I currently cannot build:

$ cmake -DCMAKE_BUILD_TYPE=Debug -DTILE_MODE=LOCAL_TILES ../crawl/crawl-ref

-- Host identification is: x86_64-pc-linux-gnu
-- Target acitecure is: x86_64-pc-linux-gnu
-- Using Lua from system
-- Using zlib from system
-- Using sqlite from system
-- Using libpng from system
-- Using sdl2 from system
-- Using sdl2-image from system
-- Using Freetype2 from system
CMake Error at CMakeLists.txt:463 (add_library):
  add_library cannot create ALIAS target "freetype" because target
  "Freetype::Freetype" is imported but not globally visible.


-- Proportional font could not be found on system. Using our font.
-- Monospace font could not be found on system. Using our font.
-- Configuring incomplete, errors occurred!

This is on an up-to-date arch linux x86-64 system.

NO_TILES builds work fine. Haven't tried WEB_TILES.

[mossheim]

@aidanholm i also ran into that issue. i fixed it and a few other things i found and made a PR against @Ozaq 's branch (https://github.com/Ozaq/crawl/pull/1). are people still interested in this? i can help if there's interest, i'd really like to see this happen.

[aidanholm]

@brianlheim yes, there is absolutely still strong interest in moving away from the horrible makefile that we are currently stuck with, but it would need to be a rather feature-complete conversion; cross-compiling with mingw and using vendored dependencies are hard requirements. There are also people building on msys and probably some on WSL as well.

[mossheim]

ok! just want to make sure i understand:

• cross-compiling -- what are the source and target platforms?
• using vendored dependencies -- it looks like there is already an option to use the dependencies provided via submodules in this repo, and that's how i was able to build it on arch linux; do you need more configurability than that?
• msys -- i don't see this tested in CI, is it? if not i'd like to add that. i have a windows environment i can use to develop but i'd feel a lot better if there were CI builds running as well
• WSL -- any user space/distribution in particular? IIUC WSL is meant to be essentially like a Linux VM so there shouldn't be much of a difference if any between an Ubuntu native and Ubuntu WSL build, for instance

regarding CI, Travis has macOS and Windows images available now, and the Windows image, although still in 'early adoption' stage, can use MSYS2. i would recommend adding jobs for those while the newer build system settles. the coverage doesn't have to be as thorough as the linux jobs you already have.

[mossheim]

cross-compiling -- what are the source and target platforms?

is it this build (https://travis-ci.org/github/crawl/crawl/jobs/564379148)? so linux compiling a binary to be used inside a windows mingw environment?

[gammafunk]

@brianlheim No, those binaries are windows-native and don't require a special environment to run. The mingw toolchain description from the homepage:

MinGW provides a complete Open Source programming tool set which is suitable for the development of native MS-Windows applications, and which do not depend on any 3rd-party C-Runtime DLLs. (It does depend on a number of DLLs provided by Microsoft themselves, as components of the operating system; most notable among these is MSVCRT.DLL, the Microsoft C runtime library. Additionally, threaded applications must ship with a freely distributable thread support DLL, provided as part of MinGW itself).

[mossheim]

okay... so just cross compiling a binary to be used on windows then. am i correct in thinking that's what is meant by "cross-compiling with mingw" above though?

[gammafunk]

Yes, that's what aidanholm was referring to, and that's very important for us as it's how we produce native windows builds, including for release.

The MSYS2 option is what we sort of recommend for windows people wanting to do development, but I'm not sure if that's truly our recommended windows dev environment any more. Typical windows compilation using MSVC was fixed somewhat recently and still works, to my knowledge.

[mossheim]

okay! are MSVC and MSYS2 options hard requirements for CMake support in this PR? or merely aspirational?

[gammafunk]

I will leave a full answer to that question to people more qualified to answer, but I think it's fair to say that we need to support at least one method of building crawl natively from windows. I guess MSVC would be the best if we had to choose one, since it works across recent versions of windows? I can't comment on the current state of compilation under windows, though. Other devs probably can do so more easily.

[mossheim]

sounds good! i've set up a few simple travis jobs that use cmake on my own fork, one is an MSVC build. will tweak around with it and see if there are any major issues.

[Goratrix]

As someone who uses MSYS2 to compile crawl, I have to say I vastly prefer it to MSVC, would be sad to have to move to MSVC. MSYS2 is very easy to work with, especially for people who work in Windows but are also familiar with Unix. It's self-contained, doesn't 'invade' the OS unlike MS tools (especially MSVC) do, and things like reinstalling the MSYS2 environment from scratch are really easy. Also, MSYS2 itself is an envirnoment, i.e. a shell, package manager, tools, basically a 'fake Unix' within windows, but most of that is not that relevant here. The actual complier toolchain is Mingw-w64. Basically MSYS2 is a full 'development' platform, and Mingw-w64 (contained within MSYS2) is a 'distribution' platform. And that's how it is being used here, for easy compliation of windows binaries with minimal overhead.

[aidanholm]

If msys is a fully featured alternative to MSVC that is reasonably easy for windows users to compile with, I'd probably prefer to remove MSVC support: crawl already has an immense maintenance burden, and we already use msys for official binaries. If there are no devteam members using it and few enough users, I don't think the added burden is worth it (from e.g. binaries built in weird ways). I'd say the same for Xcode: it looks like the project file for that is already broken, so presumably noone is using it.

I'd be interested in knowing what the benefits are of MSVC; presumably a more familiar dev environment for windows users?

[Goratrix]

I'd say it's definitely easier to compile in MSYS2 (and generally easier to install, maintain, get working without issues), but harder to do do full-fledged development (including debugging, etc...). Also thanks to MSYS2 being POSIX-compliant, you can just directly use a ton of Unix stuff and knowledge and approach.

The sort of full development that would be easier in MSVC probably doesn't matter much for non-devteam contributors, people like me who just want to compile and fix small bugs here and there...

I mean, my steps to get crawl compiled after a recent reinstall were to install MSYS2 and then do this sequence:
pacman -Syu
pacman -Syu
pacman -S base-devel git
pacman -S mingw-w64-x86_64-toolchain
pacman -S mingw-w64-x86_64-python-yaml
pacman -S zip
git clone https://github.com/crawl/crawl.git
cd crawl/crawl-ref/source
git submodule update --init
make package-windows-zips

That's it, it took a few minutes. And all of it just exists in its own folder without touching the rest of the Windows OS in any way. If I think about installing MSVC (which in itself is quite a burden), and then look at this: https://github.com/crawl/crawl/blob/master/crawl-ref/INSTALL.md#maintenance-notes it seems like a nightmare. But maybe people who use MSVC could chime in here.

[rawlins]

Re MSVC vs MSYS2: dropping MSYS2 isn't on the table (no devteam member uses MSVC and we can't even support it -- there's not even any currently active devteam member with a windows machine AFAIK, we just use VMs). Part of the appeal of cmake was that it would hopefully make handling MSVC easier exactly given that we can't directly support it. Nothing is a hard requirement for cmake in the sense that the plan has been to aim to merge some form of cmake building without removing make support (because of the complexity of the task we don't have any plans to convert online servers to cmake any time soon after this PR gets merged), but MSYS2 is definitely a hard requirement for a complete dcss build system.

re travis, we are still running travis jobs but have begun moving to github actions for most CI improvements, I think. There is a mac build via github actions. See https://github.com/crawl/crawl/tree/master/.github/workflows for the setup, though this is still evolving.

[rawlins]

This PR hasn't seen any changes in a couple years. I definitely think resurrecting the project is possible but I may soon close the PR as stale if it doesn't get picked up soon.