Skip to content
Newer
Older
100644 471 lines (358 sloc) 19.2 KB
e97a251 @deplinenoise Added more documentation. Switched to spaces rather than tabs.
authored Aug 5, 2010
1 // vim: ts=4 expandtab
a783af6 @deplinenoise Added start of a manual.
authored Aug 1, 2010
2
d96a2c8 @deplinenoise Added section on how to build/install tundra.
authored Aug 19, 2010
3 The Tundra Build System
4 =======================
a783af6 @deplinenoise Added start of a manual.
authored Aug 1, 2010
5 Andreas Fredriksson <dep@defmacro.se>
6
7 == Introduction ==
8
9 Tundra is a high-performance code build system designed to give the best
10 possible incremental build times even for very large software projects.
11
12 Its design was motivated by the games industry where projects are huge and
13 iterative rebuilding common. In games, teams of 20-30 developers working on the
14 same multi-MLOC codebase is not uncommon. Each second wasted by a build system
15 to produce a incremental build wastes productivity. But Tundra works well for
16 smaller projects too.
17
18 === Design Philosphy ===
19
20 ==== Simple and Fast ====
21
22 Tundra is written in ANSI C (C89), with very few files (the whole codebase
d96a2c8 @deplinenoise Added section on how to build/install tundra.
authored Aug 19, 2010
23 including scripts and tools is about 10000 lines of code). Aside from Lua,
24 there are no external dependencies. Most data structures in the C code are
25 simple arrays, either stack based or allocated from a linear allocator. All
26 allocations are discarded at once in pages when the program exits. This makes
27 the code simple and the program fast as very little time is spent worrying
28 about dynamic memory.
a783af6 @deplinenoise Added start of a manual.
authored Aug 1, 2010
29
30 ==== Utilize multi-core hardware ====
31
32 All serious developers are using multi-core machines, so it is important that
33 the build system can run multiple jobs at once. But it is equally important to
34 utilize these cores to do other things like file signing and implicit
35 dependency scanning. Tundra is one of very few (if any) build systems that do
36 this whenever possible. This design choice give a nice speedup to incremental
37 builds.
38
430a474 @deplinenoise Added section about codegen.
authored Aug 1, 2010
39 ==== Reliably support code generation ====
40
41 Most build systems work perfectly until you start generating source files
42 dynamically in the build process. If these files are then included into other
43 files many build systems start misbehaving, especially if multiple jobs are run
44 concurrently. This is very hard to diagnose.
45
46 Tundra does away with this by adding the possibility of segmenting the build
47 into multiple, sequenced passes. Build jobs are only allowed to implicitly
48 depend on (that is, +#include+) files generated in previous passes. This keeps
49 file generation entirely predictable and it becomes easy to support a
50 multi-layered system where the code generation tool itself is built in a first
51 pass, it is run to generate output in a second pass and then the full build
52 commences. Tundra will still go wide within each pass, but each pass is fully
53 synchronous with regards to the others so using the +-j+ option is fully
54 supported.
55
a783af6 @deplinenoise Added start of a manual.
authored Aug 1, 2010
56 ==== Separate configuration and building ====
57
58 Tundra uses Lua as its scripting language, which gives it a high-performance,
59 powerful data collection frontend for free. However, to get the maximum speed
60 from builds, all build actions are carried out in multi-threaded C code with
61 carefully crafted memory allocation patterns. This ensures that pure build
62 speed is unaffected by bad scripting and it is easier to diagnose a slow build.
63
64 ==== Don't guess ====
65
d96a2c8 @deplinenoise Added section on how to build/install tundra.
authored Aug 19, 2010
66 Tundra doesn't go out of its way to support auto-configuration of toolsets or
67 guessing what compiler you want to use. For a game project, there may be
68 millions of builds between adding a new platform or configuration to a build
69 system. Tools are also very brittle and need a specific configuration to
70 produce working builds. The time it will take to support a new toolset while
71 offset the time it takes to tell the build system about it by many magnitudes.
a783af6 @deplinenoise Added start of a manual.
authored Aug 1, 2010
72
73 Tundra therefore must be told explicitly in configuration what tools to use and
74 how to run them, it never tries to auto-configure itself for your platform. If
d96a2c8 @deplinenoise Added section on how to build/install tundra.
authored Aug 19, 2010
75 you need this capability, Tundra is probably not for you.
a783af6 @deplinenoise Added start of a manual.
authored Aug 1, 2010
76
77 == Hello, world ==
78
79 A Tundra project needs atleast two files, one is +tundra.lua+ which specifies
80 what build configurations are available for this project. Here is a sample
81 minimal +tundra.lua+ file that creates a configuration +macosx-gcc+ which pulls
82 in the default +gcc+ and +mono+ toolsets:
83
84 -------------------------------------------------------------------------------
85 Build {
86 Units = "units.lua",
87 Configs = {
88 {
89 Name = "macosx-gcc",
90 DefaultOnHost = "macosx",
91 Tools = { "gcc", "mono" },
92 },
93 },
94 }
95 -------------------------------------------------------------------------------
96
97 As you can see, it refers to another file, +units.lua+. The +Units+ file is a
98 declarative input file describing the targets that are available:
99
100 -------------------------------------------------------------------------------
101 Program {
e97a251 @deplinenoise Added more documentation. Switched to spaces rather than tabs.
authored Aug 5, 2010
102 Name = "HelloWorld",
103 Sources = { "hello.c" },
a783af6 @deplinenoise Added start of a manual.
authored Aug 1, 2010
104 }
105 Default "HelloWorld"
106 -------------------------------------------------------------------------------
107
108 If we now run Tundra on this input, it will build our +HelloWorld+ executable:
109
110 -------------------------------------------------------------------------------
111 $ tundra
112 Cc tundra-output/macosx-gcc-debug-default/hello.o
113 Program tundra-output/macosx-gcc-debug-default/HelloWorld
114 *** build success, 2 jobs run
115 -------------------------------------------------------------------------------
116
117 The next mandatory step is of course to display the famous greeting:
118
119 -------------------------------------------------------------------------------
120 $ tundra-output/macosx-gcc-debug-default/HelloWorld
121 hello, world
122 -------------------------------------------------------------------------------
123
d96a2c8 @deplinenoise Added section on how to build/install tundra.
authored Aug 19, 2010
124 More examples can be found in the `examples` directory in the Tundra
125 distribution.
126
127 == Installation ==
128
129 === Installing a binary package ===
130
131 Download a binary package from http://github.com/deplinenoise/tundra/downloads
132
133 Unpack it somewhere, and optionally stick it in your `PATH` environment
134 variable if you use Tundra a lot.
135
136 === Installing from source ===
137
138 Tundra can be obtained from the official GIT repository:
139 `git://git@github.com:deplinenoise/tundra.git`.
140
141 To build tundra, you can either use tundra itself, or CMake.
142
143 To build with tundra, first install a binary package and then run:
144 -------------------------------------------------------------------------------
145 $ tundra release
146 $ cp tundra-output/macosx-clang-release-default/tundra .
147 -------------------------------------------------------------------------------
148
149 Obviously your output directory must be adjusted to whatever platform you are
150 actually building on. This example used the `macosx-clang` toolset.
151
152 To build with CMake:
153 -------------------------------------------------------------------------------
154 $ mkdir build-dir
155 $ cd build-dir
156 $ cmake ..
157 $ make
158 $ cp tundra ..
159 -------------------------------------------------------------------------------
160
161 The last line of both examples copies the resulting binary into the tundra root
162 dir (where it should live to be executed). If you prefer to run the executable
163 from the build directory (because you're hacking on some feature) you can set
164 the `TUNDRA_HOME` environment variable to the root directory of the tundra
165 source distribution instead.
166
52cb760 @deplinenoise Added high-level overview.
authored Aug 6, 2010
167 == How Tundra works ==
168
169 A Tundra build can divided into a few distinct phases:
170
171 - Run the project's +tundra.lua+ script to set options
172 - Load toolsets, syntax files and other information as required by the configuration script
173 - Run the referred +Units+ file in syntax mode to define the project's build units
174 - Parse the resulting declarations and generate DAG nodes
175 - Pass the DAG to the native build engine
176
e97a251 @deplinenoise Added more documentation. Switched to spaces rather than tabs.
authored Aug 5, 2010
177 == The +tundra.lua+ file ==
178
179 The file +tundra.lua+ is read by Tundra when you invoke it. This is a regular
180 Lua source file. Its purpose is to call the global +Build+ function with a
181 declarative input describing the build session to Tundra. The following
182 sections are a reference of what you can place in the +Build+ block.
183 Declarations within the block can appear in any order.
184
185 === Units (required) ===
186
187 The build block must contain a reference to a secondary file containing unit
188 declarations. This file is separate because it uses a custom, extensible syntax
189 set which is suitable to define build system input. The +Units+ option must be
190 a string designating the filename to read. A common name is +"units.lua"+, but
191 any valid filename is OK.
192
193 === Configs (required) ===
194
195 The +Configs+ key should be set to an array of configurations this build
196 supports. Each configuration is in turn a table with the following syntax:
197
198 -------------------------------------------------------------------------------
199 {
200 Name = "myplatform-mytoolset",
201 Tools = { ... },
202 DefaultOnHost = "..." ,
203 Inherit = ...,
204 Env = { ... },
205 }
206 -------------------------------------------------------------------------------
207
31903eb @deplinenoise Documented environment vars, toolsets.
authored Aug 19, 2010
208 ==== Per-config Name property (required) ====
e97a251 @deplinenoise Added more documentation. Switched to spaces rather than tabs.
authored Aug 5, 2010
209
210 The name of this configuration. Configuration names must be formatted in a
211 dashed `platform-toolset` format. These two tokens form the first two in the
212 quad `platform-toolset-variant-subvariant` system Tundra uses to id builds.
213
31903eb @deplinenoise Documented environment vars, toolsets.
authored Aug 19, 2010
214 ==== Per-config Tools property (required) ====
e97a251 @deplinenoise Added more documentation. Switched to spaces rather than tabs.
authored Aug 5, 2010
215
216 A list of tools this configuration uses. A tool specification is either a
217 string, indicating that the defaults for that tool are to be used, or a table
218 `{ "toolname"; Foo=1, Bar=".." }` passing arbitrary options to the tool to
219 configure it. Tools are loaded from the tool directory list.
220
221 Projects can add their own tool scripts via a +ToolsetDirs+ array property in
222 the +Build+ block.
223
31903eb @deplinenoise Documented environment vars, toolsets.
authored Aug 19, 2010
224 ==== Per-config DefaultOnHost property (optional) ====
e97a251 @deplinenoise Added more documentation. Switched to spaces rather than tabs.
authored Aug 5, 2010
225
226 If present, this config will be built by default when the host platform matches
227 the string. This is convenient to have the host's native configuration build
228 in the default variant when you just type `tundra` in the shell.
229
31903eb @deplinenoise Documented environment vars, toolsets.
authored Aug 19, 2010
230 ==== Per-config Env property (optional) ====
e97a251 @deplinenoise Added more documentation. Switched to spaces rather than tabs.
authored Aug 5, 2010
231
31903eb @deplinenoise Documented environment vars, toolsets.
authored Aug 19, 2010
232 If present, must be set to a table. This table contains key-value bindings to
233 apply to the environment for this configuration. This typically includes things
234 such as include paths (`CPPPATH`), C preprocessor defines (`CPPDEFS`) and C
235 compiler options (`CCOPTS`).
e97a251 @deplinenoise Added more documentation. Switched to spaces rather than tabs.
authored Aug 5, 2010
236
237 -----------------------------------------------------------------------------
238 Build {
239 Configs = {
31903eb @deplinenoise Documented environment vars, toolsets.
authored Aug 19, 2010
240 { Name = "foo-bar", Env = { CPPDEFS = { "FOO", "BAR=BAZ", ... } }, },
e97a251 @deplinenoise Added more documentation. Switched to spaces rather than tabs.
authored Aug 5, 2010
241 ...
242 },
243 }
244 -----------------------------------------------------------------------------
245
31903eb @deplinenoise Documented environment vars, toolsets.
authored Aug 19, 2010
246 ==== Per-config Inherit property (optional) ====
e97a251 @deplinenoise Added more documentation. Switched to spaces rather than tabs.
authored Aug 5, 2010
247
31903eb @deplinenoise Documented environment vars, toolsets.
authored Aug 19, 2010
248 If present, must be set to a table. This table will be scanned for values if
249 they are not present in the Config table itself. This is useful to group common
250 settings between configs in external tables:
e97a251 @deplinenoise Added more documentation. Switched to spaces rather than tabs.
authored Aug 5, 2010
251
252 -----------------------------------------------------------------------------
31903eb @deplinenoise Documented environment vars, toolsets.
authored Aug 19, 2010
253 local foo_common = { Env = { CPPDEFS = { "Bar", "Baz" } } }
254
e97a251 @deplinenoise Added more documentation. Switched to spaces rather than tabs.
authored Aug 5, 2010
255 Build {
256 Configs = {
31903eb @deplinenoise Documented environment vars, toolsets.
authored Aug 19, 2010
257 { Name = "foo-bar", Inherit = foo_common, ... },
258 { Name = "foo-baz", Inherit = foo_common, ... },
e97a251 @deplinenoise Added more documentation. Switched to spaces rather than tabs.
authored Aug 5, 2010
259 ...
260 },
261 }
262 -----------------------------------------------------------------------------
263
264 === Passes (optional) ===
265
266 The build block can contain an array of passes which can be used to place
267 barriers between groups of build jobs. This is required if files are generated
268 that can be discovered only as implicit dependencies.
269
270 Here's an example of how +Passes+ can be used within the +Build+ block:
271 -------------------------------------------------------------------------------
272 Passes = {
273 Foo = { Name="User friendly name", BuildOrder = 1 },
274 Bar = { Name="Some other name", BuildOrder = 2 },
275 -- ...
276 },
277 -------------------------------------------------------------------------------
278
d8ed5aa @deplinenoise Described the environment.
authored Aug 1, 2010
279 == The Environment ==
280
281 Tundra uses a hierarchical key-value environment to store information used to
282 build the commands to run. This design is similar to the SCons environment.
283 Values are always stored as lists (in this way the environment is similar to
284 Jam variables).
285
286 === The basic environment ===
287
288 With no tools or platform settings loaded, the following keys are always available:
289
290 - +OBJECTROOT+ - specifies the directory in which variant-specific build
291 directories will be created (default: +tundra-output+)
292
293 - +SEP+ - The path separator used on the host platform
294
295 === Interpolation ===
296
297 Basic interpolation is written +$(FOO)+ and just fetches the value associated
298 with +FOO+ from the environment structure. If +FOO+ is bound to multiple
299 values, they are joined together with spaces.
300
301 === Interpolation Options ===
302
303 Tundra includes a number of interpolation shortcuts to build strings from the
304 environment. For example, to construct a list of include paths
305 from a environment variable +CPPPATH+, you can say +$(CPPPATH:p-I)+.
306
307 .Interpolation Syntax
308 [width="90%",cols="1,9",options="header"]
309 |=============================================================================
e97a251 @deplinenoise Added more documentation. Switched to spaces rather than tabs.
authored Aug 5, 2010
310 |Syntax |Effect
311 |+$(VAR:f)+ |Convert to forward slashes (+/+)
312 |+$(VAR:b)+ |Convert to backward slashes (+\+)
313 |+$(VAR:u)+ |Convert to upper case
314 |+$(VAR:l)+ |Convert to lower case
315 |+$(VAR:p<prefix>)+ |Prefix all values with the string +<prefix>+
316 |+$(VAR:s<suffix>)+ |Suffix all values with the string +<suffix>+
317 |+$(VAR:[<index>])+ |Select the item at the (one-based) +index+
318 |+$(VAR:j<sep>)+ |Join all values with +<sep>+ as a separator rather than space
319 |+$(VAR:A<suffix>)+ |Suffix all values with +<suffix>+ unless it is already there
320 |+$(VAR:P<prefix>)+ |Prefix all values with +<prefix>+ unless it is already there
d8ed5aa @deplinenoise Described the environment.
authored Aug 1, 2010
321 |=============================================================================
322
323 These interpolation options can be combined arbitrarily by tacking on several
324 options. If an option parameter contains a colon the colon must be escaped with
325 a backslash or it will be taken as the start of the next interpolation option.
326
327 === Interpolation Examples ===
328
329 Assume there is an environment with the following bindings:
330
331 [width="90%",cols="1,9"]
332 |=============================================================================
e97a251 @deplinenoise Added more documentation. Switched to spaces rather than tabs.
authored Aug 5, 2010
333 |+FOO+ |+"String"+
334 |+BAR+ |+{ "A", "B", "C" }+
d8ed5aa @deplinenoise Described the environment.
authored Aug 1, 2010
335 |=============================================================================
336
337 Then interpolating the following strings will give the associated result:
338
339 [width="90%",cols="1,9",options="header"]
340 |=============================================================================
e97a251 @deplinenoise Added more documentation. Switched to spaces rather than tabs.
authored Aug 5, 2010
341 |Expression |Resulting String
342 |`$(FOO)` |`String`
343 |`$(FOO:u)` |`STRING`
344 |`$(FOO:l)` |`string`
345 |`$(FOO:p__)` |`__String`
346 |`$(FOO:p__:s__)` |`__String__`
347 |`$(BAR)` |`A B C`
348 |`$(BAR:u)` |`A B C`
349 |`$(BAR:l)` |`a b c`
350 |`$(BAR:p__)` |`__A __B __C`
351 |`$(BAR:p__:s__:j!)` |`__A__!__B__!__C__`
352 |`$(BAR:p\::s!)` |`:A! :B! :C!`
353 |`$(BAR:AC)` |`AC BC C`
d8ed5aa @deplinenoise Described the environment.
authored Aug 1, 2010
354 |=============================================================================
355
356 === Nested Interpolation ===
357
358 Nested interpolation is possible, but should be used with care as it can be
359 hard to debug and understand. Here's an example of how the generic C toolchain
360 inserts compiler options dependening on what variant is currently active:
361
362 `$(CCOPTS_$(CURRENT_VARIANT:u))`
363
364 This works becase the inner expansion will evalate `CURRENT_VARIANT` first
365 (say, it has the value +debug+). That value is then converted to upper-case and
366 spliced into the former which yields a new expression `$(CCOPTS_DEBUG)` which
367 is then expanded in turn.
368
369 Used with care this is a powerful way of letting users customize variables per
370 configuration and then glue everything together with a simple template.
371
31903eb @deplinenoise Documented environment vars, toolsets.
authored Aug 19, 2010
372 == Environment Variables ==
373
374 These environment variables apply to C-based toolsets:
375
376 - `CPPPATH` - A list of search directories for include files
377 - `CPPDEFS` - A list of preprocessor definitions
378 - `LIBS` - A list of libraries to link with
379 - `LIBPATH` - A list of search directories for library files
380 - `CC` - The C compiler
381 - `C++` - The C++ compiler
382 - `LIB` - The program that makes static libraries (archives)
383 - `LD` - The linker
384 - `CCOPTS` - Common compiler options for all configurations
385 - `CCOPTS_<config>` - Compiler options for variant `<config>`, such as `CCOPTS_DEBUG`, `CCOPTS_RELEASE`.
386 - `CCCOM` - Command line for C compilation
387 - `CXXCOM` - Command line for C++ compilation
388 - `PCHCOMPILE` - Command line for precompiled header compilation
389 - `PROGOPTS` - Options specific to linking programs
390 - `PROGCOM` - Command line to link a program
391 - `LIBOPTS` - Options specific to creating a static library (archive)
392 - `LIBCOM` - Command line to create a static library (archive)
393 - `SHLIBOPTS` - Options specific to creating a shared library
394 - `SHLIBCOM` - Command line to create a shared library
395 - `FRAMEWORKS` - (OS X) Frameworks to include and link with
396 - `AUX_FILES_PROGRAM`, `AUX_FILES_SHAREDLIB` - List of patterns that expand to auxilliary files to clean for programs, shared libraries. Useful to clean up debug and map files.
397
398 These environment variables apply to .NET-based toolsets:
399
400 - `CSC` - The C# compiler
401 - `CSC_WARNING_LEVEL` - The C# warning level
402 - `CSLIBS` - Assembly references
403 - `CSRESOURCES` - Resource file references
404 - `CSCOPTS` - Common options
405 - `CSPROGSUFFIX` - The suffix of generated programs, by default `.exe`
406 - `CSLIBSUFFIX` - The suffix of generated libraries, by default`.dll`
407 - `CSRESGEN` - The resource compiler
408 - `CSCLIBCOM` - Command line to generate a library
409 - `CSCEXECOM` - Command line to generate an executable
410
411 == Toolsets ==
412
413 This section tries to document the stock toolsets that come included with Tundra.
414
415 === generic-cpp ===
416
417 This isn't really a toolset you would import explicity, it is a base layer the
418 other tools drag in to set up defaults. It has functionality to set up
419 preprocessor scanners, registers functions to implicitly compile source files
420 to object files and such. All other C toolsets import this toolset.
421
422 === gcc ===
423
424 The `gcc` toolset is a simple GCC toolset that only uses basic options and does
425 nothing fancy. It is suitable for run-of-the-mill UNIX clones such as Linux,
426 BSD but also works well for command-line programs on Mac OS X.
427
428 It formats include paths with `-I`, preprocessor defines with `-D` and so on.
429 It tries to run `ar` to create static libraries and there is no support for
430 dynamic libraries.
431
432 === gcc-osx and clang-osx ===
433
434 `gcc-osx` extends the `gcc` toolset by adding Mac OS X specific options for
435 frameworks and shared libraries (dylib). `clang-osx` is just like `gcc-osx` but
436 uses the CLang frontend rather than GCC.
437
438 === msvc ===
439
440 This toolset uses a `cl.exe` from the environment. It is suitable for direct
441 use if you want to run with a local MSVC compiler that is already in your path.
442
443 === msvc-vs2008 ===
444
445 This toolset imports the `msvc` toolset but can locate and set up the Visual
446 Studio 2008 compiler from the registry and explicitly select between 32 and
447 64-bit versions of the compilers. This gives two advantages:
448
449 - You can just run tundra without setting up the environment with a compiler
450 (e.g. through the "Visual Studio Command Prompt" shortcut)
451 - You can build for multiple target architectures at the same time, for example
452 build both x86 and x64 code in batch.
453
454 This toolset supports two options:
455
456 - `HostArch`: one of `x86`, `x64` or `itanium`; selects the host architecture
457 of the compiler binaries. Defaults to x64 on 64-bit machines, x86 on 32-bit
458 machines.
459 - `TargetArch`: one of `x86`, `x64` or `itanium`; selects the target architecture
460 of the compiler binaries. Defaults to `x86`.
461
462 Here's an example of how this toolset can be configured for an explicit target
463 architecture:
464
465 -------------------------------------------------------------------------------
466 Tools = { { "msvc-vs2008"; TargetArch = "itanium", HostArch = "x86" }, ... }
467 -------------------------------------------------------------------------------
468
a783af6 @deplinenoise Added start of a manual.
authored Aug 1, 2010
469 // vim: set syntax=asciidoc
470
Something went wrong with that request. Please try again.