README-mamake.md

mamake and the MAM language

MAM (Make Abstract Machine) is a simple rule-based make language that is implemented in just seven four-letter commands and five attributes, yet allows unlimited flexibility as it can execute arbitrary shell code. The program implementing MAM, mamake, is a portable C90 program written in a single file, mamake.c. This allows ksh 93u+m, or other programs using this build system, to be built using only a standard C compiler and utilities installation without any other dependencies or complications.

MAM was originally designed by Glenn Fowler at AT&T and intended as an abstraction layer for make implementations such as AT&T nmake. The original documentation for MAM specified a more extensive and slightly different language than was actually implemented in mamake.c. Notably, the bind command described there is completely different. This file documents the MAM implementation that is actually in use.

Since fixing and maintaining AT&T nmake proved impractical, mamake is used here as a full make replacement, gradually adding some features to the language to facilitate human maintenance of the Mamfiles.

Table of contents

• General overview of the MAM language
  • Strict and legacy modes
• MAM variables
  • Special expansion syntax
  • Automatic variables
• Commands
  • Comments
  • Rules
  • Referencing prerequisites or previously defined rules
  • Setting MAM variables
  • Shell actions
    • Viewpathing
    • Execution
    • Strict level 2+ change
  • Declaring common code for shell actions
  • Binding libraries
    • …while scanning and sorting leaf directories
    • …while building the current directory
  • Repeatedly iterating through a block
• Appendix: Main changes from the AT&T version

General overview of the MAM language

MAM is a simple declarative language, easy to parse for machines and easy to read for humans, in which targets are defined that correspond to files that need to be generated or updated, or that are prerequisites. mamake reads build scripts from a file, Mamfile by default, from start to finish. As it encounters make target…done blocks, it ‘makes’ (updates) the targets using the instructions within those blocks. Those instructions may declare dependencies and/or specify shell command actions.

If a target is specified on the command line, mamake will update the actions contained within that target and ignore the rest.

Strict and legacy modes

By default, mamake remains fully backward compatible with Mamfiles as originally generated by AT&T nmake. If the MAMAKE_STRICT variable is set, some backward incompatible changes and deprecation warnings are activated to ensure correct operation and to facilitate human maintenance of the Mamfiles. A numeric value assigned to MAMAKE_STRICT indicates the backward incompatibility level. The empty value is equivalent to level 1. These are called the "strict levels" and their absence the "legacy mode", also known as strict level 0.

Each time backward incompatible changes are introduced to mamake that would break previous Mamfiles, those are made subject to a MAMAKE_STRICT value of one higher than the previous highest one; details of those changes are documented throughout this file and listed in the appendix below. This makes it possible to test or backport old code using the current build system. Current Mamfiles should use the highest strict level available. The current highest available strict level is 3.

MAM variables

MAM variables are imported from the environment or set via setv (see below). They are referenced using a ${...} syntax not dissimilar to sh(1), though the braces are not optional. If an undefined variable is expanded and the variable name is valid in sh(1) syntax, the expansion is left in place unexpanded, otherwise it is removed. At strict level 2 and up, it is left unexpanded even if it is not a valid sh(1) variable name; this allows POSIX shell expansions like ${foo#*bar}.

By default, the expansion of MAM variable references is recursive, i.e., the value may itself contain other variable references. Beware: there is no reference loop detection; any variable referencing itself directly or indirectly will cause mamake to crash. At strict level 2 and up, this (mis)feature is disabled and variables always expand to their literal values, and variable references in setv only work for previously defined variables.

Note that, in shell actions (see exec below), MAM variables are expanded before the script ever reaches the shell. Consequently, the use of single shell quotes '…' does not stop their expansion as you might expect; in fact, they ensure that only MAM variable expansion happens, avoiding any potential conflicts with the shell expansion syntax.

Special expansion syntax

In ${variable?str?x?y?}, if the string value of the variable is identical to str or if c is *, then the value x is substituted, otherwise y. The x and y values may result from nested variable references. The last ? is optional.

In ${variable-x}, the value of variable is substituted if it is defined and non-empty, otherwise the value of x is substituted. In ${variable+x}, x is substituted if the value of variable is defined and non-empty, otherwise the reference is removed. Note that, unlike in sh(1), no distinction is made between an undefined variable and a defined variable with an empty value.

Automatic variables

The following variables are set and updated automatically. They are inspired by similar variables in make implementations, but since mamake is different, so are these variables.

${@} is the name of the rule currently being made.

${<} is the name of the prerequisite rule (make…done or prev) that was last processed within the current rule.

${^} is a space-separated list of names of all the current rule's previously processed prerequisites.

${?} is a space-separate list of the current rule's previously processed prerequisites that have been updated by a shell action (see exec below) during the current mamake run. Prequisites that were already up to date, or prerequisites that do not contain a shell action, are not included.

Commands

MAM commands have the following basic form:

command [ argument [ operand string ] ]

The command name consists of four lower-case letters. Unrecognized commands or attributes are an error. The argument is a single word. The operand string is any arbitrary text until the end of the line.

Comments

note is the comment command and is ignored. In the legacy mode, info and meta are also ignored.

Rules

make target [ attribute ... ]
done [ target ]

A make...done block defines the rule named target using the other commands described here. Unless the virtual attribute is used, target names the pathname of the file generated or referenced by the rule.

mamake processes the commands within the block if the target is out of date or if the rule has the virtual attribute (see below).

The target may be repeated as the operand to the done command. In that case, it is matched against the current make target and any mismatch will produce a "mismatched done statement" error. If it is omitted, the current make target is assumed.

Dependencies may be defined in two ways:

1. By nesting make...done blocks: the enclosing rule is the parent and the enclosed rules are the prerequisites.
2. By using the prev command (see Referencing previously defined rules below) to reference a previous make...done block. The dependency is defined as if that block were repeated at the prev command's location.

If the block contains one or more exec commands (see Shell actions below), the done command executes the shell script defined by them.

A make...done block may lack any exec action and, if it does not have any dependencies of its own, it may even be empty; this has the effect of merely declaring a dependency on a prerequisite file, such as a source code file that comes with the distribution or a file generated by a previously run Mamfile.

Making a prerequisite that is currently being made, or one that has already been made, produces a warning; at strict level 3 and up, this is an error.

One or more attributes may be specified by appending them to the make command. (At strict levels < 2, they may also be appended to the done command; the effect is the same either way. At strict level 1, this produces a deprecation warning.) Attributes apply to the current rule only and do not propagate down to nested rules. The following attributes are available:

• dontcare: Marks files that do not need to exist. If the file exists then its last-modified timestamp is checked and propagated, otherwise it is silently ignored.
• ignore: The timestamp associated with the target is ignored in dependency resolution.
• implicit: Marks the current rule as an implicit prerequisite of the enclosing parent rule. An implicit prerequisite can make the parent rule out of date without triggering the parent action. Implicit prerequisites usually correspond to #include prerequisites. For example, if foo.o is generated from foo.c and foo.c includes foo.h, then foo.h should be marked as an implicit prerequisite of foo.c so that touching foo.h does not make foo.c out of date while making foo.o out of date.
• notrace: Disables echoing (xtrace) of shell action commands. This does not disable the trace header for the containing rule (see Shell actions below).
• virtual: Marks a rule that is not associated with any file. The commands within are executed every time the rule is processed. By convention, a virtual rule with target install performs pre-installation.

At strict level 1 and up, specifying the following attributes is deprecated and will produce a warning; at strict level 2 and up, specifying these is an error.

• archive: Ignored. Historically used to mark the generation of an ar(1) archive.
• generated: Marks rules that produce output files generated by a shell action. The explicit assignment of this attribute is ignored at strict level 1. The exec command implicitly assigns this attribute. If a rule has this attribute, other rules dependent on this rule will avoid applying viewpathing based on this rule.
• joint: Ignored. Historically used to mark one of a group of rules that are built by a single shell action.

Referencing prerequisites or previously defined rules

prev target [ attribute ... ]

The prev command is used in two ways:

1. If target matches a previously defined rule, prev adds a dependency on that rule to the current rule. This can be used to make a rule a prerequisite of multiple make...done blocks without repeating the rule. No attributes should be given for this use of prev, because the attributes of the referenced rule are used. Superfluous attributes are an error at strict level >= 1 and ignored in the legacy mode.

2. If target does not match a previously defined rule, the following applies. In the legacy mode, prev creates an empty dummy rule and ignores the attributes; this is for backward compatibility. At strict level 1 and up, prev creates a rule that declares a dependency on a prerequisite file named by target in a manner equivalent to an empty make...done block, with the optional attributes applied to the new rule, and a nonexistent prerequisite is an error unless a virtual or dontcare attribute is given. Declaring a dependency on a prerequisite that is currently being made (i.e.: directly or indirectly within that prerequisite's block) produces a warning; at strict level 3 and up, this is an error.

Setting MAM variables

setv variable [ defaultvalue ]

Defines a new MAM variable, optionally assigning the initial defaultvalue. If the variable already has a value, the setv command is ignored; assigning a new value is not possible. When mamake starts, it imports all environment variables as MAM variables, so any variable's default value can be overridden by exporting an environment variable by its name.

If the strict level is less than 2 and the defaultvalue begins and ends with double quotes ("), those quotes are discarded, though double quotes elsewhere in the value are not treated specially. The value is otherwise taken entirely literally.

When the variable is CC, mamake runs the mamprobe script to probe the C compiler for flags and features, or uses that script's stored results if not outdated. The results are stored as a series of setv commands in a file in the directory ${INSTALLROOT}/lib/probe/C/mam, the file name being a hash of full path to the compiler indicated by ${CC}. That results file is then read and included in the current Mamfile as if it followed the setv CC command.

Shell actions

exec - code

One or more exec commands within a make...done block define a shell script that is run to generate the target. The argument following exec is ignored; by convention it is -. Each exec command appends a line of code to the shell script for the current rule. It is customary for a rule's exec commands to be contiguous, but not necessary.

Before adding each line of code to the script, MAM variable references (see MAM variables above) are expanded; their literal values are inserted into the code line (beware: no quoting is applied!). Because variables are expanded when the line is encountered, the value of the automatic variables for any exec line depends on the position of the line in the rule.

Viewpathing

After MAM variable expansion, viewpathing is applied. The first colon-separated element of ${VPATH} is considered the object code directory and the second the source code directory; viewpathing provides the first with a vew to the second. Viewpathing applies two transformations.

The first is prerequisite replacement. Each word (separated by whitespace, ;, (, ), `, |, & or =) is searched for in the current rule's prerequisites, and if it matches the name of a non-generated prerequisite, it is replaced by the canonical path to it in the source directory, ensuring that things like prerequisite headers are found.

The second is include flag duplication. After every argument that looks like a compiler include directory path (i.e., starting with -I) with a relative path name (i.e., a directory path that does not start with a /), another argument starting with -I is inserted with that path name prefixed by the path to the source directory. This mechanism ensures that headers are found both in the object directory and in the source directory. It is processed regardless of the command; for example, it also works for compiler flags passed to iffe(1).

Note that shell quotes are not treated specially. If an argument starting with -I ends in a shell quote without preceding whitespace, that trailing quote is repeated along with the prefixed path and causes a syntax error.

Execution

When mamake encounters the done command, the script is executed by the shell whose path is in the SHELL environment variable or, absent that, by sh(1). Each shell action is run in a new instance of the shell.

Before executing the script, an empty line followed by a trace header in the following format is written to standard error:

# path/to/Mamfile: startline-endline: rule

During script execution, shell action comands are traced using the shell's xtrace option, unless the rule has the notrace attribute.

Strict level 2+ change

At strict level 2 and up, mamake turns off global pathname expansion (globbing) using set -f to make safer MAM variable expansion and shell field splitting possible; this avoids unexpected pathname expansion if a value contains ?, * or [. A shell action can override this using set +f; this should only be done for individual commands in a (subshell).

Declaring common code for shell actions

shim - code

One or more shim commands declare a ‘shim’: a common section of sh(1) code that will be automatically inserted in front of subsequent shell actions upon execution. Like exec, shim combines multiple lines of code into one section, with MAM variables expanded at declaration time and viewpathing applied at execution time. The effect of shim is global.

One use case is defining a shell function that each shell action can call. For example, such a function might invoke the compiler with a series of compiler flags common to all compiler invocations, with other flags added via arguments to the function, so the common flags do not need to be repeated in every shell action.

Only one shim is active at a time, but it can be redefined. When the next exec command is encountered, the shim is marked ready for use. The next time a shim command is encountered after that, it starts a new shim from scratch that affects subsequently executed shell actions. A single shim - deactivates the shim.

Binding libraries

bind -llibraryname [ dontcare ]

These commands are scanned for while sorting leaf directories for recursive building, and executed as normal commands while building the current directory.

…while scanning and sorting leaf directories

Any leaf directories with names that start with INIT will always be built before all others. For all other leaf directories, the presence of any bind command of the form bind -lfoo anywhere in a leaf directory's Mamfile causes the leaf directory named libfoo (if it exists) to be a prerequisite of that leaf directory. The prerequisite leaf directory does not have to be in the same parent directory, as long as it is processed as part of the same scan. At this stage, attributes are ignored.

…while building the current directory

An argument of -llibraryname causes a MAM variable mam_liblibraryname to be defined (see MAM variables above). The variable will contain either the compiler argument for linking to the library libraryname (either the -llibraryname flag, or the full path in case of a static library) or, if the dontcare attribute is specified, possibly the empty string. Any library dependencies are also included (see below). This can be used both for AST libraries shipped with the distribution and for system libraries. For each corresponding *.a library archive dependency built previously, its time stamp is checked and the current target is marked as outdated if it is newer, as if a prev had been executed for it.

The variable set by bind is global, but the marking of the target as outdated applies to the current rule only, so it may be necessary to repeat a bind command when statically linking executables that depend on a library, otherwise they may not be relinked when the library changes. The mam_liblibraryname variable will not be regenerated when repeating a bind.

There is also a mechanism to communicate library dependency information across Mamfiles and mamake invocations. If a file named libraryname.req in the current directory or an ${INSTALLROOT}/lib/lib/libraryname file exists, mamake processes each of the words in the form -llibraryname in its contents as if they were arguments to bind commands and the resulting values are appended to the value of mam_liblibraryname as dependencies separated by spaces. mamake does not create these dependency files; they are expected to be generated by Mamfile shell actions (see Shell actions above).

If no such dependency file exists, and the dontcare attribute is added, then mamake compiles a small test program on the fly to check if the library exists; if this fails, the mam_liblibraryname variable will be emptied.

Any bind command whose argument does not start with -l is ignored.

Repeatedly iterating through a block

loop variable word [ word ... ]
done

loop reads the lines contained between it and the corresponding done repeatedly with a named variable set to each of the words. The lines are processed as part of the rule containing the loop. The variable is restored to its previous state after the loop completes.

Note that loop causes repeated reading and processing of Mamfile lines, not necessarily repeated execution. For instance, a loop can be used to consolidate repetitive make…done rules. However, each rule is only made once and subsequent rules by the same name are an error at strict level 3 and up, or skipped over at strict < 3. So it only makes sense to do this if the contained make target names are modified by the expansion of the iteration variable.

loop requires a seekable input file (i.e.: not a pipe).

Appendix: Main changes from the AT&T version

Compared to the original AT&T version, ksh 93u+m made a number of changes to mamake that facilitate correct operation and make it easier to maintain Mamfiles by hand. The following lists the important changes.

• Introduced the notion of ‘strict mode’ levels that tidy things up by activating some backward incompatible changes and deprecation warnings.
• Indentation and word separators may use any whitespace (e.g. tabs), not only spaces.
• Fixed a bug that stopped a rule marked virtual (not associated with any file) from being executed if a file by that rule's name exists.
• Unrecognized commands and rule attributes throw an error instead of being silently ignored.
• It has been made optional to repeat the make target after done.
• The notrace attribute was added to disable xtrace for a rule's shell action.
• The automatic variables ${@}, ${<}, ${^} and ${?} have been added.
• An iteration block command, loop…done, has been added.
• A command to set common code for shell actions, shim, has been added.
• Attempting to make a rule that has already been made produces a warning.
• Attempting to declare a dependency on a rule currently being made produces a warning.
• At strict level 1 and up:
  • Appending attributes to done instead of make is deprecated and produces a warning.
  • The ignored archive and joint attributes are deprecated.
  • Explicitly specifying the generated attribute is deprecated.
  • The dummy info and meta commands are unavailable instead of ignored.
  • The prev command may be used instead of an empty make...done block to declare a simple prerequisite with possible attributes.
  • When prev references a previously processed target, attributes are an error instead of being ignored.
  • The legacy silent and ignore command prefixes are unavailable.
• At strict level 2 and up:
  • Appending attributes to done instead of make is an error.
  • The archive and joint attributes are unavailable.
  • Explicitly specifying the generated attribute is an error.
  • All variable references are expanded to their literal values without scanning the values for recursive variable references.
  • setv does not remove leading and trailing " from the value.
  • Shell actions have pathname expansion (globbing) disabled by default.
• At strict level 3 and up:
  • Attempting to make a rule that has already been made is an error.
  • Attempting to declare a dependency on a rule currently being made is an error.