Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Browse files

reformat to fixed lines; minor updates

  • Loading branch information...
commit 4ec5a8ebdbb9576ca9aa16cc6645f4c7cf2a7705 1 parent 5ddbca4
@stevedonovan authored
Showing with 362 additions and 88 deletions.
  1. +362 −88 readme.md
View
450 readme.md
@@ -1,14 +1,50 @@
## LuaMacro - a macro preprocessor for Lua
-This is a library and driver script for preprocessing and evaluating Lua code. Lexical macros can be defined, which may be simple C-preprocessor style macros or macros that change their expansion depending on the context.
-
-It is a new, rewritten version of the [Luaforge](http://luaforge.net/projects/luamacro/) project of the same name, which required the [token filter patch](http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/#tokenf) by Luiz Henrique de Figueiredo. This patch allowed Lua scripts to filter the raw token stream before the compiler stage. Within the limits imposed by the lexical filter approach this worked pretty well. However, the token filter patch is unlikely to ever become part of mainline Lua, either in its original or [revised](http://lua-users.org/lists/lua-l/2010-02/msg00325.html) form. So the most portable option becomes precompilation, but Lua bytecode is not designed to be platform-independent and in any case changes faster than the surface syntax of the language. So using LuaMacro with LuaJIT would have required re-applying the patch, and remaining within the ghetto of specialized, experimental use.
-
-This implementation uses a [LPeg](http://www.inf.puc-rio.br/~roberto/lpeg.html) lexical analyser originally by [Peter Odding](http://lua-users.org/wiki/LpegRecipes) to tokenize Lua source, and builds up a preprocessed string explicitly, which then can be loaded in the usual way. This is not as efficient as the original, but it can be used by anyone with a Lua interpreter, whether it is Lua 5.1, 5.2 or LuaJIT 2. An advantage of fully building the output is that it becomes much easier to debug macros when you can actually see the generated results. (Another example of a LPeg-based Lua macro preprocessor is [Luma](http://luaforge.net/projects/luma/))
-
-It is not possible to discuss macros in Lua without mentioning Fabien Fleutot's [Metalua](metalua.luaforge.net/) which is an alternative Lua compiler which supports syntactical macros that can work on the AST (Abstract Syntax Tree) itself of Lua. This is clearly a technically superior way to extend Lua syntax, but again has the disadvantage of being a direct-to-bytecode compiler. (Perhaps it's also a matter of taste, since I find it easier to think about extending Lua on the lexical level.)
-
-My renewed interest in Lua lexical macros came from some discussions on the Lua mailing list about numerically optimal Lua code using LuaJIT. We have been spoiled by modern optimizing C/C++ compilers, where hand-optimization is often discouraged, but LuaJIT is new and requires some assistance. For instance, unrolling short loops can make a dramatic difference, but Lua does not provide the key concept of constant value to assist the compiler. So a very straightforward use of a macro preprocessor is to provide named constants in the old-fashioned C way. Very efficient code can be generated by generalizing the idea of 'varargs' into a statically-compiled 'tuple' type.
+This is a library and driver script for preprocessing and evaluating Lua code.
+Lexical macros can be defined, which may be simple C-preprocessor style macros or
+macros that change their expansion depending on the context.
+
+It is a new, rewritten version of the
+[Luaforge](http://luaforge.net/projects/luamacro/) project of the same name, which
+required the [token filter
+patch](http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/#tokenf) by Luiz Henrique de
+Figueiredo. This patch allowed Lua scripts to filter the raw token stream before
+the compiler stage. Within the limits imposed by the lexical filter approach this
+worked pretty well. However, the token filter patch is unlikely to ever become
+part of mainline Lua, either in its original or
+[revised](http://lua-users.org/lists/lua-l/2010-02/msg00325.html) form. So the most
+portable option becomes precompilation, but Lua bytecode is not designed to be
+platform-independent and in any case changes faster than the surface syntax of the
+language. So using LuaMacro with LuaJIT would have required re-applying the patch,
+and would remain within the ghetto of specialized, experimental use.
+
+This implementation uses a [LPeg](http://www.inf.puc-rio.br/~roberto/lpeg.html)
+lexical analyser originally by [Peter
+Odding](http://lua-users.org/wiki/LpegRecipes) to tokenize Lua source, and builds
+up a preprocessed string explicitly, which then can be loaded in the usual way.
+This is not as efficient as the original, but it can be used by anyone with a Lua
+interpreter, whether it is Lua 5.1, 5.2 or LuaJIT 2. An advantage of fully building
+the output is that it becomes much easier to debug macros when you can actually see
+the generated results. (Another example of a LPeg-based Lua macro preprocessor is
+[Luma](http://luaforge.net/projects/luma/))
+
+It is not possible to discuss macros in Lua without mentioning Fabien Fleutot's
+[Metalua](metalua.luaforge.net/) which is an alternative Lua compiler which
+supports syntactical macros that can work on the AST (Abstract Syntax Tree) itself
+of Lua. This is clearly a technically superior way to extend Lua syntax, but again
+has the disadvantage of being a direct-to-bytecode compiler. (Perhaps it's also a
+matter of taste, since I find it easier to think about extending Lua on the lexical
+level.)
+
+My renewed interest in Lua lexical macros came from some discussions on the Lua
+mailing list about numerically optimal Lua code using LuaJIT. We have been spoiled
+by modern optimizing C/C++ compilers, where hand-optimization is often discouraged,
+but LuaJIT is new and requires some assistance. For instance, unrolling short loops
+can make a dramatic difference, but Lua does not provide the key concept of
+constant value to assist the compiler. So a very straightforward use of a macro
+preprocessor is to provide named constants in the old-fashioned C way. Very
+efficient code can be generated by generalizing the idea of 'varargs' into a
+statically-compiled 'tuple' type.
tuple(3) A,B
@@ -16,7 +52,8 @@ The assigment `A = B` is expanded as:
A_1,A_2,A_3 = B_1,B_2,B_3
-I will show how the expansion can be made context-sensitive, so that the loop-unrolling macro `do_` changes this behaviour:
+I will show how the expansion can be made context-sensitive, so that the
+loop-unrolling macro `do_` changes this behaviour:
do_(i,1,3,
A = 0.5*B
@@ -28,7 +65,10 @@ expands to:
A_2 = 0.5*B_2
A_3 = 0.5*B_3
-Another use is crafting DSLs, particularly for end-user scripting. For instance, people may be more comfortable with `forall x in t do` rather than `for _,x in ipairs(t) do`; there is less to explain in the first form and it translates directly to the second form. Another example comes from this common pattern:
+Another use is crafting DSLs, particularly for end-user scripting. For instance,
+people may be more comfortable with `forall x in t do` rather than `for _,x in
+ipairs(t) do`; there is less to explain in the first form and it translates
+directly to the second form. Another example comes from this common pattern:
some_action(function()
...
@@ -44,15 +84,26 @@ we can write:
...
end
-A criticism of traditional lexical macros is that they don't respect the scoping rules of the language itself. Bad experiences with the C preprocessor lead many to regard them as part of the prehistory of computing. The macros described here can be lexically scoped, and can be as 'hygenic' as necessary, since their expansion can be finely controlled with Lua itself.
+A criticism of traditional lexical macros is that they don't respect the scoping
+rules of the language itself. Bad experiences with the C preprocessor lead many to
+regard them as part of the prehistory of computing. The macros described here can
+be lexically scoped, and can be as 'hygenic' as necessary, since their expansion
+can be finely controlled with Lua itself.
-For me, a more serious charge against 'macro magic' is that it can lead to a private dialect of the language (the original Bourne shell was written in C 'skinned' to look like Algol 68.) This often indicates a programmer uncomfortable with a language, who wants it to look like something more familiar. Relying on a preprocessor may mean that programmers need to immerse themselves in the idioms of the new language.
+For me, a more serious charge against 'macro magic' is that it can lead to a
+private dialect of the language (the original Bourne shell was written in C
+'skinned' to look like Algol 68.) This often indicates a programmer uncomfortable
+with a language, who wants it to look like something more familiar. Relying on a
+preprocessor may mean that programmers need to immerse themselves more in the idioms of
+the new language.
-That being said, macros can extend a language so that it can be more expressive for a particular task, particularly if the users are not professional programmers.
+That being said, macros can extend a language so that it can be more expressive for
+a particular task, particularly if the users are not professional programmers.
### Basic Macro Substitution
-To install LuaMacro, expand the archive and make a script or batch file that points to `luam.lua`, for instance:
+To install LuaMacro, expand the archive and make a script or batch file that points
+to `luam.lua`, for instance:
lua /home/frodo/luamacro/luam.lua %*
@@ -65,7 +116,8 @@ Any Lua code loaded with `luam` goes through four distinct steps:
* compilation
* execution
-The last two steps happen within Lua itself, but always occur, even though the Lua compiler is fast enough that we mostly do not bother to save the generated bytecode.
+The last two steps happen within Lua itself, but always occur, even though the Lua
+compiler is fast enough that we mostly do not bother to save the generated bytecode.
For example, consider this `hello.lua`:
@@ -81,9 +133,13 @@ To run the program:
$> luam -lhello-def hello.lua
Hello, World!
-So the module `hello-def.lua` is first loaded (compiled and executed, but not preprocessed) and only then `hello.lua` can be preprocessed and then loaded.
+So the module `hello-def.lua` is first loaded (compiled and executed, but not
+preprocessed) and only then `hello.lua` can be preprocessed and then loaded.
-Naturaly, there are easier ways to use LuaMacro, but I want to emphasize the sequence of macro loading, preprocessing and script loading. `luam` has a `-d` flag, meaning 'dump', which is very useful when debugging the output of the preprocessing step:
+Naturaly, there are easier ways to use LuaMacro, but I want to emphasize the
+sequence of macro loading, preprocessing and script loading. `luam` has a `-d`
+flag, meaning 'dump', which is very useful when debugging the output of the
+preprocessing step:
$> luam -d -lhello-def hello.lua
print("Hello, World!")
@@ -93,18 +149,27 @@ Naturaly, there are easier ways to use LuaMacro, but I want to emphasize the seq
require_ 'hello-def'
print(HELLO)
-You cannot use the Lua `require` function at this point, since `require` is only executed when the program starts executing and we want the macro definitions to be available during the current compilation. `require_` is the macro version, which loads the file at compile-time.
+You cannot use the Lua `require` function at this point, since `require` is only
+executed when the program starts executing and we want the macro definitions to be
+available during the current compilation. `require_` is the macro version, which
+loads the file at compile-time.
-There is also `include_`, which is analogous to `#include` in `cpp`. It takes a file path in quotes, and directly inserts the contents of the file into the current compilation. Although tempting to use, it will not work here because again the macro definitions will not be available at compile-time.
+There is also `include_`, which is analogous to `#include` in `cpp`. It takes a
+file path in quotes, and directly inserts the contents of the file into the current
+compilation. Although tempting to use, it will not work here because again the
+macro definitions will not be available at compile-time.
-`hello3.lua` fits much more into the C preprocessor paradigm, which uses the `def_` macro:
+`hello3.lua` fits much more into the C preprocessor paradigm, which uses the `def_`
+macro:
def_ HELLO "Hello, World!"
print(HELLO)
-(Like `cpp`, such macro definitions end with the line; however, there is no equivalent of `\` to extend the definition over multiple lines.)
+(Like `cpp`, such macro definitions end with the line; however, there is no
+equivalent of `\` to extend the definition over multiple lines.)
-With 2.1, an alternative syntax `def_ (name body)` is also available, which can be embedded inside a macro expression:
+With 2.1, an alternative syntax `def_ (name body)` is also available, which can be
+embedded inside a macro expression:
def_ OF_ def_ (of elseif _value ==)
@@ -116,7 +181,9 @@ Or even extend over several lines:
end
)
-`def_` works pretty much like `#define`, for instance, `def_ SQR(x) ((x)*(x))`. A number of C-style favourites can be defined, like `assert_` using `_STR_`, which is a predefined macro that 'stringifies' its argument.
+`def_` works pretty much like `#define`, for instance, `def_ SQR(x) ((x)*(x))`. A
+number of C-style favourites can be defined, like `assert_` using `_STR_`, which is
+a predefined macro that 'stringifies' its argument.
def_ assert_(condn) assert(condn,_STR_(condn))
@@ -129,7 +196,9 @@ Or even extend over several lines:
end
assert(X == 1)
-LuaMacro keeps track of Lua block structure - in particular it knows when a particular lexical scope has just been closed. This is how the `_END_CLOSE_` built-in macro works
+LuaMacro keeps track of Lua block structure - in particular it knows when a
+particular lexical scope has just been closed. This is how the `_END_CLOSE_`
+built-in macro works
def_ block (function() _END_CLOSE_
@@ -137,42 +206,93 @@ LuaMacro keeps track of Lua block structure - in particular it knows when a part
do_something_later()
end
-When the current scope closes with `end`, LuaMacro appends the necessary ')' to make this syntax valid.
+When the current scope closes with `end`, LuaMacro appends the necessary ')' to
+make this syntax valid.
-A common use of macros in both C and Lua is to inline optimized code for a case. The Lua function `assert()` always evaluates its second argument, which is not always optimal:
+A common use of macros in both C and Lua is to inline optimized code for a case.
+The Lua function `assert()` always evaluates its second argument, which is not
+always optimal:
def_ ASSERT(condn,expr) if condn then else error(expr) end
ASSERT(2 == 1,"damn! ".. 2 .." is not equal to ".. 1)
-If the message expression is expensive to execute, then this can give better performance at the price of some extra code. `ASSERT` is now a statement, not a function, however.
+If the message expression is expensive to execute, then this can give better
+performance at the price of some extra code. `ASSERT` is now a statement, not a
+function, however.
+
+### Conditional Compilation
+
+'@' has a predefined meaning for Lua scripts preprocessed with luam. `@name` means
+`name_`. (It is itself a macro, and so can be redefined if desired.)
+
+ @include 'test.inc'
+ @def A 10
+ ...
+
+This makes macro 'preprocessor' statements stand out more. Conditional compilation
+works as you would expect from C:
+
+ -- test-cond.lua
+ @if A
+ print 'A defined'
+ @else
+ print 'A not defined'
+ @end
+
+Now, what is `A`? It is a Lua expression which is evaluated at _preprocessor_
+time, and if it returns any value except `nil` or `false` it is true, using
+the usual Lua rule. Assuming `A` is just a global variable, how can it be set?
+
+ $ luam test-cond.lua
+ A defined
+ $ luam -VA test-cond.lua
+ A not defined
+
+
+
+
+
### Using macro.define
-`macro.define` is less convenient than `def_` but much more powerful. The extended form allows the substitution to be a _function_ which is called in-place at compile time:
+`macro.define` is less convenient than `def_` but much more powerful. The extended
+form allows the substitution to be a _function_ which is called in-place at compile
+time:
macro.define('DATE',function()
return '"'..os.date('%c')..'"'
end)
-Any text which is returned will be tokenized and inserted into the output stream. The explicit quoting here is needed to ensure that `DATE` will be replaced by the string "04/30/11 09:57:53". ('%c' gives you the current locale's version of the date; for a proper version of this macro, best to use `os.date` [with more explicit formats](http://www.lua.org/pil/22.1.html) .)
+Any text which is returned will be tokenized and inserted into the output stream.
+The explicit quoting here is needed to ensure that `DATE` will be replaced by the
+string "04/30/11 09:57:53". ('%c' gives you the current locale's version of the
+date; for a proper version of this macro, best to use `os.date` [with more explicit
+formats](http://www.lua.org/pil/22.1.html) .)
-This function can also return nothing, which allows you to write macro code purely for its _side-effects_.
+This function can also return nothing, which allows you to write macro code purely
+for its _side-effects_.
-Non-operator characters like `@`,`$`, etc can be used as macros. For example, say you like shell-like notation `$HOME` for expanding environment variables in your scripts.
+Non-operator characters like `@`,`$`, etc can be used as macros. For example, say
+you like shell-like notation `$HOME` for expanding environment variables in your
+scripts.
macro.define '$(x) os.getenv(_STR_(x))'
-A script can now say `$(PATH)` and get the expected expansion, Make-style. But we can do better and support `$PATH` directly:
+A script can now say `$(PATH)` and get the expected expansion, Make-style. But we
+can do better and support `$PATH` directly:
macro.define('$',function(get)
local var = get:name()
return 'os.getenv("'..var..'")'
end)
-If a macro has no parameters, then the substitution function receives a 'getter' object. This provides methods for extracting various token types from the input stream. Here the `$` macro must be immediately followed by an identifier.
+If a macro has no parameters, then the substitution function receives a 'getter'
+object. This provides methods for extracting various token types from the input
+stream. Here the `$` macro must be immediately followed by an identifier.
-We can do better, and define `$` so that something like `$(pwd)` has the same meaning as the Unix shell:
+We can do better, and define `$` so that something like `$(pwd)` has the same
+meaning as the Unix shell:
macro.define('$',function(get)
local t,v = get()
@@ -186,9 +306,15 @@ We can do better, and define `$` so that something like `$(pwd)` has the same me
(The getter `get` is callable, and returns the type and value of the next token.)
-It is probably a silly example, but it illustrates how a macro can be overloaded based on its lexical context. Much of the expressive power of LuaMacro comes from allowing macros to fetch their own parameters in this way. It allows us to define new syntax and go beyond 'pseudo-functions', which is more important for a conventional-syntax language like Lua, rather than Lisp where everything looks like a function anyway.
+It is probably a silly example, but it illustrates how a macro can be overloaded
+based on its lexical context. Much of the expressive power of LuaMacro comes from
+allowing macros to fetch their own parameters in this way. It allows us to define
+new syntax and go beyond 'pseudo-functions', which is more important for a
+conventional-syntax language like Lua, rather than Lisp where everything looks like
+a function anyway.
-It is entirely possible for macros to create macros; that is what `def_` does. Consider how to add the concept of `const` declarations to Lua:
+It is entirely possible for macros to create macros; that is what `def_` does.
+Consider how to add the concept of `const` declarations to Lua:
const N,M = 10,20
@@ -203,7 +329,9 @@ Here is one solution:
end
end)
-The key to making these constants well-behaved is `define_scoped`, which installs a block handler which resets the macro to its original value, which is usually `nil`. This test script shows how the scoping works:
+The key to making these constants well-behaved is `define_scoped`, which installs a
+block handler which resets the macro to its original value, which is usually `nil`.
+This test script shows how the scoping works:
require_ 'const'
do
@@ -217,11 +345,14 @@ The key to making these constants well-behaved is `define_scoped`, which install
assert(N == nil and M == nil)
-If we were designing a DSL intended for non-technical users, then we cannot just say to them 'learn the language properly - go read PiL!'. It would be easier to explain:
+If we were designing a DSL intended for non-technical users, then we cannot just
+say to them 'learn the language properly - go read PiL!'. It would be easier to
+explain:
forall x in {10,20,30} do
-than the equivalent generic `for` loop. `forall` can be implemented fairly simply as a macro:
+than the equivalent generic `for` loop. `forall` can be implemented fairly simply
+as a macro:
macro.define('forall',function(get)
local var = get:name()
@@ -230,9 +361,11 @@ than the equivalent generic `for` loop. `forall` can be implemented fairly simpl
return ('for _,%s in ipairs(%s) do'):format(var,rest)
end)
-That is, first get the loop variable, skip `in`, grab everything up to `do` and output the corresponding `for` statement.
+That is, first get the loop variable, skip `in`, grab everything up to `do` and
+output the corresponding `for` statement.
-Useful macros can often be built using these new forms. For instance, here is a simple list comprehension macro:
+Useful macros can often be built using these new forms. For instance, here is a
+simple list comprehension macro:
macro.define('L(expr,select) '..
'(function() local res = {} '..
@@ -242,23 +375,42 @@ Useful macros can often be built using these new forms. For instance, here is a
For example, `L(x^2,x in t)` will make a list of the squares of all elements in `t`.
-(`macro.forall` defines more sophisticated `forall` statements and list comprehension expressions, but the principle is the same.)
+(`macro.forall` defines more sophisticated `forall` statements and list
+comprehension expressions, but the principle is the same.)
-There is a second argument passed to the substitution function, which is a 'putter' object - an object for building token lists. For example, a useful shortcut for anonymous functions:
+There is a second argument passed to the substitution function, which is a 'putter'
+object - an object for building token lists. For example, a useful shortcut for
+anonymous functions:
M.define ('\\',function(get,put)
- local args, body = get:names('('), get:list()
+ local args = get:names('(')
+ local body = get:list()
return put:keyword 'function' '(' : names(args) ')' :
keyword 'return' : list(body) : space() : keyword 'end'
end)
-The `put` object has methods for appending particular kinds of tokens, such as keywords and strings, and is also callable for operator tokens. These always return the object itself, so the output can be built up with chaining.
-
-Consider `\x,y(x+y)`: the `names` getter grabs a comma-separated list of names upto the given token; the `list` getter grabs a general argument list. It returns a list of token lists and by default stops at ')'. This 'lambda' notation was suggested by Luiz Henrique de Figueiredo as something easily parsed by any token-filtering approach - an alternative notation `|x,y| x+y` has been [suggested](http://lua-users.org/lists/lua-l/2009-12/msg00071.html) but is generally impossible to implement using a lexical scanner, since it would have to parse the function body as an expression. The `\\` macro also has the advantage that the operator precedence is explicit: in the case of `\\(42,'answer')` it is immediately clear that this is a function of no arguments which returns two values.
-
-(Although I would not necessarily suggest that lambdas are a good thing in production code, they can be useful in iteractive exploration and within tests.)
-
-Macros with explicit parameters can define a substitution function, but this function receives the values themselves, not the getter and putter objects. These values are _token lists_ and must be converted into the expected types using the token list methods:
+The `put` object has methods for appending particular kinds of tokens, such as
+keywords and strings, and is also callable for operator tokens. These always return
+the object itself, so the output can be built up with chaining.
+
+Consider `\x,y(x+y)`: the `names` getter grabs a comma-separated list of names upto
+the given token; the `list` getter grabs a general argument list. It returns a list
+of token lists and by default stops at ')'. This 'lambda' notation was suggested
+by Luiz Henrique de Figueiredo as something easily parsed by any token-filtering
+approach - an alternative notation `|x,y| x+y` has been
+[suggested](http://lua-users.org/lists/lua-l/2009-12/msg00071.html) but is
+generally impossible to implement using a lexical scanner, since it would have to
+parse the function body as an expression. The `\\` macro also has the advantage
+that the operator precedence is explicit: in the case of `\\(42,'answer')` it is
+immediately clear that this is a function of no arguments which returns two values.
+
+I would not necessarily suggest that lambdas are a good thing in
+production code, but they _can_ be useful in iteractive exploration and within tests.
+
+Macros with explicit parameters can define a substitution function, but this
+function receives the values themselves, not the getter and putter objects. These
+values are _token lists_ and must be converted into the expected types using the
+token list methods:
macro.define('test_(var,start,finish)',function(var,start,finish)
var,start,finish = var:get_iden(),start:get_number(),finish:get_number()
@@ -279,7 +431,7 @@ Since no `put` object is received, such macros need to construct their own:
Consider this loop-unrolling macro:
do_(i,1,3,
- y = y + 1
+ y = y + i
)
which will expand as
@@ -306,7 +458,8 @@ For each iteration, it needs to define a local macro `i` which expands to 1,2 an
return put
end)
-Ignoring the macro stack manipulation for a moment, it works by inserting `set_` macro assignments into the output. That is, the raw output looks like this:
+Ignoring the macro stack manipulation for a moment, it works by inserting `set_`
+macro assignments into the output. That is, the raw output looks like this:
set_ i 1
y = y + i
@@ -317,9 +470,14 @@ Ignoring the macro stack manipulation for a moment, it works by inserting `set_`
undef_ i
_DROP_ 'do_'
-It's important here to understand that LuaMacro does not do _recursive_ substitution. Rather, the output of macros is pushed out to the stream which is then further substituted, etc. So we do need these little helper macros to set the loop variable at each point.
+It's important here to understand that LuaMacro does not do _recursive_
+substitution. Rather, the output of macros is pushed out to the stream which is
+then further substituted, etc. So we do need these little helper macros to set the
+loop variable at each point.
-Using the macro stack allows macros to be aware that they are expanding inside a `do_` macro invocation. Consider `tuple`, which is another macro which creates macros:
+Using the macro stack allows macros to be aware that they are expanding inside a
+`do_` macro invocation. Consider `tuple`, which is another macro which creates
+macros:
tuple(3) A,B
A = B
@@ -366,24 +524,35 @@ And here is the definition:
end
end)
-The first expansion case happens if we are not within a `do_` macro; a simple list of names is outputted. Otherwise, we know what the loop variable is, and can directly ask for its value.
+The first expansion case happens if we are not within a `do_` macro; a simple list
+of names is outputted. Otherwise, we know what the loop variable is, and can
+directly ask for its value.
### Operator Macros
-You can of course define `@` to be a macro; a new feature allows you to add new operator tokens:
+You can of course define `@` to be a macro; a new feature allows you to add new
+operator tokens:
macro.define_tokens {'##','@-'}
-which can then be used with `macro.define`, but also now with `def_`. It's now possible to define a list comprehension syntax that reads more naturally, e.g. `{|x^2| i=1,10}` by making `{|` into a new token.
+which can then be used with `macro.define`, but also now with `def_`. It's now
+possible to define a list comprehension syntax that reads more naturally, e.g.
+`{|x^2| i=1,10}` by making `{|` into a new token.
-Up to now, making a Lua operator token such as `.` into a macro was not so useful. Such a macro may now return an extra value which indicates that the operator should simply 'pass through' as is. Consider defining a `with` statement:
+Up to now, making a Lua operator token such as `.` into a macro was not so useful.
+Such a macro may now return an extra value which indicates that the operator should
+simply 'pass through' as is. Consider defining a `with` statement:
with A do
.x = 1
.y = 2
end
-I've deliberately indicated the fields using a dot (a rare case of Visual Basic syntax being superior to Delphi). So it is necessary to overload '.' and look at the previous token: if it isn't a case like `name.` or `].` then we prepend the table. Otherwise, the operator must simply _pass through_, to prevent an uncontrolled recursion.
+I've deliberately indicated the fields using a dot (a rare case of Visual Basic
+syntax being superior to Delphi). So it is necessary to overload '.' and look at
+the previous token: if it isn't a case like `name.` or `].` then we prepend the
+table. Otherwise, the operator must simply _pass through_, to prevent an
+uncontrolled recursion.
M.define('with',function(get,put)
M.define_scoped('.',function()
@@ -400,7 +569,11 @@ I've deliberately indicated the fields using a dot (a rare case of Visual Basic
Again, scoping means that this behaviour is completely local to the with-block.
-A more elaborate experiment is `cskin.lua` in the tests directory. This translates a curly-bracket form into standard Lua, and at its heart is defining '{' and '}' as macros. You have to keep a brace stack, because these tokens still have their old meaning and the table constructor in this example must still work, while the trailing brace must be converted to `end`.
+A more elaborate experiment is `cskin.lua` in the tests directory. This translates
+a curly-bracket form into standard Lua, and at its heart is defining '{' and '}' as
+macros. You have to keep a brace stack, because these tokens still have their old
+meaning and the table constructor in this example must still work, while the
+trailing brace must be converted to `end`.
if (a > b) {
t = {a,b}
@@ -408,14 +581,18 @@ A more elaborate experiment is `cskin.lua` in the tests directory. This translat
### Pass-Through Macros
-Normally a macro replaces the name (plus any arguments) with the substitution. It is sometimes useful to pass the name through, but not to push the name into the token stream - otherwise we will get an endless expansion.
+Normally a macro replaces the name (plus any arguments) with the substitution. It
+is sometimes useful to pass the name through, but not to push the name into the
+token stream - otherwise we will get an endless expansion.
macro.define('fred',function()
print 'fred was found'
return nil, true
end)
-This has absolutely no effect on the preprocessed text ('fred' remains 'fred', but has a side-effect. This happens if the substitution function returns a second `true` value. You can look at the immediate lexical environment with `peek`:
+This has absolutely no effect on the preprocessed text ('fred' remains 'fred', but
+has a side-effect. This happens if the substitution function returns a second
+`true` value. You can look at the immediate lexical environment with `peek`:
macro.define('fred',function(get)
local t,v = get:peek(1)
@@ -426,9 +603,13 @@ This has absolutely no effect on the preprocessed text ('fred' remains 'fred', b
return nil,true
end)
-Pass-through macros are useful when each macro corresponds to a Lua variable; they allow such variables to have a dual role.
+Pass-through macros are useful when each macro corresponds to a Lua variable; they
+allow such variables to have a dual role.
-An example would be Python-style lists. The [Penlight List](http://stevedonovan.github.com/Penlight/api/modules/pl.List.html) class has the same functionality as the built-in Python list, but does not have any syntactical support:
+An example would be Python-style lists. The [Penlight
+List](http://stevedonovan.github.com/Penlight/api/modules/pl.List.html) class has
+the same functionality as the built-in Python list, but does not have any
+syntactical support:
> List = require 'pl.List'
> ls = List{10,20,20}
@@ -438,7 +619,9 @@ An example would be Python-style lists. The [Penlight List](http://stevedonovan.
> = ls
{10,11,20,21,30}
-It would be cool if we could add a little bit of custom syntax to make this more natural. What we first need is a 'macro factory' which outputs the code to create the lists, and also suitable macros with the same names.
+It would be cool if we could add a little bit of custom syntax to make this more
+natural. What we first need is a 'macro factory' which outputs the code to create
+the lists, and also suitable macros with the same names.
-- list <var-list> [ = <init-list> ]
M.define ('list',function(get)
@@ -461,11 +644,16 @@ It would be cool if we could add a little bit of custom syntax to make this more
values[i] = 'List('..tostring(values[i] or '')..')'
end
local lcal = M._interactive and '' or 'local '
- local res = lcal..table.concat(vars,',')..' = '..table.concat(values,',')..tostring(endt)
+ local res = lcal..table.concat(vars,',')..' =
+'..table.concat(values,',')..tostring(endt)
return res
end)
-Note that this is a fairly re-usable pattern; it requires the type constructor (`List` in this case) and a type-specific macro function (`list_check`). The only tricky bit is handling the two cases, so the `names` method finds the end using a function, not a simple token. `names`, like `list`, returns the list and the token that ended the list, so we can use `endt` to check.
+Note that this is a fairly re-usable pattern; it requires the type constructor
+(`List` in this case) and a type-specific macro function (`list_check`). The only
+tricky bit is handling the two cases, so the `names` method finds the end using a
+function, not a simple token. `names`, like `list`, returns the list and the token
+that ended the list, so we can use `endt` to check.
list a = {1,2,3}
list b
@@ -532,9 +720,14 @@ This can be used interactively, like so (it requires the Penlight list library.)
### Preprocessing C
-With the 2.2 release, LuaMacro can preprocess C files, by the inclusion of a C LPeg lexer based on work by Peter Odding. This may seem a semi-insane pursuit, given that C already has a preprocessor, (which is widely considered a misfeature.) However, the macros we are talking about are clever, they can maintain state, and can be scoped lexically.
+With the 2.2 release, LuaMacro can preprocess C files, by the inclusion of a C LPeg
+lexer based on work by Peter Odding. This may seem a semi-insane pursuit, given
+that C already has a preprocessor, (which is widely considered a misfeature.)
+However, the macros we are talking about are clever, they can maintain state, and
+can be scoped lexically.
-One of the irritating things about C is the need to maintain separate include files. It would be better if we could write a module like this:
+One of the irritating things about C is the need to maintain separate include
+files. It would be better if we could write a module like this:
// dll.c
@@ -586,22 +779,81 @@ The macro `export` is straightforward:
return out
end)
-It looks ahead and if it finds a `{}` block it writes the block as text to a file stream; otherwise writes out the function signature. `get:upto '}'` will do the right thing here since it keeps track of brace level. To allow any other macro expansions to take place, `substitute_tostring` is directly called.
+It looks ahead and if it finds a `{}` block it writes the block as text to a file
+stream; otherwise writes out the function signature. `get:upto '}'` will do the
+right thing here since it keeps track of brace level. To allow any other macro
+expansions to take place, `substitute_tostring` is directly called.
-`tests/cexport.lua` shows how this idea can be extended, so that the generated header is only updated when it changes.
+`tests/cexport.lua` shows how this idea can be extended, so that the generated
+header is only updated when it changes.
To preprocess C with `luam`, you need to specify the `-C` flag:
luam -C -lcexport dll.lc > dll.c
-Have a look at [lc](modules/macro.lc.html) which defines a simplified way to write Lua bindings in C.
+Have a look at [lc](modules/macro.lc.html) which defines a simplified way to write
+Lua bindings in C.
+
+This was used for the [winapi](https://github.com/stevedonovan/winapi) project to
+preprocess [this
+file](https://github.com/stevedonovan/winapi/blob/master/winapi.l.c) into standard C.
+
+### A Simple Test Framework
+
+LuaMacro comes with yet another simple test framework - I apologize for this in
+advance, because there are already quite enough. But consider it a demonstration
+of how a little macro sugar can make tests more readable, even if you are
+uncomfortable with them in production code (see `tests/test-test.lua`)
+
+ require_ 'assert'
+ assert_ 1 == 1
+ assert_ "hello" matches "^hell"
+ assert_ x.a throws 'attempt to index global'
+
+The last line is more interesting, since it's transparently wrapping
+the offending expression in an anonymous function. The expanded output looks
+like this:
+
+ T_ = require 'macro.lib.test'
+ T_.assert_eq(1 ,1)
+ T_.assert_match("hello" ,"^hell")
+ T_.assert_match(T_.pcall_no(function() return x.a end),'attempt to index global')
+
+(This is a generally useful pattern - use macros to provide a thin layer of sugar
+over the underlying library. The `macro.assert` module is only 75 lines long, with
+comments - its job is to format code to make using the implementation easier.)
+
+Remember that the predefined meaning of @ is to convert `@name` into `name_`. So we
+could just as easily say `@assert 1 == 1` and so forth.
+
+Lua functions often return multiple values or tables:
+
+ two = \(40,2)
+ table2 = \({40,2})
+ @assert two() == (40,2)
+ @assert table2() == {40,2}
+
+Any use of `==` with floating-point numbers is going to bite you, eventually.
+Consider this passing assertion:
+
+ @assert 3.1412 == 3.14
+
+If the test value is explicitly in fixed point format, then we'll only match up
+to that precision. This line expands as follows:
+
+ T_.assert_match(("%4.2f"):format( 3.1412 ),"3.14")
+
+Not a very well _tested_ feature, but it does illustrate something that's fairly
+awkward to write out in Lua code.
+
+
-This was used for the [winapi](https://github.com/stevedonovan/winapi) project to preprocess [this file](https://github.com/stevedonovan/winapi/blob/master/winapi.l.c) into standard C.
### Implementation
-It is not usually necessary to understand the underlying representation of token lists, but I present it here as a guide to understanding the code.
+It is not usually necessary to understand the underlying representation of token
+lists, but I present it here as a guide to understanding the code.
#### Token Lists
@@ -609,11 +861,14 @@ The token list representation of the expression `x+1` is:
{{'iden','x'},{'+','+'},{'number','1'}}
-which is the form returned by the LPeg lexical analyser. Please note that there are also 'space' and 'comment' tokens in the stream, which is a big difference from the token-filter standard.
+which is the form returned by the LPeg lexical analyser. Please note that there are
+also 'space' and 'comment' tokens in the stream, which is a big difference from the
+token-filter standard.
The `TokenList` type defines `__tostring` and some helper methods for these lists.
-The following macro is an example of the lower-level coding needed without the usual helpers:
+The following macro is an example of the lower-level coding needed without the
+usual helpers:
local macro = require 'macro'
macro.define('qw',function(get,put)
@@ -632,7 +887,10 @@ The following macro is an example of the lower-level coding needed without the u
return res
end)
-We're using the getter `next` method to skip any whitespace, but building up the substitution without a putter, just manipulating the raw token list. `qw` takes a plain list of words, separated by spaces (and maybe commas) and makes it into a list of strings. That is,
+We're using the getter `next` method to skip any whitespace, but building up the
+substitution without a putter, just manipulating the raw token list. `qw` takes a
+plain list of words, separated by spaces (and maybe commas) and makes it into a
+list of strings. That is,
qw(one two three)
@@ -642,12 +900,28 @@ becomes
#### Program Structure
-The main loop of `macro.substitute` (towards end of `macro.lua`) summarizes the operation of LuaMacro:
-
-There are two macro tables, `imacro` for classic name macros, and `smacro` for operator style macros. They contain macro tables, which must have a `subst` field containing the substitution and may have a `parms` field, which means that they must be followed by their arguments in parentheses.
-
-A keywords table is chiefly used to track block scope, e.g. `do`,`if`,`function`,etc means 'increase block level' and `end`,`until` means 'decrease block level'. At this point, any defined block handlers for this level will be evaluated and removed. These may insert tokens into the stream, like macros. This is how something like `_END_CLOSE_` is implemented: the `end` causes the block level to decrease, which fires a block handler which passes `end` through and inserts a closing `)`.
-
-Any keyword may also have an associated keyword handler, which works rather like a macro substitution, except that the keyword itself is always passed through first. (Allowing keywords as regular macros would generally be a bad idea because of the recursive substitution problem.)
-
-The macro `subst` field may be a token list or a function. if it is a function then that function is called, with the parameters as token lists if the macro defined formal parameters, or with getter and setter objects if not. If the result is text then it is parsed into a token list.
+The main loop of `macro.substitute` (towards end of `macro.lua`) summarizes the
+operation of LuaMacro:
+
+There are two macro tables, `imacro` for classic name macros, and `smacro` for
+operator style macros. They contain macro tables, which must have a `subst` field
+containing the substitution and may have a `parms` field, which means that they
+must be followed by their arguments in parentheses.
+
+A keywords table is chiefly used to track block scope, e.g.
+`do`,`if`,`function`,etc means 'increase block level' and `end`,`until` means
+'decrease block level'. At this point, any defined block handlers for this level
+will be evaluated and removed. These may insert tokens into the stream, like
+macros. This is how something like `_END_CLOSE_` is implemented: the `end` causes
+the block level to decrease, which fires a block handler which passes `end` through
+and inserts a closing `)`.
+
+Any keyword may also have an associated keyword handler, which works rather like a
+macro substitution, except that the keyword itself is always passed through first.
+(Allowing keywords as regular macros would generally be a bad idea because of the
+recursive substitution problem.)
+
+The macro `subst` field may be a token list or a function. if it is a function then
+that function is called, with the parameters as token lists if the macro defined
+formal parameters, or with getter and setter objects if not. If the result is text
+then it is parsed into a token list.
Please sign in to comment.
Something went wrong with that request. Please try again.