User-macro calls given as user macro arguments always compile to function calls

[anko]

As @stasm mentions in anko/eslisp-fancy-function#1, if a call to a user-defined macro is given as a parameter of another user-defined macro, it always compiles to a function call, not to the results of that macro.

Example:

(macro fun0 (lambda (body)
  (return `(lambda () ,body))))
(macro ok (lambda ()
  (return '(return true))))
(fun0 (ok))

Expected output:

(function () {
    return true;
});

Actual output:

(function () {
    ok();
});

---

This does not affect built-in macros, because they can access the compilation environment with which they can optionally compile their arguments (for example like this) in a way that resolves and executes macros.

User macros currently don't have that choice. They operate on lists they've received as arguments, but the process by which they're compiled doesn't take macros into account. If it did, we'd have the opposite problem. There needs to be a user choice.

To give an example, if a is a user-defined macro and it is called with (a (b)), it receives 1 argument, which is an array containing an atom b. If it returns that argument as-is, it is interpreted as a function call b();, which the macro may have wanted to return. But another interpretation is to compile the list taking the macro table into account, in which case it may turn out that b is a macro that returns different code to be used there instead.

---

There obviously should be no such limitation. I could use some help from more experienced lispers here. What would Batman do? Is this what macroexpand/macroexpand-all are for?

[stasm]

If it did, we'd have the opposite problem. There needs to be a user choice. […]

Interesting. I'm sure I'm missing something here, but why would the user want the inverse behavior?

To give an example, if a is a user-defined macro and it is called with (a (b)), it receives 1 argument, which is an array containing an atom b. If it returns that argument as-is, it is interpreted as a function call b();, which the macro may have wanted to return.

If the user has defined/imported the b macro in the same scope, why would they want to return the atom b? My expectation would be that the b macro gets expanded: either before it's passed to the a macro, or after.

[anko]

If it did, we'd have the opposite problem. There needs to be a user choice. […]

Interesting. I'm sure I'm missing something here, but why would the user want the inverse behavior?

Apologies for the excess conciseness. 😄

An example might help:

(macro hello
       (lambda () (return 'hi)))

(macro passthrough
       (lambda (arg) (return arg)))

(passthrough (hello))

At the moment, that compiles to hello();. There's no way to get hi;, because the (hello) passed as an argument to passthrough is never macro-expanded.

If the results of a macro were expanded after the call returns, there would be no way for a macro to output a call to a function with the same name as a macro that is currently defined. In this case, the above would output hi;, but there would be no way for any implementation of passthrough to output hello();. (This is what I mean by the "opposite problem".)

If the (hello) part of that last-line call was expanded before the call happens, DSLs that involve inspecting their arguments would become impossible. (For example, in (lambda (x) (return x)) resolving macros in the arguments before they're passed to lambda is clearly not workable.)

---

This is why I'm thinking adding macroexpand might do the trick. With it, user macros could do (return arg) when they want to return it without doing macro-expansion, and do (return ((. this macroexpand) arg)) to return it with macros expanded.

[stasm]

If the results of a macro were expanded after the call returns, there would be no way for a macro to output a call to a function with the same name as a macro that is currently defined. In this case, the above would output hi;, but there would be no way for any implementation ofpassthroughto outputhello();`. (This is what I mean by the "opposite problem".)

Thanks for the explanation. Perhaps this can be mitigated by setting proper expectations instead of code? Since macros share the same namespace as functions and are expanded before the code is run, the user should expect that a macro with the same name as a function will be expanded and the function will not be called.

[dead-claudia]

I think we will need a macroexpand function for this. That's probably the best way to handle this.

[lhorie]

Macro expansion should be recursive

In CommonLisp:

(defmacro add-1 (a b) `(+ ,a ,b))
(defmacro add (a b) `(add-1 ,a ,b))
(add 1 2) ; expands to (add-1 1 2), which in turn expands to (+ 1 2), which yields 3
'(add 1 2) ; no expansion, yields (add 1 2)
(macroexpand '(add 1 2)) ; expands recursively, but does not eval. yields (+ 1 2)
(macroexpand-1 '(add 1 2)) ; expands once. yields (add-1 1 2)

My understanding of macroexpand is that it's a debugging tool you call from the REPL to troubleshoot a macro, not something you would write in a production macro. I'm not even sure you can have a non-macro form w/ the same name as a macro form in any traditional lisp. In CL, for example, the last definition wins:

(defun add (a b) (- 1 2))
(defmacro add (a b) `(+ ,a ,b))
(add 1 2) ; 3

(defmacro add (a b) `(+ ,a ,b))
(defun add (a b) (- 1 2))
(add 1 2) ; -1

Sweet.js has a concept called let macros that lets you define non-recursive macros, but that only applies for when a symbol has the same name as a macro.

In addition, Sweet.js has primitive support for modules, which allow you to define "private" macros.

I use both of those sweet.js features in Mithril's template compiler (it's basically a function inliner that replaces m(...) calls with their outputs), but the contortionism required to accomplish that is downright scary...

[vendethiel]

Macro expansion definitely should be recursive. Common Lisp is a Lisp-2 (a namespace for variables, and one for functions + macros, which have very similar behavior, and can call one another... But arguably CL is pretty weird in its "everything-is-late-bound" behavior.

Macroexpand(-1) are debugging tools, AFAIK (some implementations may give you even more advanced tools, like SBCL's macroexpand-all, which will even expand lets etc, using SBCL's code walker directly, IIRC)

[vendethiel]

Also, Sweet.js's "let macros" do some weird stuff to make the macro not expand... Only inside itself. It'll be expanded everywhere else normally, AFAIK

[lhorie]

FWIW, I just tried in Racket REPL and also got "last definition wins" behavior

(define-syntax-rule (add a b) (+ a b))
(define add (lambda (a b) (- a b)))
(add 1 2) ; -1

(define add (lambda (a b) (- a b)))
(define-syntax-rule (add a b) (+ a b))
(add 1 2) ; 3

Sweet.js let macros do feel like a bolted-on hack, but it's the only way I was able to output a variable called m while also having a macro called m.

I think there is a subtle but important difference between lisp macros and sweet.js macros: lisp macros occupy space in the runtime's v-table, but sweet.js macros do not. The main implication is that if you can dynamically change a macro as is possible w/ a lisp REPL, then by definition you cannot have macros w/ the same name as runtime values. Conversely, to be able to output "shadowed" variables, macro-expansion-time must be at compile-time, and that must be distinctly separated from runtime.

Given that eslisp doesn't aim to be a "true" lisp (in the dynamic code-is-always-data sense), perhaps an approach closer to sweetjs might make more sense.

[vendethiel]

code-is-always-data

Even that stops somewhere. As I've heard, old lisps literally stored functions as cons cells, but that proved to be too slow for "real-life stuff". That's why later lisps change it

[vendethiel]

Sweet.js let macros do feel like a bolted-on hack, but it's the only way I was able to output a variable called m while also having a macro called m.

I think macros are confusing enough we don't really need this though, do we? Care to explain your use case a bit more, maybe? I might've not understood it.

[lhorie]

My use case is that one of my API functions (m) has no side-effects, and it's typically used with hard-coded values, so it can be pre-computed ahead-of-time. So I wrote a macro named m that overshadows the function m so that m("#foo.bar") would become {tag: "div", attrs: {id: "foo", class: "bar"}, children: []} after a compilation pass. The problem is that m is also the namespace for the library and constructs like m.component should not be expanded into anything, so simply using case macros without a let macro would cause syntax errors.

[vendethiel]

Oh, I see now. Well, but the m identifier in a Lisp-2 is not special (a function /= a variable, so having a m identifier isn't a big deal). In a Lisp-1, a good module system could probably help? (require [mithril :use [m component])?

[lhorie]

Regardless of whether it's a lisp 1 or 2, there's still a difference between a runtime macro and a compile-time macro. Eslisp macros run at compile-time, so they become subjected to the possibility of variable shadowing.

My suggestion, based off of what I said above, is to have macro be always recursive like in other lisps, and add a letmacro form that does the equivalent of a sweetjs let macro (i.e. don't recursively expand if a symbol has the same name as the letmacro, otherwise expand recursively as normal).