Macro hygiene is hard to use correctly in nested macro expansion

[c42f]

With nested macro expansion, the inner macro sees an expression generated by the outer macro which may include Expr(:escape). However, macro writers generally test macros only with a single level of expansion, not including Expr(:escape).

This means that macros which pattern match their input are incorrect by default when used in a nested expansion.

As a simple example of how pervasive this problem is, consider that Base.@view cannot generally be used within the AST generated by another macro:

julia> macro m(ex)
           quote
               @view $(esc(ex))
           end
       end

julia> A = [1,2,3]

julia> @m A[1:2]
ERROR: LoadError: ArgumentError: Invalid use of @view macro: argument must be a reference expression A[...].
Stacktrace:
 [1] @view(::LineNumberNode, ::Module, ::Any) at ./views.jl:123
in expression starting at REPL[9]:3

The problem here is that @view gets provided with esc(:(A[1:2])) as an argument, which is not an Expr(:ref) as naturally expected by the authors of @view

This problem occurs whenever macros try to pattern match their input rather than simply substituting it into a larger expression. The pattern matching must be aware that Expr(:escape) could occur anywhere. Anybody writing macros directly against the Expr API (by using the head field, etc) is going to handle this incorrectly.

This usability issue has also been discussed at length in #23221. However that issue doesn't describe the problem very clearly as a problem of usability, so I thought I'd restate it here.

Here's another interesting case:

julia> macro m2(ex)
           quote
               @show $(esc(ex))
           end
       end

julia> @m2  1
$(Expr(:escape, 1)) = 1

What to do?

A possible way forward is to treat this as an Expr API problem: if pattern matching within macros is incorrect by default, maybe we need better ways to pattern match expressions — for example as in MacroTools or MLStyle — ensuring that any appearance of Expr(:escape) doesn't break the matching process, and returning matched pieces with a correctly nested level of escape.

A larger overhaul of the macro system as in #6910 has also been mentioned in relation to this. In that PR, quoteed code created within macros is transformed during lowering, such that every quoted symbol made by the macro is unescaped with Expr(:hygenic, sym). I'm not sure whether it solves the problem completely or simply shifts it around to create new and exciting footguns for macro writers.

[tpapp]

Thanks for opening an issue about this. This is one of the reasons I try to avoid writing nontrivial macros in Julia.

I think that sticking to a Common Lisp-style "we do our own gensyms, thank you" approach would solve this; but I am not sure how that would compose with the existing hygiene mechanism (even if the user could disable the hygiene transformation for particular macros).

[caginyunus]

I believe we need practical examples showing how this works. I have asked a related question on the discourse:
https://discourse.julialang.org/t/calling-a-macro-within-a-macro/57766

[jtrakk]

#6910 is a relevant PR from David Moon. He mentions in a comment

The root problem here is that Julia macros only allow expressions as arguments.