Macro Constructors

Macro Constructors for Dependents

The aim of macro constructors is to allow users to provide callbacks for dependents without having to manually specify their individual dependencies. This works by inspecting the callback's AST for symbols that are bound to a scope outside the function's, and injecting code into the caller to determine which of these are dependents at runtime.

This page is mostly aimed to serve as documentation for development of macro constructors, not a tutorial for their usage. For that, see examples/m45_macro_ctors.jl.

Throughout this page, the user-facing, function-based constructors will be referred to as regular constructors. Note that this doesn't include internal constructors, which aren't exported, like _Point(...).

Structure

Structurally, macros sit between the end user and the regular constructors. This means that they do not interact with other parts of the library or the user's code, encapsulating their logic from the rest of the library. As a consequence, if the API of the regular constructors changes, the macros can be adjusted easily.

Macro constructors can mostly be defined in a couple of lines, using a few helper functions. First, these helpers are described, as the majority of the processing is handled by them. Then, some examples are included to show how macro constructors are assembled. Finally, some internals have been documented as a reference at the end, to aid potential modifications later.

Helpers

All of the below helper functions can be found in src/Helpers/dependency_lookup.jl.

Callback Validation

_validate_callback_expr(callback::Expr, arg_count::Integer)::Expr

Handles the basic validation and "normalization" of callback expressions, and returns the validated form. If the callback is found to be invalid, an ErrorException is thrown using error() and a user-friendly error message is shown.

This helps catch a couple of issues that could be harmful in later stages, and would only be caught by Julia at compile-time, not parse-time.

Validation performs the following checks:

• callback must be a function definition expression (according to MacroTools.isdef)
• the number of arguments accepted by callback must be equal to arg_count
• the arguments of callback cannot be keyword arguments
• two arguments of callback cannot share the same name Symbol (argument names cannot be repeated)
• arguments of callback cannot be explicitly typed (i.e. arg::T-like)
• arguments of callback cannot be slurped (i.e. args...-like)
• arguments of callback cannot have default values

Note that "arguments" here refer to the non-dependent arguments of the callback, which are specified by the user when using a macro constructor. The arguments for the dependents themselves are added later.

Normalization ensures the following:

• callback is a function definition in MacroTools longdef form
• callback is an anonymous function (function name is removed)
• callback.args[1] is an Expr of type :tuple, other function headers are converted to this

Keyword Arguments

Keyword arguments (kwargs) work a bit differently in macros than in functions, namely (a; b)-like argument lists are not allowed. The workaround chosen is to recognize the following syntax as keyword arguments for macros:

macro my_macro(arg1, arg2, kw_args...)
    # ...
end

color = (1,0,0)
@my_macro(1, 2, width = 15, color)
@my_macro 1 2 width = 15 color

In both of the above calls, width and color act as keyword arguments (color being shorthand for color = color). Note the absence of the ; symbol, as using it changes the order the macro receives arguments. This would lead to more complex argument processing, as everything would have to be received as a single args... slurp, and iterated manually.

To process the above syntax, use the following function:

_parse_macro_kw_args(kw_names::Vector{Symbol}, kw_args...)::Dict{Symbol, Any}

Here, kw_names is a vector of valid keyword arguments names, and kw_args... is simply the forwarded list of possible keyword arguments.

The function parses symbols and expressions of type :(=) into a dictionary, where if the argument is a symbol, it'll be both the key and the value, while if it's an assignment, the lhs will be the key, the rhs the value.

Additional validation is also performed on kwargs:

• arguments must be of type Symbol, or Expr with the head :(=)
• arguments must have their key present in kw_names
• arguments cannot be actual kwargs (i.e. @my_macro(; k = v))
• two arguments cannot share the same key

If any of the above is violated, an ErrorException is thrown, with a user-friendly error message.

There's an edge case where the above technique falls short: macros that have regular arguments with default values. Take the following example:

macro my_macro(my_arg = 1, kw_args...)
    # ...
end

@my_macro() # this works (my_arg == 1, kw_args == [])
@my_macro(2, width = 2) # this works (my_arg == 2, kw_args == [:(width = 2)])
@my_macro(width = 2) # this breaks (my_arg == :(width = 2), kw_args == [])

In the last case, because width = 2 is not an actual kwarg, but an expression we want to reinterpret as one, Julia assumes it's a value for arg. This would lead to inconsistent behavior with regular function kwargs.

For a quick fix, use:

_kw_arg_or_default(arg, default, kw_args::Tuple)::Tuple

This function takes an argument that may or may not be a kwarg, a default value for it, and the tuple of true kwargs. In the above example, these would be my_arg, 1 and kw_args, respectively. It then returns a tuple, whose first element is the correct value for arg, the second is another tuple, the intended state of kw_args.

If arg is not a kwarg (e.g. it has been defaulted or the user provided a proper value for it) then simply (arg, kw_args) is returned. If it is an intended kwarg, then the returned value is (default, (kw_args..., arg)).

This can then be used in the beginning of a macro to "fix" the placement of parameters:

macro my_macro(my_arg = 1, kw_args...)
    (my_arg, kw_args) = _kw_arg_or_default(my_arg, 1, kw_args)
    # ...
end

This unfortunately includes the repetition of the default value, so make sure to update any calls to this function inside the macro body when the default value of an argument changes.

Creating the Returned Code

_create_ctor_wrapper(callback, mod::Module, base_ctor, get_ctor_args = tuple; ctor_kw_args...)::Expr

This function handles creating all the boilerplate code needed for macro constructors, so it takes in quite a few arguments. To understand all of them, it's probably easiest to look at their usage in Examples.

• callback: this is simply the (normalized) callback function definition expression
• mod: this is the calling module, used for early macro-expansion of callback (this should (almost?) always be the implicit __module__ argument macros receive)
• base_ctor: this is the regular constructor that the arguments should be forwarded to in the expanded code (i.e. the regular constructor wrapped by the macro)
• get_ctor_args: this determines the forwarding order of arguments to base_ctor. It is a function that should transform its received callback and deps arguments into a tuple (of any size), which will then be used as the arguments tuple of the call to base_ctor. Its default value simply returns (callback, deps) which is how regular constructors with no additional arguments expect it.
• ctor_kw_args...: the kwargs to forward to base_ctor

Examples

Point

For the trivial case, here's how the macro constructor that wraps Point(callback::Expr, deps::DependentsT) is defined:

macro Point(callback::Expr)
    callback = _validate_callback_expr(callback, 0)
    return _create_ctor_wrapper(callback, __module__, Juliagebra.Point)
end

We simply validate callback (expecting 0 arguments) and then create a wrapper around the call to Juliagebra.Point. Because it only takes in the callback and a list of dependents, get_ctor_args does not need to be specified.

Parametric Surface

One macro constructor that makes use of get_ctor_args is, for example, the one wrapping:

ParametricSurface(callback::Function, width, height, uStart, uEnd, vStart, vEnd, dependents::DependentsT; transparent::Bool=false, color=(0.8,0.0,0.3))

This is defined as:

macro ParametricSurface(callback::Expr, width, height, uStart, uEnd, vStart, vEnd, kw_args...)
    parsed_kw_args = _parse_macro_kw_args([:transparent, :color], kw_args...)
    callback = _validate_callback_expr(callback, 2)

    return _create_ctor_wrapper(callback, __module__, Juliagebra.ParametricSurface,
        (cb, deps) -> (cb, width, height, uStart, uEnd, vStart, vEnd, deps);
        parsed_kw_args...)
end

First, it parses its kwargs collected in kw_args..., allowing the keys transparent and color. Then, it validates callback, expecting 2 args ($u$ and $v$). Finally, it creates a wrapper call to Juliagebra.ParametricSurface, with the surface parameters placed between the callback argument, and the generated deps argument, following the original signature. The final call will look something like this:

ParametricSurface(callback, width, height, uStart, uEnd, vStart, vEnd, captured_deps)

Segment Sequence

This is currently the only macro constructor which uses _kw_arg_or_default, so here's its implementation as demonstration:

macro SegmentSequence(callback::Expr, break_every = 2, kw_args...)
    (break_every, kw_args) = _kw_arg_or_default(break_every, 2, kw_args)

    parsed_kw_args = _parse_macro_kw_args([:color, :width, :type, :reversed], kw_args...)
    callback = _validate_callback_expr(callback, 0)
    return _create_ctor_wrapper(callback, __module__, Juliagebra.SegmentSequence, (cb, deps) -> (cb, deps, break_every); parsed_kw_args...)
end

Note that after the initial _kw_arg_or_default call, the arguments can be used just like in any other macro. It's also important that kw_args are passed as a Tuple, they're not splatted, like in most cases.

If it had more than one default-valued arguments, the _kw_arg_or_default call can simply be repeated sequentially for all of them.

Internals

This part of the documentation is for those looking to extend/modify/bugfix the internal workings of the macro constructors. For adding or modifying just the macro constructors themselves, everything up until this point should be enough.

The Generated Boilerplate

When using _create_ctor_wrapper, here's the rough outline of the things that happen:

First, we generate a list of symbols from the callback definition that may point to dependents. This happens at macro-expansion time, so we don't have access to the type of these symbols yet, further checking is delegated to the caller, at runtime. The logic for this is automatically injected into by the macro.

Then, we generate the wrapper code for both the callback, and the constructor call that the caller will execute. The problem here is that we don't know, at macro-expansion time, how many dependents the callback actually relies on, so we can't determine the needed argument count for the callback. This is not something we can perform at runtime (if we find a way later, this part would benefit from a rewrite), so this part of the process requires some "hacky" solutions, as described later.

Free Variable Symbol Collection

_collect_free_vars(def::Expr, mod::Module)::Set{Symbol}

Given a def function definition expression, and a mod module for macro-expansion, generates a set of symbols, which includes every used symbol inside def whose definition is not inside def itself (the set of free variables). This can include global and local variables captured from the defining scope context, both of which may potentially hold dependents.

To keep track of which symbols are defined inside def, an AST traversal is performed, that keeps track of inner scopes and their defined symbols. This handles a lot of cases both for scoping (e.g. loops, lets, etc.) and for symbol definitions (e.g. assignments, different forms of function definition, local/global/const declarations, etc.). It should cover most cases encountered during callback definition, but it can be extended with more cases as needed.

NOTE: As the rest of the system counts this function's returned set as the superset for the potential dependencies, this is perhaps the most critical part of the macro constructor system. False positives, and more importantly, false negatives, highly influence the correctness of the dependent list generated, so this should be kept up-to-date when a new, missing AST edge case is discovered.

Function names from call expressions are stripped, to reduce the number of non-dependent symbols returned.

This function uses the following two helpers. These are not too important, but are mentioned here for sake of completeness.

_process_lhs!(lhs, current_scope::Set{Symbol}, walk_fn)
_process_fn_sig!(sig, inner_scope::Set{Symbol}, outer_scope::Set{Symbol}, walk_fn)

_process_lhs! handles the traversal of lhs treating its entirety as a "symbol to be defined". Because of Julia's expression system, this may be a nested expression, which defines several other local symbols, or references free variables, etc. It may also end up being a non-symbol introducing assignment (e.g. vec[i] = 0).

_process_fn_sig! handles the case of function declarations inside the function body. This also takes care of traversing default values, which may reference free variables.

Wrapping the Callbacks

Note that some variables have been renamed, and escape calls (i.e. $(esc(...))) have been stripped from the snippets in this section to improve readability.

As described earlier, the generation of callbacks is a bit tricky. Exact argument count is not known until runtime, but it's too late to change the AST of the function definition at that point. The only option to do that would be to @eval it afterwards, but Julia has no local alternative for @eval, so this would mean losing captures from the local scope's context.

The workaround instead is to receive arguments slurped (i.e. args...), and place a let block around the body, which captures the arguments actually received. This adds some runtime overhead, but it should be negligible in most cases, especially because usually the set of dependencies and the set of free variables only differs in a few captured constants/variables.

The final generated code is roughly of the form:

captured_deps = []

free_sym_idx_1 = 0
if @isdefined(free_sym) && free_sym isa PlanDNA
    push!(captured_deps, free_sym)
    free_sym_idx_1 = length(captured_deps)
end

# ...

wrapped_callback = (base_cb_arg1, base_cb_arg2, dep_args...) -> begin
    let #= ... =#;
        # actual callback body comes here
    end
end

regular_constructor(callback, captured_deps) # plus other potential args (e.g. width, color, etc.)

So, for every free symbol found, a unique index is generated (technically in the form of a gensym). Here, free_sym_idx_1 is generated for free_sym. This index generation happens at macro-expansion time. Then, during the actual execution, the caller checks (through the injected code) if free_sym refers to a dependent, and if so, inserts it into captured_deps and sets its own index to the running length.

This is repeated for every single free symbol collected from the AST of the callback.

Then, the actual callback function is generated, which first has the fixed arguments defined by the user (e.g. $u$ and $v$ for surfaces), then the slurped collection of dependents. This avoid having to know the exact number of dependents at macro-expansion time.

Around the actual body of the callback, a let block is placed, which has bindings of the form:

free_sym =
    free_sym_idx_1 > 0 ?
        dep_args[free_sym_idx_1] :
        @isdefined(free_sym) ?
            free_sym :
            @warn "Failed to find symbol <...>"

If during the init-phase free_sym was determined not to be a dependent (its index is $0$), it is simply captured from the defining context (if it can be, otherwise a warning is shown). If it was found to be a dependent, we can index into dep_args using its index determined during initialization (the order of dependent callback arguments matches the order provided to the regular constructor in captured_deps).

For $n$ free variables, $n$ number of bindings (and index initializers) are generated, in the same form as seen above.