Feature: re-entrant macros.

[conartist6]

I have a feature to run by you @kentcdodds. As I'm sure I've mentioned I've been slowly working towards v1 of my file watching build tool, macrome. I had stepped away from work on it for a while and as I've started thinking more about the final product over the past few days I've realized that I'm particularly excited about an integration that is possible with babel-plugin-macros (which I shall detail below).

Macrome is a really barebones build system. In essence it's two parts:

1. A watcher. It is a rich layer built on top of facebook's watchman. It is aware of version control operations and allows you to react to changes on the disk.
2. A tracking engine. When macrome writes files it tags them with an /* @macrome */ leading comment. It knows how to efficiently track down files with its headers, enabling it to efficiently locate and remove stale content.

The purpose of these two parts is to enable development workflows that make the code the source of truth, and particularly aim to unhide a lot of magic that has snuck its way into how modern frontend development is done. By magic I'm talking about things like experimental language features, JSX, CSS modules, macros, and other things. So my functionality is to take a file that has JSX or a babel macro in it, and spit out a copy of the real code on disk, most likely co-located with the original file. The theory is it's the plain javascript that's your product, and that you should be able to use as much magic as you want yet still understand what the output is, i.e. what code the client will be running. It's super useful to have access to that before the structure of it is all erased because it will be an invaluable asset for static analysis and for new developers, both of whom likely know about the basic rules of javascript but not every crazy thing that developers have invented.

OK but I'm not even to the proposal yet! Sorry. All that is old hat. It's more or less built, and it would be crazy useful even as just that. But I think I have an even better idea: create explicit support for overwriting input with output, i.e. doing codemods. This is something I've considered for a while, but it was only recently that I realized just how useful it could be when coupled with macros and the power of the watching engine.

TL;DR Let me describe the simplest re-entrant macro. Its purpose is to tag certain sites in code that integrate with outside tooling. Translations will be a very common example of. Errors should be also. These are little pieces of the code that also will need to be in a database somewhere, but you've got a problem: when the code changes, how can you tell which snippets are the same? You need to either manually create and track IDs or to treat some part of your data as a key, which has its own drawbacks. Enter the new way:

// translatable.js
import { id} from 'babel-plugin-macros';

export default translate(id, 'The quick brown fox jumped over the lazy dog');

And the magic is when you save the file, it overwrites itself. This is why the macro code must be re-entrant, it can't send itself into an infinite loop. What you should get is:

// translatable.js
import { id} from 'babel-plugin-macros';

export default translate(id`15`, 'The quick brown fox jumped over the lazy dog');

And now hopefully you see why macros needs an integration with some bigger central system. Unless it is always generation UUIDv4s, this macro needs a map somewhere that tracks which IDs it has given out. Macrome can provide that central location. Of course you could always have had a central database anyway, but then you get into some trickiness when developing your code. When do you create the database entry? How do you avoid conflicts between developers? The biggest companies have solutions to these problems, and most everyone else just stays away. Incidentally, how do you avoid conflicts between developers? I'm not sure of the specifics, but I'm sure you can do it with more macros. You just need some way to flag to CI that it needs to generate a new value. The obvious way would be to commit with your id macro usage not expanded to a template string, but if macrome is watching it would just reexpand it. Instead you could do something like id.dev`15`. The macro would then have different behavior on CI: behaving as if no id had yet been provided and also rewriting itself to import the id macro in place of id.dev. This is the power of having the code itself be the source of truth.

Finally though it doesn't so much involve macros, imagine the power of a re-entrant code generator for documentation. Let's say you have your README.md file. You tell macrome to process it with a re-entrant embedding generator. Now you can write this:

# Usage
```js
\\ ./tests/basic-usage.test.js
```

And once you save the file macrome will go and dig up the real content of that file and paste it in there for you. Now you have documentation that is also test code! This implies some things I have to think further about though in terms of how macrome might allow generators to specify dependencies, i.e. I need to be rebuilt if one of my dependencies does. Fortunately I think I already need such a system to support watching configs, as I want to be able to create a tool that is not prone to needing to be restarted.

[kentcdodds]

Hey @conartist6, I'm afraid I don't have time to bring myself up to speed with what your proposing. Can you tersely describe the feature you need and the amount of complexity you think it will add to the codebase?

[conartist6]

Oh I don't need anything really. I'm just thinking out loud, stating a direction I'd like to head and some reason behind it. If I recall the means for communication with macros evaluators is already there. I've fed boolean variables into macros and I don't see why I couldn't feed in a configured callback.

[kentcdodds]

Ah, gotcha. Sounds awesome!