Create Static Enough Metaprogramming proposal

[mraleph]

This moves content from #4271 into a markdown file in the repository to make discussion and revisions easier.

I have incorporated some of the feedback from discussions on the issue - but I continue to maintain focus on this as a toolchain feature. I have added some remarks that analyzer can't constant fold everything anyway because it does not have access to the compilation environment.

I would like to collect a few rounds of feedback and then rejuvenate the prototype implementation to get something experimental working across all platforms in the SDK so that we can have an idea of how well this could work in a real world.

[eernstg]

LGTM. (Very interesting indeed!)

I could suggest the addition of 'a' or 'the' in many locations, but decided that this wouldn't be the top priority at this time.

[jakemac53]

This definitely does drastically simplify the proposal - but it also makes it a lot less useful. Ultimately trying to accomplish this was the downfall of the original macros proposal though.

But, I think it is important to look at the original motivations for macros, and which of those use cases are covered by this.

Ultimately, I think you can end up with some fairly decent solutions for automatic encoding/decoding, equality, toString, etc. But not copyWith or constructors due to the signatures being dependent on the shape of the class (you could generate the bodies of these but that's only half the boilerplate).

Edit: I see you have an interesting copyWith idea using records, that is fairly reasonable actually.

[mraleph]

I have added a section trying to cover some of the use cases mentioned peaked from an old spreadsheet I managed to find.

I could not exactly figure out what some of those entail, could you help me a bit (look for TODO: unclear what this means. and if you have context please add it in the comment). I am interested about auto listenable and render accessors. I think union types (ala freezed) means more concise way of declaring sealed class hierarchies - which I think should just be its own feature (paired with primary constructors).

I think a number of use cases require property wrappers and/or ability to redirect methods - this should probably be its own feature. If property wrappers are available then @konst can be used to handle the boilerplate.

Finally some cases (e.g. proxies, functional widgets and reduced widget classes boilerplate) require ability to create class declarations. I think it is actually an okay feature to give via @konst reflection. As long as classes are anonymous and are otherwise not visible.

[jakemac53]

What is the baseline?

[mraleph]

I don't have numbers right now but I can try to revive the prototype and update this later.

[jakemac53]

This would be 🔥 (and help a lot with the reduction of boilerplate)

[mraleph]

Yeah, it would be useful in many places. Also it would solve a performance problem we occasionally encounter where inheritance or generics cause polymorphism and severely reduce performance.

[mraleph]

I have made some updates based on comments. PTAL.

[munificent]

I still have the same reservations as before, but landing this in the repo so we can discuss more sounds great. It's a very solid well-thought out proposal and the background information it provides is really helpful for any metaprogramming-related features we might do.

[osa1]

I'm confused about how @konst applies to statements and functions. Given this:

When @konst is applied to loop iteration variables it instructs the compiler to expand the loop at compile time by first computing the sequence of values for that iteration variable, then cloning the body for each value in order and substituting iteration variable with the corresponding constant.

(in particular, "expand the loop at compile time" suggests code generation based on constant values)

I would expect this function

void bar<@konst T>(@konst T v) {

}

to be monomorphised based on Ts passed. (and generate one more copy for non-const T)

Otherwise when I do things like this code in the JSON examples:

Map<String, Object?> toJson<@konst T>(T value) => {
    for (@konst final field in TypeInfo.of<T>().fields)
      if (!field.isStatic) field.name: field.getFrom(value),
  };

The loop cannot be unfold at compile time, because the unfolded loop needs to be different for each T.

But the text doesn't mention monomorphisation at all. Am I confused?

If this allows monomorphisation (and it has to, if I get it right), then that's a pretty big capability for the langauge that's also worth mentioning.

[osa1]

Shouldn't the Object? here be TypeInfo<Object?> (or TypeInfo, TypeInfo<dynamic>)? Can the field types be anything else other than a TypeInfo?

[mraleph]

You mean FieldInfo<T, TypeInfo<Object?>>? That would not work for the intended purpose, we would like use these type-arguments to ensure proper typing of access operations.

final class FieldInfo<HostType, FieldType> {
  @konst external FieldType getFrom(HostType o);
  @konst external void setOn(HostType o, FieldType fieldValue);
}

If you have field String x in class C its FieldInfo will have reified type FieldInfo<C, String>, which guarantees type safety of fi.getFrom(...) and fi.setOn(...).

[osa1]

I also find it confusing to see what function calls (or other syntax) is part of the generated code vs. compile-time evaluation. For example, if I change the toJson example from the proposal to this:

Map<String, Object?> toJson<@konst T>(T value) => {
    for (@konst final field in TypeInfo.of<T>().fields)
      if (!field.isStatic) processFieldName(field.name): processFieldValue(field.getFrom(value)),
  };

(processFieldName and processFieldValue are new)

Just by looking at this code (without seeing processFieldName and processFieldValue definitions) I can't tell whether these function calls are evaluated in compile time or runtime.

I wonder if we could make stages explicit. There are many languages that do this that we could look for inspiration. For example Common Lisp's backquote and comma, MetaOCaml's (.< expr >.) and escape (.~expr) make the stages explicit.

[osa1]

Something else that I just realized is that if you evaluate some code in runtime in some cases and compile time in others, you can get runtime error in some cases and compile-time error in others.

For example, consider this silly code: (modified from the operator == example in the proposal)

mixin DataClass<@konst T> {
  @override
  blah(Object? other) {
    final typeInfo = TypeInfo.of<T>();
    for (@konst final field in typeInfo.fields) {
      final value2 = field.getFrom(other as T);
      ...
    }
    return true;
  }
}

Here the line other as T will fail then other's type is not right, but when the failure happens will depend on when you evaluate other as T.

Does the proposal address this? (sorry if I can't find it..)

[mraleph]

@osa1

But the text doesn't mention monomorphisation at all. Am I confused?

If this allows monomorphisation (and it has to, if I get it right), then that's a pretty big capability for the langauge that's also worth mentioning.

There is specifically this paragraph which talks about it though it does not explicitly call it monomorphisation.

When @konst is applied to parameters (including type parameters) it turns
functions into templates: compiler will require that annotated parameter is a
constant known at compile time and clone the function for a specific combination
of parameters. The original function is removed from the program: it is
impossible to invoke it dynamically or tear it off. To annotate this as
@konst developer will need to place @konst on the declaration of the
function itself.

I also find it confusing to see what function calls (or other syntax) is part of the generated code vs. compile-time evaluation.

That's fair, though its somewhat intentional:

1. to avoid any syntactic changes altogether;
2. to keep code runnable as-is in environments which choose to not implement this feature;

I am open for suggestions on how to make this better if you have any ideas. I think it would need some syntactic designator, but I don't immediately see anything that would be very readable.

I wonder if we could make stages explicit.

I think MetaOCaml's or other similar systems approach (Template Haskell, Scala macros, etc) are somewhat different because they operate on pieces of code into which you splice other pieces of code (quotes and splices are two building blocks), so syntax is naturally very clear around what is code and what is a hole in that code into which you can splice more code.

This proposal describes (optional) compile time specializaton - so I think the fact that syntax looks the same is kinda by design.

Something else that I just realized is that if you evaluate some code in runtime in some cases and compile time in others, you can get runtime error in some cases and compile-time error in others.

Yep, I think this is by design as well. Though maybe not written as explicitly.