New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Static Metaprogramming #1482
Comments
Would static function composition be in the scope of this feature? At the moment, higher-order functions in Dart are fairly limited since they require knowing the full prototype of the decorated function. An example would be a void Function() debouce(Duration duration, void Function() decorated) {
Timer? timer;
return () {
timer?.cancel();
timer = Timer(duration, () => decorated());
};
} which allows us to, instead of: class Example {
void doSomething() {
}
} write: class Example {
final doSomething = debounce(Duration(seconds: 1), () {
});
} but that comes with a few drawbacks:
With static meta-programming, our class Example {
@Debounce(Duration(seconds: 1))
void doSomething() {
print('doSomething');
}
@Debounce(Duration(seconds: 1))
void doSomethingElse(int value, {String named}) {
print('doSomethingElse $value named: $named');
}
} |
There is a delicate balance re: static function composition, but there are certainly many useful things that could be done with it. I think ultimately it is something we would like to support as long as we can make it obvious enough that this wrapping is happening. The specific balance would be around user confusion - we have a guiding principle that we don't want to allow overwriting of code in order to ensure that programs keep their original meaning. There are a lot of useful things you could do by simply wrapping a function in some other function (some additional ones might include uniform exception handling, analytics reporting, argument validation, etc). Most of these things would not change the meaning really of the original function, but the code is being "changed" in some sense by being wrapped. Ultimately my sense is this is something we should try to support though. I think the usefulness probably outweighs the potential for doing weird things. |
I like Lisp approach (in my opinion, the utmost language when it comes to meta-programming). Instead of defining a |
For something like debounce, a more aspect-like approach seems preferable. Say, if you could declaratively wrap a function body with some template code: class Example {
void doSomething() with debounce(Duration(seconds: 1)) {
print('doSomething');
}
void doSomethingElse(int value, {String named}) with debounce(Duration(seconds: 1)) {
print('doSomethingElse $value named: $named');
}
}
template debounce<R>(Duration duration) on R Function {
template final Stopwatch? sw;
template late R result;
if (sw != null && sw.elapsed < duration) {
return result;
} else {
(sw ??= Stopwatch()..start()).reset();
return result = super;
}
} This defines a "function template" (really, a kind of function mixin) which can be applied to other functions. (Maybe we just need AspectD for Dart.) |
But an important part of function composition is also the ability to inject parameters and ask for more parameters. For example, a good candidate is functional stateless-widgets, to add a @statelessWidget
Widget example(BuildContext context, {required String name}) {
return Text(name);
} and the resulting prototype after composition would be: Widget Function({Key? key, required String name}) where the final code would be: class _Example extends StatelessWidget {
Example({Key? key, required String name}): super(key: key);
final String name;
@override
Widget build(BuildContext) => originalExampleFunction(context, name: name);
}
Widget example({Key? key, required String name}) {
return _Example(key: key, name: name);
} |
I definitely agree we don't want to allow for changing the signature of the function from what was written. I don't think that is prohibitive though as long as you are allowed to generate a new function/method next to the existing one with the signature you want. The original function might be private in that case. |
That's what functional_widget does, but the consequence is that the developer experience is pretty bad. A major issue is that it breaks the "go to definition" functionality because instead of being redirected to their function, users are redirected to the generated code It also causes a lot of confusion around naming. Because it's common to want to have control on whether the generated class/function is public or private, but the original function to always be private. By modifying the prototype instead, this gives more control to users over the name of the generated functions. |
Allowing the signature to be modified has a lot of disadvantages as well. I think its probably worse to see a function which is written to have a totally different signature than it actually has, than to be navigated to a generated function (which you can then follow through to the real one). You can potentially blackbox those functions in the debugger as well so it skips right to the real one if you are stepping through. |
I suppose this will allow generating |
Yes. |
@tatumizer This issue is just for the general problem of static metaprogramming. What you describe would be one possible solution to it, although we are trying to avoid exposing a full AST api because that can make it hard to evolve the language in the future. See https://github.com/dart-lang/language/blob/master/working/static%20metaprogramming/intro.md for an intro into the general design direction we are thinking of here which I think is not necessarily so far off from what you describe (although the mechanics are different). |
Great intro & docs. Hopefully we'll stay (far far) away from annotations to develop/work with static meta programming?! |
The main reason we use this as an example is its well understood by many people, and it is also actually particularly demanding in terms of features to actually implement due to the public api itself needing to be generated :).
Can you elaborate? Default values for parameters are getting some important upgrades in null safe dart (at least the major loophole of being able to override them accidentally by passing |
I believe the issue is that we cannot easily differentiate between freezed supports this, but only because it relies on factory constructors and interface to hide the internals of |
Right, this is what I was describing which null safety actually does fix at least partially. You can make the parameter non-nullable (with a default), and then null can no longer be passed at all. Wrapping functions are required to copy the default value, basically it forces you to explicitly handle this does cause some extra boilerplate but is safe. For nullable parameters you still can't differentiate (at least in the function wrapping case, if they don't provide a default as well) |
Metaprogramming is a broad topic. How to rationalize? We should start with what gives the best bang for buck (based on use cases). Draft topics for meta programming 'output' code:
Also on output code:
Would be great if this could work without saving the file, a IDE-like syntax (hidden) code running continuously if syntax is valid. I refuse to use build_runner's |
Metaprograming opens doors to many nice features Other language that does a great job at implementing macros is Haxe you can use Haxe language to define macros I guess there are many challenges to implement this. |
can we extend classes with analyzer plugin? |
I'm not sure if I like the idea having this added to Dart because the beauty of Dart is its simplicity. The fact that it isn't as concise as other languages it in reality an advantage because it makes Dart code really easy to read and to reason about. |
I agree with this. |
@escamoteur Writing less code does not make it more complicated necessarily. It can, I agree, if someone does not fully understand the new syntax. But the trade-off is obvious: time & the number of lines saved vs the need for someone to learn a few capabilities. Generated code is normal simple code. I just suggested real-time code generation instead of running the builder every time or watching it to save. That way you get real time goto. But if you are using notepad then of course you need to run a process. |
Just to be 100% clear, we are intensely focused on these exact questions. We will not ship something which does not integrate well with all of our tools and workflows. You should be able to read code and understand it, go to definition, step through the code in the debugger, get good error messages, get clear and comprehensible stack traces, etc. |
In my honest opinion: things must be obvious, not magical.
^ this |
But there is nothing beautiful about writing data classes or running complicated and and slow code-generation tools. I'm hoping this can lead to more simplicity not less. Vast mounds of code will be removed from our visible classes. StatefulWidget can maybe just go away? (compiler can run the split macro before it builds?). Things can be auto-disposed. Seems like this could hit a lot of pain points, not just data classes and serialization.. |
Since dart currently offers code generation for similar jobs-to-be-done, I'd suggest evaluating potential concerns with that consideration:
On the other hand, besides being an upgrade from codegen for developers, metaprogramming could provide healthier means for language evolution beyond getting data classes done. Quoting Bryan Cantrill:
PS @jakemac53 the |
this would be fantastic if it allowed, the longed-for serialization for JSON natively without the need for manual code generation or reflection in time of execution Today practically all applications depend on serialization for JSON, a modern language like dart should already have a form of native serialization in the language, being obliged to use manual codegen or typing serialization manually is something very unpleasant |
My approach on a macro mechanism. Basically tagging a certain scope with a macro annotation, that refers to one or multiple classes to 1:1 replace the code virtually... like a projection. It's very easy to understand and QOL can be extended by providing utility classes. #ToStringMaker() // '#' indicates macro and will rewrite all code in next scope
class Person {
String name;
int age;
}
// [REWRITTEN CODE] => displayed readonly in IDE
// class Person {
// String name;
// int age;
//
// toString() => 'Person(vorname:$name, age:$age)'
// }
class ToStringMaker extends Macro {
// fields and constructor can optionally obtain parameters
@override
String generate(String code, MacroContext context) { // MacroContext provides access to other Dart files in project and other introspection features
var writer = DartClassWriter(code); // DartClassWriter knows the structure of Dart code
writer.add('String toString() => \'${writer.className}(${writer.fields.map(field => '${field.name}:\${field.name}').join(', ')})\'');
return writer.code; // substitute code for referenced scope
}
} |
Unfortunately this does not solve one of the core use cases for the |
@jakemac53 Got it; now that I think of it, I would have a need for that as well (specifically, to import a flutter dependency that is guaranteed to be transitive, since my package will depend on flutter). I was thinking of just forcing users to Will what I gave in the sample above (i.e., getting an |
Regarding getting access to all identifiers in scope, I think the way I would approach that is through a general mechanism for introspecting on libraries. However, providing access to the import scope is actually a significantly more complex issue than it appears on the surface. In general we have a carefully structured set of phases, which restrict the information in a given phase to that which was produced in previous phases only. But, library imports could actually be added in any phase, since they are implicitly added whenever an identifier is included in generated code. Since you can produce code which references identifiers in any phase, we could never expose the set of available libraries in a complete manner. It is possible that we could provide access to only the hand-authored library imports, but it gets quite complex to try and do so, because library augmentations are very intertwined with the libraries themselves (and I think in practice the implementations are likely to just modify in place the original library as things are added to it). I do think that in general it is likely we end up with some form of library level introspection though, for instance a |
@GregoryConrad it might be useful to understand exactly what you want to achieve in order to see if there is some solution that we could come up with to satisfy the constraints, that is compatible with the phased approach. |
@jakemac53 Sure, I've got a package I'm working on that creates unnested widget trees. As far as I have been able to tell, it is entirely possible to do with this macros proposal today given some way to get an Widget build(BuildContext context) {
return Unnested()
.container(color: Colors.red)
.sizedBox_shrink()
.listView(...);
}
Widget build(BuildContext context) {
return Unnested()
.padding(padding: const EdgeInsets.all(4))
.myCustomWidget(someParams: fooBar)
.end();
} The above relies upon a basic config file, i.e.: import 'package:flutter/material.dart';
import 'package:my_app/my_custom_widget.dart';
// other imports...
@unnested
class Unnested {} Another possible implementation of this config file could be the following: @unnested(imports: ['package:flutter/material.dart', 'package:my_app/my_custom_widget.dart'])
class Unnested {} I have no preference one way or the other. I just would like this to be possible seeing as how convenient this package would be, especially since it would play with macros perfectly. I can imagine use cases for other macros too. Say a package provides some functionality, but requires a user to define some types to work with the package (perhaps a database package that serializes data classes). A macro could be made that adds functionality to the macro's calling library based on all of the available types. You might say that a macro could just be applied to the type definition itself, but what happens if the type is defined in a different package? Then that is not possible and the only way for a macro to know about the type is to get it in a list of possible identifiers, since you can't assume the macro's package will depend on every other possible package that exposes types that may be relevant. |
I am not super familiar with flutter so correct me if I am misunderstanding anything, but basically this is looking for any widgets defined in any imported library, and then generating a method in the So you would need access to the list of imported libraries in the declarations phase (phase 2). At that point the types all exist already so you could see them in the libraries, and the only question really is how you should get access to the imported libraries themselves. So the following would be able to work feasibly (given some sort of library introspection mechanism by uri): import 'package:flutter/material.dart';
import 'package:my_app/my_custom_widget.dart';
// other imports...
// Not ideal that you have to duplicate these
@unnested(imports: ['package:flutter/material.dart', 'package:my_app/my_custom_widget.dart'])
class Unnested {} Or alternatively, you could use extensions on // You would annotate each library import
@unnested()
import 'package:flutter/material.dart';
@unnested()
import 'package:my_app/my_custom_widget.dart';
// other imports...
// Generated extension methods look like this:
extension UnnestedMaterial on Unnested {
Unnested padding(...) { ... }
} |
Correct; there will be one method created per Widget constructor.
Exactly what I was thinking! Thus why I brought up the 2 possible APIs returning Either option you gave above seems reasonable to me. As long as one of them is supported in the final implementation of macros, I'd be happy! |
Sounds good, I can't make any promises, but I filed an issue to try and capture the requirements so I don't forget about it. |
Was looking at the proposed API again; think I noticed some missing functionality.
/// The base class for all declarations.
abstract class Declaration {
/// An identifier pointing to this named declaration.
Identifier get identifier;
} Shouldn't any Concretely, say I have the following: class Annotation {
const Annotation(Object obj);
}
const _someObj = Object();
void fn(
@Annotation(_someObj) int someIntParameter,
) {} I'd expect |
The metadata annotation API is not yet designed, #1930. It is going to be tricky and likely have some restrictions, because of how it will interact with phases. Specifically metadata which themselves rely on macros (even transitively) need to be thought about carefully (or banned). |
Yes ever since the decision to focus on soundness, records, patterns, and class modifiers, all work on macros has been on hold in order to deliver those features instead. They were deemed easier and more likely to actually ship in time for Dart 3, and we wanted to ensure we were providing some concrete user value in Dart 3 along with the switch to no longer supporting Dart <2.12 (ie pre-null safe code). I personally have spent almost the entire last year just preparing our internal codebase for null safety by assisting internal teams with their migrations. However, we do plan to pick back up work soon on macros 👍 . |
is it possible to test macros in web apps? |
No - it isn't possible to use them on any platform really right now (for a bit you could technically get the VM to work for certain macros, but that has bit rotted a bit, hopefully should be back soon). |
Depends what you mean, but it is very unlikely they would ship to stable in 2023. Possibly in an experimental form, but there are no specific dates. This is a feature that we have to get right the first time, and it will likely exist in an experimental state for a while. |
I prepare for this feature along time ago. Static metaprogramming will be the game changer. |
is it ready for experimentation? 🙏🏻 |
The feature has ten times the thumbs up then any other feature on the roadmap. It has been 2 and a half years. Hope we have something soon as this is really what Dart needs to overcome it's shortcomings (missing reflection, data classes etc.) |
You are absolutely right. I've been building just average-size apps with Flutter and i like the framework really really much with the control you have, but every time i wait half a minute for the build_runner just to build my models i ask myself why i did not simply choose Kotlin for my project. The Dart dx is horrible atm. Macros would help with a lot of problems. |
Yeah I agree with you, Dart should have this feature along time ago. I work with Java also and Java Annotation in Spring Boot surprises me alot. I start looking for Dart Annotation too but what I found is build_runner, which is too slow for code generation. Hope Dart team will make this work! |
I don't think that this should improve source generation time that much, at least I don't expect that from the start. We need this to make it easy to automate stuff that is hardcoded today. |
AFAICR it was suggested that the time of generation would be way quicker than what we have today, and this is probably the major complain from code generation users.
We can already do it with current |
Is it possible to augment a class or hook a function call? |
I don't know what you mean by "hook a function call". Regarding "augment a class", augmentations are a distinct feature from static metaprogramming, although it will be widely used by it, so it's out of the scope of what I was saying. Currently, we can use part files, which are more rudimentary, but in practical effects can achieve the same, although more verbosely. |
If I understood correctly, with augmentation you don't have to extend a class to add stuff to it. Hooking a function call is add a layer before or after a function execution with access to variables. |
Exactly. With part files we can do it by extending/mixing a class. So for practical purposes we can do the same. Although, as I said, augmentations are a different feature.
This is very doable with current language by overriding and calling super.
Sure, but it does not add anything that we can't do, it only improves things that we can already do. |
I kindly ask you to no pollute the thread as you do not understand augmentation in static metaprogramming. Static metaprogramming enables things that are NOT possible with build_runner, so I hope it will be added soon. |
Metaprogramming refers to code that operates on other code as if it were data. It can take code in as parameters, reflect over it, inspect it, create it, modify it, and return it. Static metaprogramming means doing that work at compile-time, and typically modifying or adding to the program based on that work.
Today it is possible to do static metaprogramming completely outside of the language - using packages such as build_runner to generate code and using the analyzer apis for introspection. These separate tools however are not well integrated into the compilers or tools, and it adds a lot of complexity where this is done. It also tends to be slower than an integrated solution because it can't share any work with the compiler.
Sample Use Case - Data Classes
The most requested open language issue is to add data classes. A data class is essentially a regular Dart class that comes with an automatically provided constructor and implementations of
==
,hashCode
, andcopyWith()
(calledcopy()
in Kotlin) methods based on the fields the user declares in the class.The reason this is a language feature request is because there’s no way for a Dart library or framework to add data classes as a reusable mechanism. Again, this is because there isn’t any easily available abstraction that lets a Dart user express “given this set of fields, add these methods to the class”. The
copyWith()
method is particularly challenging because it’s not just the bodyof that method that depends on the surrounding class’s fields. The parameter list itself does too.
We could add data classes to the language, but that only satisfies users who want a nice syntax for that specific set of policies. What happens when users instead want a nice notation for classes that are deeply immutable, dependency-injected, observable, or differentiable? Sufficiently powerful static metaprogramming could let users define these policies in reusable abstractions and keep the slower-moving Dart language out of the fast-moving methodology business.
Design
See this intro doc for the general design direction we are exploring right now.
The text was updated successfully, but these errors were encountered: