Will modules facilitate partial classes?

[Levi-Lesches]

From #1749 (comment):

For me it is seems logical that for friends all visibility related features work as they were the same module - not just access, but also overrides. In particular because access and overrides are the same feature - building the interface of a class.

Would this discussion benefit "partial classes" (#252)? Previously, a few questions came up in discussion and in my head:

1. How would the compiler know when to stop looking for more partial declarations?
2. Where should the "canonical" declaration go? In other words, what do you import?
3. Which libraries should get to reference private members? What if each partial declaration has its own private members?

Modules would answer all of those questions: the partial class may be split between different libraries, but the class as a whole would belong to the module. I saw that partial classes are not yet on the roadmap, maybe this can change that? Not to mention, both partial classes and modules are a huge step towards metaprogramming. Thoughts?

[munificent]

Modules would answer all of those questions: the partial class may be split between different libraries, but the class as a whole would belong to the module.

That sounds right to me too.

I saw that partial classes are not yet on the roadmap, maybe this can change that?

We're currently investigating static metaprogramming and macros. Macros would let you programmatically introduce new members right into a class declaration, so, if anything, that probably reduces the desire for partial classes. I think we'll want to see how that pans out before we would consider also adding partial classes to the language.

[Levi-Lesches]

Macros would let you programmatically introduce new members right into a class declaration

What's the currently proposed way of doing that? Are the members being inserted directly into the AST/kernel/etc, or are they being generated as source code somewhere? The need for partial classes arise when you consider the features a lot of people want with metaprogramming (and is why I'm interested in them in the first place):

• all generated code should be inspectable. That means by a human as well as the IDE's "goto" feature
• generated code should not override human-written code
• even stricter, generated code should not be emitted in the same file as human-written code

In my proposal at #1565, I use partial classes to accommodate for all of the above.

// person.dart -- handwritten
part "person.g.dart";

@data
partial class Person {
  final String name;
  final int age;

  String get greeting => "Hi, I'm $name and I'm $age years old";
}

// person.g.dart
part of "person.dart";

partial class Person {
  const Person({
    this.name,
    this.age,
  });
  
  @override
  String toString() => "Person($name, $age)";

  @override
  bool operator ==(Object other) => other is Person && other.name == name && other.age == age;
  
  Map toJson() => {
    "name": name,
    "age": age,
  };

  Person copyWith({String? name, int? age}) => Person(
    name: name ?? this.name, 
    age: age ?? this.age,
  );
}

The two definitions would be combined when compiled, but the source code remains separate so it's clear what was generated and what was written by a human. At least, that's the system I went with; I couldn't think of a better one that hits all the points above.

[munificent]

What's the currently proposed way of doing that? Are the members being inserted directly into the AST/kernel/etc, or are they being generated as source code somewhere?

With the macros proposal, they are generated programmatically. You write some Dart code that uses and API to build up objects representing code in memory and then use that same API to attach those pieces of code as new declarations in classes, libraries, etc. It looks a lot like reflection, except that it's all happening at compile time.

• all generated code should be inspectable. That means by a human as well as the IDE's "goto" feature

The developer experience is really important for macros. It's not fully fleshed out yet, but our expectation is that, yes, editors and IDEs will have some way to go to definition into code generated by macros. Probably this means dumped the programmatically generated code out to disk somewhere.

• generated code should not override human-written code

That's a general goal of macros, yes, but there are valid use cases for replacing or at least wrapping the hand-authored code. Drawing the line between "too magical" and "too limited" is hard.

• even stricter, generated code should not be emitted in the same file as human-written code

Since macros work programmatically, the generated code isn't really in "files" per se. The macro directly augments the declarations as first-class objects in memory.

I agree it's important that users can tell which code is hand-authored and which code is produced by macros, but physically separating them in multiple files is just a means to that end. As long as the user experience is clear, I'm not too picky about how bytes are arranged on disk.

[Levi-Lesches]

As long as the user experience is clear, I'm not too picky about how bytes are arranged on disk.

Right, if macros can be done well without literally generating code in files, I'd be all for that too. It's just that after discussion in four to five issues, I can't find another way to let people see and understand the generated code without actually getting to see it interact with the rest of their project. For the most part, people seem to have no problem understanding that importing data.dart will import their classes in that file, and they know where to look to find those definitions. Something similar like import or part with a .g,dart will let devs know where to look if they want to see all the generated definitions being used in that file, and they can compare the generated code against their hand-written approach. Again, that's just the established conventions. If there can be a new way of doing it that's just as good, then let's go with that. I'm just wondering what that will be.

Since macros work programmatically, the generated code isn't really in "files" per se. The macro directly augments the declarations as first-class objects in memory.

The developer experience is really important for macros. It's not fully fleshed out yet, but our expectation is that, yes, editors and IDEs will have some way to go to definition into code generated by macros. Probably this means dumped the programmatically generated code out to disk somewhere.

I'm probably reading it wrong, but it sounds like you're thinking that macros can work in the backend without generating code, but will dump their definitions as code when asked? Shouldn't we stick with code to simplify that process and make sure devs can always verify visually that they're producing the correct behavior? Also because if code is generated when the IDE asks, that puts more work on the back of the IDEs that now have to manually generate the code and know when to safely delete the generated files.

I get that keeping macros in the backend can be technically more efficient, but it might just be better for the dev experience if it were less magic and more like hand-written code.

[munificent]

I'm probably reading it wrong, but it sounds like you're thinking that macros can work in the backend without generating code, but will dump their definitions as code when asked?

Possibly, yes. I'm basically just saying that we don't know how macro output will be visualized, just that it's a requirement that it can be.

Shouldn't we stick with code to simplify that process and make sure devs can always verify visually that they're producing the correct behavior? Also because if code is generated when the IDE asks, that puts more work on the back of the IDEs that now have to manually generate the code and know when to safely delete the generated files.

Yeah, I think I am generally leaning towards something file-based too. I agree it makes a lot of stuff simpler (though it makes some other stuff harder).

[Levi-Lesches]

...though it makes some other stuff harder

Hopefully partial classes can help with those, as in my example above

[lrhn]

I'd be a little worried by having classes declared across multiple libraries.

Usually, so far, a class been declared and exported from a single library. If we allow different parts of a partial class to live in different libraries, then

• Do we want one of them to be primary and the rest just supplying parts, but not being visible in the export scope - or is the class visible in all the libraries?
• Let's assume a primary declaration and multiple additions. Then the primary export of the class must know all the additions in order to fully define the class. And the parts must point to the primary declaration they're adding to, so the libraries are definitely strongly connected. (If we don't have a primary, I think it's going to have to be strongly connected anyway).
• The different parts can have incompatible private names (that's a feature, and nothing new, you get the same by extending a class from another library).

Nothing insurmountable, I'm sure we can make that work. I'm just not sure the complexity is worth the effort.
If the libraries have to be strongly connected anyway , I'm not sure I see the advantage over just requiring all the parts to be in the same library. (But then, I'm not opposed to part files, I just want them to have individual imports too).

[Levi-Lesches]

If the libraries have to be strongly connected anyway, I'm not sure I see the advantage over just requiring all the parts to be in the same library.

The motivation is code generation. With the constraint of generating code in a separate file, I tried to work out what that would look like when adding members to classes, and I experimented with extensions, mixins, subclasses, but nothing actually works. Extensions can't declare fields or constructors nor can they override members. A cannot mix-in B if B is declared on A. And subclassing doesn't actually augment the existing class, and _$ syntax doesn't feel natural. You can see the Q&A section of #1565 for my suggestion of how partial classes can make code generation much simpler.

• Do we want one of them to be primary and the rest just supplying parts, but not being visible in the export scope - or is the class visible in all the libraries?
• Let's assume a primary declaration and multiple additions. Then the primary export of the class must know all the additions in order to fully define the class. And the parts must point to the primary declaration they're adding to, so the libraries are definitely strongly connected. (If we don't have a primary, I think it's going to have to be strongly connected anyway).
• The different parts can have incompatible private names (that's a feature, and nothing new, you get the same by extending a class from another library).

I tried to answer that in the top comment, but of course I wouldn't know the exact implementation details:

Modules would answer all of those questions: the partial class may be split between different libraries, but the class as a whole would belong to the module.

So:

• There would be no primary class, the class declaration is equally split among all partial declarations with the same name
• You'd import the class by importing the module as a whole. If modules don't end up allowing that, then I suppose any import of one of the partial declarations would search through the rest of the module to compile the class. So if Foo is partially declared in a.dart and b.dart, which are both part of the data module, then any of these could be made to import the same class.
  • import "package:myProject/data";
  • import "package:myProject/data/a.dart";
  • import "package:myProject/data/b.dart";
• This depends on how private variables are ultimately handled in modules, so like you said it would probably be the same as extending.

[munificent]

If the libraries have to be strongly connected anyway , I'm not sure I see the advantage over just requiring all the parts to be in the same library. (But then, I'm not opposed to part files, I just want them to have individual imports too).

The main advantage is that the generated library can have its own encapsulated imports which the main library doesn't need to worry about.

[lrhn]

The main advantage is that the generated library can have its own encapsulated imports which the main library doesn't need to worry about.

I'd just let parts have that too.

[munificent]

I'd just let parts have that too.

Heh, at that point, I think we're just arguing about nomenclature. A "part file" that has its own top level namespace and imports is what I'd call a "library". :)

[lrhn]

Yes, if the "part of" library exports and imports that "library", and they're able to share private names, and both contribute to the same partial class (so they'd have to be strongly connected). That's what I would call "one library".

Using parts only you can't have two classes with the exact same name, and you can't do conditional imports on parts.Other than that, nothing prevents a package from being one big implementation library, spread into multiple parts, and one public API library exporting the public parts. And then we're back full-circle to defining compilation units/modules as sets-of-libraries or single libraries as sets-of-parts.
The single library with sets of parts only has one entry point. That can be good or bad.

Definitely nomenclature bike-shedding. We just need a list of what such two files can do together, and what otherwise unrelated libraries cannot, then we can decide which fits best.