Structuring win32 package types

[timsneath]

With growing support for Win32, COM and WinRT types, this package could become cumbersome: either by exporting so many types that Dart analysis is compromised, or because too many changes lead to a breaking change that requires a major version bump. I'm therefore opening this issue to discuss a more radical overhaul of the package structure.

Fortunately, Dart has good support for a hierarchy of dependent packages, so it would be possible to split it up with little fuss.

The Win32 APIs are somewhat difficult to separate out, unfortunately. They have lots of interdependencies and are designed without hierarchy. The philosophy so far has been to be somewhat selective in which APIs are added, with the focus on core APIs (user32, kernel32) that are likely to be used by other Dart apps, rather than including obsolete APIs like fax services or XPS).

The code generator tooling (tools\generator) could be extracted into a separate package; perhaps it could even support build_runner rather than being a batch file.

The WinRT APIs are highly structured and hierarchical, with few interdependencies between each other. It's possible to imagine a separate package for each of the top-level namespaces (e.g. Windows.Storage), each of which depends on the core win32 library and perhaps the Windows.Foundation package.

Curious on your thoughts, @halildurmus?

[halildurmus]

With growing support for Win32, COM and WinRT types, this package could become cumbersome: either by exporting so many types that Dart analysis is compromised, or because too many changes lead to a breaking change that requires a major version bump. I'm therefore opening this issue to discuss a more radical overhaul of the package structure.

Yep, it's getting out of hand.

The Win32 APIs are somewhat difficult to separate out, unfortunately. They have lots of interdependencies and are designed without hierarchy.

Does this also apply to COM APIs? Will this package only expose Win32 and COM APIs after we separate out WinRT APIs?

The code generator tooling (tools\generator) could be extracted into a separate package; perhaps it could even support build_runner rather than being a batch file.

That would be great!

I imagine every package for WinRT top-level namespace would specify which types (Class, Interface, Enum, and Struct) it wants to be generated and the generator would use that information instead of using winrt_ prefixed JSON files inside the data\. The same thing can also be done for Win32 and COM APIs. The com_types.json file and win32_ prefixed JSON files would be specified in this package. One more thing to consider, do we do the same for exclusions (lib\src\model\exclusions.dart)? Does any of this make sense?

The WinRT APIs are highly structured and hierarchical, with few interdependencies between each other. It's possible to imagine a separate package for each of the top-level namespaces (e.g. Windows.Storage), each of which depends on the core win32 library and perhaps the Windows.Foundation package.

I love this idea.

Do you imagine these packages will be in a single repository (winrt?) or a repository will be created for each one (windows-foundation, windows-storage etc.)?

[timsneath]

COM APIs

Does this also apply to COM APIs? Will this package only expose Win32 and COM APIs after we separate out WinRT APIs?

If we start with the package consumer as the target audience, there are probably two principles at play:

• As a package consumer, I want to directly depend on smaller packages (low perf impact, fewer breaking changes)
• As a package consumer, I want to easily discover packages, and intuit which package contains which API

If we can achieve these goals while also broadening the range of modern APIs exposed, that would be excellent.

The challenge with Win32/COM is that there's no built-in namespace. The win32metadata has attempted to retrofit a namespace, but it's too broad to be useful (for example, Win32.System.* includes everything from kernel APIs to IIS to HoloLens). There is also no attempt to partition based around dependencies, so a System API may also depend on APIs from the Win32.Graphics, Win32.Devices and Win32.Shell namespaces :(

I tried at one point to export everything (the old "v3" constants in the generator were vestiges of that approach), but that was unmanageable. And selectively exporting based on namespace never generated all the dependencies. That's why I've reverted back to the current approach of curating which APIs are exposed based on usage. I'm less worried about Win32 APIs, because there are fewer of them, but COM is certainly a problem here. At some point, I'd like to use the metadata to generate a true dependency chain, which might allow a more automated approach, but that's a way off.

So in the interim, my suggestion would be that we:

• Just include Win32 C-style APIs in the core package, since lots of them have broad applicability
• Be selective about which COM APIs we expose in the core package, based on real-world needs
• Prefer WinRT APIs over COM APIs as your work matures and becomes generally-applicable

Open to other ideas, of course.

Packages

pub.dev

In terms of pub.dev, I propose we'd have the following packages:

• win32 (the core package, just C-style and perhaps COM types)
• wingen (the generator\ package, roughly mapping to ffigen)
• windows_graphics, windows_gaming, windows_storage, etc. (one for each of the 19 WinRT top-level namespaces, possibly with the exception of Windows.Foundation?)

As a package consumer, you'd either take a dependency on win32 or windows_* (with the latter having a transitive dependency on win32).

Rather than publishing them from the sneath.org verified publisher, we'd publish them from a new win32.pub publisher that would have multiple owners (the two of us for now). I'd transfer ownership of the existing package to that publisher. (The longer-term plan is that pub.dev publishing becomes a GitHub Action, which reduces some of the toil of managing ~20 packages.)

GitHub repo

Do you imagine these packages will be in a single repository (winrt?)

I suppose there are two choices:

1. We keep them all together in a single repo, as with https://github.com/flutter/plugins, which makes it easier to make atomic changes that affect multiple packages (e.g. a change to the generator that impacts other packages).
2. We separate them out, perhaps with you owning the WinRT repos and me owning the Win32 repos?

What's important to me is that you get the appropriate credit. It's not lost on me that today the repos and packages are listed under my name, but realistically you're doing a huge amount of the work and I want that to be reflected appropriately. Basically, I want you to feel rewarded for working on this, and I'm realizing that I don't know enough about your motivations! This started as a lightweight hobby project for me, but you're bringing a tremendous amount of professionalism to it and it's a huge encouragement to me!

[halildurmus]

So in the interim, my suggestion would be that we:

* Just include Win32 C-style APIs in the core package, since lots of them have broad applicability

* Be selective about which COM APIs we expose in the core package, based on real-world needs

* Prefer WinRT APIs over COM APIs as your work matures and becomes generally-applicable

I absolutely agree with these.

In terms of pub.dev, I propose we'd have the following packages:

* `win32` (the core package, just C-style and _perhaps_ COM types)

* `wingen` (the `generator\` package, roughly mapping to `ffigen`)

* `windows_graphics`, `windows_gaming`, `windows_storage`, etc. (one for each of the 19 WinRT top-level namespaces, possibly with the exception of `Windows.Foundation`?)

Awesome!

Rather than publishing them from the sneath.org verified publisher, we'd publish them from a new win32.pub publisher that would have multiple owners (the two of us for now). I'd transfer ownership of the existing package to that publisher. (The longer-term plan is that pub.dev publishing becomes a GitHub Action, which reduces some of the toil of managing ~20 packages.)

Yeah, we will definitely need to set up a GitHub Action at some point 😄.

GitHub repo

I suppose there are two choices:

1. We keep them all together in a single repo, as with https://github.com/flutter/plugins, which makes it easier to make atomic changes that affect multiple packages (e.g. a change to the generator that impacts other packages).

2. We separate them out, perhaps with you owning the WinRT repos and me owning the Win32 repos?

What's important to me is that you get the appropriate credit. It's not lost on me that today the repos and packages are listed under my name, but realistically you're doing a huge amount of the work and I want that to be reflected appropriately. Basically, I want you to feel rewarded for working on this,

I haven't really thought about it until now. I must admit, I like the idea of owning the WinRT repos myself. I think that would increase the sense of reward for working on it.

and I'm realizing that I don't know enough about your motivations! This started as a lightweight hobby project for me, but you're bringing a tremendous amount of professionalism to it and it's a huge encouragement to me!

Almost two and half years ago, I stumbled upon this project (I think it was in the summer of 2020). Up until that day, I didn't know FFI was a thing (a programming language can communicate with another language so cool!). The whole thing was so interesting for me. I wanted to learn more about this.

I was watching you evolve this project in your free time every weekend, trying to learn how these things work. I also read your posts on Medium about this, could feel your excitement about this. You have done multiple massive refactorings (e.g. switching from manually parsing IDL files to using the newly released Win32.winmd). I was looking forward to weekend to see what are you going to do this time 😅. It had become a weekly routine for me. I know you attempted multiple times to fix the WinRT picker. That also caught my eye, I was obsessed with it, wanted to figure it out. I tried to get it to work multiple times with no luck.

About 9 months ago, I decided to give it another shot. I think I looked into this 4 day back-to-back (5-6 hours in total). On the fourth day, I finally figured it out. I remember I was so happy that day, the sense of reward was massive. After that, I realized that with this Pointer bug fixed there are more opportunities to take this WinRT projection further,

And here I am. I must admit, I couldn't possibly imagine that I could contribute to it so much. The whole experience was so great, loved collaborating with you. I feel like I learned a lot during this time. I look forward to what we will accomplish this year 👀.

[timsneath]

I haven't really thought about it until now. I must admit, I like the idea of owning the WinRT repos myself. I think that would increase the sense of reward for working on it.

Done! That would be great: I'm keen that you get the fame you deserve for this!

I'm so touched by what you wrote at the end. It's so nice to hear your story of how you discovered this project. I know what you mean about the sense of reward: it feels quite groundbreaking to make Dart do something that nobody else has ever done before.

I got started on this because I was one of a few people who have a Windows development background in our team, and because I enjoy the puzzle of writing code even though my job doesn't directly call for it. But as this project has grown, it's taken on a bit of a life of its own, and I'm so thrilled to have a collaborator who is willing to co-own it. You have written code that I couldn't write, helped keep the quality bar high, and given me motivation to keep going. Thank you! And I agree -- 2023 might be the point where this project fully blossoms! Have a great new year.

[halildurmus]

Thank you, happy new year!

[timsneath]

Just to explore this a bit more, I created a test repo: https://github.com/timsneath/windows_devices. This contains just the contents of the Windows.Devices namespace. This depends on a private branch of win32 (v5), which has the requisite files removed.

The changes necessary for the windows_devices package are relatively benign - just changing imports.

The changes to the core package are a little more painful: mostly because it introduces a circular dependency, since IMapView and other foundation classes use a couple of classes from Windows.Devices (BatteryReport and PedometerStepKind, most prominently). Ideally we wouldn't have to depend on the child packages from the foundation package, but I don't know if that's possible.

[halildurmus]

The changes to the core package are a little more painful: mostly because it introduces a circular dependency, since IMapView and other foundation classes use a couple of classes from Windows.Devices (BatteryReport and PedometerStepKind, most prominently). Ideally we wouldn't have to depend on the child packages from the foundation package, but I don't know if that's possible.

I think you meant to write PedometerReading instead of BatteryReport no?

I think I could get rid of pedometerstepkind_helpers.dart and generalize IMap, IMapView, and IKeyValuePair interfaces to handle all enum type arguments (just like in IVector). I'm hoping to look into this today.

[halildurmus]

Fixed this in #638.

[timsneath]

I was actually thinking of BatteryReport, which was one of the classes that was missing when I removed all traces of Windows.Devices classes from win32. But it's referenced from Windows.Gaming.Gamepad, which would itself move to a separate package when this work is complete. So it's not truly a circular reference; the windows_gaming package would take a dependency on windows_devices, which is the same as in the Windows metadata:

(There are a few circular dependencies remaining, for example, Windows.Foundation depends on Windows.Storage, which in turn depends on Windows.Foundation, but circular package dependencies are not prohibited in Dart, so that's not a problem.)

[halildurmus]

pub.dev

In terms of pub.dev, I propose we'd have the following packages:

* `win32` (the core package, just C-style and _perhaps_ COM types)

* `wingen` (the `generator\` package, roughly mapping to `ffigen`)

* `windows_graphics`, `windows_gaming`, `windows_storage`, etc. (one for each of the 19 WinRT 
   top-level namespaces, possibly with the exception of `Windows.Foundation`?)

I made a lot of progress over the weekend in separating out the WinRT bits from this package and I have realized that it might be a good idea to split the generator package into two packages (win32gen for Win32 and COM APIs and winrtgen for WinRT APIs). winrtgen would depend on the win32gen package to gain access to the shared logic, and it would live inside the same folder as WinRT packages. WDYT?

[timsneath]

@halildurmus Firstly, an apology that I've been a bit radio silent these last couple of weeks. As you might have noticed, we've been a little busy with Flutter Forward!

Thrilled you've been making good progress in the meantime. And well done for doing some of the necessary surgery to implement this model.

I don't mind where it resides, but I wonder if it's a bad idea to separate win32gen and winrtgen so completely. They seem very tightly coupled (often a change needs to be made across both parts, due to the class inheritance), and I worry that it might require a lot of coordination to keep two co-dependent packages in sync with one another. Can we find a common home for them in a single package?

[halildurmus]

@halildurmus Firstly, an apology that I've been a bit radio silent these last couple of weeks. As you might have noticed, we've been a little busy with Flutter Forward!

No apology is needed. I already figured that you were probably very busy with the Flutter Forward event which was great BTW (thanks to the great work of the entire Dart/Flutter team).

Thrilled you've been making good progress in the meantime. And well done for doing some of the necessary surgery to implement this model.

Thank you! It's all coming together slowly but nicely.

I don't mind where it resides, but I wonder if it's a bad idea to separate win32gen and winrtgen so completely. They seem very tightly coupled (often a change needs to be made across both parts, due to the class inheritance), and I worry that it might require a lot of coordination to keep two co-dependent packages in sync with one another.

I have been experiencing the drawbacks you mentioned above while I was trying to refactor the winrtgen package. After thinking about this for a while, I decided to remove the dependency on win32gen and instead copy only the necessary bits for the WinRT projection (a few classes and some utility functions) into winrtgen. I wouldn't call it a complete duplication as I was able to remove the unnecessary bits for WinRT projection and merge some classes. I also plan to do more refactoring this week.

The thing is, it looks really complicated with Win32, COM and WinRT projection all in one package and it will get more and more complex as the package evolves. In some cases, the changes I make to the WinRT projection affect the COM projection, so I have to do more checking and the code gets complex and ugly. While I admit that there is some common logic between these projections, it's actually not that great. I also think that as WinRT projection evolves in the future, the differences between them will outweigh the similarities.

With all parts of the Win32/COM projection removed, winrtgen looks really clean and tidy. You can check out the winrtgen package here and also the win32 package with all the WinRT bits removed in the dev branch.

As I said, I'm planning to do more refactoring this week in the generation classes as well as the projection tests. I would like to hear your thoughts on this, especially after I perform these surgeries. I think you are gonna love it (hopefully 😄).

[timsneath]

Sounds good -- look forward to seeing the updated version!

[halildurmus]

Done 🚀