Improve modular compilation pipeline with kernel

[jakemac53]

The general idea here is to create a new initializeModularCompiler which can re-use Components from previous compiles (based on an input digest which we provide through the bazel worker protocol).

We should hide this behind a flag as well so we can roll it out safely.

cc @jensjoha

[jakemac53]

Note that we also need to resolve the issues around config specific imports still - but I think the pending change from @sigmundch might give us all we need (ability to provide a custom libraries.json)

[jakemac53]

https://dart-review.googlesource.com/c/sdk/+/97003 gets things compiling with configurable imports, although I am having some runtime failures now when using the test package (couldn't compile it at all before though, so that is a step forward).

[vsmenon]

@jensjoha @kmillikin @jakemac53 - do we know what's needed here?

[vsmenon]

Any ETA on this?

[jensjoha]

I think it depends on what the goal is.

I have a number of CLs out that I believe is in a "landable state" that make the kernel pipeline be "on par" with the non-kernel pipeline for some incremental changes. I've had no reviews yet though.

I also have something that makes the kernel pipeline "on par" (or faster) in all cases that I've tested, but I've basically been told that those changes cannot be made because of restrictions in package build.

[vsmenon]

For 2.3, just "on par" that we can move package:build onto. That's enough to switch over from analyzer-ddc to kernel-ddc (and lets us use the Flutter kernel transform for widget inspection).

Today, with @jakemac53 's current build-with-kernel branch, it seems about twice as slow for a small incremental edit - pretty noticeable on demos.

Beyond 2.3, definitely interested in ideas for even faster. Do you have a pointer to CLs that are out?

[jensjoha]

https://dart-review.googlesource.com/c/sdk/+/95386 changes DDC and bazel worker to use the incremental compiler which for some changes makes it on par. It builds on a few also un-landed and un-reviewed CFE changes though.

The thing about "small incremental edits" is that it very much depend on what file is changed though. If it is a file that is (transitively) imported by lots of other files, invalidating it will pull lots of other stuff with it and thus not really be "small".

The current pipeline has tricks for not doing that when the structure of things doesn't change.

I applied a somewhat similar trick to package build for kernelm which worked (and made the kernel version always faster in my tests), but which I was wasn't allowed.

[dgrove]

Now that https://dart-review.googlesource.com/c/sdk/+/95386 has landed, what is remaining here?

[jakemac53]

I believe we can close this now, for the canonical edits I have tested with the flutter gallery we are on par or better than analyzer/ddc. Just waiting on a dev release to send out the PR for the build_* changes and publish something there.

There are still additional wins we can investigate in the future but those would be longer term and aren't blocking.