Parallel type-checking in compilation (behind a feature flag)

[safesparrow]

This is the joint work of @nojaf and myself

This is a feature we have been working on and discussing for the last couple of months.

Extended documentation can be found in src/Compiler/Driver/GraphChecking/Docs.md
https://github.com/dotnet/fsharp/blob/0c5c1ffb93dce47004c32b9d71b9ea9cb22e4ead/src/Compiler/Driver/GraphChecking/Docs.md

The summary of the PR is:

1. A new test compilation option GraphBasedChecking is added.
2. Graph-based type-checking works in the following way:
   1. Use parsed AST to infer what files might/might not depend on each other.
   2. Perform type-checking in parallel, using the dependency graph.

The net result is a lot faster type-checking, with exact improvements being highly dependent on the structure of the project and the use of signature files.

Basic performance test:
FSharp.Compiler.Service, Server GC, ReadyToRun, release mode, 8-core/16-thread CPU:

Sequential:
Realdelta: 18.4

Parallel checking of files with backing signature files:
Realdelta: 8.1

Graph-based:
Realdelta: 6.1

The impact is much more relevant in projects without backing signature files, where the existing feature cannot bring any benefits.

TODO:

• Remove any unwanted changes from the branch introduced for testing
• Add more timing results, with projects without .fsi files used.
• Cleanup

Pinging @vzarytovskii @0101 @T-Gro @baronfel.

Thanks to @nojaf who has been working with me on this change for months now 👏

[safesparrow]

There are quite a few temporary changes that we will need to revert - have commented on all that I could identify.

[vzarytovskii]

Looks nice, will give it a proper review this week, will also think about testing matrix in CI!

[vzarytovskii]

Could you also please mark new public APIs as experimental (via the attribute), repurpose CI leg(s) for parallel project checking to also include these changes, and expose a flag in VS settings to enable it?

[nojaf]

@vzarytovskii, I've added a CI leg similar to --test:ParallelCheckingWithSignatureFilesOn.
Let me know if that needs any tweaks.

[T-Gro]

The new CI leg passes, good job.

The failing test is one I introduced (issue with different process locking a .csv file with timing), shall I have a look and fix it?

[nojaf]

I don't think that failure is related to our changes. I triggered the CI again, if it still pops up I can take a look.

[0101]

Looks good!

[nojaf]

Thank you @0101 for the review! Much appreciated!

[vzarytovskii]

This looks good to me.

Since it's a separate compilation path under flag, I think we should get it in and start testing it on compiler itself, tests, pr ci (and turn off in signed builds).

I'm fine with merging it in.

[vzarytovskii]

One thing I would like to ask is to make sure docs are up to date, but it can be done in separate pr.

[nojaf]

One thing I would like to ask is to make sure docs are up to date, but it can be done in separate pr.

@safesparrow I added some notes on the algorithm. I think it is up to date otherwise.
Could you also please review the whole file please?

[safesparrow]

One thing I would like to ask is to make sure docs are up to date, but it can be done in separate pr.

@safesparrow I added some notes on the algorithm. I think it is up to date otherwise. Could you also please review the whole file please?

I reviewed it and made changes. I think it's good enough now.

[safesparrow]

@vzarytovskii @T-Gro @0101 We think this is now ready to merge. If you think it is too, could you please merge it?

[T-Gro]

I think so too.
Let's use it :)

[safesparrow]

Cheers @T-Gro

Just to note that as discussed above the nondeterministic pdb generation means that the feature auto-disables in default deterministic builds, so to start using it you'll have to disable determinism.

Fixing this is our plan but so far it's proving difficult to pinpoint what exactly causes it.