Source-file level parallelism for compilation

[tothambrus11]

Currently, the compiler processes modules as a whole and type-checks, lowers and transforms all code sequentially within modules. In the future, when a module has dozens of source files, we may want to not only incrementalize, but also parallelize the compilation on a source file level.

Historically, the limitation has been that type-checking requires consistent unique IDs of equivalent types in a program, so source-file level paralellism was out of question. However, some time ago we discovered a technique of merging identitites and types of separately compiled modules, so maybe we could use the same technique here. The aim of this thread is to discuss what the implications of this compilation mode would be, and how much complications it would cause.

If source-file-level type merging is possible, we could imagine delaying the type-checking of function bodies, and doing that only when the compilation of a particular source file has been requested. It would be also interesting to know how much we can support per-file separate compilation in the emitter and transformation phases. In particular, we would probably need to disable dead code elimination and inlining across file boundaries.

[kyouko-taiga]

While that is certainly an interesting goal, I am not in a hurry to start any serious investigation into parallelizing the compilation of a single module until we get hard numbers showing that it is necessary.

Note that the approach we use for importing types from another module requires serialization and deserialization, which is not exactly fast ... So I don't consider it a very promising solution for parallelizing compilation. There are other hurdles too. One is that it is very difficult to reason about the typer in a parallel setup because the order in which things are processed is in fact significant.