Loading symbols from TASTy files directly

[nicolasstucki]

Before this PR we used to parse the classfiles first and when we found a classfile that has a TASTY attribute we switched and loaded the tasty. Now we find the .tasty files directly and load the symbols directly from them. We still load the class files to check that the UUID in that classfile matches the UUID in the TASTy file.

When looking for classes in the classpath, we prioritize the TASTy files over classfiles. This implies that the symbol loader will receive the .tasty files for Scala 3 code and .class for Scala 2 and Java code.

A variant of the ClassfileParser called ClassfileTastyUUIDParser was added to have a way to check the UUID in the TASTY attribute of the classfile. The ClassfileParser could not be used directly because it eagerly tries to initialize parts of the symbols that are already loaded from the TASTy file, causing some conflicts.

Open question: should we only check the TASTy UUID under some flag to avoid loading both the .tasty and the .class files? The second commit introduces this check.

[bishabosha]

This is also going to be very useful for TASTy-only class paths, such as the coming changes to include build-pipelining

[smarter]

@bishabosha Can I assign this PR to you?

[bishabosha]

does this count as an API change?

[nicolasstucki]

Good catch. This falls into undefined behavior in the documentation. Therefore we can change it.

This comes from a call to quotes.reflect.SourceFile.current. In the inspector, we explicitly state that this method will not work

https://github.com/lampepfl/dotty/blob/c62909009506171b03e092c4a391483b044d3b49/scaladoc/src/scala/tasty/inspector/Inspector.scala#L27

[bishabosha]

I did a benchmark on my machine of memory usage by compiling org-lichess/lila (140k scala LOC) and did not observe any significant change between either including the check or not, and the old way of loading tasty

[bishabosha]

test performance please

[nicolasstucki]

I restarted the tests to make sure we do not have any conflicts before we merge this.

[mbovel]

This PR might have caused a performance regression:

See https://dotty-bench.epfl.ch.

[bishabosha]

there's just a lot more IO being done here now, i wonder if we can work more on assumption or caching rather than constantly asking the file system for proof of things

[bishabosha]

I can't really reproduce any large spikes caused by these changes

[nicolasstucki]

Are we sure that the CPU clock speed is still limited? In the past we had some issues with that when the machine rebooted.

[bishabosha]

Why should this be 3.4? It doesnt have any effect on the compatibility - also it will be needed to bring pipelining to 3.3

[nicolasstucki]

It does not introduce any binary compatibility, but some external tools might start behaving differently. For example, if a build tool does not properly invalidate/remove the old .tasty files (only .class files) we could encounter an issue. We decided that we should wait and see what happens in 3.4. If some issues are found, then the build tools can be fixed. Later, when we are sure that all tools work properly we can backport it to 3.3.

[bishabosha]

@Kordyjan this would be eventually necessary if we ever want to have build pipelining support for scala 3.3, it seems from @nicolasstucki comment we should only wait to see if it's a problem or not. The proposed flowchart in #18636 would not reject this PR (or we need clarity on what breaking forward compatibility means)

[bishabosha]

Alternatively, I am proposing a new file extension for "signature" tasty files used in pipelining or other purposes, which would depend on some of the changes here, so that would be more simple to nominate for backport (older scala version would simply ignore the file, and pipelining flags would be ignored)

[bishabosha]

I have noticed a 3% slowdown compiling lichess-org/lila with 3.3.1 vs current main build, didn't verify if this is a cause -

Scala 3.3.1	(clean; System.gc; compile runs after cold boot sbt)
time (s)	Memory (GB)
107	5.72
71	5.79
68	6.28
64	5.31
65	6.10
64	6.34

Scala 3.4.0-RC1-bin-SNAPSHOT	(clean; System.gc; compile runs after cold boot sbt)
time (s)	Memory (GB)
107	4.97
74	6.40
68	6.04
68	5.71
67	7.23
66	6.54