Experimental support for build pipelining [ci: last-only] #7712
Conversation
retronym
changed the title
retronym
force-pushed
the
topic/pipeline-trace-macro-omnibus-rebase
branch
2 times, most recently
from
3c05e29
to
cf83014
Compare
|
Very cool to see this! For outline typing to work correctly, you'll need to at least disable it for trees that may contain super-accessors, see https://github.com/lampepfl/dotty/pull/4767/files#diff-7c63b7bfffa9340fb48ac9b61fa56beeR1366 for the list of things where it needs to be disabled in Dotty (might not be complete, but was good enough to make parallelization work on Dotty itself as well as a few smaller projects). Also note that the compiler phases that sbt inject are not thread-safe, this was recently fixed but isn't part of any released version of sbt: sbt/zinc#626, (in my PR I worked around this by synchronizing the callbacks, but that's only possible because we inlined the sbt phases in Dotty itself, for scalac you'll need to use a custom |
|
Thanks for the pointer. The exception for super is annoying, but such is life. I think my implementation will work for
Could you elaborate on that with an example? |
|
Wrote this a while ago so I don't have an example handy, but in Dotty, lambdas are desugared early into a Block node that wraps a DefDef with the body of the lambda and a Closure node, typechecking that DefDef may affect type inference in the enclosing method so it's not a black box, but this check should be refined to lambdas that are enclosed in a method that must be typechecked, and in any case that doesn't matter for Scala 2 since you don't have the same encoding for lambdas. |
…th JARs Backports: - scala#7366 - scala#7644
.sig files, containing the pickle bytes, are output in place of .class files emitted by Scalac. Java defined .class files are emitted after stripping them of code.
If a new hidden setting is enabled. Build tools that want to implement build pipelining can use this to feed to downstream classpaths.
Under this mode, the RHS of defs and vals are only typechecked if the result type of the definition is inferred and the definition's signature is forced. The movitivation is to create a fast path to extract the pickles for the API for use on the classpath of downstream compiles, which could include parallel compilation of chunks of the the current set of source files.
retronym
force-pushed
the
topic/pipeline-trace-macro-omnibus-rebase
branch
from
e31e6a8
to
b955c4f
Compare
|
I think I'll move the @jvican Do you have any comments about the other parts of this PR? |
|
Oh, I'm sorry for my delay reviewing this, it did fell through the cracks. I'm having a deeper look at it today and will provide more thorough comments. |
There was a problem hiding this comment.
Great to see this making it into the compiler! The experiments I've done with our implementation of pipelined compilation in bloop have been positive overall, giving for some builds such as Spark's > 30% speedup (the implementation is compatible with 2.11, 2.12 and 2.13). However, I'm excited to see an implementation inside the compiler. The compiler is in a much better position to provide a fast and efficient pipelining implementation.
I've left a few comments about the current approach. There are some conflicts between the changes here and some places in the incremental compiler bridge that we should ideally fix so that we can continue to provide the same bridge sources for all minor versions in 2.12 (and not cut a special one for 2.12.9 which would need some changes to the logic that resolves compiler versions).
The implementation seems sounds so I'm just commenting on high-level stuff, providing my insight from the build tool side and specifying what we would need to make this accessible to the whole community of tools.
| // sub-project compilation is able to get a cache hit. A more comprehensive solution would be to | ||
| // involve build tools in the policy: they could close entries with refcount of zero when that | ||
| // entry's JAR is about to be overwritten. | ||
| private val deferCloseMs = Integer.getInteger("scalac.filebasedcache.defer.close.ms", 1000) |
There was a problem hiding this comment.
From my side, I'd like a mechanism to enable this class loader caching but be in charged of the invalidation. I feel it would be better to expose an API here instead of relying on concrete top-level caches that cannot be modified easily from the outside.
Can we add an API that we can override in global that allows us to handle the lifetime of these caches?
With this API, I'd be able to override it in Zinc and use it in Bloop to correctly invalidate these caches and free up the file pointers when I know a user operation will write to an open cached file (e.g. when the project of a compiler plugin is recompiled, when a dependent project that uses straight-to-jar compilation is used, etc etc). This will enable 100% correct usage of class loader caching with zero user intervention, which is what I'm aiming for.
There was a problem hiding this comment.
My idea was that you could override findMacroClassLoader and findPluginClassloader and DIY caching / invalidation.
There was a problem hiding this comment.
Now that this is already present in analyser I'm confident we'll be able to implement our own invalidation. Thanks!
|
|
||
| def stripClassFile(classfile: Array[Byte]): OutputFile = { | ||
| val input = new ClassNode() | ||
| new ClassReader(classfile).accept(input, ClassReader.SKIP_DEBUG | ClassReader.SKIP_FRAMES | ClassReader.SKIP_CODE) |
retronym
force-pushed
the
topic/pipeline-trace-macro-omnibus-rebase
branch
from
b955c4f
to
7707a76
Compare
- Scope the build strategies inside object PipelineMain - Prefer Future.traverse to Future.sequence
retronym
force-pushed
the
topic/pipeline-trace-macro-omnibus-rebase
branch
from
a17594d
to
0b1974c
Compare
We need to typecheck the RHS of type-ascribed definitions if they contain Super trees that require super-accessors to be added to the enclosing template. Rather than take that on right now, I'm removing this feature to focus on the other form of build pipelining.
…e-macro-omnibus-rebase
small followup to scala#7712. the community build found that a couple of projects (mima, twotails) were using the old constructor. since it's easy to do, let's keep both source compat (with the default arguent) and bincompat (with the extra constructor, which we can toss for 2.13)
small followup to scala#7712. the community build found that a couple of projects (mima, classpath-shrinker) were using the old constructor. since it's easy to do, let's keep both source compat (with the default arguent) and bincompat (with the extra constructor, which we can toss for 2.13)
small followup to scala#7712. the community build found that a couple of projects (mima, classpath-shrinker) were using the old constructor. since it's easy to do, let's keep both source compat (with the default arguent) and bincompat (with the extra constructor, which we can toss for 2.13)
|
@retronym this caused some easily-fixable-looking community build breakage, prospective fix submitted at #7822 but it also caused the following, which I don't know what to make of. this happens while compiling the plugin's tests, and it's reproducible outside dbuild just by cloning the twotails repo and doing looking at the twotails build, the only thing unusual I see is: |
restore source compat and bincompat small followup to scala#7712. the community build found that a couple of projects (mima, classpath-shrinker) were using the old constructors. since it's easy to do, let's keep both source compat (with the default arguments) and bincompat (with the extra constructors, which we can toss for 2.13)
restore source compat and bincompat small followup to scala#7712. the community build found that a couple of projects (mima, classpath-shrinker) were using the old constructors. since it's easy to do, let's keep both source compat (with the default arguments) and bincompat (with the extra constructors, which we can toss for 2.13)
Supporting infrastracture for build tools that want to implement build pipelining.
This means that in a multi-module builds, a downstream project could be compiled immediately after its upstream projects have finished typechecking (more precisely, have finished the "pickler" phase), rather waiting for the backed of the the compiler to produce
.classfiles.ClassfileParseris extended to support.sigfiles, containing just the Scala Pickle (which is usually parsed from the argument of a class file annotation).A utility is provided to extract
.sigfiles for a given classpath entry. This also strips out code from non-Scala originated classfiles.PipelineMainis a proof-of-concept driver that demonstrates how this all fits together. It can accept a set of compiler argument files, infer the inter-project dependencies, and schedule pipelined compilation.