In-source sbt compiler bridge [ci: last-only] #10472

Documentation TBD. Rewritten from sbt/zinc@f8a907a

We need to call it just once and this is already being done. Rewritten from sbt/zinc@9cee296

This commit merges 1.0 branch into class-based-dependencies branch. The class-based-dependencies branch has been obtained from the sbt repo by rerunning the filter-branch command that has been used originally to spawn zinc repo out of sbt repo: git filter-branch --index-filter 'git rm --cached -qr -- . && git reset -q $GIT_COMMIT -- compile interface util/classfile util/classpath util/datatype util/relation' --prune-empty That preserved some merges from the original class-based-dependencies branch that didn't make sense so I rebased the filtered branch on top of 7e9ccc10af0c53f17ecfe87cead3ef03fd95642f. This commit is the first commit in zinc's history after filtering. Once the rebase was done and I got a cleaner history, I've merged 1.0 into my class-based-dependencies branch. This commit is exactly that merge. The merge itself is very complicated because very non-trivial changes happened in both zinc and my class-based-dependencies branch. What you see here represents the best effort at merging. I expect mistakes to be discovered and fixed in subsequent commits. I got it as far as to compile everything and make most of the test pass. I highlight below some of the least obvious changes I've made: - Analysis.groupBy operation has been nuked. It's been broken in class-based-dependencies and it's not clear who needs it - AnalysisTest is removed as it was mostly testing the Analysis.groupBy - changes to definition.json and other.json are ports of changes from definition and other (from the interface subproject). Consult hisotry of those files to understand those changes - TestAnalyzingCompiler has been adapted to handle invalidation at class level instead of source file level - TestAnalysisCallback has been adapted to changes how apis, products and dependencies are tracked. A lot of code has been copied over from AnalysisCallback. I'm not sure if that the best course of action but I don't understand the role of TestAnalysisCallback exactly to have a better opinion. # Conflicts: # compile/api/src/main/scala/xsbt/api/ShowAPI.scala # compile/api/src/test/scala/xsbt/api/NameHashingSpecification.scala # compile/inc/src/main/scala/sbt/inc/Dependency.scala # compile/integration/src/main/scala/sbt/compiler/AggressiveCompile.scala # compile/integration/src/main/scala/sbt/compiler/IncrementalCompiler.scala # compile/interface/src/main/scala/xsbt/Dependency.scala # compile/persist/src/main/scala/sbt/inc/AnalysisFormats.scala # compile/persist/src/main/scala/xsbt/api/AnalyzedClassFormat.scala # compile/persist/src/main/scala/xsbt/api/SourceFormat.scala # compile/src/test/scala/sbt/compiler/javac/JavaCompilerSpec.scala # incrementalcompiler/src/main/scala/sbt/internal/inc/MixedAnalyzingCompiler.scala # incrementalcompiler/src/main/scala/sbt/internal/inc/javac/AnalyzingJavaCompiler.scala # interface/definition # interface/other # interface/src/main/java/xsbti/DependencyContext.java # interface/src/test/scala/xsbti/TestCallback.scala # internal/compiler-bridge/src-2.10/main/scala/xsbt/API.scala # internal/compiler-bridge/src-2.10/main/scala/xsbt/CompilerInterface.scala # internal/compiler-bridge/src-2.10/main/scala/xsbt/ExtractAPI.scala # internal/compiler-bridge/src-2.10/main/scala/xsbt/ExtractUsedNames.scala # internal/compiler-bridge/src/test/scala/xsbt/DependencySpecification.scala # internal/compiler-bridge/src/test/scala/xsbt/ExtractAPISpecification.scala # internal/compiler-bridge/src/test/scala/xsbt/ExtractUsedNamesSpecification.scala # internal/compiler-bridge/src/test/scala/xsbt/ScalaCompilerForUnitTesting.scala # internal/compiler-interface/src/main/java/xsbti/AnalysisCallback.java # internal/incrementalcompiler-apiinfo/src/main/scala/xsbt/api/APIUtil.scala # internal/incrementalcompiler-apiinfo/src/main/scala/xsbt/api/HashAPI.scala # internal/incrementalcompiler-apiinfo/src/main/scala/xsbt/api/NameHashing.scala # internal/incrementalcompiler-classfile/src/main/scala/sbt/internal/inc/classfile/Analyze.scala # internal/incrementalcompiler-core/src/main/scala/sbt/internal/inc/APIDiff.scala # internal/incrementalcompiler-core/src/main/scala/sbt/internal/inc/APIs.scala # internal/incrementalcompiler-core/src/main/scala/sbt/internal/inc/Analysis.scala # internal/incrementalcompiler-core/src/main/scala/sbt/internal/inc/Compile.scala # internal/incrementalcompiler-core/src/main/scala/sbt/internal/inc/Incremental.scala # internal/incrementalcompiler-core/src/main/scala/sbt/internal/inc/IncrementalAntStyle.scala # internal/incrementalcompiler-core/src/main/scala/sbt/internal/inc/IncrementalCommon.scala # internal/incrementalcompiler-core/src/main/scala/sbt/internal/inc/IncrementalDefaultImpl.scala # internal/incrementalcompiler-core/src/main/scala/sbt/internal/inc/IncrementalNameHashing.scala # internal/incrementalcompiler-core/src/main/scala/sbt/internal/inc/MemberRefInvalidator.scala # internal/incrementalcompiler-core/src/main/scala/sbt/internal/inc/Relations.scala # internal/incrementalcompiler-core/src/test/scala/sbt/inc/AnalysisTest.scala # internal/incrementalcompiler-core/src/test/scala/sbt/inc/TestCaseGenerators.scala # internal/incrementalcompiler-persist/src/main/scala/sbt/internal/inc/TextAnalysisFormat.scala # internal/incrementalcompiler-persist/src/main/scala/xsbt/api/SourceFormat.scala # internal/incrementalcompiler-persist/src/test/scala/sbt/inc/TextAnalysisFormatSpecification.scala Rewritten from sbt/zinc@b0a8a91

The merge in b0a8a91aaa5b6f55714986f7bf389ec8aa091ec0 messed up code in Dependency.scala. This commit just copies over the code of the `traverse` method as-is from class-based-dependencies. This commit makes all tests in compilerBridge to pass. Rewritten from sbt/zinc@49f8786

The fix was to just copy over the main code and adapt it to Scala 2.10 compiler API. Rewritten from sbt/zinc@469a3ac

Rewritten from sbt/zinc@19d8b9d

The b0a8a91aaa5b6f55714986f7bf389ec8aa091ec0 merge commit didn't resolve merge conflicts for AnalyzeSpecification.scala and for ClassToAPISpecifaction.scala. This commit oves them to the right subprojects and adapts them to Scalatest APIs. Both specifications depend on TestCallback so it had to be moved to a subproject commonly referenced. I've moved it to the compiler-interface subproject. As a consequence, I had to add a dependency on scala-library in `test` configuration to compile TestCallback written in Scala. Rewritten from sbt/zinc@ae5d0fe

The `declaredClasses` relation duplicates the data stored and functionality provided by `classes` relation and it's not being used in incremental compiler. Let's remove it. A historical note: the declaredClasses relation has been introduced at the time when classes relation fulfilled a different role. At some point, classes has been refactored to fulfill exactly the same role as declaredClasses relation. Rewritten from sbt/zinc@deac13a

Remove dependency on CallbackGlobal from ExtractAPI that doesn't require it anymore and update the docs. Remove dependency on CallbackGlobal from ClassName and GlobalHelpers that do not need it either. Rewritten from sbt/zinc@f475de2

This commit enables control of whether a compiler instance should be reused between compiling groups of Scala source files. Check comments in the code for why this can be useful to control. Rewritten from sbt/zinc@a184722

Add a pending test that shows a problem with instability of representing self variables. This test covers the bug described in scala#2504. In order to test API representation of a class declared either in source file or unpickled from a class file, ScalaCompilerForUnitTesting has been extended to extract APIs from multiple compiler instances sharing a classpath. Rewritten from sbt/zinc@3f26571

The reason for instability is a bit tricky so let's unpack what the previous code checking if there's self type declared was doing. It would check if `thisSym` of a class is equal to a symbol representing the class. If that's true, we know that there's no self type. If it's false, then `thisSym` represents either a self type or a self variable. The second (type test) was supposed to check whether the type of `thisSym` is different from a type of the class. However, it would always yield false because TypeRef of `thisSym` was compared to ClassInfoType of a class. So if you had a self variable the logic would see a self type (and that's what API representation would give you). Now the tricky bit: `thisSym` is not pickled when it's representing just a self variable because self variable doesn't affect other classes referring to a class. If you looked at a type after unpickling, the symbol equality test would yield true and we would not see self type when just a self variable was declared. The fix is to check equality of type refs on both side of the type equality check. This makes the pending test passing. Also, I added another test that checks if self types are represented in various combinations of declaring a self variable or/and self type. Fixes scala#2504. Rewritten from sbt/zinc@81cbabf

Rewritten from sbt/zinc@012f615

When `B2.scala` replaces `B.scala` in the new test `types-in-used-names-a`, the name hash of `listb` does not change because the signature of `C.listb` is still `List[B]`, however users of `C.listb` have to be recompiled since the subtyping relationships of its type have changed. This commit does this by extending the definition of "used names" to also include the names of the types of trees, even if these types do not appear in the source like `List[B]` in `D.scala` (since `B` has been invalidated, this will force the recompilation of `D.scala`). This commit does not fix every issue with used types as illustrated by the pending test `types-in-used-names-b`, `B.scala` is not recompiled because it uses the type `T` whose hash has not changed, but `T` is bounded by `S` and `S` has changed, so it should be recompiled. This should be fixable by including the type bounds underlying a `TypeRef` in `symbolsInType`. The test `as-seen-from-a` that did not work before shows that we may not have to worry about tracking prefixes in `ExtractAPI` anymore, see the discussion in scala#87 for more information. Rewritten from sbt/zinc@350afa7

The previous approach to value class API (introduced by sbt/sbt#2261 and refined by sbt/sbt#2413 and sbt/sbt#2414) was to store both unerased and erased signatures so that changes to value classes forced recompilations. This is no longer necessary thanks to scala#87: if a class type is used, then it becomes a dependency of the current class and its name is part of the used names of the current class. Since the name hash of a class changes if it stops or start extending AnyVal, this is enough to force recompilation of anything that uses a value class type. If the underlying type of a value class change, its name hash doesn't change, but the name hash of <init> change and since every class uses the name <init>, we don't need to do anything special to trigger recompilations either. Rewritten from sbt/zinc@1e7e99e

Rewritten from sbt/zinc@7769b58

This change refactors definition.json to make `ClassLike` a non-recursive type. Before the introduction of class-based dependency tracking, an api of a nested class has been stored as a member of the enclosing class so a class could have other class as a member. This was expressed by the fact that `ClassLike` was recursive: its Structure had a collection of `Definition` to represents the members and `ClassLike` was a subtype of `Definition`. With introduction of class-based dependency tracking, each class has been extracted and stored separately. The inner class would still be stored as a member of outer class but members of inner classes were skipped. An empty inner class was stored just to mark the fact that there's a member with a given name which is important for name hashing correctness when there's no dependency on a class directly but a rename could introduce one. Storing an empty class was a hack and this commit fixes the type hierarchy by introducing the following changes: - introduce ClassDefinition that is a subtype of Definition; all members of a class are subtypes of ClassDefinition (ClassLike is not a subtype of ClassDefinition) - change Structure to refer to ClassDefinition instead of Definition for members - move ClassLike higher up in type hierarchy so its a direct subtype of Definition - introduce ClassLikeDef which represents an inner class as a member of the outer class; ClassLikeDef carries only information about class declaration itself but not about its members and that is enforced statically NameHashing has been simplified because it doesn't have to keep track of the entire path for definitions it hashes. Hashes of names are tracked individually per class so location is simply name of the class and it's type (we want to distinguish between objects and classes). NameHashingSpecification has been refactored to not rely on nested classes for testing the desired scenarios. The semantics of tests has been preserved even if a different API structure is used in tests. Rewritten from sbt/zinc@1a696ed

Rewritten from sbt/zinc@f002cbe

traverse(tree: Tree) used to call super.traverse(tree) at the end. sbt/sbt@0f61629 brought the traversing call to inside of the pattern matching. For the case of MacroExpansionOf(original), it amounts to not traveling the macro-expanded code. See sbt/src/sbt-test/source-dependencies/macro-nonarg-dep for the repro. Rewritten from sbt/zinc@acf4fac

For example, when the `--sourcepath` option is provided and the refchecks phase compiles an annotation found on a referenced symbol from the sourcepath. `compileLate` assumes that all non-sentinel compiler phases can be down cast to `GlobalPhase`. This commit changes the two phases in SBT to extend this instead of `Phase`. This has the knock on benefit of simplifying the phases by letting the `GlobalPhase.run` iterator over the list of compilation units and feed them to us one by one. I checked that the test case failed before making each change. Rewritten from sbt/zinc@a27be44

FPORT: Fixes incremental compiler missing member ref from macro expansion Rewritten from sbt/zinc@ddefbb0

FPORT: Avoid CCE when scalac internally uses compileLate. Fixes scala#2452 Rewritten from sbt/zinc@d0d97f4

For an inherited class, ExtractAPI would form a (source) class name by calling `Symbol.className` on the inherited class. However, that created a name of a class as seen at declaration site and not at inheritance site. Let's consider an example: class A { class AA } class B extends A Before this change, ExtractAPI would create an API representation of `AA` twice: once seen from A, and then the second time seen from B as an inherited member. However, in both cases it would use `A.AA` as a name. This commit fixes naming so an inherited representation of `AA` has a name `B.AA`. This commit also clarifies how classes declared in package objects are named. If you have: package pkg1.pkg2 package object pkg3 { class Foo } then the fully qualified name of the class corresponding to `pkg3` package object is pkg1.pkg2.pkg3.package. The full name of the `Foo` class is pkg2.pkg2.pkg3.Foo. Rewritten from sbt/zinc@459df6b

Use debuglog for logging in the API phase. That method takes care of checking whether debugging is enabled for a given phase with `-Ylog` option. Previously, the verbose log was spammed with details of the API phase execution, making reading verbose log difficult. Rewritten from sbt/zinc@5391d89

Fix API extraction of inherited classes Rewritten from sbt/zinc@1f07510

Rewritten from sbt/zinc@8813305

Rewritten from sbt/zinc@f872f9c

This avoids an NPE when accessing position info in case `sourcePath` or `sourceFile` are `null`. See sbt/sbt#2766 for a stack trace. Rewritten from sbt/zinc@e2a249a

Rewritten from sbt/zinc@3fb6f44

Ref sbt/zinc#187 Rewritten from sbt/zinc@23ed4ef

Ref scala#193 Rewritten from sbt/zinc@09a32f4

Rewritten from sbt/zinc@46f84e3

Rewritten from sbt/zinc@e2836a9

- remove multiple collection repacking and switch to mutable (faster) ones - Postpone decoding of symbol name to the end (it is called only once for each name) - create cache that keeps enclosing class, names used in that class and already processed symbols in for that class Rewritten from sbt/zinc@b77a2b3

- use associated file from symbol instead of querying for one - cache current source file with all attributes - use faster extracting symbols from type (no collections is created) - invoke callback asynchronously (in executor with one thread). Rewritten from sbt/zinc@55a5d38

…back). Remove async calls in Dependecy phase. Rewritten from sbt/zinc@1edf1f3

Now it is possible to disable incremental compilation metadata creation in IncOptions and directly in AnalysisCallback. Rewritten from sbt/zinc@f898c2b

They are not relevant and their effect on performance is minimal, but it's good to have them for consistency. We do check for `NoSymbol` as well as `null` when it comes to symbols. It feels weird if we don't for types. Rewritten from sbt/zinc@fad19ce

These helpers are only used in `Dependency` for now, but I feel that they could become handy in other Zinc scalac phases and remove the overplaty way of guarding against inexistent or useless symbols and types. Rewritten from sbt/zinc@c53439a

@romanowski

Instead of gathering all symbols in a set and using `foreach`, reuse a foreach symbol traverser that will go through all the symbols of a type and execute an operation on them. This finishes up similar work done by @romanowski. Apparently, this instance of `symbolsInType` was forgotten. As this function is not useful anymore, it is removed for the posterity. Rewritten from sbt/zinc@207491d

The name `foreachSymbolInType` is misleading in the sense that gives the impression to the reader that a certain operation will be applied to every type symbol in a type, whereas it only does so for symbol that don't correspond to packages. This commit renames this method to a longer but more concise version of this function. I haven't skimped on the length of the name because, IMO, helpers should prefer correctness over shortness. Rewritten from sbt/zinc@9019f8d

This commit is an aesthetic change to the code of `ExtractUsedNames` in the following ways: * It clarifies the purpose of the `termSymbol` attached to `Import` nodes and removes (now) unnecessary comments from the source. * It adds curly braces around a long method for readability purposes. * It removes `tpnme` which was introduced for 2.8 Scala support. Zinc does not anymore offer support for 2.8.x and 2.9.x, hence the removal. * It moves `emptyName` to `GlobalHelpers` under `isEmptyName` and instead of guarding against `NoSymbol` it reuses the helper `ignoredSymbol`. Rewritten from sbt/zinc@16e1670

`SafeLazy` has been traditionally implemented in `zincApiInfo` because it is part of the sbt API and is accessible to all the subprojects that depend on it. Before this commit, `SafeLazy` was a runtime dependency (using reflection) of the compiler bridge. In this regard, Zinc was assuming that the sbt API was accessible at runtime and therefore invoked it to use an implementation of lazy that would remove references to the thunks once they've been forced. This was done to free memory as soon as possible since those thunks usually depend on classes of compiler internals and would not be GC'ed otherwise. However, the compiler bridge is not supposed to depend on sbt APIs since its code is compiled by the Scala compiler that the user picks in SBT. Its only dependency is the compiler interface, which is implemented in Java and compiled beforehand with javac. This commit removes the runtime dependency of the compiler bridge to the sbt API and avoids the method invocations using reflection. This was done for the following reasons: * Simplicity. It is not obvious why `SafeLazy` is invoked reflectively. See sbt/zinc#113. * Performance. Even though the JVM should make this simple use of reflection fast, there's a very small overhead of using reflection in the compiler bridge because `lzy` is (most likely) hot. The fix consists of a Java implementation of `SafeLazy` that uses the non-thread-safe lazy val implementation described [here](http://docs.scala-lang.org/sips/pending/improved-lazy-val-initialization.html). It is complemented with a proxy written in Scala that will create an indirection layer for things like by-name and strict evaluation. This implementation of lazy val assumes that `SafeLazy` will never be called asynchronously. If this is the case, it's up to the Zinc maintainer to make sure that safe publishing is implemented at the call-site or to change the implementation to avoid races and uninitialized fields. Rewritten from sbt/zinc@c1f821b

Future error reporting would be more sophisticated because ideally we want to report concise error messages with good contextual information. This commit takes the first step by putting common error messages in an object `Feedback` stored into `GlobalHelpers`. Temporary error messages have not been added since they will be removed in future commits (things like `super` not being handled in `ExtractAPI`, for instance). Rewritten from sbt/zinc@e288c18

Fixes scala#222 Rewritten from sbt/zinc@a271e7c

Add cache verifier. Add classpath mappers. Add mapper for whole MiniSetup after setup os loaded. Fixes small problems with dependencies phase (e.g. reduce numbers of NoSymbol checked) and do treat refinement class as top-level class (since it does not have runtime representation. Rewritten from sbt/zinc@52aa327

Rewritten from sbt/zinc@4b4e5b2

…pilation Add option to disable incremental compilation Rewritten from sbt/zinc@7cad702

Make incremental compilation artifacts reusable (aka cached compilation) Rewritten from sbt/zinc@b5c2817

This commit does two things: * In 2.8, `computePhaseDescriptors` was accessed using reflection because it was private and inaccessible to `CachedCompiler`. However, it was turned into protected later on by scala@1016d68. As current Zinc only supports Scala versions 2.10, 2.11 and 2.12, the access to `computePhaseDescriptors` is no longer reflective. * Remove unused imports that cluttered the source file. Rewritten from sbt/zinc@33407c5

This commit introduces JMH benchmarks as requested by sbt/zinc#225. The strategy for the JMH benchmarks is the following: 1. Clone repository to be compiled and checkout at concrete commit (for reproducibility). 2. Generate sbt task for every subproject we want to test. This task will tell us information about the compiler options: sources and classpath (scalac options are integrated in next commit). 3. Execute the sbt tasks and get the output. 4. Instantiate a compiler fully integrated with the compiler bridge that has all the parameters given by the sbt output. 5. Run and enjoy. The JMH benchmarks run this for every iteration, for now. However, they only measure the running time of the **whole** compiler pipeline. There is no reliable way to do it only for our Zinc phases since they are tightly coupled to the compiler, so this is the best we can do. The JMH benchmarks are only 2.12 compatible. This commit introduces a benchmark for Shapeless and shows how easy it is to extend for other codebases. It also adds some tests to make sure that the required scaffolding works. For that, we have to modify the accessibility of some methods in our infrastructure, particularly `ScalaCompilerForUnitTesting`. Rewritten from sbt/zinc@4a09b04

`API` guards against null and `NoSymbol`s. This commit reuses those checks in `GlobalHelpers` and rewrites `isTopLevel` so that it's easier to see what the performed checks are. This rewrite also reorders the checks for performance reasons. The way `TopLevelTraverser` works is implicitly checking for `isNestedClass` because it only traverses the trees that are inside package whose symbols should have that package as an owner. `isTopLevel` is checked this way, so it's a redundant check. However, it's not safe to remove it from here because the semantics of `TopLevelTraverser` could change and people would expect the implementation of `isTopLevel` to be stable. Therefore, I've moved that check at the end of the check chain so that only valid top level symbols pay the price of that check and we spare a check in those symbols that are more likely to fail checks before. Aside from this, `isStatic` is now checked before `isImplClass` because non static symbols are more frequent and will therefore fail the check before. Aside from this, `isImplClass` will always return `false` in 2.12 because the encoding of traits has changed. Rewritten from sbt/zinc@5fbd749

@romanowski

`Dependency` was architectured in two phases. First, a traverser would run in the tree of the compilation unit and populate all the found dependencies. Second, the caller of the traverser would go through these data structures and process them. Since performance is a major feature of Zinc, this commit changes the previous design and mixes the previously described steps into one. This is possible because they don't depend on each other and processing dependencies does not require any kind of ordering. This is likely to have a significant impact in the performance of `Dependency` memory-wise for the following reasons: * At the moment of processing the class dependency, that object is already hot in the cache and its access is for free. * We economise the amount of memory required for holding the populated temporary data structures. For larger compilation units, this data would be large and likely to be promoted from young generations to older generations, thus consuming unnecessary GC time. Note that `addClassDependency` and `addTopLevelImportDependency` cache the added dependency unlike the previous version, which would just add to the hash set even though the key was already there. This rewrite of `Dependency` tries to touch as less lines as possible to preserve the git history for future archeologists of this codebase. It also adds a lot of new comments explaining the underlying algorithm. @romanowski has performed a similar job before when he rewrote the temporary data structures to use `Iterator`s instead of the underlying sets. I'm unsure of the real impact of this change in comparison with his, but it should at least be a little bit faster. Rewritten from sbt/zinc@035c046

Previous comment was referring to some implementation details that are not used anymore. Remove and specify what the phase actually does. Rewritten from sbt/zinc@3f09fd5

@romanowski

Addresses feedback from @romanowski on not repeating this computation every time a top level import is processed by `Dependency`. Rewritten from sbt/zinc@dbba375

The following rewrites minor parts of `ExtractUsedNames` to: * Use names instead of strings. This way, we can reuse the set for names as a cache and remove `symbolsCache` which ends up consuming unnecessary memory (we don't care from which symbols we get a name, if two symbols have the same name, only one of them will be used). * Never add names that are repeated, check for their existence first. This way we will always spare one unnecessary write to a set. As writes are expensive and reads are not, it's better to check with `contains` before `add`ing to a set. Note that this is not a performance overhead because `add` invokes `contains` before `+=`. Check the implementation of mutable `Set`. * Never add original trees directly. Same as for names. * Document the current implementation of cached `EnclosingNonLocalClass` and specify what `currentOwner` is. Rewritten from sbt/zinc@0ca3830

Rewritten from sbt/zinc@0dee711

Rewritten from sbt/zinc@8fef6c8

Rewritten from sbt/zinc@d8a8de9

This commit introduces changes to Scala 2.10 sources from the following PRs: * sbt/zinc#225 * sbt/zinc#216 * sbt/zinc#206 * sbt/zinc#221 It also removes a stub for 2.8 compatibility in `DelegatingReporter`. Support for Scala 2.8 compatibility is not already maintained it. Rewritten from sbt/zinc@5d46c1b

This is a revert of the change that was made in: sbt/zinc@b0a8a91#diff-babf52667b7dd64d625e0bcddfd8da48R211 Here's my analysis of what happened. Prior to creating this commit Greg was working on class-based name-hashing in a branch in sbt/sbt, targeting sbt 0.13. With this commit Greg was re-syncing his changes with the changes that had happened in the sbt/zinc repo, to be able to PR sbt/zinc. However, there was an important difference in sources between sbt/sbt and sbt/zinc. The Scala compiler/reflect APIs change over time, for instance in scala 2.11 the "atPhase" method was deprecated in favour of the brand-new, more precise, but otherwise identical "enteringPhase" method. But sbt/sbt didn't care about deprecations and just used "atPhase" for all versions of Scala it supported (2.8/2.9/etc). In sbt/zinc, instead, the sources had been split into 2.10 and 2.11+ sources, and the 2.11+ sources had switched to the newer, non-deprecated APIs. So "atPhase" had been replaced with "enteringPhase". So when Greg was re-syncing the changes in his branch with the changes in sbt/zinc he had to deal with (a) the sources being split and (b) the compiler API differences. At this point I think 1 of the 2 following plausible scenarios happened. Either: 1. For one reason or another Greg accidentally, and incorrectly, thought it was the correct implementation, perhaps he forgot that atPhase and enteringPhase are synonymous. So he made the 2.11+ code invoke enteringPhase(currentRun.typerPhase.next) instead of enteringPhase(currentRun.typerPhase), thinking it was the correct equivalent to atPhase(currentRun.typerPhase) in the 2.10 code. 2. Greg intentionally, purposely, made, all of a sudden, a judgement call, while syncing, dealing with merge conflicts and file path changes, that ExtractAPI#annotations MUST phase travel to exiting the typer phase (via enteringPhase(typer.next)). And then he (a) changed _only_ the 2.11+ code, leaving the the 2.10 code travel to the wrong place _and_ (b) didn't mention it in his "highlight of less obvious changes" in the commit message. I find 1 to be much, much more plausible, so let's revert to what has worked since its inception in 2010: sbt/sbt@af4f41e#diff-d31107a2fcd3c8482cbac4ee198375b6R311 and today in the latest stable release, 0.13.13: https://github.com/sbt/sbt/blob/v0.13.13/compile/interface/src/main/scala/xsbt/ExtractAPI.scala#L189 Rewritten from sbt/zinc@feb062e

Having `DependencySource` and `EnclosingNonLocalClass` as wrapper in both `ExtractUsedNames` and `Dependency` is nice, but it has a runtime overhead. The issue arises from the fact that caching is only performed at the `currentOwner` level, and hence when we traverse the tree and change its value, the cached instance is recomputed because the `currentOwner` does not equal the owner of the previous cached instance. However, both cached instances may resolve to the same non local class and recomputing it is a waste of time. Let's say we have this code: ```scala object A { lazy val x = { ... } val y = { ... } def z(...) = { ... } } ``` The traverser changes `currentOwner` when it enters `x`, `y` and `z` and for every action we pay the initialization cost of another `DependencySource` | `EnclosingNonLocalClass` instance. This commit checks that the resolved non local class is the same, which spare us several writes to disk. It also removes the runtime overhead of `DependencySource` and `EnclosingNonLocalClass` by inlining its fields as members of the enclosing `Traverser` class. Rewritten from sbt/zinc@2f1002f

The previous test was flaky because there was no guarantee that the prefix of `BB` was correctly catched by the type traverser (the name of `a` could have been recognised in the type of `foo` and not in `bar`). The following PR makes sure that the prefix `c` is also captured. Rewritten from sbt/zinc@83bcf50

The following commit fixes the issues with as seen from that are explained in this fantastic issue by Greg: scala#174, as well as scala#239 that handles structural types. These issues are related to the fact that our previous approach was only inspecting types, when some type information like type bounds is only present in symbols. To get that information, we need a more precise search that looks into the core of the Scalac types for the required information. Of course, type bounds is not all we're interested about. The issue is that type members, method parameters and type parameters can have type information in its definition that is necessary at the use site to detect and propagate changes. This information is also tied to the fact that type members can have different materializations depending on the prefix (both because of type members and path-dependent types). `types-in-used-names-b` and `as-seen-from-b` are a perfect example of this. This commit turns them into passing tests. Having more in-depth look at the algorithm behind it will help us understand what it does. In essence, the new type traverser is the responsible of adding dependencies on every `TypeRef` and `SingleType`. They contain concrete information about types (they are materialized), so their presence must be recorded. We also have the presence of other types like `PolyType` and `MethodType`. These types are used for defining type parameters for classes (think List[A]) and method type parameters (think def foo[T](t: T)). They are nested, meaning that their return type can also be a `PolyType` or a `MethodType`. To handle them, we traverse the symbols in their definition -- for method types we traverse the types of the parameters, while for poly types we add directly the dependency on the symbol --so that the name of the type parameters are also recorded-- and then we continue checking for their information if they are not a class, that is, if they are an abstract type with a definition that we may need to traverse (existential type, refined type, bounds, etc). In the case of `TypeBounds`, we traverse them if they are not the default specified by the SLS (`Nothing` for low bound, `Any` for high). Refined types need special handling since we need to check their declarations, that can introduce new type members or vals. If they do have them, we add a dependency right away on those definitions. As usual, `ThisType` and `ConstantType` need to be inspected by checking their underlying representation (`C` in `C.this` and `12` in `Int(12)`). `ExistentialType`, the last type on the traverser before falling back to `mapOver`, has a list of symbols called `quantified` that needs to be traversed since they are the symbol information that constrain the existential type. As in the case of `TypeBounds`, we guard against the default types `Nothing` for low bound and `Any` for high bound, so that unnecessary names that are always present in source files don't appear. This change triggers a very weird behaviour in 2.10, in which for some reason the names `Nothing` and `Any` appear. This does not seem to come from the new TypeDependencyTraverser and I've been able to track its appearance to the case in the traverser where we check for `hasSymbol` and add with `addSymbol`. I've added a TODO, which is not urgent, to find out what's happening, since this only affect one concrete snippet of the whole test code. Benchmark: ``` [info] # Run complete. Total time: 00:25:51 [info] [info] Benchmark (_tempDir) Mode Cnt Score Error Units [info] HotScalacBenchmark.compile /tmp/sbt_abdb5ed2 sample 18 20893.226 ± 625.622 ms/op [info] HotScalacBenchmark.compile:compile·p0.00 /tmp/sbt_abdb5ed2 sample 19797.115 ms/op [info] HotScalacBenchmark.compile:compile·p0.50 /tmp/sbt_abdb5ed2 sample 21005.074 ms/op [info] HotScalacBenchmark.compile:compile·p0.90 /tmp/sbt_abdb5ed2 sample 21894.267 ms/op [info] HotScalacBenchmark.compile:compile·p0.95 /tmp/sbt_abdb5ed2 sample 22045.262 ms/op [info] HotScalacBenchmark.compile:compile·p0.99 /tmp/sbt_abdb5ed2 sample 22045.262 ms/op [info] HotScalacBenchmark.compile:compile·p0.999 /tmp/sbt_abdb5ed2 sample 22045.262 ms/op [info] HotScalacBenchmark.compile:compile·p0.9999 /tmp/sbt_abdb5ed2 sample 22045.262 ms/op [info] HotScalacBenchmark.compile:compile·p1.00 /tmp/sbt_abdb5ed2 sample 22045.262 ms/op [info] HotScalacBenchmark.compile:·gc.alloc.rate /tmp/sbt_abdb5ed2 sample 18 289.838 ± 8.669 MB/sec [info] HotScalacBenchmark.compile:·gc.alloc.rate.norm /tmp/sbt_abdb5ed2 sample 18 6500730176.000 ± 13633760.029 B/op [info] HotScalacBenchmark.compile:·gc.churn.PS_Eden_Space /tmp/sbt_abdb5ed2 sample 18 289.082 ± 24.260 MB/sec [info] HotScalacBenchmark.compile:·gc.churn.PS_Eden_Space.norm /tmp/sbt_abdb5ed2 sample 18 6480403569.778 ± 464987965.594 B/op [info] HotScalacBenchmark.compile:·gc.churn.PS_Old_Gen /tmp/sbt_abdb5ed2 sample 18 12.679 ± 12.697 MB/sec [info] HotScalacBenchmark.compile:·gc.churn.PS_Old_Gen.norm /tmp/sbt_abdb5ed2 sample 18 290767194.667 ± 290528363.065 B/op [info] HotScalacBenchmark.compile:·gc.churn.PS_Survivor_Space /tmp/sbt_abdb5ed2 sample 18 7.321 ± 2.865 MB/sec [info] HotScalacBenchmark.compile:·gc.churn.PS_Survivor_Space.norm /tmp/sbt_abdb5ed2 sample 18 165547052.444 ± 66661097.019 B/op [info] HotScalacBenchmark.compile:·gc.count /tmp/sbt_abdb5ed2 sample 18 101.000 counts [info] HotScalacBenchmark.compile:·gc.time /tmp/sbt_abdb5ed2 sample 18 21332.000 ms [info] WarmScalacBenchmark.compile /tmp/sbt_abdb5ed2 sample 3 52769.937 ± 6743.004 ms/op [info] WarmScalacBenchmark.compile:compile·p0.00 /tmp/sbt_abdb5ed2 sample 52412.023 ms/op [info] WarmScalacBenchmark.compile:compile·p0.50 /tmp/sbt_abdb5ed2 sample 52747.567 ms/op [info] WarmScalacBenchmark.compile:compile·p0.90 /tmp/sbt_abdb5ed2 sample 53150.220 ms/op [info] WarmScalacBenchmark.compile:compile·p0.95 /tmp/sbt_abdb5ed2 sample 53150.220 ms/op [info] WarmScalacBenchmark.compile:compile·p0.99 /tmp/sbt_abdb5ed2 sample 53150.220 ms/op [info] WarmScalacBenchmark.compile:compile·p0.999 /tmp/sbt_abdb5ed2 sample 53150.220 ms/op [info] WarmScalacBenchmark.compile:compile·p0.9999 /tmp/sbt_abdb5ed2 sample 53150.220 ms/op [info] WarmScalacBenchmark.compile:compile·p1.00 /tmp/sbt_abdb5ed2 sample 53150.220 ms/op [info] WarmScalacBenchmark.compile:·gc.alloc.rate /tmp/sbt_abdb5ed2 sample 3 125.382 ± 13.840 MB/sec [info] WarmScalacBenchmark.compile:·gc.alloc.rate.norm /tmp/sbt_abdb5ed2 sample 3 7055970890.667 ± 1078954896.900 B/op [info] WarmScalacBenchmark.compile:·gc.churn.PS_Eden_Space /tmp/sbt_abdb5ed2 sample 3 117.215 ± 73.864 MB/sec [info] WarmScalacBenchmark.compile:·gc.churn.PS_Eden_Space.norm /tmp/sbt_abdb5ed2 sample 3 6596470733.333 ± 4281843293.325 B/op [info] WarmScalacBenchmark.compile:·gc.churn.PS_Survivor_Space /tmp/sbt_abdb5ed2 sample 3 2.279 ± 1.015 MB/sec [info] WarmScalacBenchmark.compile:·gc.churn.PS_Survivor_Space.norm /tmp/sbt_abdb5ed2 sample 3 128269752.000 ± 72721263.065 B/op [info] WarmScalacBenchmark.compile:·gc.count /tmp/sbt_abdb5ed2 sample 3 73.000 counts [info] WarmScalacBenchmark.compile:·gc.time /tmp/sbt_abdb5ed2 sample 3 8746.000 ms [info] ColdScalacBenchmark.compile /tmp/sbt_abdb5ed2 ss 10 44611.286 ± 963.131 ms/op [info] ColdScalacBenchmark.compile:·gc.alloc.rate /tmp/sbt_abdb5ed2 ss 10 152.054 ± 2.753 MB/sec [info] ColdScalacBenchmark.compile:·gc.alloc.rate.norm /tmp/sbt_abdb5ed2 ss 10 7249761568.800 ± 95126804.264 B/op [info] ColdScalacBenchmark.compile:·gc.churn.PS_Eden_Space /tmp/sbt_abdb5ed2 ss 10 144.481 ± 9.964 MB/sec [info] ColdScalacBenchmark.compile:·gc.churn.PS_Eden_Space.norm /tmp/sbt_abdb5ed2 ss 10 6889406191.200 ± 490961958.245 B/op [info] ColdScalacBenchmark.compile:·gc.churn.PS_Old_Gen /tmp/sbt_abdb5ed2 ss 10 ≈ 10⁻³ MB/sec [info] ColdScalacBenchmark.compile:·gc.churn.PS_Old_Gen.norm /tmp/sbt_abdb5ed2 ss 10 21136.000 ± 101049.368 B/op [info] ColdScalacBenchmark.compile:·gc.churn.PS_Survivor_Space /tmp/sbt_abdb5ed2 ss 10 2.848 ± 0.335 MB/sec [info] ColdScalacBenchmark.compile:·gc.churn.PS_Survivor_Space.norm /tmp/sbt_abdb5ed2 ss 10 135792956.800 ± 16291050.509 B/op [info] ColdScalacBenchmark.compile:·gc.count /tmp/sbt_abdb5ed2 ss 10 248.000 counts [info] ColdScalacBenchmark.compile:·gc.time /tmp/sbt_abdb5ed2 ss 10 29901.000 ms [success] Total time: 1553 s, completed Feb 26, 2017 3:06:29 AM [success] Total time: 0 s, completed Feb 26, 2017 3:06:29 AM ``` Rewritten from sbt/zinc@929b758

This commit takes care of speeding up analysis of type dependencies as much as possible. In both `ExtractUsedNames` and `Dependency`, we have a cache function associated with a source symbol. This source symbol is the "key" of the cache in the sense that from it we detect how a dependency should be tracked. `Dependency`, for instance, adds a dependency from `X` to `Y`, where X is the origin symbol and `Y` is the destination symbol. However, only `X` determines how to a dependency should be added (and on which data structure). The same happens for `ExtractAPI`, but whose case is simpler because there is no destination symbol: only the origin symbol is the necessary to cache -- we have a set of names for a given symbol. Our previous type analysis had a type cache, but this type cache only lasted one type traversal. The algorihtm was very pessimistic -- we cleared the `visited` cache with `reinitializeVisited` after every traversal so that members would be correctly recognized if the origin symbol changed. However, the origin symbol usually stays the same, especially when traversing bodies of methods and variables, which contain a high proportion of types. Taking this into account, we arrive to the conclusion that we can keep type caches around as long as the `currentOwner` doesn't change, because dependencies are only registered for top-level classes in both cases (`ExtractAPI` and `Dependency`). The introduced solution allows every phase to implement their own `TypeTraverser` and override the function that takes care of adding a dependency. This is necessary because the functions to add dependencies depend on the context (origin symbols and more stuff), which ultimately varies in `ExtractAPI` and `Dependency`. The following benchmark has been obtained by the same formula as the commit mentioned before, and benchmarks the compilation of the Scala standard library. BEFORE ``` [info] Benchmark (_tempDir) Mode Cnt Score Error Units [info] HotScalacBenchmark.compile /tmp/sbt_b9131bfb sample 18 21228.771 ± 521.207 ms/op [info] HotScalacBenchmark.compile:compile·p0.00 /tmp/sbt_b9131bfb sample 20199.768 ms/op [info] HotScalacBenchmark.compile:compile·p0.50 /tmp/sbt_b9131bfb sample 21256.733 ms/op [info] HotScalacBenchmark.compile:compile·p0.90 /tmp/sbt_b9131bfb sample 21931.177 ms/op [info] HotScalacBenchmark.compile:compile·p0.95 /tmp/sbt_b9131bfb sample 22112.371 ms/op [info] HotScalacBenchmark.compile:compile·p0.99 /tmp/sbt_b9131bfb sample 22112.371 ms/op [info] HotScalacBenchmark.compile:compile·p0.999 /tmp/sbt_b9131bfb sample 22112.371 ms/op [info] HotScalacBenchmark.compile:compile·p0.9999 /tmp/sbt_b9131bfb sample 22112.371 ms/op [info] HotScalacBenchmark.compile:compile·p1.00 /tmp/sbt_b9131bfb sample 22112.371 ms/op [info] HotScalacBenchmark.compile:·gc.alloc.rate /tmp/sbt_b9131bfb sample 18 284.115 ± 6.036 MB/sec [info] HotScalacBenchmark.compile:·gc.alloc.rate.norm /tmp/sbt_b9131bfb sample 18 6474818679.556 ± 42551265.360 B/op [info] HotScalacBenchmark.compile:·gc.churn.PS_Eden_Space /tmp/sbt_b9131bfb sample 18 283.385 ± 23.147 MB/sec [info] HotScalacBenchmark.compile:·gc.churn.PS_Eden_Space.norm /tmp/sbt_b9131bfb sample 18 6455703779.556 ± 483463770.519 B/op [info] HotScalacBenchmark.compile:·gc.churn.PS_Old_Gen /tmp/sbt_b9131bfb sample 18 12.857 ± 12.406 MB/sec [info] HotScalacBenchmark.compile:·gc.churn.PS_Old_Gen.norm /tmp/sbt_b9131bfb sample 18 297978002.222 ± 287556197.389 B/op [info] HotScalacBenchmark.compile:·gc.churn.PS_Survivor_Space /tmp/sbt_b9131bfb sample 18 6.901 ± 2.092 MB/sec [info] HotScalacBenchmark.compile:·gc.churn.PS_Survivor_Space.norm /tmp/sbt_b9131bfb sample 18 158212212.444 ± 50375116.805 B/op [info] HotScalacBenchmark.compile:·gc.count /tmp/sbt_b9131bfb sample 18 105.000 counts [info] HotScalacBenchmark.compile:·gc.time /tmp/sbt_b9131bfb sample 18 21814.000 ms [info] WarmScalacBenchmark.compile /tmp/sbt_b9131bfb sample 3 55924.053 ± 16257.754 ms/op [info] WarmScalacBenchmark.compile:compile·p0.00 /tmp/sbt_b9131bfb sample 54895.051 ms/op [info] WarmScalacBenchmark.compile:compile·p0.50 /tmp/sbt_b9131bfb sample 56438.555 ms/op [info] WarmScalacBenchmark.compile:compile·p0.90 /tmp/sbt_b9131bfb sample 56438.555 ms/op [info] WarmScalacBenchmark.compile:compile·p0.95 /tmp/sbt_b9131bfb sample 56438.555 ms/op [info] WarmScalacBenchmark.compile:compile·p0.99 /tmp/sbt_b9131bfb sample 56438.555 ms/op [info] WarmScalacBenchmark.compile:compile·p0.999 /tmp/sbt_b9131bfb sample 56438.555 ms/op [info] WarmScalacBenchmark.compile:compile·p0.9999 /tmp/sbt_b9131bfb sample 56438.555 ms/op [info] WarmScalacBenchmark.compile:compile·p1.00 /tmp/sbt_b9131bfb sample 56438.555 ms/op [info] WarmScalacBenchmark.compile:·gc.alloc.rate /tmp/sbt_b9131bfb sample 3 117.417 ± 27.439 MB/sec [info] WarmScalacBenchmark.compile:·gc.alloc.rate.norm /tmp/sbt_b9131bfb sample 3 6999695530.667 ± 608845574.720 B/op [info] WarmScalacBenchmark.compile:·gc.churn.PS_Eden_Space /tmp/sbt_b9131bfb sample 3 111.263 ± 90.263 MB/sec [info] WarmScalacBenchmark.compile:·gc.churn.PS_Eden_Space.norm /tmp/sbt_b9131bfb sample 3 6633605792.000 ± 5698534573.516 B/op [info] WarmScalacBenchmark.compile:·gc.churn.PS_Old_Gen /tmp/sbt_b9131bfb sample 3 0.001 ± 0.040 MB/sec [info] WarmScalacBenchmark.compile:·gc.churn.PS_Old_Gen.norm /tmp/sbt_b9131bfb sample 3 74741.333 ± 2361755.471 B/op [info] WarmScalacBenchmark.compile:·gc.churn.PS_Survivor_Space /tmp/sbt_b9131bfb sample 3 2.478 ± 7.592 MB/sec [info] WarmScalacBenchmark.compile:·gc.churn.PS_Survivor_Space.norm /tmp/sbt_b9131bfb sample 3 147881869.333 ± 475964254.946 B/op [info] WarmScalacBenchmark.compile:·gc.count /tmp/sbt_b9131bfb sample 3 73.000 counts [info] WarmScalacBenchmark.compile:·gc.time /tmp/sbt_b9131bfb sample 3 9581.000 ms [info] ColdScalacBenchmark.compile /tmp/sbt_b9131bfb ss 10 45562.453 ± 836.977 ms/op [info] ColdScalacBenchmark.compile:·gc.alloc.rate /tmp/sbt_b9131bfb ss 10 147.126 ± 2.229 MB/sec [info] ColdScalacBenchmark.compile:·gc.alloc.rate.norm /tmp/sbt_b9131bfb ss 10 7163351651.200 ± 57993163.779 B/op [info] ColdScalacBenchmark.compile:·gc.churn.PS_Eden_Space /tmp/sbt_b9131bfb ss 10 137.407 ± 6.810 MB/sec [info] ColdScalacBenchmark.compile:·gc.churn.PS_Eden_Space.norm /tmp/sbt_b9131bfb ss 10 6692512710.400 ± 429243418.572 B/op [info] ColdScalacBenchmark.compile:·gc.churn.PS_Survivor_Space /tmp/sbt_b9131bfb ss 10 2.647 ± 0.168 MB/sec [info] ColdScalacBenchmark.compile:·gc.churn.PS_Survivor_Space.norm /tmp/sbt_b9131bfb ss 10 128840603.200 ± 7324571.862 B/op [info] ColdScalacBenchmark.compile:·gc.count /tmp/sbt_b9131bfb ss 10 245.000 counts [info] ColdScalacBenchmark.compile:·gc.time /tmp/sbt_b9131bfb ss 10 29462.000 ms [success] Total time: 1595 s, completed Feb 26, 2017 1:42:55 AM [success] Total time: 0 s, completed Feb 26, 2017 1:42:55 AM ``` AFTER ``` [info] Benchmark (_tempDir) Mode Cnt Score Error Units [info] HotScalacBenchmark.compile /tmp/sbt_c8a4806b sample 18 20757.144 ± 519.221 ms/op [info] HotScalacBenchmark.compile:compile·p0.00 /tmp/sbt_c8a4806b sample 19931.333 ms/op [info] HotScalacBenchmark.compile:compile·p0.50 /tmp/sbt_c8a4806b sample 20786.971 ms/op [info] HotScalacBenchmark.compile:compile·p0.90 /tmp/sbt_c8a4806b sample 21615.765 ms/op [info] HotScalacBenchmark.compile:compile·p0.95 /tmp/sbt_c8a4806b sample 21676.163 ms/op [info] HotScalacBenchmark.compile:compile·p0.99 /tmp/sbt_c8a4806b sample 21676.163 ms/op [info] HotScalacBenchmark.compile:compile·p0.999 /tmp/sbt_c8a4806b sample 21676.163 ms/op [info] HotScalacBenchmark.compile:compile·p0.9999 /tmp/sbt_c8a4806b sample 21676.163 ms/op [info] HotScalacBenchmark.compile:compile·p1.00 /tmp/sbt_c8a4806b sample 21676.163 ms/op [info] HotScalacBenchmark.compile:·gc.alloc.rate /tmp/sbt_c8a4806b sample 18 290.476 ± 7.069 MB/sec [info] HotScalacBenchmark.compile:·gc.alloc.rate.norm /tmp/sbt_c8a4806b sample 18 6476081869.778 ± 18700713.424 B/op [info] HotScalacBenchmark.compile:·gc.churn.PS_Eden_Space /tmp/sbt_c8a4806b sample 18 290.409 ± 20.336 MB/sec [info] HotScalacBenchmark.compile:·gc.churn.PS_Eden_Space.norm /tmp/sbt_c8a4806b sample 18 6478102528.000 ± 468310673.653 B/op [info] HotScalacBenchmark.compile:·gc.churn.PS_Old_Gen /tmp/sbt_c8a4806b sample 18 13.261 ± 12.790 MB/sec [info] HotScalacBenchmark.compile:·gc.churn.PS_Old_Gen.norm /tmp/sbt_c8a4806b sample 18 301324965.333 ± 290518111.715 B/op [info] HotScalacBenchmark.compile:·gc.churn.PS_Survivor_Space /tmp/sbt_c8a4806b sample 18 6.735 ± 2.338 MB/sec [info] HotScalacBenchmark.compile:·gc.churn.PS_Survivor_Space.norm /tmp/sbt_c8a4806b sample 18 150953349.778 ± 54074639.209 B/op [info] HotScalacBenchmark.compile:·gc.count /tmp/sbt_c8a4806b sample 18 101.000 counts [info] HotScalacBenchmark.compile:·gc.time /tmp/sbt_c8a4806b sample 18 21267.000 ms [info] WarmScalacBenchmark.compile /tmp/sbt_c8a4806b sample 3 54380.549 ± 24064.367 ms/op [info] WarmScalacBenchmark.compile:compile·p0.00 /tmp/sbt_c8a4806b sample 53552.873 ms/op [info] WarmScalacBenchmark.compile:compile·p0.50 /tmp/sbt_c8a4806b sample 53687.091 ms/op [info] WarmScalacBenchmark.compile:compile·p0.90 /tmp/sbt_c8a4806b sample 55901.684 ms/op [info] WarmScalacBenchmark.compile:compile·p0.95 /tmp/sbt_c8a4806b sample 55901.684 ms/op [info] WarmScalacBenchmark.compile:compile·p0.99 /tmp/sbt_c8a4806b sample 55901.684 ms/op [info] WarmScalacBenchmark.compile:compile·p0.999 /tmp/sbt_c8a4806b sample 55901.684 ms/op [info] WarmScalacBenchmark.compile:compile·p0.9999 /tmp/sbt_c8a4806b sample 55901.684 ms/op [info] WarmScalacBenchmark.compile:compile·p1.00 /tmp/sbt_c8a4806b sample 55901.684 ms/op [info] WarmScalacBenchmark.compile:·gc.alloc.rate /tmp/sbt_c8a4806b sample 3 120.159 ± 52.914 MB/sec [info] WarmScalacBenchmark.compile:·gc.alloc.rate.norm /tmp/sbt_c8a4806b sample 3 6963979373.333 ± 137408036.138 B/op [info] WarmScalacBenchmark.compile:·gc.churn.PS_Eden_Space /tmp/sbt_c8a4806b sample 3 113.755 ± 135.915 MB/sec [info] WarmScalacBenchmark.compile:·gc.churn.PS_Eden_Space.norm /tmp/sbt_c8a4806b sample 3 6588595392.000 ± 5170161565.753 B/op [info] WarmScalacBenchmark.compile:·gc.churn.PS_Old_Gen /tmp/sbt_c8a4806b sample 3 0.002 ± 0.048 MB/sec [info] WarmScalacBenchmark.compile:·gc.churn.PS_Old_Gen.norm /tmp/sbt_c8a4806b sample 3 90400.000 ± 2856554.534 B/op [info] WarmScalacBenchmark.compile:·gc.churn.PS_Survivor_Space /tmp/sbt_c8a4806b sample 3 2.623 ± 7.378 MB/sec [info] WarmScalacBenchmark.compile:·gc.churn.PS_Survivor_Space.norm /tmp/sbt_c8a4806b sample 3 151896768.000 ± 399915676.894 B/op [info] WarmScalacBenchmark.compile:·gc.count /tmp/sbt_c8a4806b sample 3 73.000 counts [info] WarmScalacBenchmark.compile:·gc.time /tmp/sbt_c8a4806b sample 3 10070.000 ms [info] ColdScalacBenchmark.compile /tmp/sbt_c8a4806b ss 10 45613.670 ± 1724.291 ms/op [info] ColdScalacBenchmark.compile:·gc.alloc.rate /tmp/sbt_c8a4806b ss 10 147.106 ± 4.973 MB/sec [info] ColdScalacBenchmark.compile:·gc.alloc.rate.norm /tmp/sbt_c8a4806b ss 10 7165665000.000 ± 68500786.134 B/op [info] ColdScalacBenchmark.compile:·gc.churn.PS_Eden_Space /tmp/sbt_c8a4806b ss 10 138.633 ± 12.612 MB/sec [info] ColdScalacBenchmark.compile:·gc.churn.PS_Eden_Space.norm /tmp/sbt_c8a4806b ss 10 6749057403.200 ± 438983252.418 B/op [info] ColdScalacBenchmark.compile:·gc.churn.PS_Survivor_Space /tmp/sbt_c8a4806b ss 10 2.716 ± 0.298 MB/sec [info] ColdScalacBenchmark.compile:·gc.churn.PS_Survivor_Space.norm /tmp/sbt_c8a4806b ss 10 132216236.800 ± 11751803.094 B/op [info] ColdScalacBenchmark.compile:·gc.count /tmp/sbt_c8a4806b ss 10 247.000 counts [info] ColdScalacBenchmark.compile:·gc.time /tmp/sbt_c8a4806b ss 10 29965.000 ms [success] Total time: 1593 s, completed Feb 26, 2017 11:54:01 AM [success] Total time: 0 s, completed Feb 26, 2017 11:54:01 AM ``` Machine info: ``` jvican in /data/rw/code/scala/zinc [22:24:47] > $ uname -a [±as-seen-from ●▴▾] Linux tribox 4.9.11-1-ARCH #1 SMP PREEMPT Sun Feb 19 13:45:52 UTC 2017 x86_64 GNU/Linux jvican in /data/rw/code/scala/zinc [23:15:57] > $ cpupower frequency-info [±as-seen-from ●▴▾] analyzing CPU 0: driver: intel_pstate CPUs which run at the same hardware frequency: 0 CPUs which need to have their frequency coordinated by software: 0 maximum transition latency: Cannot determine or is not supported. hardware limits: 400 MHz - 3.40 GHz available cpufreq governors: performance powersave current policy: frequency should be within 3.20 GHz and 3.20 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency: Unable to call hardware current CPU frequency: 3.32 GHz (asserted by call to kernel) boost state support: Supported: yes Active: yes jvican in /data/rw/code/scala/zinc [23:16:14] > $ cat /proc/meminfo [±as-seen-from ●▴▾] MemTotal: 20430508 kB MemFree: 9890712 kB MemAvailable: 13490908 kB Buffers: 3684 kB Cached: 4052520 kB SwapCached: 0 kB Active: 7831612 kB Inactive: 2337220 kB Active(anon): 6214680 kB Inactive(anon): 151436 kB Active(file): 1616932 kB Inactive(file): 2185784 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 12582908 kB SwapFree: 12582908 kB Dirty: 124 kB Writeback: 0 kB AnonPages: 6099876 kB Mapped: 183096 kB Shmem: 253488 kB Slab: 227436 kB SReclaimable: 152144 kB SUnreclaim: 75292 kB KernelStack: 5152 kB PageTables: 19636 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 22798160 kB Committed_AS: 7685996 kB VmallocTotal: 34359738367 kB VmallocUsed: 0 kB VmallocChunk: 0 kB HardwareCorrupted: 0 kB AnonHugePages: 5511168 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 136620 kB DirectMap2M: 4970496 kB DirectMap1G: 15728640 kB jvican in /data/rw/code/scala/zinc [23:16:41] > $ cat /proc/cpuinfo [±as-seen-from ●▴▾] processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 78 model name : Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz stepping : 3 microcode : 0x88 cpu MHz : 3297.827 cache size : 4096 KB physical id : 0 siblings : 4 core id : 0 cpu cores : 2 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 22 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp bugs : bogomips : 5618.00 clflush size : 64 cache_alignment : 64 address sizes : 39 bits physical, 48 bits virtual power management: processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 78 model name : Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz stepping : 3 microcode : 0x88 cpu MHz : 3296.459 cache size : 4096 KB physical id : 0 siblings : 4 core id : 1 cpu cores : 2 apicid : 2 initial apicid : 2 fpu : yes fpu_exception : yes cpuid level : 22 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp bugs : bogomips : 5620.22 clflush size : 64 cache_alignment : 64 address sizes : 39 bits physical, 48 bits virtual power management: processor : 2 vendor_id : GenuineIntel cpu family : 6 model : 78 model name : Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz stepping : 3 microcode : 0x88 cpu MHz : 3399.853 cache size : 4096 KB physical id : 0 siblings : 4 core id : 0 cpu cores : 2 apicid : 1 initial apicid : 1 fpu : yes fpu_exception : yes cpuid level : 22 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp bugs : bogomips : 5621.16 clflush size : 64 cache_alignment : 64 address sizes : 39 bits physical, 48 bits virtual power management: processor : 3 vendor_id : GenuineIntel cpu family : 6 model : 78 model name : Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz stepping : 3 microcode : 0x88 cpu MHz : 3210.327 cache size : 4096 KB physical id : 0 siblings : 4 core id : 1 cpu cores : 2 apicid : 3 initial apicid : 3 fpu : yes fpu_exception : yes cpuid level : 22 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp bugs : bogomips : 5620.33 clflush size : 64 cache_alignment : 64 address sizes : 39 bits physical, 48 bits virtual power management: ``` In comparison with df308723, the new changes improve the running time of Zinc by half a second in hot and warm benchmarks, and a decrease of 100ms for cold benchmarks, which seems to be product of the variation given the number of ms/op. It is a success taking into account that now we're traversing more types and symbols than before, so these changes allow us to do more work and still decrease the running time of Zinc. These changes are likely to have a bigger effect on huge industrial codebases in which the ratio of types is very high, and with a lot of rich types like poly types, method types, refinements and existential types that have lots of constraints. Rewritten from sbt/zinc@1cb2382

@eed3si9n

These options have been removed per @eed3si9n's suggestion. They are not valuable anymore since they served to transition from the previous algorithm to the new one. This commit removes `antStyle` and `nameHashing` completely, the first one being inexistent and the second one being always true. Rewritten from sbt/zinc@18a8bda

Commit 929b7589bbf02140b17de04106b7162eebc5db21 introduced a decrease of 20 seconds -- from 30 seconds to 10 seconds -- because it was being optimistic. This timer is the responsible of making this test pass, and though 10 seconds is a very good time and we've passed it, it can happen that compilation lasts more than expected and fails the tests. For that reason, and to find a good tradeoff, this commit raises the timer from 10 seconds to 20 seconds, 1/3 less of what we had originally before this PR. Rewritten from sbt/zinc@f758e3d

Rewritten from sbt/zinc@6353aea

Fixes scala#233 Rewritten from sbt/zinc@bb94aee

This issue was fixed in https://github.com/sbt/zinc/pull/239/files#diff-65c95a9d18dc8a76c6182e1c6377fc65R154. For some reason, 2.10 compiler is detecting root packages to have symbols fields (which 2.11 do not recognise as such), so we need to protect from them in `ExtractUsedNames` since `_root_` should definitely not be registered as a name, because it's present in any Scala source file. Rewritten from sbt/zinc@3b3be86

…used vals to _ Rewritten from sbt/zinc@63695f4

…y.scala and related classes Benchmark results: With Scala collections: [info] Benchmark (_tempDir) Mode Cnt Score Error Units [info] HotScalacBenchmark.compile /tmp/sbt_51938d03 sample 18 21437.554 ± 949.595 ms/op [info] HotScalacBenchmark.compile:compile·p0.00 /tmp/sbt_51938d03 sample 19964.887 ms/op [info] HotScalacBenchmark.compile:compile·p0.50 /tmp/sbt_51938d03 sample 21407.728 ms/op [info] HotScalacBenchmark.compile:compile·p0.90 /tmp/sbt_51938d03 sample 22541.867 ms/op [info] HotScalacBenchmark.compile:compile·p0.95 /tmp/sbt_51938d03 sample 24595.399 ms/op [info] HotScalacBenchmark.compile:compile·p0.99 /tmp/sbt_51938d03 sample 24595.399 ms/op [info] HotScalacBenchmark.compile:compile·p0.999 /tmp/sbt_51938d03 sample 24595.399 ms/op [info] HotScalacBenchmark.compile:compile·p0.9999 /tmp/sbt_51938d03 sample 24595.399 ms/op [info] HotScalacBenchmark.compile:compile·p1.00 /tmp/sbt_51938d03 sample 24595.399 ms/op [info] HotScalacBenchmark.compile:·gc.alloc.rate /tmp/sbt_51938d03 sample 18 280.535 ± 11.459 MB/sec [info] HotScalacBenchmark.compile:·gc.alloc.rate.norm /tmp/sbt_51938d03 sample 18 6446299745.333 ± 14785544.668 B/op [info] HotScalacBenchmark.compile:·gc.churn.PS_Eden_Space /tmp/sbt_51938d03 sample 18 276.572 ± 22.333 MB/sec [info] HotScalacBenchmark.compile:·gc.churn.PS_Eden_Space.norm /tmp/sbt_51938d03 sample 18 6355643405.333 ± 460709368.527 B/op [info] HotScalacBenchmark.compile:·gc.churn.PS_Old_Gen /tmp/sbt_51938d03 sample 18 11.677 ± 12.619 MB/sec [info] HotScalacBenchmark.compile:·gc.churn.PS_Old_Gen.norm /tmp/sbt_51938d03 sample 18 275401288.000 ± 296971980.543 B/op [info] HotScalacBenchmark.compile:·gc.churn.PS_Survivor_Space /tmp/sbt_51938d03 sample 18 6.754 ± 1.900 MB/sec [info] HotScalacBenchmark.compile:·gc.churn.PS_Survivor_Space.norm /tmp/sbt_51938d03 sample 18 155936858.667 ± 45709221.663 B/op [info] HotScalacBenchmark.compile:·gc.count /tmp/sbt_51938d03 sample 18 98.000 counts [info] HotScalacBenchmark.compile:·gc.time /tmp/sbt_51938d03 sample 18 19073.000 ms [info] WarmScalacBenchmark.compile /tmp/sbt_51938d03 sample 3 55118.747 ± 18980.181 ms/op [info] WarmScalacBenchmark.compile:compile·p0.00 /tmp/sbt_51938d03 sample 54358.180 ms/op [info] WarmScalacBenchmark.compile:compile·p0.50 /tmp/sbt_51938d03 sample 54693.724 ms/op [info] WarmScalacBenchmark.compile:compile·p0.90 /tmp/sbt_51938d03 sample 56304.337 ms/op [info] WarmScalacBenchmark.compile:compile·p0.95 /tmp/sbt_51938d03 sample 56304.337 ms/op [info] WarmScalacBenchmark.compile:compile·p0.99 /tmp/sbt_51938d03 sample 56304.337 ms/op [info] WarmScalacBenchmark.compile:compile·p0.999 /tmp/sbt_51938d03 sample 56304.337 ms/op [info] WarmScalacBenchmark.compile:compile·p0.9999 /tmp/sbt_51938d03 sample 56304.337 ms/op [info] WarmScalacBenchmark.compile:compile·p1.00 /tmp/sbt_51938d03 sample 56304.337 ms/op [info] WarmScalacBenchmark.compile:·gc.alloc.rate /tmp/sbt_51938d03 sample 3 119.464 ± 35.856 MB/sec [info] WarmScalacBenchmark.compile:·gc.alloc.rate.norm /tmp/sbt_51938d03 sample 3 7018134616.000 ± 329861736.207 B/op [info] WarmScalacBenchmark.compile:·gc.churn.PS_Eden_Space /tmp/sbt_51938d03 sample 3 109.472 ± 43.096 MB/sec [info] WarmScalacBenchmark.compile:·gc.churn.PS_Eden_Space.norm /tmp/sbt_51938d03 sample 3 6431816405.333 ± 2656847732.046 B/op [info] WarmScalacBenchmark.compile:·gc.churn.PS_Survivor_Space /tmp/sbt_51938d03 sample 3 2.692 ± 2.508 MB/sec [info] WarmScalacBenchmark.compile:·gc.churn.PS_Survivor_Space.norm /tmp/sbt_51938d03 sample 3 158121976.000 ± 114580201.514 B/op [info] WarmScalacBenchmark.compile:·gc.count /tmp/sbt_51938d03 sample 3 76.000 counts [info] WarmScalacBenchmark.compile:·gc.time /tmp/sbt_51938d03 sample 3 9389.000 ms [info] ColdScalacBenchmark.compile /tmp/sbt_51938d03 ss 10 48069.209 ± 3267.219 ms/op [info] ColdScalacBenchmark.compile:·gc.alloc.rate /tmp/sbt_51938d03 ss 10 140.486 ± 9.407 MB/sec [info] ColdScalacBenchmark.compile:·gc.alloc.rate.norm /tmp/sbt_51938d03 ss 10 7201372165.600 ± 51736683.622 B/op [info] ColdScalacBenchmark.compile:·gc.churn.PS_Eden_Space /tmp/sbt_51938d03 ss 10 133.059 ± 9.610 MB/sec [info] ColdScalacBenchmark.compile:·gc.churn.PS_Eden_Space.norm /tmp/sbt_51938d03 ss 10 6824023763.200 ± 383438006.440 B/op [info] ColdScalacBenchmark.compile:·gc.churn.PS_Survivor_Space /tmp/sbt_51938d03 ss 10 3.092 ± 0.517 MB/sec [info] ColdScalacBenchmark.compile:·gc.churn.PS_Survivor_Space.norm /tmp/sbt_51938d03 ss 10 159124314.400 ± 34649991.106 B/op [info] ColdScalacBenchmark.compile:·gc.count /tmp/sbt_51938d03 ss 10 255.000 counts [info] ColdScalacBenchmark.compile:·gc.time /tmp/sbt_51938d03 ss 10 30967.000 ms With Java collections: [info] Benchmark (_tempDir) Mode Cnt Score Error Units [info] HotScalacBenchmark.compile /tmp/sbt_55e2f965 sample 18 21213.858 ± 627.059 ms/op [info] HotScalacBenchmark.compile:compile·p0.00 /tmp/sbt_55e2f965 sample 20199.768 ms/op [info] HotScalacBenchmark.compile:compile·p0.50 /tmp/sbt_55e2f965 sample 21189.624 ms/op [info] HotScalacBenchmark.compile:compile·p0.90 /tmp/sbt_55e2f965 sample 22132.503 ms/op [info] HotScalacBenchmark.compile:compile·p0.95 /tmp/sbt_55e2f965 sample 22313.697 ms/op [info] HotScalacBenchmark.compile:compile·p0.99 /tmp/sbt_55e2f965 sample 22313.697 ms/op [info] HotScalacBenchmark.compile:compile·p0.999 /tmp/sbt_55e2f965 sample 22313.697 ms/op [info] HotScalacBenchmark.compile:compile·p0.9999 /tmp/sbt_55e2f965 sample 22313.697 ms/op [info] HotScalacBenchmark.compile:compile·p1.00 /tmp/sbt_55e2f965 sample 22313.697 ms/op [info] HotScalacBenchmark.compile:·gc.alloc.rate /tmp/sbt_55e2f965 sample 18 283.808 ± 7.726 MB/sec [info] HotScalacBenchmark.compile:·gc.alloc.rate.norm /tmp/sbt_55e2f965 sample 18 6463578940.444 ± 48501608.706 B/op [info] HotScalacBenchmark.compile:·gc.churn.PS_Eden_Space /tmp/sbt_55e2f965 sample 18 283.368 ± 18.931 MB/sec [info] HotScalacBenchmark.compile:·gc.churn.PS_Eden_Space.norm /tmp/sbt_55e2f965 sample 18 6458616490.667 ± 464534032.338 B/op [info] HotScalacBenchmark.compile:·gc.churn.PS_Old_Gen /tmp/sbt_55e2f965 sample 18 14.097 ± 12.464 MB/sec [info] HotScalacBenchmark.compile:·gc.churn.PS_Old_Gen.norm /tmp/sbt_55e2f965 sample 18 327768881.778 ± 289039158.120 B/op [info] HotScalacBenchmark.compile:·gc.churn.PS_Survivor_Space /tmp/sbt_55e2f965 sample 18 7.283 ± 3.030 MB/sec [info] HotScalacBenchmark.compile:·gc.churn.PS_Survivor_Space.norm /tmp/sbt_55e2f965 sample 18 167634607.111 ± 73095498.785 B/op [info] HotScalacBenchmark.compile:·gc.count /tmp/sbt_55e2f965 sample 18 106.000 counts [info] HotScalacBenchmark.compile:·gc.time /tmp/sbt_55e2f965 sample 18 22282.000 ms [info] WarmScalacBenchmark.compile /tmp/sbt_55e2f965 sample 3 53329.177 ± 5520.744 ms/op [info] WarmScalacBenchmark.compile:compile·p0.00 /tmp/sbt_55e2f965 sample 53016.003 ms/op [info] WarmScalacBenchmark.compile:compile·p0.50 /tmp/sbt_55e2f965 sample 53351.547 ms/op [info] WarmScalacBenchmark.compile:compile·p0.90 /tmp/sbt_55e2f965 sample 53619.982 ms/op [info] WarmScalacBenchmark.compile:compile·p0.95 /tmp/sbt_55e2f965 sample 53619.982 ms/op [info] WarmScalacBenchmark.compile:compile·p0.99 /tmp/sbt_55e2f965 sample 53619.982 ms/op [info] WarmScalacBenchmark.compile:compile·p0.999 /tmp/sbt_55e2f965 sample 53619.982 ms/op [info] WarmScalacBenchmark.compile:compile·p0.9999 /tmp/sbt_55e2f965 sample 53619.982 ms/op [info] WarmScalacBenchmark.compile:compile·p1.00 /tmp/sbt_55e2f965 sample 53619.982 ms/op [info] WarmScalacBenchmark.compile:·gc.alloc.rate /tmp/sbt_55e2f965 sample 3 122.504 ± 12.714 MB/sec [info] WarmScalacBenchmark.compile:·gc.alloc.rate.norm /tmp/sbt_55e2f965 sample 3 6970875709.333 ± 68053507.295 B/op [info] WarmScalacBenchmark.compile:·gc.churn.PS_Eden_Space /tmp/sbt_55e2f965 sample 3 116.331 ± 69.758 MB/sec [info] WarmScalacBenchmark.compile:·gc.churn.PS_Eden_Space.norm /tmp/sbt_55e2f965 sample 3 6619001184.000 ± 3344291079.994 B/op [info] WarmScalacBenchmark.compile:·gc.churn.PS_Old_Gen /tmp/sbt_55e2f965 sample 3 0.001 ± 0.027 MB/sec [info] WarmScalacBenchmark.compile:·gc.churn.PS_Old_Gen.norm /tmp/sbt_55e2f965 sample 3 48530.667 ± 1533523.185 B/op [info] WarmScalacBenchmark.compile:·gc.churn.PS_Survivor_Space /tmp/sbt_55e2f965 sample 3 2.630 ± 0.121 MB/sec [info] WarmScalacBenchmark.compile:·gc.churn.PS_Survivor_Space.norm /tmp/sbt_55e2f965 sample 3 149638445.333 ± 17459243.351 B/op [info] WarmScalacBenchmark.compile:·gc.count /tmp/sbt_55e2f965 sample 3 79.000 counts [info] WarmScalacBenchmark.compile:·gc.time /tmp/sbt_55e2f965 sample 3 9138.000 ms [info] ColdScalacBenchmark.compile /tmp/sbt_55e2f965 ss 10 45129.825 ± 1432.394 ms/op [info] ColdScalacBenchmark.compile:·gc.alloc.rate /tmp/sbt_55e2f965 ss 10 149.365 ± 3.975 MB/sec [info] ColdScalacBenchmark.compile:·gc.alloc.rate.norm /tmp/sbt_55e2f965 ss 10 7202488660.000 ± 86462765.565 B/op [info] ColdScalacBenchmark.compile:·gc.churn.PS_Eden_Space /tmp/sbt_55e2f965 ss 10 139.420 ± 4.400 MB/sec [info] ColdScalacBenchmark.compile:·gc.churn.PS_Eden_Space.norm /tmp/sbt_55e2f965 ss 10 6724772576.000 ± 286711519.947 B/op [info] ColdScalacBenchmark.compile:·gc.churn.PS_Survivor_Space /tmp/sbt_55e2f965 ss 10 3.052 ± 0.295 MB/sec [info] ColdScalacBenchmark.compile:·gc.churn.PS_Survivor_Space.norm /tmp/sbt_55e2f965 ss 10 147178454.400 ± 13834656.130 B/op [info] ColdScalacBenchmark.compile:·gc.count /tmp/sbt_55e2f965 ss 10 252.000 counts [info] ColdScalacBenchmark.compile:·gc.time /tmp/sbt_55e2f965 ss 10 29945.000 ms [success] Total time: 1575 s, completed Mar 6, 2017 3:47:55 PM [success] Total time: 0 s, completed Mar 6, 2017 3:47:55 PM Rewritten from sbt/zinc@3435826

Rewritten from sbt/zinc@476bbf3

Rewritten from sbt/zinc@0328ba4

Rewritten from sbt/zinc@87e53ec

So it also means that we unified usages of implicit and macro names. Better serialization is required. Rewritten from sbt/zinc@f43fecf

Now only files that use sealed class as pattern match target gets recompiled when we add/modify children of sealed class/trait. Rewritten from sbt/zinc@acdd58d

This commit makes minor changes to `ExtractUsednames` in the spirit of better readability and some microoptimization to leave it the way it was before (see the rewrite of `add` by a `contains` + `add`). It also makes some changes to the API: sets some classes to final and renames `ClassFileManagers` to `ClassFileManager` for consistency with the rest of the API. In the future, I'm happy to consider a name change, but for now it's better to stick to the convention of `ClassFileManager` being acting like a "companion object". Rewritten from sbt/zinc@1246df8

Rewritten from sbt/zinc@6119146

Original type trees have to be traversed if present to correctly handle dependencies specified in explicit, user-defined types that are expanded by the compiler, in cases such as type projections. The example `fuzzy-types` is undercompiling because the prefix `FactoryProvider` is totally lost at `Usage` after typer. This is one of these cases where Scalac is expanding types and not leaving any trace to keep track of the original dependency. `FactoryProvider.type#MyFactory#Product` becomes `foo.B with foo.Nil#Product`, and therefore the type dependency traverser doesn't see `FactoryProvider`. Traversing original type trees fixes the issue. Rewritten from sbt/zinc@662e602

Implement used scopes for used names and implement PatMat scope Rewritten from sbt/zinc@c9d567f

Fix scala#101: Remove SimpleType Rewritten from sbt/zinc@172cc90

Remove the `lazy` attribute for the `parent` field and the `declared` field of the `Structure` type. The `inherited` field require the `lazy` attribute, but `parent` and `declared` should not need it. Rewritten from sbt/zinc@7ee45c7

Remove the `lazy` attribute for the `selfType` field and the `structure` field of the `ClassLike` definition. Rewritten from sbt/zinc@d1b18bd

This reverts commit d1b18bd62a0e083bd16afe321ef6fb2de92f08b6. Rewritten from sbt/zinc@f9055e3

This reverts commit 7ee45c714b62dc02f83b157f762ca9dcb495a171. Rewritten from sbt/zinc@ae0d474

The same as sbt/librarymanagement#87. Rewritten from sbt/zinc@b893f80

Rewritten from sbt/zinc@fd323ab

Rewritten from sbt/zinc@baad700

This fixes sbt console. git checkout 0328ba478e2fce1fb17919ba60bfccffe64e30f0^ \ internal/compiler-bridge/src/main/scala/xsbt/ConsoleInterface.scala Rewritten from sbt/zinc@c92ab3c

Rewritten from sbt/zinc@07647b9

Rewritten from sbt/zinc@1abf6ca

…r changes The name hashing algorithm is designed to take implicit conversions into account: when a name "foo" is changed somewhere in a dependency of X, you have to recompile X if it uses the name "foo", even if the usage of "foo" in X is completely unrelated, just because this might have an effect on available implicit conversions. However, there is one case where we can be sure that implicit conversions will not kick in: when we call a constructor. A constructor name is always "<init>", this PR now replaces this name by "pkgA;pkgB;className;init;", this mean that we no longer recompile classes when an unrelated constructor in a used class changed (see the new test `constructors-unrelated` for an example). Rewritten from sbt/zinc@aca8dfa

Remove unused `GlobalCompat` for compatibility with 2.8.1 and extract the run subclassing out of the main logic of the `run` method. Rewritten from sbt/zinc@8a86b55

The callback implementation should be independent of the compiler interface and the source code should reflect so. Breaking change in comparison with the previous API: `Compiler` is now called `ZincCompiler` to make the naming more clear and better distinguish between `CachedCompiler` and its underlying `ZincCompiler`. Docs have been added and simplifications to methods have been provided and double checked with previous Scala versions. For instance, `dropRun` has been confirmed to only exist in 2.10.x series and removed in 2.11.x. Therefore, its implementation has been moved to the compatibility traits for 2.10.x series. We then remove a reflective call for versions above 2.10.6. Rewritten from sbt/zinc@e39dd51

`OutputSetting` is like `OutputGroup` by just returns strings instead of files. I see no reason why `OutputGroup` should not be used instead, therefore simplifying the Zinc API and allowing users to have access to files and not mere strings. If they want strings, they always can do `getAbsolutePath`. N.B. This is good for machine independence. Rewritten from sbt/zinc@1f2d12c

* Remove dangerous use of output groups as fields in compilation. In the case of `SingleOutput`, the source directory was invented and incorrect (the root). Users of the API processing this file could create chaos in people's computers. The new design forces consumers of the `Compilation` API to make up their minds and identify whether the compilation was used with a single output or several outputs. In order to do that, we return `Output` instead of an array of `OutputGroup`. * Rename `outputDirectory` and `sourceDirectory` to be Java friendly. * Augment `Output` interface with Java-friendly methods. Scala users can use pattern matching instead. Rewritten from sbt/zinc@761ea0e

Rewritten from sbt/zinc@73ff81a

* Make methods in `Output` subclasses public * Changes to the API. Java idiomatic renames: * sourceDirectory -> getSourceDirectory * outputDirectory -> getOutputDirectory Rewritten from sbt/zinc@df6f5a4

Rewritten from sbt/zinc@d061e9e

Previously, the main class detection was handled by https://github.com/sbt/zinc/blob/1.0/internal/zinc-apiinfo/src/main/scala/xsbt/api/Discovery.scala which looks for a main method with the correct signature in the extracted API. This is imperfect because it relies on ExtractAPI dealiasing types (because Discovery will look for a main method with a parameter type of `java.lang.String` and won't recognize `scala.Predef.String`), dealiasing means that the extracted API looses information and thus can lead to undercompilation. This commit partially fixes this by adding a new callback to AnalysisCallback: void mainClass(File sourceFile, String className) that is used to explicitly register main entry points. This way, tools do not need to interpret the extracted API, this is much better since it makes it easier for zinc to evolve the API representation. This commit does not actually changes ExtractAPI to not dealias, this can be done in a later PR. Note that there is another usecase for xsbt.api.Discovery that this PR does not replace: discovering tests. This is more complicated because different test frameworks have different ways to discover tests. For more information, grep for "Fingerprint" in https://github.com/sbt/sbt and https://github.com/sbt/junit-interface Rewritten from sbt/zinc@f10c53c

In reference to sbt/sbt#2884 I'm seeing the console helpers (cpHelpers) being statically 'Object', and therefore not being that helpful: scala> cpHelpers res0: Object = sbt.internal.ConsoleProject$Imports@610be000 scala> cpHelpers.taskKeyEvaluate <console>:37: error: value taskKeyEvaluate is not a member of Object cpHelpers.taskKeyEvaluate ^ scala> cpHelpers.asInstanceOf[sbt.internal.ConsoleProject.Imports].taskKeyEvaluate _ res3: sbt.TaskKey[Nothing] => sbt.internal.ConsoleProject.Evaluate[Nothing] = $$Lambda$4294/1575143649@5a54d62c This is because I misinterpreted the Scala 2.8 compatibility layer I tore out in 1abf6ca3bfe0628321e1562a9b4cfe58e19ab7b7. Rewritten from sbt/zinc@0e4bb2c

Rewritten from sbt/zinc@d7f75ab

Rewritten from sbt/zinc@9c5c75f

Rewritten from sbt/zinc@7ded74a

Rewritten from sbt/zinc@92197af

Rewritten from sbt/zinc@cb4034d

* Change `Reporter` to log problems (zxsbti.Problem`). * Move `convert` to object (needs to be accessible from both logger and bridge). Rewritten from sbt/zinc@ec7b873

Rewritten from sbt/zinc@510d64d

Syncs up with sbt/util#84. Rewritten from sbt/zinc@6ef476b

Rewritten from sbt/zinc@68ac7a2

Rewritten from sbt/zinc@2b29cce

Rewritten from sbt/zinc@35f904a

Rewritten from sbt/zinc@7320e79

Follow-up on scala#314 - I _still_ misinterpreted.. Turns out the ".asInstanceOf[AnyRef].getClass.getName" implementation was the _original_ implementation. Then Mark switched to using bindValue in sbt/sbt@4b8f0f3. Since Scala 2.11.0 (scala#1648 in particular) bindValue was removed. So we'll use NamedParam and quietBind, both which exist since Scala 2.9.0. Fixes sbt/sbt#2884, tested with local releases. Rewritten from sbt/zinc@33d2e68

Rewritten from sbt/zinc@25734c3

xsbti Java classes were ported into compiler bridge, instead of the compiler interface by mistake. Since there's not code utilizing this interface yet, this was never caught. Rewritten from sbt/zinc@e2896e1

Rewritten from sbt/zinc@505ac2e

### background In sbt 0.13 days, we could ignore the relationship between two classes defined in the same `*.scala` source file, because they will be compiled anyway, and the invalidation was done at the source file level. With class-based namehashing, the invalidation is done at the class level, so we can no longer ignore inheritance relationship coming from the same source, but we still have old assumptions scattered around the xsbt-dependency implementation. ### what we see without the fix ``` [info] Compiling 1 Scala source to ... .... [debug] [inv] internalDependencies: [debug] [inv] DependencyByInheritance Relation [ [debug] [inv] xx.B -> gg.table.A [debug] [inv] xx.Foo -> xx.C [debug] [inv] ] [debug] [inv] DependencyByMemberRef Relation [ [debug] [inv] xx.B -> gg.table.A [debug] [inv] xx.Hello -> gg.table.A [debug] [inv] xx.Foo -> xx.C [debug] [inv] ] .... Caused by: java.lang.AbstractMethodError: xx.Foo.buildNonemptyObjects(II)V ``` First, we see that `xx.C -> xx.B DependencyByInheritance` relationship is missing. Second, the error message seen is `java.lang.AbstractMethodError` happening on `xx.Foo`. ### what this changes This change changes two if expressions that was used to filter out dependency info coming from the same source. One might wonder why it's necessary to keep the local inheritance info, if two classes involved are compiled together anyways. The answer is transitive dependencies. Here's likely what was happening: 1. `gg.table.A` was changed, 2. causing `xx.B` to invalidate. 3. However, because of the missing same-source inheritance, it did not invalidate `xx.C`. 4. This meant that neither `xx.Foo` was invalidated. 5. Calling transform method on a new `xx.Foo` causes runtime error. By tracking same-source inheritance, we will now correctly invalidate `xx.C` and `xx.Foo`. I think the assumption that's broken here is that "we don't need to track inheritance that is happening between two classes in a same source." ### Is this 2.11 only issue? No. The simple trait-trait inheritance reproduction alone will not cause problem in Scala 2.12 because of the [compile-to-interface](http://www.scala-lang.org/news/2.12.0/#traits-compile-to-interfaces) traits. However, not all traits will compile to interface. This means that if we want to take advantage of the compile-to-interface traits, we still should keep track of the same-source inheritance, but introduce some more logic to determine whether recompilation is necessary. Fixes sbt/zinc#417 Rewritten from sbt/zinc@05482d1

It looks like scalac encodes access rights of objects in their names. To make sure that we get the right simple names, we need to use `unexpandedName` instead of `name` which will decipher these access rights and return their simple names insted (with all the previous `$$` prefixes stripped out). Rewritten from sbt/zinc@6dc7586

Rewritten from sbt/zinc@797855f

Rewritten from sbt/zinc@41310b4

Rewritten from sbt/zinc@641a491

Quoting from 1e7e99e7e19e1c45f5a52aa31c399bd33c007582: If the underlying type of a value class change, its name hash doesn't change, but the name hash of <init> change and since every class uses the name <init>, we don't need to do anything special to trigger recompilations either This was true until aca8dfac0b839cb8e93a7702f6ec2de09773b1b3 where we started giving unique names to constructors. This broke the value-class-underlying type but this wasn't noticed because the test was broken in the same commit (and has now been fixed in the previous commit in this PR). Rewritten from sbt/zinc@0215c84

* add new compiler bridge Rewritten from sbt/zinc@af6e535

Fixes scala#395, sbt/sbt#3427 In scala#5903 Scala compiler's REPL-related classes went through some changes, including move to a different package. This implements a new compiler bridge tracking the changes. To verify that the new bridge compiles under 2.13, we need to compile it using sbt 1.0.3, which in turn requires a bridge compatible with Scala 2.13.0-M2. To work around this chicken-egg, I've manually created a bridge and published it to Maven Central as "org.scala-sbt" % "compiler-bridge_2.13.0-M2" % "1.1.0-M1-bootstrap2". Rewritten from sbt/zinc@c60ff23

Splitting compiler bridge tests to another subproject because while the bridge itself can be compiled with just compiler-interface, util-interface, and Scala Compiler as dependencies, the testing introduces more (such as IO). This creates problem for new Scala versions where IO or test libraries do not exist yet (e.g. Scala 2.13.0-M2). This also removes the Mima test due to the lack of 2.13 bridge for Zinc 1.0.0. Compiler bridge just needs to compile itself against the interface and Scala compiler, so there's no need to run Mima test. Rewritten from sbt/zinc@91cb532

* 1.0.x: (28 commits) Split compiler bridge tests to another subproject Implement compiler bridge for 2.13.0-M2 Add yourkit acknoledgement in the README "sbt '++ 2.13.0-M2!' compile" does not work with sbt 1.0.0 Add header to cached hashing spec Add headers to missing files Fix scala#332: Add sbt-header back to the build Update sbt-scalafmt to 1.12 Make classpath hashing more lightweight Fix scala#442: Name hash of value class should include underlying type source-dependencies/value-class-underlying: fix test Ignore null in generic lambda tparams Improve and make scripted parallel Fix scala#436: Remove annoying log4j scripted exception Fix scala#127: Use `unexpanded` name instead of `name` Add pending test case for issue/127 source-dependencies / patMat-scope workaround Fixes undercompilation on inheritance on same source Add real reproduction case for sbt/zinc#417 Add trait-trait-212 for Scala 2.12.3 ... Conflicts: internal/zinc-apiinfo/src/main/scala/sbt/internal/inc/ClassToAPI.scala project/build.properties zinc/src/main/scala/sbt/internal/inc/MixedAnalyzingCompiler.scala The ClassToAPI conflict is due to: * sbt/zinc#393 (a 1.x PR), conflicting with * sbt/zinc#446 (a 1.0.x PR). The build.properties conflict is due to different PRs bumping sbt.version from 1.0.0 to 1.0.2 to 1.0.3. (scala#413, scala#418, scala#453). The MixedAnalyzingCompiler conflict is due to: * sbt/zinc#427 (a 1.x PR), conflicting with * sbt/zinc#452 (a 1.0.x PR). Rewritten from sbt/zinc@e245d95

Remove the file exclude files before (a) compiler-bridge's exclude file is redundant since we no longer mima-check it, and (b) it interacts badly with multiple versions. Rewritten from sbt/zinc@3495f5a

…ala 2.12 compatibility Rewritten from sbt/zinc@9f99972

Rewritten from sbt/zinc@af90375

Rewritten from sbt/zinc@3a20f0b

Rewritten from sbt/zinc@e71db51

Rewritten from sbt/zinc@351ab3d

- hand inline enteringPhase to avoid closure allocation - avoid looking up getter/setters for non fields - optimize for common case of no annotations Rewritten from sbt/zinc@be05038

This is not to improve performance, but rather to give cleaner profiles. Rewritten from sbt/zinc@8ca9eff

Rewritten from sbt/zinc@3e4b3b4

Rewritten from sbt/zinc@52959c9

Rewritten from sbt/zinc@947f39a

Rewritten from sbt/zinc@ccd17b6

This reverts commit 52959c9e40be56d17fdf92c8d3c2662911d3e4bd. Fixes sbt/zinc#538 The use of `java.util.HashMap` causes `java.util.ConcurrentModificationException` on JDK 9 and JDK 10. This is likely because `processType` recursively end up calling `processType` while modifying `typeCache`. Rewritten from sbt/zinc@7a1995d

This reverts commit 3e4b3b4705ce67e88dac909a7c97e13cf242139f. Rewritten from sbt/zinc@6e29291

Conflicts: build.sbt internal/zinc-classpath/src/main/scala/sbt/internal/inc/classpath/ClasspathUtilities.scala Rewritten from sbt/zinc@2cf88f5

Rewritten from sbt/zinc@5d854ae

Rewritten from sbt/zinc@9add603

Rewritten from sbt/zinc@b283e61

`reporter.warning` is already used elsewhere, so use it uniformly. Rewritten from sbt/zinc@6550c06

Rewritten from sbt/zinc@72804b4

When a case class is defined with no companion, the companion is synthesized by the compiler and every object creation of that case class is proxied to the synthesized apply method of the newly synthesized companion. From this perspective, if we have a `case class A(x: Int)` and a use site `A(1)`, `ExtractUsedNames` will extract the name to the apply method in `A(1)` and mark it as used. However, this is wrong. When the user changes the signature of the case class, only the signature of the case class constructor changes and this change is not propagated to the apply signature (`ExtractAPI` traverses trees, and the synthesized module has no trees as it is added in namer). Therefore, when we compare changed names in the old and new API, Zinc concludes that only references to `A;<init>;` must be recompiled and since the use site that contained `A(1)` had no such reference, then it's ignored. To fix this problem, we protect ourselves from this point of indirection and extract the proper name of the case class constructor iff the companion case class is indeed synthesized by the compiler. Note that when the user defines `object A` alongside the definition of `A`, Zinc does the right thing. Note that this fixes sbt/sbt#4316 and also fixes issues with default parameters in case classes. Rewritten from sbt/zinc@4463be9

The previous commit was a good fix for the wrong problem. After some more investigation, the problem at hand is that we don't traverse synthetic top-level trees in `API` and therefore we don't extract the used names from them. This was causing us to ignore any names in the synthetic companion and, on top of that, also synthesized top-level members like package object definitions! Now, thanks to removing the filtering out that was happening in `isTopLevel`, our previous issue is fixed and with it also a pending test checking that package objects are recognized when introduced in a change `packageobject-and-traits`. This commit also adds a `checkRecompilations 2` in one of the tests related to package objects to make sure that test is behaving correctly. Rewritten from sbt/zinc@663ec5a

Follow up to sbt/util#173 Rewritten from sbt/zinc@ed1b515

Registers only non-local generated classes in the callback by extracting information about its names and using the names to generate class file paths. Mimics the previous logic that was present in `Analyzer`, despite the fact that now we construct the names that the compiler will give to every non-local class independently of genbcode. Why do we do this? The motivation is that we want to run the incremental algorithm independently of the compiler pipeline. This independence enables us to: 1. Offload the incremental compiler logic out of the primary pipeline and run the incremental phases concurrently. 2. Know before the compilation is completed whether another compilation will or will not be required. This is important to make incremental compilation work with pipelining and enables further optimizations; for example, we can start subsequent incremental compilations before (!) the initial compilation is done. This can buy us ~30-40% faster incremental compiler iterations. This method only takes care of non-local classes because local clsases have no relevance in the correctness of the algorithm and can be registered after genbcode. Local classes are only used to contruct the relations of products and to produce the list of generated files + stamps, but names referring to local classes **never** show up in the name hashes of classes' APIs, hence never considered for name hashing. As local class files are owned by other classes that change whenever they change, we could most likely live without adding their class files to the products relation and registering their stamps. However, to be on the safe side, we will continue to register the local products in `Analyzer`. Rewritten from sbt/zinc@856d416

Rewritten from sbt/zinc@ac79f2e

…d-api Notify analysis callback of generated non-local classes before genbcode Rewritten from sbt/zinc@1ff44cd

Otherwise we may end up losing internal dependencies because the compiler failed to do a good job at setting `associatedFile` in the classfile parser. This is what happens in scala#590 with default arguments, whose test passes after this implementation. Rewritten from sbt/zinc@fd8329e

We want to make sure that if this is ever run, we don't repeat the computation for the same class over and over again across the same compiler run. So we install a per-cache run. Rewritten from sbt/zinc@ec0e2b3

Rewritten from sbt/zinc@7b2e998

Rewritten from sbt/zinc@b9caefd

Rewritten from sbt/zinc@9f833af

Rewritten from sbt/zinc@b4e54c3

Rewritten from sbt/zinc@0f54b6d

Rewritten from sbt/zinc@1c93dec

Rewritten from sbt/zinc@a6aab45

Rewritten from sbt/zinc@596d58c

Rewritten from sbt/zinc@31484ad

Rewritten from sbt/zinc@17bf07b

Rewritten from sbt/zinc@15f631e

The commit includes several changes, with all my feedback about this new feature. The changes include: 1. Performance changes, most notably `Analyze`. 2. Separation of logic and implemntation, most notably `Compile`. 3. A lot of additions with regards to documentation and invariants that must be met for straight-to-jar compilation to work correctly. Rewritten from sbt/zinc@0130097

This refactors the compiler bridge unit test to use the normal Zinc facility, namely AnalyzingCompiler that's built on Scala 2.12, but is capable of driving the compiler bridge built on non-2.12. This allows us to run the unit tests without any additional dependencies published for Scala 2.13.0-M5. Rewritten from sbt/zinc@57bcaa0

Rewritten from sbt/zinc@20c3a04

Rewritten from sbt/zinc@4dd951f

Rewritten from sbt/zinc@a7cb4be

This will close file handles to JARs within recent versions of Scalac. References scala#7366 Rewritten from sbt/zinc@1f91729

Rewritten from sbt/zinc@1ae7188

Rewritten from sbt/zinc@b7e7957

Fixes scala/bug#11679 Rewritten from sbt/zinc@2097464

Fixes sbt/zinc#734 Ref scala/bug#11865 When `ArrayIndexOutOfBoundsException` is encountered, this will use startLine and startColumn as fallback. This is probably cheaper than trying to figure out the EOL position. Rewritten from sbt/zinc@4c33afe

This implements Zinc support for passing source code as a VirtualFile datatype. This allows build tools to pass source files as in-memory datatype. Another motivation for doing so is to remove machine dependence from the internal state of incremental compilation (Analysis). By making Analysis free of absolute paths, we should be able to cache the file and resume compilation from another machine. Notes: Anlyzer needs to reverse guess the source file from a `*.class` file, which is currently done by comparing the package names against the directory names of source. This supports the behavior by adding `names()` method to VirtualFileRef. scalac would report `/private/var/folders/hg/.../classes/S.class`, but javac thinks it's at `/private/var/folders/hg/..../classes/S.class`. ``` [info] [debug] > classToSouce.get(/var/folders/hg/2602nfrs2958vnshglyl3srw0000gn/T/sbt_c2a71c77/target/scala-2.12/classes/S.class) = None [info] [debug] > classToSource = List((/private/var/folders/hg/2602nfrs2958vnshglyl3srw0000gn/T/sbt_c2a71c77/target/scala-2.12/classes/S.class,S), (/private/var/folders/hg/2602nfrs2958vnshglyl3srw0000gn/T/sbt_c2a71c77/target/scala-2.12/classes/JJ.class,JJ)) ``` Rewritten from sbt/zinc@db028c3

Prior to zinc 712, this code looked like: ```scala // This lookup could be improved if a hint where to look is given. outputDirs.map(new File(_, classFilePath)).find(_.exists()).map(AbstractFile.getFile(_)) ``` When I ported this to NIO Path, it started to error on Windows because `<` are not allowed as path character. This change should at least get back to the state, hopefully. Rewritten from sbt/zinc@a7d9dde

Also fix a bug in fullName with encoded names. Rewritten from sbt/zinc@b645682

Avoid slow path for associated file more often Rewritten from sbt/zinc@2f1bad9

Calling isDirectory on every reference is pretty excessive! Rewritten from sbt/zinc@b3f4cf5

Rewritten from sbt/zinc@8b2db7a

Rewritten from sbt/zinc@00b73bd

Rewritten from sbt/zinc@bedf6b8

For modular pipelining we need both early output (JAR file containing Scala sig files) and early Analysis (Zinc internal information). This adds `IncOptions#earlyOutput` and `Lookup#storeEarlyAnalysis` so the early artifacts can be generated during compile phases. Rewritten from sbt/zinc@1b7081d

Early analysis store is passed into the Setup. CompileProgress is used to notify the early output timing. Rewritten from sbt/zinc@9676419

The deletion part required some tweaking to make sure this works for dir-based output. Rewritten from sbt/zinc@50601d5

Rewritten from sbt/zinc@0504f70

Rewritten from sbt/zinc@8c788d8

* Rename VirtualFileWrap/XsbtPlainFile/XsbtVirtualFile to AbstractZincFile/ZincPlainFile/ZincVirtualFil; * Default FileConverter#toVirtualFile(VirtualFileRef): VirtualFile * Rework BasicVirtualDirectory & BasicMemoryFile to avoid needless Option and BasicVirtualDirectory boxing, & use structural sharing; * Cleanup some of VirtualFile's docs. Rewritten from sbt/zinc@2c0e4e0

…d-code-parts Cleanup parts of the VirtualFile-related code Rewritten from sbt/zinc@fd60d33

... fixing OutputSpec. Rewritten from sbt/zinc@d254d7d

Fixes sbt/zinc#779 As it stands, compiler-interface and compiler bridge implementation are of internal concern of Zinc implementation. We _should_ be able to remove method or change the signatures. The word "interface" here is between Scalac and Zinc internal, not to the world. In reality, the situation is more complicated because we have Dotty compiler out there that is bound to a specific version of compiler-interface. So when I released sbt 1.4.0-M1 this resulted in NoSuchMethodErrors: ``` [error] ## Exception when compiling 1 sources to /private/tmp/hello-dotty/target/scala-0.24/classes [error] java.lang.RuntimeException: java.lang.reflect.InvocationTargetException [error] xsbt.CompilerInterface.newCompiler(CompilerInterface.java:35) .... [error] Caused by: java.lang.NoSuchMethodError: xsbti.compile.SingleOutput.getOutputDirectory()Ljava/io/File; [error] at xsbt.CachedCompilerImpl.<init>(CachedCompilerImpl.java:35) ``` To smooth things out, one approach we've discussed is to create a separate compiler-interface (in a different package) that is less dependent on Zinc specifics. Related to that, in sbt/zinc#661 I've created Java interface for `CompilerInterface1`, and we can use pattern matching to see which capability the compiler bridge implementation supports. This commit brings in those Java interfaces as well. In any case, this commit brings back the old `java.io.File`-based methods, and locally I was able to get hello world from Dotty. Rewritten from sbt/zinc@b4df9a3

Rewritten from sbt/zinc@28c2924

Avoid: * throwing if the "wrong" `run` method on CachedCompiler2 is called * casting the CachedCompiler value returned by GlobalsCache to CachedCompiler2 The easiest/safest way to do this is by making CachedCompiler2 extend CachedCompiler, as much as I'd prefer not to. Fortunately implementing the old methods in our bridge's CachedCompiler implementation (CachedCompiler0) is trivial. Rewritten from sbt/zinc@28cd5ee

In 2.12 scala.reflect.io.PlainNioFile is private[scala] with no nioPath val, so first we need to define the alias in the scala package and then we need to unwrap it with Java reflection... :sadface: Rewritten from sbt/zinc@172d638

Reluctantly restore compatibility of xsbti.* Rewritten from sbt/zinc@1179789

(cherry picked from commit c046a7183a28163e619a27ece712578167052323) Rewritten from sbt/zinc@5c8ed0f

Rewritten from sbt/zinc@ae4bca3

…File Use the real 2.12 scala.reflect.io.PlainNioFile Rewritten from sbt/zinc@561a506

Drop CompilerCache Rewritten from sbt/zinc@325970a

Fixes sbt/zinc#857 If the symbols are not from VirtualFile, we should be able to ignore it. Rewritten from sbt/zinc@d3141f7

Historically Streams have not performed nearly as well as Iterator. I'm not sure how relevant that is the case for the latest versions of scala but there were a number of places where it was easy to switch from Streams to Iterators so this commit does just that. There is very little expressivity loss in the switch from Iterator to Stream. From a performance perspective, I have been testing the zinc overhead in a small project by looping 100 iterations of no-op Test / compile. While not a scientific benchmark by any means, I pretty consistently found zinc to run about 10% faster (~63ms average after this change compared to ~70 ms before). Rewritten from sbt/zinc@bd9530c

Fixes scala#831 This is a future proofing step for the compiler bridge. As suggested by Guillaume this introduces an indirection using `java.util.ServiceLoader` so the class name of the compiler interface implementation can be configured. This also a smoke test to invoke these entry points using the brige. Rewritten from sbt/zinc@a3adcfd

Rewritten from sbt/zinc@7df6749

This was in the original, to-upstream source, but it got lost somewhere. Rewritten from sbt/zinc@e085744

Rewritten from sbt/zinc@105fcaa

Make xsbt.CompilerInterface class name configurable Rewritten from sbt/zinc@951c722

Add close method to interactive console Rewritten from sbt/zinc@d3d58f3

Fixes sbt/zinc#871 This is a minor refactoring to add `classpath` parameter to `AnalyzingCompiler#compile(...)` like all other methods. Rewritten from sbt/zinc@864444c

Rewritten from sbt/zinc@8d49d9c

…e caller of the enclosing named method" Rewritten from sbt/zinc@de0a288

While waiting for compilation, I decided to clean up some of the warnings that were emitted during said compilation. Rewritten from sbt/zinc@8cc98be

Scaladoc are not generated with 1.4.0-M2: sbt/sbt#5798. There seemed to be three problems: 1. the sources were not actually specified in ScaladocBridge 2. Three was an assumption that the compiler would return only AbstractZincFiles in DelegatingReporter.makePosition but, in doc at least, it returns scala.reflect.io.PlainFile These are straightforward to fix though I am somewhat concerned that the pattern match in DelegatingReporter has an unhandled case. Rewritten from sbt/zinc@53bc41a

8b2db7a792f3dbf47d31d6e543b353b4e1a42834 introduced a regression because it is not always the case that the source passed in to DelegateReporter.makePosition is an instance of AbstractZincFile. 53bc41acc7d878617f0d8b6f9400d606e8191637 partially fixed this by handling PlainFile but the proper fix is to handle AbstractFile which is what was done prior to 8b2db7a792f3dbf47d31d6e543b353b4e1a42834 This was detected by running the community build in the shapeless project. Rewritten from sbt/zinc@a05e58a

Previously, we would complete symbols before trying to find the associated classfile. This is problematic for symbols where we don't have the classfile (for instance, when compiling with Pants' `strict_deps` enabled). This initialization was introduced in scala#758, to workaround a scalac bug where the associated classfile of a symbol wouldn't be set before the symbol is completed. This bug has been fixed in Scala 2.12.12 and later (see scala/scalac#8889). Fixes scala#949 Rewritten from sbt/zinc@3b2cb10

Previously, we would complete symbols before trying to find the associated classfile. This is problematic for symbols where we don't have the classfile (for instance, when compiling with Pants' `strict_deps` enabled). This initialization was introduced in scala#758, to workaround a scalac bug where the associated classfile of a symbol wouldn't be set before the symbol is completed. This bug has been fixed in Scala 2.12.12 and later (see scala/scalac#8889). Fixes scala#949 Rewritten from sbt/zinc@d27f5fa

[1.4.x] Don't initialize symbols before finding classfile Rewritten from sbt/zinc@d5594f1

ExtractAPI stringifies trees in annotation argument position. Rewritten from sbt/zinc@4bbbd5d

Rewritten from sbt/zinc@0500da6

Rewritten from sbt/zinc@a2a38b4

Rewritten from sbt/zinc@d152289

Rewritten from sbt/zinc@b2a9ef1

Before 2.13, `isStatic` is implemented by `isNonBottomSubClass(StaticAnnotationClass)` which forces the annotation symbol's type. In 2.13, Java annotations are identified by flags. This check doesn't force the info, and the flags are missing if the info is still a `ClassfileLoader`. This leads to spurious API changes (annotation goes missing) if the depending if the info is already forced or not. A fix for this will be in 2.13.7, but we should still work around it in Zinc to make sure zinc works correctly on 2.13.0-6. Rewritten from sbt/zinc@c25d956

Rewritten from sbt/zinc@922885f

Rewritten from sbt/zinc@97ae167

Rewritten from sbt/zinc@4256606

Rewritten from sbt/zinc@ffc856a

Rewritten from sbt/zinc@60e3e99

Rewritten from sbt/zinc@5c86093

Rewritten from sbt/zinc@f7c4266

Rewritten from sbt/zinc@abda423

Rewritten from sbt/zinc@200f12c

Rewritten from sbt/zinc@f0e92fe

Rewritten from sbt/zinc@a853dd7

See also sbt/sbt#7306 Rewritten from sbt/zinc@415ce1a

add`@tailrec` annotation Rewritten from sbt/zinc@8234ea9

``` git clone git@github.com:sbt/zinc.git zinc-bridge-only ➜ zinc-bridge-only git log --oneline -n1 | cat dfdb20994 Update scalafmt-core to 3.7.8 ``` Filter-branch to move bridge sources to `internal/compiler-bridge` across the entire history (they were in `internal/compile-bridge` and `compile/interface` in early history). Add a reference to the original zinc commit to every commit message. Then filter-branch to keep only `internal/compiler-bridge`. This also removes merge commits. ``` ➜ zinc-bridge-only git filter-branch -f --tree-filter 'if [ -d internal/compile-bridge ]; then mv internal/compile-bridge internal/compiler-bridge; elif [ -d compile/interface ]; then mkdir -p internal; mv compile/interface internal/compiler-bridge; fi' --msg-filter 'cat; echo; echo Rewritten from sbt/zinc@$GIT_COMMIT' bridge-only ➜ zinc-bridge-only git filter-branch -f --prune-empty --subdirectory-filter internal/compiler-bridge bridge-only ``` Clean out the repo. ``` ➜ zinc-bridge-only git clean -f -d ➜ zinc-bridge-only for t in `git tag`; do git tag -d $t; done ➜ zinc-bridge-only git br -D develop ➜ zinc-bridge-only git remote remove origin ➜ zinc-bridge-only git gc --prune=now ➜ zinc-bridge-only git rm -r src/main/scala_2.10 ➜ zinc-bridge-only git rm -r src/main/scala-2.11 ➜ zinc-bridge-only git rm -r src/main/scala-2.12 ➜ zinc-bridge-only git rm -r src/main/scala_2.11-12 ``` Move bridge sources to `src/sbt-bridge` ``` ➜ zinc-bridge-only mkdir -p src/sbt-bridge/scala/tools/xsbt/ ➜ zinc-bridge-only git mv src/main/scala/xsbt/* src/sbt-bridge/scala/tools/xsbt/ ➜ zinc-bridge-only git mv src/main/scala_2.13/xsbt/* src/sbt-bridge/scala/tools/xsbt/ ➜ zinc-bridge-only git mv src/main/scala_2.13/scala/ZincCompat.scala src/sbt-bridge/scala/tools/xsbt ➜ zinc-bridge-only git mv src/main/resources src/sbt-bridge/ ➜ zinc-bridge-only rm -rf src/main ```

``` ➜ scala13 git log --oneline -n1 | cat b913236 Merge pull request scala#10399 from lrytz/t12774 ➜ scala13 git checkout -b sbt-bridge-2023 Switched to a new branch 'sbt-bridge-2023' ➜ scala13 git remote add zinc-bridge-only ../zinc-bridge-only ➜ scala13 git fetch zinc-bridge-only ➜ scala13 git merge --allow-unrelated-histories zinc-bridge-only/bridge-only ```

In-source sbt compiler bridge [ci: last-only] #10472

In-source sbt compiler bridge [ci: last-only] #10472

Commits on Mar 14, 2016

Commits on Mar 16, 2016

Commits on Mar 25, 2016

Commits on Mar 29, 2016

Commits on Mar 30, 2016

Commits on Apr 8, 2016

Commits on Apr 9, 2016

Commits on Apr 15, 2016

Commits on Apr 26, 2016

Commits on May 2, 2016

Commits on May 3, 2016

Commits on May 4, 2016

Commits on May 5, 2016

Commits on Jul 16, 2016

Commits on Oct 25, 2016

Commits on Nov 12, 2016

Commits on Nov 23, 2016

Commits on Dec 23, 2016

Commits on Jan 7, 2017

Commits on Jan 23, 2017

Commits on Jan 25, 2017

Commits on Jan 27, 2017

Commits on Feb 2, 2017

Commits on Feb 6, 2017

Commits on Feb 7, 2017

Commits on Feb 8, 2017

Commits on Feb 13, 2017

Commits on Feb 16, 2017

Commits on Feb 17, 2017

Commits on Feb 21, 2017

Commits on Feb 24, 2017

Commits on Feb 26, 2017

Commits on Feb 27, 2017

Commits on Mar 2, 2017

Commits on Mar 7, 2017

Commits on Mar 8, 2017

Commits on Mar 11, 2017

Commits on Mar 14, 2017

Commits on Mar 15, 2017

Commits on Mar 16, 2017

Commits on Mar 18, 2017

Commits on Mar 19, 2017

Commits on Mar 28, 2017

Commits on Apr 4, 2017

Commits on Apr 15, 2017

Commits on May 3, 2017

Commits on May 4, 2017

Commits on May 5, 2017

Commits on May 21, 2017

Commits on May 26, 2017

Commits on Jun 9, 2017

Commits on Jul 3, 2017

Commits on Jul 7, 2017

Commits on Jul 8, 2017

Commits on Jul 10, 2017

Commits on Jul 13, 2017

Commits on Jul 14, 2017

Commits on Jul 15, 2017

Commits on Jul 18, 2017

Commits on Aug 22, 2017

Commits on Sep 5, 2017

Commits on Oct 7, 2017

Commits on Oct 9, 2017

Commits on Oct 12, 2017

Commits on Oct 13, 2017

Commits on Oct 16, 2017

Commits on Oct 22, 2017

Commits on Nov 3, 2017

Commits on Nov 8, 2017

Commits on Nov 16, 2017

Commits on Nov 21, 2017

Commits on Nov 23, 2017

Commits on Jan 2, 2018

Commits on Feb 11, 2018

Commits on Feb 21, 2018

Commits on Feb 22, 2018

Commits on Mar 11, 2018

Commits on May 6, 2018