Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
The alternative, flat representation of classpath elements #4176
This pull request provides an alternative, more efficient implementation of classpath handling in the compiler. This feature will be delivered in 2.11.5 but it's off by default. To try it out, you must enable it with
Below is the original description of this PR with all remaining details.
This is another PR created for the flat classpath representation. This time the target is changed to 2.11.x branch.
It's finished work started and abandoned by @gkossakowski
The main goal of Grzegorz was to improve the efficiency of classpath. At the end it turned out that the efficiency improvement is not significant so Grzegorz stopped working on that.
On the other hand it was worth to continue these works due to several reasons:
New (flat) classpath has dedicated classes for various file types characteristic for Scala with JVM backend - what allows us to create more efficient, more low-level operations.
One of the most important things took into account during creating this implementation was adding it as an optional feature, which can be marked as experimental and turned on using flag. At the beginning users will use an old classpath implementation by default and they will be able to use the new implementation when:
I took into account also planned better support for JSR-223. So it supports for instance ManifestResources introduced here: #2238
I tested flat classpath in several ways:
The flat classpath with turned on caching allowed me to reduce allocated memory in old gen from about 6GB (for current classpath) to about 750MB. So in the case of IDE and presentation compilers the gain is really significant.
Tests using a computer with SSD:
Tests using other computer with the scala project directory moved to some external HDD connected via USB cable:
Note: I changed the classPath method in Global to the dynamic dispatch. And now there's returned a base class of the old and the new classpath with common interface based on a part of API of the old classpath. I tested it with sbt and there's no problem with compiling projects etc. Previously Grzegorz created certain hack for sbt's compiler interface where current classpath was calling the flat one, when it was needed. I changed this ugly solution to this mentioned dynamic dispatch and removed a hack. We can return to something like this, if it'd turn out that it's necessary.
Thanks to Grzegorz for hints and his initial implementation.
Lastly, the below disclaimer is required by the lawyers:
THIS PROGRAM IS SUBJECT TO THE TERMS OF THE BSD 3-CLAUSE LICENSE.
THE FOLLOWING DISCLAIMER APPLIES TO ALL SOFTWARE CODE AND OTHER MATERIALS CONTRIBUTED IN CONNECTION WITH THIS SOFTWARE:
I just finished reviewing all commits. I'm really impressed with quality of this PR. Really good work on both documentation (in commit messages and in the code) and the code itself! Also the structure of commits is really good.
Here're the things that hold me from giving LGTM and merging:
I just kicked community build that will use scala compiler version submitted by this PR: https://jenkins-dbuild.typesafe.com:8499/job/Community-2.11.x-manual/102/console
However, even if we find some issues with it we can fix them in subsequent PRs. For now only the two tasks I outlined above are needed to be done for merging this PR.
I'll read your questions/remarks and answer to them soon.
added a commit
this pull request
Dec 5, 2014
It wasn't until now. I created sbt project with the following config:
resolvers in Global += Resolver.sonatypeRepo("snapshots") scalaVersion in Global := "2.11.5-SNAPSHOT" //scalacOptions in Global += "-YclasspathImpl:flat" // allow 20 different compiler instances to run in parallel concurrentRestrictions in Global := Tags.limitAll(20) :: Nil lazy val project1 = project lazy val project2 = project ... lazy val project19 = project lazy val project20 = project
Each project has one source file and no extra dependencies apart from standard Java and Scala dependencies.
I attached Yourkit profiler, ran
If you measure memory consumption caused just by compilation (by subtracting the memory consumed by sbt itself) you get:
The difference in memory consumption is not as dramatic as reported originally in the PR but still very noticeable. The gain we get depends on classpath's size. Standard classpath is small.
For embedded newlines, it's preferable to use
This means if I use the method, my code won't run on earlier point releases, which is Mima's whole point. Kind of surprised this got by the censors, since it's trivial to implement as an extension method. It might have been preferable to add it, if necessary, on the nsc side.