Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to scalameta 3.7.0 with default settings. #675

Merged
merged 10 commits into from
Apr 9, 2018

Conversation

olafurpg
Copy link
Contributor

@olafurpg olafurpg commented Apr 4, 2018

This commit is a major milestone in that Scalafix now uses the latest
SemanticDB with default settings. Previously, scalafix relied on two
features that are now no longer default:

  • -P:semanticdb:denotations:all, to lookup information about any symbol
    scalafix relied on semanticdb-scalac to persist full information for
    all referenced symbols in each compilation unit. Now, scalafix
    can load information directly from the classpath thanks to metacp
    and the new --dependency-classpath command line flag.
  • -P:semanticdb:signatures:old, these were pretty-printed versions of
    types that have been replaced with the new specification of Type in
    https://github.com/scalameta/scalameta/blob/master/semanticdb/semanticdb3/semanticdb3.md#type

There's quite a bit happening in this PR, it's roughly broken down by:

  • New --dependency-classpath cli flag be processed by metacp.
  • New SymbolTable to lookup information about symbols from the metacp
    processed classpath.
  • Refactor ExplicitResultTypes to use s.Type instead of the deprecated
    s.SymbolInformation.signatur
  • Remove tests for Symbol.Multi, which are no longer emitted by
    semanticdb-scalac.

These changes required updates to testkit, the cli and sbt-scalafix.

Future work:

  • The tests pass but I noticed a couple errors while running
    ExplicitResultTypes on Slick. Given that this PR already does quite a
    lot I prefer to get it merged asap and fix those errors later.
  • ExplicitResultTypes crashes on java paths #674
  • --dependency-classpath may not be necessary. We might be able to away
    with only --classpath and be smarter about which files to load from
    the provided files to fix.

This commit is a major milestone in that Scalafix now uses the latest
SemanticDB with default settings. Previously, scalafix relied on two
features that are now no longer default:

* -P:semanticdb:denotations:all, to lookup information about any symbol
  scalafix relied on semanticdb-scalac to persist full information for
  all referenced symbols in each compilation unit. Now, scalafix
  can load information directly from the classpath thanks to metacp
  and the new --dependency-classpath command line flag.
* -P:semanticdb:signatures:old, these were pretty-printed versions of
  types that have been replaced with the new specification of `Type` in
  https://github.com/scalameta/scalameta/blob/master/semanticdb/semanticdb3/semanticdb3.md#type

There's quite a bit happening in this PR, it's roughly broken down by:

* New --dependency-classpath cli flag be processed by metacp.
* New `SymbolTable` to lookup information about symbols from the metacp
  processed classpath.
* Refactor ExplicitResultTypes to use s.Type instead of the deprecated
  s.SymbolInformation.signatur
* Remove tests for Symbol.Multi, which are no longer emitted by
  semanticdb-scalac.

These changes required updates to testkit, the cli and sbt-scalafix.

Future work:

* The tests pass but I noticed a couple errors while running
  ExplicitResultTypes on Slick. Given that this PR already does quite a
  lot I prefer to get it merged asap and fix those errors later.
* --dependency-classpath may not be necessary. We might be able to away
  with only --classpath and be smarter about which files to load from
  the provided files to fix.
lazy val scalafixSbt1 = scalafixSbt(
scala212,
sbt1,
_.dependsOn(testUtils212 % Test)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this formatting change intended?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, for consistency with the changes below

def resolveClasspath: Configured[Classpath] =
classpath match {
def resolveClasspath: Configured[Classpath] = {
classDirectory match {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a blocker but I don't like that this name hints that the classpath is a single directory entry, which is not what I usually see for a classpath.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree this is confusing. Note that the cli still accepts --classpath as an alias for --class-directory. I renamed for consistency with sbt classDirectory to distinguish it from dependencyClasspath.

The public facing API is fully compatible with the previous behavior despite this change so we have wiggle room to improve on this in the future.

* @param sclasspath Regular classpath to process.
* @param out The output stream to print out error messages.
*/
def toMetaClasspath(sclasspath: Classpath, out: PrintStream): Classpath = {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fact that these methods take a printstream looks really awkward.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awkward sure, but what do you propose instead? This is not a public API

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use System.out directly.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scalafix runs via nailgun, in IDEs with custom logging, etc,

It's not OK to use System.out

.withScalaLibrarySynthetics(true)
val reporter = metacp.Reporter().withOut(out)
val mclasspath = scala.meta.cli.Metacp.process(settings, reporter).get
mclasspath
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it okay to crash if metacp returns None?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've refactored the code to propagate the error and report a helpful message on failure.

@@ -55,12 +55,20 @@ case class ScalafixOptions(
sourceroot: Option[String] = None,
@HelpMessage(
"java.io.File.pathSeparator separated list of directories or jars containing " +
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a list or a single directory?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It remains a list, unchanged from the previous behavior of --classpath. I've renamed it back to --classpath and added more details about --dependency-classpath.

* This utility is necessary because SymbolTable contains only global symbols and
* SemanticdbIndex has local symbols.
*/
class CombinedSymbolTable(index: SemanticdbIndex, table: SymbolTable) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to pass both index and table here, given that EagerInMemorySemanticdbIndex already takes a symbol table?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, I've merged this into EagerSemanticdbIndex avoiding the need for CombinedSymbolTable.

class LazySymbolTable(mclasspath: Classpath) extends SymbolTable {

// Map from symbols to the paths to their corresponding semanticdb file where they are stored.
private val unloadedSymbols = TrieMap.empty[String, AbsolutePath]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"unloaded" as in "not yet loaded" or as in "previously loaded, but no longer available"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed to notYetLoadedSymbols

val fs = newJarFileSystem(path)
// NOTE(olafur): We don't fs.close() because that can affect another place where `FileSystems.getFileSystems`
// was used due to a `FileSystemAlreadyExistsException`. I don't know what the best solution is for reading the
// same zip file from multiple concurrent places.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this is/was the source of deadlocks?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was my first suspicion as well! I continuously ran the test suite to see problems. This wasn't the cause of the deadlock, I checked the stack traces and they led to metacp.

shortenNames: Boolean,
pos: Position)(implicit index: SemanticdbIndex): Option[(Type, Patch)] = {
try {
val table = index.asInstanceOf[EagerInMemorySemanticdbIndex].table
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Woah, this cast looks scary.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seconded, why do we know more than the compiler here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SemanticdbIndex trait should only have a single implementation, I've kept the abstract trait to separate the public API from the implementation. I admit this is not an ideal situation but the alternative is to expose implementation details in the public API. We have the same situation in other places, for example for linter suppression. private[scalafix] does not bite enough IMO compared to forcing a cast.

/*
* Returns a scala.meta.Tree given a scala.meta.Symbol.
*
* NOTE: this method has become an utter mess and is in need of a clean reimplementation.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, not gonna review then.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The implementation is unchanged from the previous TypeSyntax, the diff is mostly reformatting with minor changes.

result <- TypeSyntax.prettify(
tpe,
ctx,
config.unsafeShortenNames,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is this config passed? https://scalacenter.github.io/scalafix/docs/users/configuration shows how to add rules to the scalafix config, but does not show if it is possible to pass rule specific config.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's configuration for ExplicitResultTypes documented here https://scalacenter.github.io/scalafix/docs/rules/ExplicitResultTypes

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn’t able to put that together from the docs for some reason but I get it now, thanks.

* This utility is necessary because SymbolTable contains only global symbols and
* SemanticdbIndex has local symbols.
*/
class CombinedSymbolTable(index: SemanticdbIndex, table: SymbolTable) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is combined symbol table needed if shortenNames is false?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's needed both when shortenNames is true and false.

private val unloadedSymbols = TrieMap.empty[String, AbsolutePath]
private val loadedSymbols = TrieMap.empty[String, s.SymbolInformation]
private val semanticdb = "META-INF/semanticdb"
private val semanticIdx = "META-INF/semanticdb.semanticidx"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would this be a good place to add

private val manifest = "META-INF/MANIFEST.MF"

in case there is a classpath manifest in a jar?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that should be handled elsewhere, for example scala.meta.Classpath should be able to handle.

if (root.isDirectory) {
loadIndex(root, Files.newInputStream(root.resolve(semanticIdx).toNIO))
} else if (PathIO.extension(root.toNIO) == "jar") {
withJarFileSystem(root) { jarRoot =>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not familiar enough with the NIO api to know the answer to this, but are you sure that these apis handle symlinks?

import org.langmeta.semanticdb.Symbol
import scala.meta.internal.semanticdb3.SingletonType.{Tag => x}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This saves two characters and makes the code harder to read. Why not just Tag.?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've replaced x with Tag but kept the import local, Tag is used in s.Type.Tag/s.SingletonType.Tag and a few others I think.

}

def isFunctionN(symbol: String): Boolean = {
symbol.startsWith("scala.Function") &&
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if there this is a perforamance issue, but from an accuracy issue how about

"scala.Function(\\d)+#".r 

to make sure we capture Function1 but not FunctionPure (not that such a thing currently exists).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there are no other scala.Function* symbols so I thought this would be enough, regexes are such a heavy machine.

}
symbol match {
case Symbol.Global(owner, Signature.Term(_) | Signature.Type(_)) =>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read this as "A symbol is stable only if it is A Type or Term owned by Symbol.None or owned by a chain of n Global Terms". Is that correct?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method suffers from the same problems as symbolToTree, it can be ignored.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the introduction of SymbolInformation one should no longer infer any information from the structure of symbols. Symbols are purely an ID and nothing more.

@@ -4,7 +4,7 @@ object ExplicitResultTypesPathDependent {
class Path {
class B { class C }
implicit val x: Path.this.B = new B
implicit val y: x.C = new x.C
implicit val y: Path.this.x.C = new x.C
def gimme(yy: x.C) = ???; gimme(y)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

??? indeed.

Copy link
Contributor

@ShaneDelmore ShaneDelmore left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a few questions but don't have enough familiarity with the codebase to call any of them a blocker.

* revert --class-directory, stick to --classpath
* better error handling for metacp interaction
* add --metacp-cache-directory
* add --metacp-no-par for pending --par flag in metacp
Copy link
Contributor Author

@olafurpg olafurpg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the review! I've addressed all comments, but I'm still investigating the CI failures.

lazy val scalafixSbt1 = scalafixSbt(
scala212,
sbt1,
_.dependsOn(testUtils212 % Test)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, for consistency with the changes below

def resolveClasspath: Configured[Classpath] =
classpath match {
def resolveClasspath: Configured[Classpath] = {
classDirectory match {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree this is confusing. Note that the cli still accepts --classpath as an alias for --class-directory. I renamed for consistency with sbt classDirectory to distinguish it from dependencyClasspath.

The public facing API is fully compatible with the previous behavior despite this change so we have wiggle room to improve on this in the future.

* @param sclasspath Regular classpath to process.
* @param out The output stream to print out error messages.
*/
def toMetaClasspath(sclasspath: Classpath, out: PrintStream): Classpath = {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awkward sure, but what do you propose instead? This is not a public API

.withScalaLibrarySynthetics(true)
val reporter = metacp.Reporter().withOut(out)
val mclasspath = scala.meta.cli.Metacp.process(settings, reporter).get
mclasspath
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've refactored the code to propagate the error and report a helpful message on failure.

@@ -55,12 +55,20 @@ case class ScalafixOptions(
sourceroot: Option[String] = None,
@HelpMessage(
"java.io.File.pathSeparator separated list of directories or jars containing " +
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It remains a list, unchanged from the previous behavior of --classpath. I've renamed it back to --classpath and added more details about --dependency-classpath.

shortenNames: Boolean,
pos: Position)(implicit index: SemanticdbIndex): Option[(Type, Patch)] = {
try {
val table = index.asInstanceOf[EagerInMemorySemanticdbIndex].table
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SemanticdbIndex trait should only have a single implementation, I've kept the abstract trait to separate the public API from the implementation. I admit this is not an ideal situation but the alternative is to expose implementation details in the public API. We have the same situation in other places, for example for linter suppression. private[scalafix] does not bite enough IMO compared to forcing a cast.

}

def isFunctionN(symbol: String): Boolean = {
symbol.startsWith("scala.Function") &&
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there are no other scala.Function* symbols so I thought this would be enough, regexes are such a heavy machine.

}
symbol match {
case Symbol.Global(owner, Signature.Term(_) | Signature.Type(_)) =>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method suffers from the same problems as symbolToTree, it can be ignored.

}
symbol match {
case Symbol.Global(owner, Signature.Term(_) | Signature.Type(_)) =>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the introduction of SymbolInformation one should no longer infer any information from the structure of symbols. Symbols are purely an ID and nothing more.

/*
* Returns a scala.meta.Tree given a scala.meta.Symbol.
*
* NOTE: this method has become an utter mess and is in need of a clean reimplementation.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The implementation is unchanged from the previous TypeSyntax, the diff is mostly reformatting with minor changes.

Tests were failing with `java.nio.file.NoSuchFileException: /META-INF/semanticdb.semanticidx`
on 2.11, but not 2.12. I was unable to reproduce the problem using
`JarFile` from java.io so I'm gonna throw in the towel and use
JarFile for reading zip files instead.
The Windows CI was failing due to URI encoding issues:

```
[info] scalafix.tests.rule.SemanticTests *** ABORTED *** (0
    milliseconds)
[info]   java.nio.file.InvalidPathException: Illegal char <:> at index
2: /C:/projects/scalafix/.cross/unit/target/scala-2.12/test-classes/
```

My experience is that this is typically caused by stringly typed
programming around URIs, so let's hope this commit fixes the issue.
@olafurpg
Copy link
Contributor Author

olafurpg commented Apr 5, 2018

I'm investigating the CI build failures in scalameta/scalameta#1476 I suspect they are caused by even more incorrect usage of nio FileSystems.

The Travis 2.11 failures seem to be caused by differences is positions compared to 3.2.0

Semanticdb-scala no longer emits accurate dialects so we need to use
other tricks to compute the scala version specific directory
.withCacheDir(cacheDirectory.getOrElse(default.cacheDir))
val reporter = metacp
.Reporter()
.withOut(devNull) // out prints classpath of proccessed classpath, which is not relevant for scalafix.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh, this is unfortunate, given that we're calling process directly.

"On macOS the default cache directory is ~/Library/Caches/semanticdb. ")
metacpCacheDir: Option[String] = None,
@HelpMessage("Set this flag to disable parallel processing with metacp.")
metacpNoPar: Boolean = false,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that the help message should explain that there is a danger of deadlocks.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

"On macOS the default cache directory is ~/Library/Caches/semanticdb. ")
metacpCacheDir: Option[String] = None,
@HelpMessage("Set this flag to disable parallel processing with metacp.")
metacpNoPar: Boolean = false,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ShaneDelmore We'll need to set this flag when we'll be upgrading Scalafix internally.

}
}

class TypeSyntax private (
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you ultimately want parts of this code to end up in Metap? scalameta/scalameta#1479

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to rewrite this file from scratch first before considering that. I am not happy with the code quality here, but I am more eager to upgrade to scalameta v3.7 and this PR is already doing too much.

This should fix the CI error on Windows
Copy link
Contributor Author

@olafurpg olafurpg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review! Upgrading to scalameta v3.7.2 fixed the last blocking problem on Windows

"On macOS the default cache directory is ~/Library/Caches/semanticdb. ")
metacpCacheDir: Option[String] = None,
@HelpMessage("Set this flag to disable parallel processing with metacp.")
metacpNoPar: Boolean = false,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

}
}

class TypeSyntax private (
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to rewrite this file from scratch first before considering that. I am not happy with the code quality here, but I am more eager to upgrade to scalameta v3.7 and this PR is already doing too much.

@olafurpg
Copy link
Contributor Author

olafurpg commented Apr 9, 2018

Merging and releasing 0.6.0-M2 :shipit:

Note that ExplicitResultTypes in M2 will be a fairly broken until TypeSyntax gets a rewrite that I'm planning for this week.

@olafurpg olafurpg merged commit 3f070c3 into scalacenter:master Apr 9, 2018
@olafurpg olafurpg deleted the tpe branch April 9, 2018 12:43
@olafurpg olafurpg added this to the v0.6.0-M2 milestone Apr 11, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants