make scalafixEnable less aggressive & more future-proof #292

bjaglin · 2022-03-24T01:48:06Z

Keep the build scala version when semanticdb-scalac is available for that scala version for the scalameta version that scalafix tracks.
If not, try to find the earliest binany-compatible semanticdb-scalac (preferred over the latest last to be more repeatable) and use it instead.
Only when that fails (or resolution fails), force the scala version as previously.

(1) avoids unnecessary scala bumps which can result in

zinc cache invalidation
compilation failures because of stricter compiler checks
resolution failures because of yet-unpublished compiler plugins (fixes scalafixEnable / scalafix resets scalac version scalafix#1573)

(2) opens the possibility for using sbt-scalafix on scala versions released "in the future" without any scalameta/scalafix bump (as long as scalameta is published obviously).

sbt:sbt-scalafix> scripted sbt-scalafix/scalafixEnable
...
[warn] Forcing scalaVersion 2.11.12 in project scala211_old since no semanticdb-scalac version binary-compatible with 4.4.10 and cross-published for scala 2.11.0 was found - consider bumping scala
[info] Using semanticdb-scalac 4.1.0 in project scala211 since the version 4.4.10 tracked by scalafix 0.9.27 is not available for scala 2.11.11 - consider upgrading sbt-scalafix or bumping scala
[info] Using semanticdb-scalac 4.4.28 in project scala212 since the version 4.4.10 tracked by scalafix 0.9.27 is not available for scala 2.12.15 - consider upgrading sbt-scalafix or bumping scala

1. Keep the build scala version when semanticdb-scalac is available for that scala version for the scalameta version that scalafix tracks. 2. If not, try to find the earliest binany-compatible semanticdb-scalac (preferred over the latest last to be more repeatable) and use it instead. 3. Only when that fails (or resolution fails), force the scala version as previously. (1) avoids unnecessary scala bumps which can result in - zinc cache invalidation - compilation failures because of stricter compiler checks - resolution failures because of yet-unpublished compiler plugins (2) opens the possibility for using sbt-scalafix on scala versions released "in the future" without any scalameta/scalafix bump (as long as scalameta is published obviously).

bjaglin · 2022-03-24T02:11:44Z

src/sbt-test/sbt-scalafix/dependency/build.sbt

@@ -1,4 +1,4 @@
-scalaVersion := "2.12.7"


The downside of this PR is that trying to keep the scalaVersion untouched can cause errors as an old semanticdb-scalac may be available but not fully functional. I think that's acceptable.

👌 on Java 8

💥 on Java 11

sbt:sbt-scalafix> scripted sbt-scalafix/dependency ... [info] [info] Using semanticdb-scalac 4.0.0 in project dependency since the version 4.4.32 tracked by scalafix 0.9.34 is not available for scala 2.12.7 - consider upgrading sbt-scalafix or bumping scala ... [info] [error] ## Exception when compiling 1 sources to /tmp/sbt_fab36d90/dependency/target/scala-2.12/classes [info] [error] java.lang.NoClassDefFoundError: javax/xml/bind/DatatypeConverter [info] [error] scala.meta.internal.semanticdb.scalac.InputOps$XtensionGSourceFileInput.toMD5(InputOps.scala:41) [info] [error] scala.meta.internal.semanticdb.scalac.TextDocumentOps$XtensionCompilationUnitDocument.toTextDocument(TextDocumentOps.scala:614) [info] [error] scala.meta.internal.semanticdb.scalac.SemanticdbPipeline$SemanticdbTyperComponent$ComputeSemanticdbPhase.saveSemanticdbForCompilationUnit(SemanticdbPipeline.scala:59) [info] [error] scala.meta.internal.semanticdb.scalac.SemanticdbPipeline$SemanticdbTyperComponent$ComputeSemanticdbPhase.apply(SemanticdbPipeline.scala:65) [info] [error] scala.tools.nsc.Global$GlobalPhase.applyPhase(Global.scala:441) [info] [error] scala.tools.nsc.Global$GlobalPhase.run(Global.scala:392) [info] [error] scala.meta.internal.semanticdb.scalac.SemanticdbPipeline$SemanticdbTyperComponent$ComputeSemanticdbPhase.run(SemanticdbPipeline.scala:75) [info] [error] scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1467)

💥 on Java 17

sbt:sbt-scalafix> scripted sbt-scalafix/dependency ... [info] [info] Using semanticdb-scalac 4.0.0 in project dependency since the version 4.4.32 tracked by scalafix 0.9.34 is not available for scala 2.12.7 - consider upgrading sbt-scalafix or bumping scala ... [info] [error] ## Exception when compiling 1 sources to /tmp/sbt_b2813b6/target/scala-2.12/classes [info] [error] java.lang.reflect.InvocationTargetException [info] [error] java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ... [info] [error] java.io.IOError: java.lang.RuntimeException: /packages cannot be represented as URI [info] [error] at java.base/jdk.internal.jrtfs.JrtPath.toUri(JrtPath.java:175) [info] [error] at scala.tools.nsc.classpath.JrtClassPath.asURLs(DirectoryClassPath.scala:204)

This kind of regression should be mostly addressed by 012651c

bjaglin · 2022-03-24T07:24:29Z

cc @olafurpg

adpi2 · 2022-03-24T08:50:08Z

I like this effort to make sbt-scalafix as resilient as possible but I would challenge this:

If not, try to find the earliest binany-compatible semanticdb-scalac (preferred over the latest last to be more repeatable) and use it instead.

There are two cases:

we use a recent scalaVersion with an old sbt-scalafix: In that case the earliest version of semanticdb-scalac is better because it is the closest to sbt-scalafix so it has greater chances to work. It is repeatable.
we use an old scalaVersion with a recent sbt-scalafix: in that case however the latest version of semanticdb-scalac is probably better because it is the closest to sbt-scalafix. It is also repeatable because there should not be any new release of semanticdb-scala for that outdated scalaVersion.

Also we can adapt the hint correspondingly with either "consider upgrading sbt-scalafix" or "consider bumping scala"

bjaglin · 2022-03-24T08:54:06Z

There are two cases

Good catch, I did think about checking the case (mostly for the log message, not the earliest/latest policy) but was not sure it was worth it. Now that I think about it, it would probably solve #292 (comment). I will add this in a separate commit.

bjaglin · 2022-03-24T11:56:28Z

src/sbt-test/sbt-scalafix/scalafixEnable/build.sbt

@@ -15,7 +15,7 @@ lazy val scala211_old = project.settings(

 // 2.11.x is supported
 lazy val scala211 = project.settings(
-  // semanticdb-scalac_2.11.11 no longer available, became available as of 2.0.0


The fact that there the earliest artifact was non binary-compatible exercised more code, so the change of behavior reduces code coverage unfortunately. Extracting the logic and stubbing the coursier API seems like a lot of work though, so I think we can live with that, and revisit if/when scalameta reaches 5.x.

When there is no semanticdb-scalac available for the build scala version for the scalameta version that scalafix tracks, we can tell the user if it is because it is no longer published or not yet published. When it is no longer published, the latest version should not change, so we can use that one in a repeatable manner, in order to maximize the chances of compatiblity.

bjaglin · 2022-03-24T12:00:46Z

The scripted now results in

[warn] Forcing scalaVersion 2.11.12 in project scala211_old since no semanticdb-scalac version binary-compatible with 4.4.10 and cross-published for scala 2.11.0 was found - consider bumping scala
[info] Using semanticdb-scalac 4.4.28 in project scala212 since the version 4.4.10 tracked by scalafix 0.9.27 will not be published for scala 2.12.15 - consider upgrading sbt-scalafix
[info] Using semanticdb-scalac 4.1.9 in project scala211 since the version 4.4.10 tracked by scalafix 0.9.27 is no longer published for scala 2.11.11 - consider bumping scala

Is that what you had in mind @adpi2 ?

bjaglin · 2022-03-24T12:18:53Z

src/main/scala/scalafix/sbt/ScalafixEnable.scala

+            val recommendedSemanticdbV =
+              VersionNumber(BuildInfo.scalametaVersion)
+            val compatibleSemanticdbVs = Try(
+              coursierapi.Versions.create


To my knowledge, this is the best Java coursier API to achieve something like suggested in scalacenter/scalafix#1146 (comment)

bjaglin · 2022-03-24T12:20:43Z

src/main/scala/scalafix/sbt/ScalafixEnable.scala

+              VersionNumber(BuildInfo.scalametaVersion)
+            val compatibleSemanticdbVs = Try(
+              coursierapi.Versions.create
+                .withRepositories(project.scalafixResolvers0: _*)


That part is untested, so there is a risk that this fails on corporate environments. The failure fallback keeps the current behavior though, so apart from an added latency it shouldn't be a problem.

Maybe this is out of topic in this PR but we could use the csrConfiguration key in sbt to configure coursier with authentication and credentials.

I looked at that in the context of scalacenter/scalafix#1571, but I am not sure it would help as scalafix-core uses the Java coursier API directly, not via what lm/lm-coursier.

That's true, but it may not be so hard to translate the sbt csrConfiguration into the Java API with something similar to ToCoursier given that both csrConfiguration and the coursier Java API are stable.

bjaglin · 2022-03-24T12:22:05Z

src/main/scala/scalafix/sbt/ScalafixEnable.scala

+      partialVersion <- CrossVersion.partialVersion(scalaV).toList
+      maybeRecommendedSemanticdbScalacV <-
+        maybeRecommendedSemanticdbScalacScalaVersion.lift(partialVersion).toList
+      scalafixResolvers0 <- (p / scalafixResolvers)


this is currently looked up at the ThisBuild level elsewhere, but since usage of custom resolvers is not really tested, I chose to be overly defensive

bjaglin · 2022-03-24T16:18:30Z

A last tweak on wording and I think we are good to go:

[info] Setting semanticdbVersion to 4.4.28 in project scala212 since the version 4.4.10 tracked by scalafix 0.9.27 will not be published for scala 2.12.15 - consider upgrading sbt-scalafix
[info] Setting semanticdbVersion to 4.1.9 in project scala211 since the version 4.4.10 tracked by scalafix 0.9.27 is no longer published for scala 2.11.11 - consider bumping scala
[warn] Forcing scalaVersion to 2.11.12 in project scala211_old since no semanticdb-scalac version binary-compatible with 4.4.10 and cross-published for scala 2.11.0 was found - consider bumping scala

adpi2

LGTM

bjaglin force-pushed the dontforceversion branch from 1a16731 to 1a4d1fe Compare March 24, 2022 01:48

This was referenced Mar 24, 2022

Remove -Xfatal-warnings scalac option in scalafixEnable scalacenter/scalafix#1346

Closed

scalafixEnable / scalafix resets scalac version scalacenter/scalafix#1573

Closed

bjaglin force-pushed the dontforceversion branch from 1a4d1fe to 71e8b9f Compare March 24, 2022 02:10

bjaglin commented Mar 24, 2022

View reviewed changes

bjaglin marked this pull request as ready for review March 24, 2022 02:19

bjaglin requested review from adpi2 and mlachkar March 24, 2022 07:23

bjaglin mentioned this pull request Mar 24, 2022

sbt-scalafix doc updates scalacenter/scalafix#1572

Merged

bjaglin commented Mar 24, 2022

View reviewed changes

bjaglin force-pushed the dontforceversion branch from b9e14bd to 012651c Compare March 24, 2022 12:00

bjaglin commented Mar 24, 2022

View reviewed changes

scalafixEnable: refer to sbt keys in log messages

ac062ef

bjaglin force-pushed the dontforceversion branch from 7fc72f8 to ac062ef Compare March 24, 2022 16:17

bjaglin mentioned this pull request Mar 24, 2022

all scala 2.x patch releases are supported scalacenter/scalafix#1576

Merged

adpi2 approved these changes Mar 25, 2022

View reviewed changes

bjaglin merged commit 3bcfd20 into scalacenter:main Mar 25, 2022

This was referenced Mar 26, 2022

Do not hardcode sbt-scalafix version scala-steward-org/scala-steward#2357

Closed

sbt-scalafix doesn't respect sbt.override.build.repos scalacenter/scalafix#1049

Open

bjaglin mentioned this pull request Apr 3, 2022

dynamic sbt-scalafix version scala-steward-org/scala-steward#2475

Merged

bjaglin mentioned this pull request Apr 11, 2022

Update sbt-scalafix to 0.10.0 scalameta/munit#513

Closed

bjaglin mentioned this pull request Apr 11, 2022

scalafixEnable: preserve versions set by ++ #300

Merged

bjaglin mentioned this pull request Jul 15, 2022

scalafix 0.10.1 scalameta/munit#560

Merged

bjaglin mentioned this pull request Jul 19, 2023

Update scalameta to 4.8.5 scalacenter/scalafix#1825

Closed

This was referenced Sep 10, 2023

better error reporting when semanticdb-scalac is not found #369

Merged

Advertize scalafixEnable ekrich/sconfig#340

Merged

bjaglin mentioned this pull request Nov 10, 2023

Update scalameta to 4.8.8 scalacenter/scalafix#1846

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

make scalafixEnable less aggressive & more future-proof #292

make scalafixEnable less aggressive & more future-proof #292

bjaglin commented Mar 24, 2022 •

edited

bjaglin Mar 24, 2022 •

edited

bjaglin Mar 24, 2022 •

edited

bjaglin commented Mar 24, 2022

adpi2 commented Mar 24, 2022

bjaglin commented Mar 24, 2022

bjaglin Mar 24, 2022

bjaglin commented Mar 24, 2022 •

edited

bjaglin Mar 24, 2022

bjaglin Mar 24, 2022

adpi2 Mar 24, 2022

bjaglin Mar 24, 2022

adpi2 Mar 24, 2022

bjaglin Mar 24, 2022 •

edited

bjaglin commented Mar 24, 2022

adpi2 left a comment

make scalafixEnable less aggressive & more future-proof #292

make scalafixEnable less aggressive & more future-proof #292

Conversation

bjaglin commented Mar 24, 2022 • edited

bjaglin Mar 24, 2022 • edited

Choose a reason for hiding this comment

👌 on Java 8

💥 on Java 11

💥 on Java 17

bjaglin Mar 24, 2022 • edited

Choose a reason for hiding this comment

bjaglin commented Mar 24, 2022

adpi2 commented Mar 24, 2022

bjaglin commented Mar 24, 2022

bjaglin Mar 24, 2022

Choose a reason for hiding this comment

bjaglin commented Mar 24, 2022 • edited

bjaglin Mar 24, 2022

Choose a reason for hiding this comment

bjaglin Mar 24, 2022

Choose a reason for hiding this comment

adpi2 Mar 24, 2022

Choose a reason for hiding this comment

bjaglin Mar 24, 2022

Choose a reason for hiding this comment

adpi2 Mar 24, 2022

Choose a reason for hiding this comment

bjaglin Mar 24, 2022 • edited

Choose a reason for hiding this comment

bjaglin commented Mar 24, 2022

adpi2 left a comment

Choose a reason for hiding this comment

bjaglin commented Mar 24, 2022 •

edited

bjaglin Mar 24, 2022 •

edited

bjaglin Mar 24, 2022 •

edited

bjaglin commented Mar 24, 2022 •

edited

bjaglin Mar 24, 2022 •

edited