Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bloop fails to compile project with weird error #2053

Closed
ktonga opened this issue May 18, 2023 · 11 comments
Closed

Bloop fails to compile project with weird error #2053

ktonga opened this issue May 18, 2023 · 11 comments
Labels
bug A defect or misbehaviour.
Milestone
Bloop v1.5.7

Comments

@ktonga
Copy link
Contributor

ktonga commented May 18, 2023

It happens for only one of the modules (maybe some other after passing that one too) in a multi-module project, there are many modules before that one is attempted which finish compiling without issue. Also if I compile using sbt it doesn't happen.

I wasn't able to isolate the issue in a smaller self-contained project. I can help with any debugging if I'm guided, TBH I don't know where I could start from. To see that error I had to add the --verbose flag.

[E] Unexpected error when compiling merchant-model: 'null'
[T] java.nio.file.ClosedFileSystemException
[T] 	jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.ensureOpen(ZipFileSystem.java:1628)
[T] 	jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.getFileAttributes(ZipFileSystem.java:536)
[T] 	jdk.zipfs/jdk.nio.zipfs.ZipPath.readAttributes(ZipPath.java:767)
[T] 	jdk.zipfs/jdk.nio.zipfs.ZipPath.readAttributes(ZipPath.java:777)
[T] 	jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.readAttributes(ZipFileSystemProvider.java:276)
[T] 	java.base/java.nio.file.Files.readAttributes(Files.java:1851)
[T] 	java.base/java.nio.file.Files.size(Files.java:2468)
[T] 	sbt.internal.inc.HashUtil$.farmHash(HashUtil.scala:31)
[T] 	sbt.internal.inc.FarmHash$.ofPath(Stamp.scala:89)
[T] 	sbt.internal.inc.Stamper$.$anonfun$forHashInRootPaths$3(Stamp.scala:233)
[T] 	sbt.internal.inc.Stamper$.tryStamp(Stamp.scala:188)
[T] 	sbt.internal.inc.Stamper$.$anonfun$forHashInRootPaths$1(Stamp.scala:233)
[T] 	sbt.internal.inc.UncachedStamps.library(Stamp.scala:468)
[T] 	sbt.internal.inc.InitialStamps.$anonfun$library$1(Stamp.scala:422)
[T] 	scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:86)
[T] 	sbt.internal.inc.InitialStamps.library(Stamp.scala:422)
[T] 	sbt.internal.inc.TimeWrapBinaryStamps.$anonfun$library0$1(Stamp.scala:446)
[T] 	sbt.internal.inc.Stamper$.$anonfun$timeWrap$1(Stamp.scala:252)
[T] 	sbt.internal.inc.TimeWrapBinaryStamps.library(Stamp.scala:447)
[T] 	sbt.internal.inc.bloop.internal.BloopAnalysisCallback.$anonfun$addProductsAndDeps$16(BloopAnalysisCallback.scala:400)
[T] 	scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
[T] 	scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
[T] 	scala.collection.TraversableLike.map(TraversableLike.scala:286)
[T] 	scala.collection.TraversableLike.map$(TraversableLike.scala:279)
[T] 	scala.collection.mutable.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:50)
[T] 	scala.collection.SetLike.map(SetLike.scala:105)
[T] 	scala.collection.SetLike.map$(SetLike.scala:105)
[T] 	scala.collection.mutable.AbstractSet.map(Set.scala:50)
[T] 	sbt.internal.inc.bloop.internal.BloopAnalysisCallback.$anonfun$addProductsAndDeps$1(BloopAnalysisCallback.scala:398)
[T] 	scala.collection.TraversableOnce$folder$1.apply(TraversableOnce.scala:196)
[T] 	scala.collection.TraversableOnce$folder$1.apply(TraversableOnce.scala:194)
[T] 	scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
[T] 	scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:199)
[T] 	scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:192)
[T] 	scala.collection.AbstractTraversable.foldLeft(Traversable.scala:108)
[T] 	sbt.internal.inc.bloop.internal.BloopAnalysisCallback.addProductsAndDeps(BloopAnalysisCallback.scala:356)
[T] 	sbt.internal.inc.bloop.internal.BloopAnalysisCallback.get(BloopAnalysisCallback.scala:298)
[T] 	sbt.internal.inc.bloop.internal.BloopIncremental$.$anonfun$compileIncremental$5(BloopIncremental.scala:119)
[T] 	bloop.task.Task.$anonfun$runAsync$7(Task.scala:265)
[T] 	monix.eval.Task$Map.apply(Task.scala:4604)
[T] 	monix.eval.Task$Map.apply(Task.scala:4600)
[T] 	monix.eval.internal.TaskRunLoop$.startFull(TaskRunLoop.scala:170)
[T] 	monix.eval.internal.TaskRestartCallback.syncOnSuccess(TaskRestartCallback.scala:101)
[T] 	monix.eval.internal.TaskRestartCallback.onSuccess(TaskRestartCallback.scala:74)
[T] 	monix.eval.internal.TaskExecuteOn$AsyncRegister$$anon$1.run(TaskExecuteOn.scala:71)
[T] 	java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1395)
[T] 	java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
[T] 	java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
[T] 	java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
[T] 	java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
[T] 	java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
[E] Failed to compile 'merchant-model'
@tgodzik
Copy link
Contributor

tgodzik commented May 18, 2023

Thanks for reporting! This looks like an issue with Zinc it or the way we are using it 🤔

@tgodzik tgodzik added the bug A defect or misbehaviour. label May 23, 2023
@tgodzik tgodzik added this to the Bloop v1.5.7 milestone May 23, 2023
tgodzik added a commit to tgodzik/bloop that referenced this issue May 25, 2023
@tgodzik
improvement: Don't fail on hashing, but use a random one
3ed2320 
Previously, when hashing would fail the whole compilation would also fail and while this seems incredibly rare let's try to guard ourselves here.

Connected to scalacenter#2053
@tgodzik tgodzik mentioned this issue May 25, 2023
Merged
tgodzik added a commit to tgodzik/bloop that referenced this issue May 25, 2023
@tgodzik
improvement: Don't fail on hashing, but use a random one
964a4f8 
Previously, when hashing would fail the whole compilation would also fail and while this seems incredibly rare let's try to guard ourselves here.

Connected to scalacenter#2053
@ktonga
Copy link
Contributor Author

ktonga commented May 27, 2023

Hey @tgodzik, after testing with the snapshot as you suggested on Twitter, I got the following error logs:

2023.05.27 22:34:42 ERROR Could not calculate hash for /BC/java.base/java/lang/CharSequence.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BC/java.xml/javax/xml/parsers/DocumentBuilderFactory.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEFG/java.xml/org/w3c/dom/Node.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /B/java.base/java/io/InputStream.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEFG/java.xml/javax/xml/xpath/XPathConstants.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEFG/java.xml/org/w3c/dom/NodeList.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEFG/java.base/java/io/ByteArrayInputStream.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEF/java.xml/javax/xml/xpath/XPathExpression.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEF/java.xml/javax/xml/xpath/XPath.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEF/java.xml/javax/xml/namespace/QName.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEF/java.xml/javax/xml/parsers/DocumentBuilder.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEF/java.xml/javax/xml/xpath/XPathFactory.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEFG/java.xml/org/w3c/dom/Document.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /9ABCDEFG/java.base/java/io/File.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /9ABCDEFG/java.base/java/lang/ClassLoader.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BC/java.base/java/nio/file/Path.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEFG/java.base/java/nio/file/Paths.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /9ABCDEF/java.base/java/time/OffsetDateTime.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /89ABCDEFG/java.base/java/util/HashMap.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEF/java.base/java/time/ZoneOffset.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /9ABCDEF/java.base/java/time/LocalDate.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /9ABCDEF/java.base/java/time/ZoneId.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /879ABCDEFG/java.base/java/nio/charset/Charset.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEFG/java.base/java/nio/charset/StandardCharsets.sig because of null
2023.05.27 22:34:43 ERROR Could not calculate hash for /BCDEFG/java.base/java/util/concurrent/atomic/AtomicLong.sig because of null
2023.05.27 22:34:43 ERROR Could not calculate hash for /BCDEFG/java.base/java/util/Iterator.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /BCDEF/java.base/java/util/stream/Stream.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /BCDEFG/java.base/java/util/stream/BaseStream.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /9AB/java.base/java/lang/Integer.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /BCD/java.base/java/lang/Thread.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /BCDEFG/java.base/java/util/concurrent/Future.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /B/java.base/java/util/concurrent/CompletableFuture.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /BCDEFG/java.base/java/util/concurrent/Callable.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /89ABCDEFG/java.base/java/util/Collections.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /9ABCDEFG/java.base/java/util/concurrent/Executors.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /BCDEFG/java.base/java/util/concurrent/ExecutorService.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /BCDEFG/java.base/java/lang/Iterable.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /BCDEF/java.base/java/util/function/Consumer.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /9ABCDEF/java.base/java/time/ZonedDateTime.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /9ABCDEF/java.base/java/time/LocalTime.sig because of null
2023.05.27 22:34:46 ERROR Could not calculate hash for /9ABCDEFG/java.base/java/util/concurrent/ArrayBlockingQueue.sig because of null
2023.05.27 22:34:46 ERROR Could not calculate hash for /BCDEF/java.base/java/util/function/IntUnaryOperator.sig because of null
2023.05.27 22:34:46 ERROR Could not calculate hash for /BCDEFG/java.base/java/util/concurrent/atomic/AtomicInteger.sig because of null
2023.05.27 22:34:46 ERROR Could not calculate hash for /BCDEFG/java.base/java/util/Collection.sig because of null
2023.05.27 22:34:46 ERROR Could not calculate hash for /BCDEF/java.base/java/util/concurrent/atomic/AtomicReference.sig because of null

Hope it helps, If you need any more help just let me know.
Cheers.

@tgodzik
Copy link
Contributor

tgodzik commented May 27, 2023

Och, that's curious. It shouldn't reach into the jar, just calculate the hash of the entire jar.

@tgodzik
Copy link
Contributor

tgodzik commented May 27, 2023

Also, any idea where those .sig files are coming from? Should they be part of the JDK itself? I wonder if we should just filter them out and only calculate hashes for .class files.

@ktonga
Copy link
Contributor Author

ktonga commented May 29, 2023

Hey @tgodzik

It doesn't look like those files are in the JDK runtime FS.
I played around a bit with it and it seems it only contains .class files:

val fs = java.nio.file.FileSystems.getFileSystem(java.net.URI.create("jrt:/"))
java.nio.file.Files.list(fs.getPath("modules", "java.base", "java/lang")).forEach(println(_))

modules/java.base/java/lang/AbstractMethodError.class
modules/java.base/java/lang/AbstractStringBuilder.class
modules/java.base/java/lang/Appendable.class
modules/java.base/java/lang/ApplicationShutdownHooks$1.class
modules/java.base/java/lang/ApplicationShutdownHooks.class
modules/java.base/java/lang/ArithmeticException.class
modules/java.base/java/lang/ArrayIndexOutOfBoundsException.class
modules/java.base/java/lang/ArrayStoreException.class
modules/java.base/java/lang/AssertionError.class
modules/java.base/java/lang/AssertionStatusDirectives.class
modules/java.base/java/lang/AutoCloseable.class
modules/java.base/java/lang/Boolean.class
modules/java.base/java/lang/BootstrapMethodError.class
modules/java.base/java/lang/Byte$ByteCache.class
modules/java.base/java/lang/Byte.class
modules/java.base/java/lang/CharSequence$1CharIterator.class
modules/java.base/java/lang/CharSequence$1CodePointIterator.class
modules/java.base/java/lang/CharSequence.class
modules/java.base/java/lang/Character$CharacterCache.class
....

 java.nio.file.Files.lines(fs.getPath("modules", "java.base", "java/lang/CharSequence.sig")).forEach(println(_))

java.nio.file.NoSuchFileException: /modules/java.base/java/lang/CharSequence.sig
  at java.base/jdk.internal.jrtfs.JrtFileSystem.checkNode(JrtFileSystem.java:486)
  at java.base/jdk.internal.jrtfs.JrtFileSystem.getFileContent(JrtFileSystem.java:245)
  at java.base/jdk.internal.jrtfs.JrtFileSystem.newInputStream(JrtFileSystem.java:334)
  at java.base/jdk.internal.jrtfs.JrtPath.newInputStream(JrtPath.java:637)
  at java.base/jdk.internal.jrtfs.JrtFileSystemProvider.newInputStream(JrtFileSystemProvider.java:325)
  at java.base/java.nio.file.Files.newInputStream(Files.java:160)
  at java.base/java.nio.file.Files.newBufferedReader(Files.java:2922)
  at java.base/java.nio.file.Files.lines(Files.java:4141)
  at java.base/java.nio.file.Files.lines(Files.java:4227)
  ... 59 elided

@tgodzik
Copy link
Contributor

tgodzik commented May 30, 2023

the .sig files are supposed to be pickles needed for pipelining (not only probably) but we don't have any pickles for Java files, especially not inside a jar.

I will dig in a bit more.

@tgodzik
Copy link
Contributor

tgodzik commented May 30, 2023

Are you using any JDK specific classes or options? How is the modules defined, would you be able to share that?

Looks like the sig files should not pop up there and they are somehow 🤔

tgodzik added a commit to tgodzik/bloop that referenced this issue May 30, 2023
@tgodzik
bugfix: Don't accept files other than .class or .jar as binary deps
861762c 
Should help with scalacenter#2053

The binary is described as:
```
A binary entry represents either the jar or the
concrete class file from which the Scala compiler
knows that <code>onBinaryClassName</code> comes from.
```
@tgodzik tgodzik mentioned this issue May 30, 2023
Merged
@ktonga
Copy link
Contributor Author

ktonga commented Jun 2, 2023

Are you using any JDK specific classes or options? How is the modules defined, would you be able to share that?

Looks like the sig files should not pop up there and they are somehow 🤔

So this is the JDK im using locally:

$ java -version
openjdk version "17.0.7" 2023-04-18
OpenJDK Runtime Environment Homebrew (build 17.0.7+0)
OpenJDK 64-Bit Server VM Homebrew (build 17.0.7+0, mixed mode, sharing)

And this, I think, are all the relevant build definition bits:

val baseSettings =
  Seq(
    scalaVersion := Dependencies.Version.scala,
    organization := "...",
    logLevel     := Level.Info,
    scalacOptions ++= Seq("-release", "11"),
    Compile / scalacOptions ++= Seq(
      "-Wconf:any:warning-verbose",
      "-Wunused:nowarn",
      "-Ymacro-annotations",
      "-Xsource:3"
    ),
    PB.protocVersion := Version.protobufJava, // generate objects at same version as lib
    Compile / scalacOptions ~= {
      options: Seq[String] =>
        options.filterNot(
          Set(
            "-Wunused:imports",
            "-Wunused:params",
            "-Wunused:privates",
            "-Xlint:infer-any"
          )
        )
    },
    javacOptions ++= Seq("-source", "16", "-target", "16"),
    //      scalafmtOnCompile := true,
    testFrameworks ++= Seq(
      TestFramework("zio.test.sbt.ZTestFramework"),
      TestFramework("hedgehog.sbt.Framework")
    ),
    Global / cancelable           := true,
    Global / turbo                := true, // 🤞
    Global / fork                 := true, // https://github.com/sbt/sbt/issues/2274
    Global / onChangedBuildSource := ReloadOnSourceChanges,
    resolvers ++= Seq(
      "Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots",
      "Confluent" at "https://packages.confluent.io/maven",
      "Ovotech" at "https://kaluza.jfrog.io/artifactory/maven/"
      //        Resolver.bintrayRepo("ovotech", "maven")
    ),
    // Don't make or publish scaladoc
    Compile / doc / sources                := Nil,
    Compile / packageDoc / publishArtifact := false,
    // parallel and forked tests
    Test / fork               := true,
    Test / testForkedParallel := true,
    Test / parallelExecution  := true,
    // [for the moment] Test / turbo := true,
    Test / concurrentRestrictions := Seq(
      Tags.limit(Tags.CPU, java.lang.Runtime.getRuntime.availableProcessors()),
      Tags.limit(Tags.Test, java.lang.Runtime.getRuntime.availableProcessors())
    ),
    // incremental compilation cache
    pushRemoteCacheTo := Some(MavenCache("sbt-build-cache", file(".sbt-build-cache")))
  )

val compilerPlugins =
  List(
    kindProjector,
    betterMonadicFor
  )

val commonSettings =
  baseSettings ++
    Seq(
      Compile / compile / wartremoverErrors ++= warts,
      libraryDependencies ++= compilerPlugins,
      dependencyOverrides ++= List(
        ...
      )
    )

lazy val `merchant-model` = (project in file("modules/merchant-model"))
  .settings(
    commonSettings,
    libraryDependencies ++=
      List(
        ...
      )
  )
  .dependsOn(
    ...
  )

tgodzik added a commit to tgodzik/bloop that referenced this issue Jun 29, 2023
@tgodzik
bugfix: Don't accept files other than .class or .jar as binary deps
77ca736 
Should help with scalacenter#2053

The binary is described as:
```
A binary entry represents either the jar or the
concrete class file from which the Scala compiler
knows that <code>onBinaryClassName</code> comes from.
```
@tgodzik
Copy link
Contributor

tgodzik commented Jun 30, 2023

I finally got back to the PR with a potential fix for your issue @ktonga <- it should be in the newest snapshot version

@ktonga
Copy link
Contributor Author

ktonga commented Jun 30, 2023

I finally got back to the PR with a potential fix for your issue @ktonga <- it should be in the newest snapshot version

Thank you so much, I'll give it a go. 🙏

@ktonga
Copy link
Contributor Author

ktonga commented Jul 5, 2023

Hey! I was able to update to 1.5.7 and I'm not seeing the error any more, so I think we could close this one as fixed.

@ktonga ktonga closed this as completed Jul 5, 2023
tgodzik added a commit to tgodzik/bloop that referenced this issue Jul 13, 2023
@tgodzik
improvement: Don't fail on hashing, but use a random one
f4723c9 
Previously, when hashing would fail the whole compilation would also fail and while this seems incredibly rare let's try to guard ourselves here.

Connected to scalacenter#2053
tgodzik added a commit to tgodzik/bloop that referenced this issue Jul 13, 2023
@tgodzik
bugfix: Don't accept files other than .class or .jar as binary deps
79c0db8 
Should help with scalacenter#2053

The binary is described as:
```
A binary entry represents either the jar or the
concrete class file from which the Scala compiler
knows that <code>onBinaryClassName</code> comes from.
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug A defect or misbehaviour.
Projects
None yet
Milestone
Bloop v1.5.7
Development

No branches or pull requests
2 participants
@ktonga @tgodzik