-
Notifications
You must be signed in to change notification settings - Fork 302
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
When there is a missing jar in Maven's local repo, it will fail. #2942
Comments
FYI here's full stack trace from OP's https://github.com/panbingkun/spark/actions/runs/8105142421/job/22153031672: [info] spark-hive: mimaPreviousArtifacts not set, not analyzing binary compatibility
[error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] file:/root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar: not found: /root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar
[error]
[error] at lmcoursier.internal.shaded.coursier.Artifacts$.$anonfun$fetchArtifacts$9(Artifacts.scala:365)
[error] at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$extension$1(Task.scala:14)
[error] at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$extension$1$adapted(Task.scala:14)
[error] at lmcoursier.internal.shaded.coursier.util.Task$.wrap(Task.scala:82)
[error] at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$2(Task.scala:14)
[error] at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
[error] at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
[error] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] at java.base/java.lang.Thread.run(Thread.java:840)
[error] Caused by: lmcoursier.internal.shaded.coursier.cache.ArtifactError$NotFound: not found: /root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar
[error] at lmcoursier.internal.shaded.coursier.cache.internal.Downloader.$anonfun$checkFileExists$1(Downloader.scala:590)
[error] at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
[error] at scala.util.Success.$anonfun$map$1(Try.scala:255)
[error] at scala.util.Success.map(Try.scala:213)
[error] at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
[error] at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
[error] at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
[error] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] at java.base/java.lang.Thread.run(Thread.java:840)
[error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] file:/root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar: not found: /root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar
[error] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] at java.base/java.lang.Thread.run(Thread.java:840)
[error] (mllib-local / update) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] file:/root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar: not found: /root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar
[error] (mllib-local / mimaPreviousClassfiles) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] file:/root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar: not found: /root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar
[error] (sql-kafka-0-10 / update) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] file:/root/.m2/repository/net/sf/jopt-simple/jopt-simple/5.0.4/jopt-simple-5.0.4.jar: not found: /root/.m2/repository/net/sf/jopt-simple/jopt-simple/5.0.4/jopt-simple-5.0.4.jar
[error] (streaming-kafka-0-10 / update) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] file:/root/.m2/repository/net/sf/jopt-simple/jopt-simple/5.0.4/jopt-simple-5.0.4.jar: not found: /root/.m2/repository/net/sf/jopt-simple/jopt-simple/5.0.4/jopt-simple-5.0.4.jar
[error] (connect-client-jvm / update) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] file:/root/.m2/repository/com/lihaoyi/geny_2.13/0.7.1/geny_2.13-0.7.1.jar: not found: /root/.m2/repository/com/lihaoyi/geny_2.13/0.7.1/geny_2.13-0.7.1.jar
[error] file:/root/.m2/repository/org/scala-lang/modules/scala-collection-compat_2.13/2.9.0/scala-collection-compat_2.13-2.9.0.jar: not found: /root/.m2/repository/org/scala-lang/modules/scala-collection-compat_2.13/2.9.0/scala-collection-compat_2.13-2.9.0.jar
[error] file:/root/.m2/repository/com/lihaoyi/sourcecode_2.13/0.3.0/sourcecode_2.13-0.3.0.jar: not found: /root/.m2/repository/com/lihaoyi/sourcecode_2.13/0.3.0/sourcecode_2.13-0.3.0.jar
[error] Total time: 299 s (04:59), completed Mar 1, 2024, 1:42:34 AM I wonder if |
Yea, that's right, only |
it would be greate if we can delete the jars with command |
@lefou @ghostbuster91 Sorry to bother you, do we have a way to bypass this case? Except for |
Because I found #2929, it seems that our scene is very similar. |
I don't know. I think the proper solution would require a change in coursier. At least I didn't found anything in the docs, that could help here. Some developer note: Maven does use different implementations for Remote Repositories and Local Repositories. The local one stores extra information like lastest update attempts and other metadata and its core property is to store partial/incomplete dependency trees. Coursier has probably only one implementation for remote repositories and isn't able to detect a partial Maven repository state. So we either need to implement some local repository support or improve the existing implementation to better handle local repos. If you plan to rely on a Maven local repository with coursier, you better get prepared to contribute. |
Thank you for your patient and detailed explanation.
But when we use SBT greater than
I speculate that |
I'd be tempted to close this as "won't fix". coursier has always been assuming that the repository that contains metadata (like POM files) also contains the JARs. That allows to limit network requests (single request to download the JAR), but also to know for sure that a JAR doesn't exist (checking if we get 404 is the only way to know that a dependency doesn't have a JAR), rather than check all repositories for dependencies that don't have a JAR. I'm not sure breaking that assumption and issuing more network request for |
sbt/sbt#7506
When we use the local Maven repo as the first level cache of SBT, if there is a corrupted jar in this repo, it will directly fail.
When there is a corrupted jar in Maven's local repo, It should not fail, but rather automatically skip this cache.
The text was updated successfully, but these errors were encountered: