Ivy import of cats library fails and corrupts Ivy Cache

If I try and import the cats library (there might be others) I get the error below and afterwards my Ivy cache is corrupted and SBT fails when I run sbt update - specifically the scala and sbt libraries are corrupted and fails with
org.scala-lang#scala-library;2.11.8: configuration not found in org.scala-lang#scala-library;2.11.8: 'master(compile)'. Missing configuration: 'compile'.
If I delete scala and scala sbt libs from the ivy cache and they are downloaded again SBT works again.
I have tested this repeatedly and the behaviour is consistent. I am using ammonite repl 0.7.7.

import $ivy.org.typelevel::cats:0.7.2`, cats._

:: resolving dependencies :: org.typelevel#cats_2.11-caller;working
confs: [default]
found org.typelevel#cats_2.11;0.7.2 in chain-resolver
[0.7.2] org.typelevel#cats_2.11;0.7.2
found org.scala-lang#scala-library;2.11.8 in m2
found org.typelevel#cats-macros_2.11;0.7.2 in chain-resolver
[0.7.2] org.typelevel#cats-macros_2.11;0.7.2
found com.github.mpilquist#simulacrum_2.11;0.8.0 in chain-resolver
found org.typelevel#macro-compat_2.11;1.1.1 in chain-resolver
found org.typelevel#machinist_2.11;0.4.1 in chain-resolver
found org.scala-lang#scala-reflect;2.11.7 in m2
found org.typelevel#cats-kernel_2.11;0.7.2 in chain-resolver
[0.7.2] org.typelevel#cats-kernel_2.11;0.7.2
found org.typelevel#cats-kernel-laws_2.11;0.7.2 in chain-resolver
[0.7.2] org.typelevel#cats-kernel-laws_2.11;0.7.2
found org.scalacheck#scalacheck_2.11;1.12.5 in chain-resolver
found org.scala-sbt#test-interface;1.0 in m2
found org.typelevel#discipline_2.11;0.4 in chain-resolver
found org.typelevel#catalysts-platform_2.11;0.0.2 in chain-resolver
[0.0.2] org.typelevel#catalysts-platform_2.11;0.0.2
found org.typelevel#catalysts-macros_2.11;0.0.2 in chain-resolver
[0.0.2] org.typelevel#catalysts-macros_2.11;0.0.2
found org.typelevel#cats-core_2.11;0.7.2 in chain-resolver
[0.7.2] org.typelevel#cats-core_2.11;0.7.2
found org.typelevel#cats-laws_2.11;0.7.2 in chain-resolver
[0.7.2] org.typelevel#cats-laws_2.11;0.7.2
found org.typelevel#cats-free_2.11;0.7.2 in chain-resolver
[0.7.2] org.typelevel#cats-free_2.11;0.7.2
found org.typelevel#cats-jvm_2.11;0.7.2 in chain-resolver
[0.7.2] org.typelevel#cats-jvm_2.11;0.7.2
ammonite.runtime.tools.IvyThing$IvyResolutionException: failed to resolve ivy dependencies unresolved dependency: org.scala-lang#scala-library;2.11.8: configuration not found in org.scala-lang#scala-library;2.11.8: 'master(compile)'. Missing configuration: 'compile'. It was required from org.typelevel#cats-jvm_2.11;0.7.2 compile, unresolved dependency: org.scala-lang#scala-reflect;2.11.7: configuration not found in org.scala-lang#scala-reflect;2.11.7: 'master(compile)'. Missing configuration: 'compile'. It was required from org.typelevel#machinist_2.11;0.4.1 compile, unresolved dependency: org.scala-sbt#test-interface;1.0: configuration not found in org.scala-sbt#test-interface;1.0: 'master(compile)'. Missing configuration: 'compile'. It was required from org.scalacheck#scalacheck_2.11;1.12.5 compile
ammonite.runtime.tools.IvyThing.resolveArtifact(IvyThing.scala:108)
ammonite.runtime.Interpreter.loadIvy(Interpreter.scala:592)
ammonite.runtime.ImportHook$BaseIvy$$anonfun$resolve$2.liftedTree1$1(ImportHook.scala:166)
ammonite.runtime.ImportHook$BaseIvy$$anonfun$resolve$2.apply(ImportHook.scala:166)
ammonite.runtime.ImportHook$BaseIvy$$anonfun$resolve$2.apply(ImportHook.scala:160)
ammonite.util.Res$Success.flatMap(Res.scala:57)
ammonite.runtime.ImportHook$BaseIvy.resolve(ImportHook.scala:160)
ammonite.runtime.ImportHook$BaseIvy$$anonfun$handle$4$$anonfun$apply$3.apply(ImportHook.scala:175)
ammonite.runtime.ImportHook$BaseIvy$$anonfun$handle$4$$anonfun$apply$3.apply(ImportHook.scala:175)
ammonite.util.Res$$anonfun$1.apply(Res.scala:33)
ammonite.util.Res$$anonfun$1.apply(Res.scala:30)
scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
scala.collection.immutable.List.foldLeft(List.scala:84)
ammonite.util.Res$.map(Res.scala:30)
ammonite.runtime.ImportHook$BaseIvy$$anonfun$handle$4.apply(ImportHook.scala:175)
ammonite.runtime.ImportHook$BaseIvy$$anonfun$handle$4.apply(ImportHook.scala:174)
ammonite.util.Res$Success.flatMap(Res.scala:57)
ammonite.runtime.ImportHook$BaseIvy.handle(ImportHook.scala:174)
ammonite.runtime.Interpreter$$anonfun$resolveSingleImportHook$3.apply(Interpreter.scala:151)
ammonite.runtime.Interpreter$$anonfun$resolveSingleImportHook$3.apply(Interpreter.scala:150)
ammonite.util.Res$Success.flatMap(Res.scala:57)
ammonite.runtime.Interpreter.resolveSingleImportHook(Interpreter.scala:150)
ammonite.runtime.Interpreter$$anonfun$resolveImportHooks$2.apply(Interpreter.scala:199)
ammonite.runtime.Interpreter$$anonfun$resolveImportHooks$2.apply(Interpreter.scala:199)
ammonite.util.Res$$anonfun$1.apply(Res.scala:33)
ammonite.util.Res$$anonfun$1.apply(Res.scala:30)
scala.collection.IndexedSeqOptimized$class.foldl(IndexedSeqOptimized.scala:57)
scala.collection.IndexedSeqOptimized$class.foldLeft(IndexedSeqOptimized.scala:66)
scala.collection.mutable.ArrayBuffer.foldLeft(ArrayBuffer.scala:48)
ammonite.util.Res$.map(Res.scala:30)
ammonite.runtime.Interpreter.resolveImportHooks(Interpreter.scala:199)
ammonite.runtime.Interpreter$$anonfun$processLine$2.apply(Interpreter.scala:213)
ammonite.runtime.Interpreter$$anonfun$processLine$2.apply(Interpreter.scala:209)
ammonite.util.Catching.flatMap(Res.scala:108)
ammonite.runtime.Interpreter.processLine(Interpreter.scala:209)
ammonite.repl.Repl$$anonfun$action$4$$anonfun$apply$2.apply(Repl.scala:88)
ammonite.repl.Repl$$anonfun$action$4$$anonfun$apply$2.apply(Repl.scala:87)
ammonite.repl.Scoped$$anonfun$flatMap$1.apply(Signaller.scala:44)
ammonite.repl.Signaller.apply(Signaller.scala:29)
ammonite.repl.Scoped$class.flatMap(Signaller.scala:44)
ammonite.repl.Signaller.flatMap(Signaller.scala:11)
ammonite.repl.Repl$$anonfun$action$4.apply(Repl.scala:87)
ammonite.repl.Repl$$anonfun$action$4.apply(Repl.scala:74)
ammonite.util.Res$Success.flatMap(Res.scala:57)
ammonite.repl.Repl.action(Repl.scala:74)
ammonite.repl.Repl.loop$1(Repl.scala:98)
ammonite.repl.Repl.run(Repl.scala:117)
ammonite.Main.run(Main.scala:108)
ammonite.Main$$anonfun$main$1$$anonfun$apply$1.apply(Main.scala:249)
ammonite.Main$.ammonite$Main$$ifContinually$1(Main.scala:228)
ammonite.Main$$anonfun$main$1.apply(Main.scala:230)
ammonite.Main$$anonfun$main$1.apply(Main.scala:230)
scala.Option.foreach(Option.scala:257)
ammonite.Main$.main(Main.scala:230)
ammonite.Main.main(Main.scala:-1)
`

[jmccance]

Just ran into the same problem.

∵ amm
Loading...
Welcome to the Ammonite Repl 0.7.7
(Scala 2.11.8 Java 1.8.0_102)
@ import $ivy.`org.typelevel::cats:0.7.0`
:: loading settings :: url = jar:file:/Users/joelm/bin/amm!/org/apache/ivy/core/settings/ivysettings.xml
:: resolving dependencies :: org.typelevel#cats_2.11-caller;working
    confs: [default]
    found org.typelevel#cats_2.11;0.7.0 in central
    found org.scala-lang#scala-library;2.11.8 in chain-resolver
    [2.11.8] org.scala-lang#scala-library;2.11.8
    found org.typelevel#cats-macros_2.11;0.7.0 in central
    found com.github.mpilquist#simulacrum_2.11;0.8.0 in central
    found org.typelevel#macro-compat_2.11;1.1.1 in central
    found org.typelevel#machinist_2.11;0.4.1 in chain-resolver
    found org.scala-lang#scala-reflect;2.11.7 in m2
    found org.typelevel#cats-kernel_2.11;0.7.0 in central
    found org.typelevel#cats-kernel-laws_2.11;0.7.0 in central
    found org.scalacheck#scalacheck_2.11;1.12.5 in chain-resolver
    found org.scala-sbt#test-interface;1.0 in m2
    found org.typelevel#discipline_2.11;0.4 in chain-resolver
    found org.typelevel#catalysts-platform_2.11;0.0.2 in chain-resolver
    found org.typelevel#catalysts-macros_2.11;0.0.2 in chain-resolver
    found org.typelevel#cats-core_2.11;0.7.0 in central
    found org.typelevel#cats-laws_2.11;0.7.0 in central
    found org.typelevel#cats-free_2.11;0.7.0 in central
    found org.typelevel#cats-jvm_2.11;0.7.0 in central
ammonite.runtime.tools.IvyThing$IvyResolutionException: failed to resolve ivy dependencies unresolved dependency: org.scala-lang#scala-reflect;2.11.7: configuration not found in org.scala-lang#scala-reflect;2.11.7: 'master(compile)'. Missing configuration: 'compile'. It was required from org.typelevel#machinist_2.11;0.4.1 compile, unresolved dependency: org.scala-sbt#test-interface;1.0: configuration not found in org.scala-sbt#test-interface;1.0: 'master(compile)'. Missing configuration: 'compile'. It was required from org.scalacheck#scalacheck_2.11;1.12.5 compile
  ammonite.runtime.tools.IvyThing.resolveArtifact(IvyThing.scala:108)
  ammonite.runtime.Interpreter.loadIvy(Interpreter.scala:592)
  ammonite.runtime.ImportHook$BaseIvy$$anonfun$resolve$2.liftedTree1$1(ImportHook.scala:166)
  ammonite.runtime.ImportHook$BaseIvy$$anonfun$resolve$2.apply(ImportHook.scala:166)
  ammonite.runtime.ImportHook$BaseIvy$$anonfun$resolve$2.apply(ImportHook.scala:160)
  ammonite.util.Res$Success.flatMap(Res.scala:57)
  ammonite.runtime.ImportHook$BaseIvy.resolve(ImportHook.scala:160)
  ammonite.runtime.ImportHook$BaseIvy$$anonfun$handle$4$$anonfun$apply$3.apply(ImportHook.scala:175)
  ammonite.runtime.ImportHook$BaseIvy$$anonfun$handle$4$$anonfun$apply$3.apply(ImportHook.scala:175)
  ammonite.util.Res$$anonfun$1.apply(Res.scala:33)
  ammonite.util.Res$$anonfun$1.apply(Res.scala:30)
  scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
  scala.collection.immutable.List.foldLeft(List.scala:84)
  ammonite.util.Res$.map(Res.scala:30)
  ammonite.runtime.ImportHook$BaseIvy$$anonfun$handle$4.apply(ImportHook.scala:175)
  ammonite.runtime.ImportHook$BaseIvy$$anonfun$handle$4.apply(ImportHook.scala:174)
  ammonite.util.Res$Success.flatMap(Res.scala:57)
  ammonite.runtime.ImportHook$BaseIvy.handle(ImportHook.scala:174)
  ammonite.runtime.Interpreter$$anonfun$resolveSingleImportHook$3.apply(Interpreter.scala:151)
  ammonite.runtime.Interpreter$$anonfun$resolveSingleImportHook$3.apply(Interpreter.scala:150)
  ammonite.util.Res$Success.flatMap(Res.scala:57)
  ammonite.runtime.Interpreter.resolveSingleImportHook(Interpreter.scala:150)
  ammonite.runtime.Interpreter$$anonfun$resolveImportHooks$2.apply(Interpreter.scala:199)
  ammonite.runtime.Interpreter$$anonfun$resolveImportHooks$2.apply(Interpreter.scala:199)
  ammonite.util.Res$$anonfun$1.apply(Res.scala:33)
  ammonite.util.Res$$anonfun$1.apply(Res.scala:30)
  scala.collection.IndexedSeqOptimized$class.foldl(IndexedSeqOptimized.scala:57)
  scala.collection.IndexedSeqOptimized$class.foldLeft(IndexedSeqOptimized.scala:66)
  scala.collection.mutable.ArrayBuffer.foldLeft(ArrayBuffer.scala:48)
  ammonite.util.Res$.map(Res.scala:30)
  ammonite.runtime.Interpreter.resolveImportHooks(Interpreter.scala:199)
  ammonite.runtime.Interpreter$$anonfun$processLine$2.apply(Interpreter.scala:213)
  ammonite.runtime.Interpreter$$anonfun$processLine$2.apply(Interpreter.scala:209)
  ammonite.util.Catching.flatMap(Res.scala:108)
  ammonite.runtime.Interpreter.processLine(Interpreter.scala:209)
  ammonite.repl.Repl$$anonfun$action$4$$anonfun$apply$2.apply(Repl.scala:88)
  ammonite.repl.Repl$$anonfun$action$4$$anonfun$apply$2.apply(Repl.scala:87)
  ammonite.repl.Scoped$$anonfun$flatMap$1.apply(Signaller.scala:44)
  ammonite.repl.Signaller.apply(Signaller.scala:29)
  ammonite.repl.Scoped$class.flatMap(Signaller.scala:44)
  ammonite.repl.Signaller.flatMap(Signaller.scala:11)
  ammonite.repl.Repl$$anonfun$action$4.apply(Repl.scala:87)
  ammonite.repl.Repl$$anonfun$action$4.apply(Repl.scala:74)
  ammonite.util.Res$Success.flatMap(Res.scala:57)
  ammonite.repl.Repl.action(Repl.scala:74)
  ammonite.repl.Repl.loop$1(Repl.scala:98)
  ammonite.repl.Repl.run(Repl.scala:117)
  ammonite.Main.run(Main.scala:108)
  ammonite.Main$$anonfun$main$1$$anonfun$apply$1.apply(Main.scala:249)
  ammonite.Main$.ammonite$Main$$ifContinually$1(Main.scala:228)
  ammonite.Main$$anonfun$main$1.apply(Main.scala:230)
  ammonite.Main$$anonfun$main$1.apply(Main.scala:230)
  scala.Option.foreach(Option.scala:257)
  ammonite.Main$.main(Main.scala:230)
  ammonite.Main.main(Main.scala:-1)

[olafurpg]

I bumped into the same issue after updating today. Nuking ~/.ivy2 seems to do the trick. Unfortunately, that means you have to re-download the whole internet on the next sbt update. I do that once in awhile anyways so I don't mind.

Edit. Seems like I still get the same failed to resolve ivy dependencies unresolved dependency: org.scala-lang#scala-library;2.11.8: error in SBT.

[malzzz]

I'm also having the same issue, but with other libraries. Oddly enough, cats imports fine. I can recall at least different libraries that corrupted my ivy cache. Unfortunately I didn't take a mental note of them at the time; however, the latest one was scrimage.

Going through the errors and doing "rm -r ~/.ivy2/cache/com.i-am-totally-corrupted" is a small pita depending on how many directories have to be removed. Purging the entire cache sucks as well from a time-standpoint.

[erik-stripe]

@olafurpg I've noticed something similar -- it seems like after I use Ammonite I start observing corrupted jars (i.e. I delete part of the Ivy cache and a particular error resolves). I've stopped using Ammonite-REPL and haven't seen these errors reassert myself. I'm not 100% sure what the cause is but Ammonite-REPL is definitely implicated.

[lihaoyi]

Yes I've hit this several times; it's a pain in the neck. Needs to be fixed

[malzzz]

@lihaoyi Do you have an intuition about the source(s) of the bug? If you point me in a general direction, I can perhaps get into it. Just from a cursory look, maybe IvyThing.scala? Going solely off the comments in the file, at least parts of that seem to be a kind of black box.

I haven't had a reason to look into how Ivy functions; however, just from observing the behavior of this issue qua dealing with the issue, I think I've noticed a pattern. That is, the ivy cache seems to get corrupted only when other processes are making use of the ivy cache. If I perform an sbt update on a repo via the terminal at the same time that Intellij tries to perform an update, one of the processes will have to wait on an sbt lock to the appropriate ivy directories. Does ammonite ensure that no other processes are making use of the cache prior to ammonite's operations on it? This is merely my intuition about the issue, which lacks any concrete evidence beyond my potentially fallacious memories about the sequence of events leading up to instances of cache corruption.

[avdv]

I also hit this consistently, every time I import a library via ivy2 (which itself or any of its dependencies is not already in the cache).

Here's the current error I'm getting:

[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: commons-io#commons-io;2.4: configuration not found in commons-io#commons-io;2.4: 'master(compile)'. Missing configuration: 'compile'. It was required from com.typesafe.play#play-doc_2.11;1.2.2 compile
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::

The ivy-2.4.xml in ~/.ivy2/cache/commons-io/commons-io looks fishy to me as it is basically empty:

<?xml version="1.0" encoding="UTF-8"?>
<ivy-module version="2.0">
        <info organisation="commons-io"
                module="commons-io"
                revision="2.4"
                status="release"
                publication="20161102131814"
                default="true"
        />
        <configurations>
                <conf name="default" visibility="public"/>
        </configurations>
        <publications>
                <artifact name="commons-io" type="jar" ext="jar" conf="default"/>
        </publications>
</ivy-module>

whereas after I deleted the file and ran sbt update again, the file contains all sorts of sections, like publications:

 <publications>
      <artifact name="commons-io" type="jar" ext="jar" conf="master"/>
</publications>

[mcanlas]

Is this still an issue? Does anyone have a minimal reproducible example? Running this in amm seems to work for me.

@ import $ivy.`org.typelevel::cats:0.8.1`

[dsebban]

This is still a big issue

Error:Error while importing SBT project:<br/>...<br/><pre>[info] Resolving org.javassist#javassist;3.18.1-GA ... [info] Resolving org.apache.commons#commons-math;2.2 ... [info] Resolving jline#jline;2.12.1 ... [info] Resolving org.scala-lang#scalap;2.11.7 ... [warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] :: UNRESOLVED DEPENDENCIES :: [warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] :: org.apache.commons#commons-lang3;3.1: configuration not found in org.apache.commons#commons-lang3;3.1: 'master(compile)'. Missing configuration: 'compile'. It was required from de.flapdoodle.embed#de.flapdoodle.embed.process;1.50.0 compile [warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] [warn] Note: Unresolved dependencies path: [warn] org.apache.commons:commons-lang3:3.1 [warn] +- de.flapdoodle.embed:de.flapdoodle.embed.process:1.50.0 [warn] +- de.flapdoodle.embed:de.flapdoodle.embed.mongo:1.50.0 [trace] Stack trace suppressed: run 'last *:update' for the full output. [trace] Stack trace suppressed: run 'last *:ssExtractDependencies' for the full output. [error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.commons#commons-lang3;3.1: configuration not found in org.apache.commons#commons-lang3;3.1: 'master(compile)'. Missing configuration: 'compile'. It was required from de.flapdoodle.embed#de.flapdoodle.embed.process;1.50.0 compile [error] (*:ssExtractDependencies) sbt.ResolveException: unresolved dependency: org.apache.commons#commons-lang3;3.1: configuration not found in org.apache.commons#commons-lang3;3.1: 'master(compile)'. Missing configuration: 'compile'. It was required from de.flapdoodle.embed#de.flapdoodle.embed.process;1.50.0 compile

[lihaoyi]

Yeah it's still a problem. Hoping that @alexarchambault publishing coursier 1.0 will be the end of our ivy-is-not-behaving problems 😛

[nightscape]

I just browsed the issues of coursier, but didn't find one that shows the specific symptoms of this issue. Does anyone know the specific issue (maybe @alexarchambault)? Just to make sure that this problem is known and tracked on the coursier side.

[olafurpg]

@nightscape I don't think Ammonite uses coursier. To my best understanding, this issue is caused by the ivy resolver, and will hopefully go away once ammonite switches to Coursier v1.0.

[maesenka]

I also ran into this problem, trying to import the amazon AWS SDK library.

On each attempt, Ammonite fails on the import and each time the same entries in the cache get corrupted (commons-logging, com.fasterxml, joda-time, org.apache.httpcomponents). This makes ammonite currently unusable for my project.

[Atry]

This issue is a regression, as ammonite-repl 0.6.2 never corrupts ivy cache

[drdozer]

I'm seeing this with:

**ammonite.runtime.tools.IvyThing$IvyResolutionException: failed to resolve ivy dependencies unresolved dependency: com.typesafe#config;1.2.1: configuration not found in com.typesafe#config;1.2.1: 'master(compile)'. Missing configuration: 'compile'. It was required from com.typesafe.slick#slick_2.12;3.2.0-M2 compile**

I'd rather not download the universe -- what specifically would I need to surgically nuke to get this to work?