tests are done against scala-xml dependency and not scala-xml local code !!! #20

dacr · 2014-02-28T10:14:38Z

SBT_OPTS="-verbose"
sbt test

and you'll see for example that XMLEventReader used class is coming from .ivy2/cache/org.scala-lang.modules/scala-xml_2.11.0-M8/bundles/scala-xml_2.11.0-M8-1.0.0-RC7.jar and not from the locale one I'm testing and which was freshly recompiled ./target/scala-2.11.0-M8/classes/scala/xml/pull/XMLEventReader.class with fixes I'm trying to test.

All tests are concerned.

[Loaded scala.xml.pull.XMLEventReaderTest from file:/home/work/experiments/scala/scala-xml-dacr/target/scala-2.11.0-M8/test-classes/]
[Loaded scala.xml.pull.XMLEventReader from file:/home/work/.ivy2/cache/org.scala-lang.modules/scala-xml_2.11.0-M8/bundles/scala-xml_2.11.0-M8-1.0.0-RC7.jar]
[Loaded scala.xml.pull.XMLEventReader$POISON$ from file:/home/work/.ivy2/cache/org.scala-lang.modules/scala-xml_2.11.0-M8/bundles/scala-xml_2.11.0-M8-1.0.0-RC7.jar]
[Loaded scala.xml.pull.XMLEventReader$Parser from file:/home/work/.ivy2/cache/org.scala-lang.modules/scala-xml_2.11.0-M8/bundles/scala-xml_2.11.0-M8-1.0.0-RC7.jar]
[Loaded scala.xml.pull.XMLEventReader$Parser$$anonfun$run$1 from file:/home/work/.ivy2/cache/org.scala-lang.modules/scala-xml_2.11.0-M8/bundles/scala-xml_2.11.0-M8-1.0.0-RC7.jar]
[Loaded scala.xml.pull.XMLEventReader$Parser$$anonfun$setEvent$1 from file:/home/work/.ivy2/cache/org.scala-lang.modules/scala-xml_2.11.0-M8/bundles/scala-xml_2.11.0-M8-1.0.0-RC7.jar]

The text was updated successfully, but these errors were encountered:

dacr · 2014-02-28T10:41:37Z

Looks like the dependency comes from scala-compiler/ivy-2.11.0-M8.xml :

       <dependencies>
                <dependency org="org.scala-lang" name="scala-library" rev="2.11.0-M8" force="true" conf="compile->compile(*),master(*);runtime->runtime(*)"/>
                <dependency org="org.scala-lang" name="scala-reflect" rev="2.11.0-M8" force="true" conf="compile->compile(*),master(*);runtime->runtime(*)"/>
                <dependency org="org.scala-lang.modules" name="scala-xml_2.11.0-M8" rev="1.0.0-RC7" force="true" conf="compile->compile(*),master(*);runtime->runtime(*)"/>
                <dependency org="org.scala-lang.modules" name="scala-parser-combinators_2.11.0-M8" rev="1.0.0-RC5" force="true" conf="compile->compile(*),master(*);runtime->runtime(*)"/>
                <dependency org="jline" name="jline" rev="2.11" force="true" conf="optional->compile(*),master(*)"/>
        </dependencies>

jsuereth · 2014-02-28T14:02:00Z

@adriaanm This is related to the other thread about running scaladoc, it seems but worse. I'll see if I can dive into your classpaths and try to restore some sanity.

jsuereth · 2014-02-28T14:04:38Z

Proof of the issue, as well as culprit:

> show test:fullClasspath
[warn] Credentials file /home/jsuereth/.ivy2/.credentials does not exist
[info] List(Attributed(/home/jsuereth/projects/scala/scala-xml/target/scala-2.11.0-M8/test-classes), Attributed(/home/jsuereth/projects/scala/scala-xml/target/scala-2.11.0-M8/classes), Attributed(/home/jsuereth/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.11.0-M8.jar), Attributed(/home/jsuereth/.ivy2/cache/org.scala-lang/scala-compiler/jars/scala-compiler-2.11.0-M8.jar), Attributed(/home/jsuereth/.ivy2/cache/org.scala-lang/scala-reflect/jars/scala-reflect-2.11.0-M8.jar), Attributed(/home/jsuereth/.ivy2/cache/org.scala-lang.modules/scala-xml_2.11.0-M8/bundles/scala-xml_2.11.0-M8-1.0.0-RC7.jar), Attributed(/home/jsuereth/.ivy2/cache/org.scala-lang.modules/scala-parser-combinators_2.11.0-M8/bundles/scala-parser-combinators_2.11.0-M8-1.0.0-RC5.jar), Attributed(/home/jsuereth/.ivy2/cache/junit/junit/jars/junit-4.11.jar), Attributed(/home/jsuereth/.ivy2/cache/org.hamcrest/hamcrest-core/jars/hamcrest-core-1.3.jar), Attributed(/home/jsuereth/.ivy2/cache/com.novocode/junit-interface/jars/junit-interface-0.10.jar), Attributed(/home/jsuereth/.ivy2/cache/junit/junit-dep/jars/junit-dep-4.10.jar), Attributed(/home/jsuereth/.ivy2/cache/org.scala-tools.testing/test-interface/jars/test-interface-0.5.jar))
[success] Total time: 0 s, completed Feb 28, 2014 9:03:29 AM
> libraryDependencies
[info] List(org.scala-lang:scala-library:2.11.0-M8, junit:junit:4.11:test, com.novocode:junit-interface:0.10:test, org.scala-lang:scala-compiler:2.11.0-M8:test)

jsuereth · 2014-02-28T14:12:46Z

Ok, this fix is simple. In build.sbt:

// used in CompilerErrors test
libraryDependencies += ("org.scala-lang" % "scala-compiler" % scalaVersion.value % "test").excludeAll(ExclusionRule(organization="org.scala-lang.modules"))

Proof:

show test:fullClasspath
[warn] Credentials file /home/jsuereth/.ivy2/.credentials does not exist
[info] Updating {file:/home/jsuereth/projects/scala/scala-xml/}scala-xml...
[info] Resolving jline#jline;2.11 ...
[info] Done updating.
[info] List(Attributed(/home/jsuereth/projects/scala/scala-xml/target/scala-2.11.0-M8/test-classes), Attributed(/home/jsuereth/projects/scala/scala-xml/target/scala-2.11.0-M8/classes), Attributed(/home/jsuereth/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.11.0-M8.jar), Attributed(/home/jsuereth/.ivy2/cache/org.scala-lang/scala-compiler/jars/scala-compiler-2.11.0-M8.jar), Attributed(/home/jsuereth/.ivy2/cache/org.scala-lang/scala-reflect/jars/scala-reflect-2.11.0-M8.jar), Attributed(/home/jsuereth/.ivy2/cache/junit/junit/jars/junit-4.11.jar), Attributed(/home/jsuereth/.ivy2/cache/org.hamcrest/hamcrest-core/jars/hamcrest-core-1.3.jar), Attributed(/home/jsuereth/.ivy2/cache/com.novocode/junit-interface/jars/junit-interface-0.10.jar), Attributed(/home/jsuereth/.ivy2/cache/junit/junit-dep/jars/junit-dep-4.10.jar), Attributed(/home/jsuereth/.ivy2/cache/org.scala-tools.testing/test-interface/jars/test-interface-0.5.jar))

Locally tests seem to pass. @adriaanm if you want to go this route, I can submit a PR. I think it's what you want, but please validate the resulting classpath.

dacr · 2014-02-28T15:59:06Z

NOK - Just tested on my environment with a modified XMLEventReader source code. I got the same proof result as you for "show test:fullClasspath" sbt command, but the reality is different, when sbt is started with -verbose JVMOptions, I can still see this class loaded from file scala- xml_2.11.0-M8-1.0.0-RC7.jar and not from scala-xml-dacr/target/scala-2.11.0-M8/classes.

[Loaded scala.xml.pull.XMLEventReader from file:/home/work/.ivy2/cache/org.scala-lang.modules/scala-xml_2.11.0-M8/bundles/scala-xml_2.11.0-M8-1.0.0-RC7.jar]

and so my test case is not executed against my modified XMLEventReader class.

adriaanm · 2014-02-28T16:08:29Z

Ok, thanks for raising this! I consider this a blocker for the Scala 2.11.0 final release.

adriaanm · 2014-02-28T16:29:20Z

Implied: I'm looking into fixing this and putting in a backstop to make sure we don't break this again. Help much appreciated!

jsuereth · 2014-02-28T18:14:33Z

Ok, working build fix: https://github.com/scala/scala-xml/tree/wip/fix-build-classpath (not in my own fork, sorry!)

dacr · 2014-02-28T20:22:00Z

I've cloned your branch :
1/ no changes, tests are OK although I can see that :

$ export SBT_OPTS="-verbose" ; sbt test | grep "XMLEventReader "
[Loaded scala.xml.pull.XMLEventReader from file:/Users/dcr/.ivy2/cache/org.scala-lang.modules/scala-xml_2.11.0-M8/bundles/scala-xml_2.11.0-M8-1.0.0-RC7.jar]

(vi ~/.sbtconfig to remove SBT_OPTS overwrite)

2/ bring back my XMLEventReader and XMLEventReaderTest changes, but the dependency is still here, and the wrong XMLEventReader class is tested.

adriaanm · 2014-02-28T20:23:15Z

I haven't pushed the second half of my fix (developed with help of @jsuereth):

//// testing:
// used in CompilerErrors test
libraryDependencies += ("org.scala-lang" % "scala-compiler" % scalaVersion.value % "test").exclude("org.scala-lang.modules", s"scala-xml*")

// needed to fix classloader issues (see #20)
// alternatively, manage the scala instance as shown below (commented)
fork in Test := true

jsuereth · 2014-02-28T20:26:07Z

Yeah, I still haven't found where scala-xml 2.11.0-M8 is sneaking into the classpath. My guess is it's getting loaded in sbt's Dual-Loader when the scala compiler is run against the code and then when our child classloader (for the project) is used, it gets that parent one.

Two follow ups:

We may want to separate sbt from any attempts to share Scala classloaders between the compiler + the project. What we're doing now breaks scala modularization.
We should resolve the scala-tool configuration for scala modules using "dummy" artifacts so that when there are circular dependencies the current project doesn't get in the way of grabbing necessary jars for bootstrapping.

dacr · 2014-02-28T20:28:55Z

ok sorry I thought it was done. And thanks for the work around, (fork in Test := true).

not scala-compiler's scala-xml dependency... also put int a println to show the version we're testing to prevent regression

Fix #20: test locally built scala.xml classes

Somehow, the scala-parser-combinators classes used to run tests were not the local ones, even though there is no dependency on scala-compiler nor on anything that depends on scala-parser-combinators. After some head-scratching, I discovered scala/scala-xml#20. Now I am even more confused, but at least `fork in Test := true` does prevent this evil classloader magic.

gourlaysama · 2014-03-08T15:47:28Z

It seems like this is also happening with scala-parser-combinators. After desperately trying to make a new test pass, I realized the wrong classfiles were tested... see scala/scala-parser-combinators#13.

Forking is needed to fix scalac classloader issues (see scala/scala-xml#20) that conflate the compiler's run-time classpath and the compilation classpath. The run-time (boot)classpath leaks into the compilation classpath, so that scalac see classes that are used to run it as classes used to compile against... Forking uses a minimal classpath, so this craziness is avoided. Alternatively, we could manage the scala instance as shown at the end of this file (commented)

drewboardman · 2019-11-18T15:35:00Z

I'm still running into this bug.

SethTisue · 2019-11-19T09:11:17Z

@drewboardman can you be more specific?

there are a lot of tickets in this area; I made a (probably not complete) list at #195 (comment)

you are using sbt 1.1.2 or newer, I hope?

adriaanm referenced this issue in adriaanm/scala-xml Feb 28, 2014

WIP: compiler tests should check scala-xml version tested

5343c7d

adriaanm added a commit to adriaanm/scala-xml that referenced this issue Feb 28, 2014

Fix scala#20: test locally built scala.xml classes

982e0cb

not scala-compiler's scala-xml dependency... also put int a println to show the version we're testing to prevent regression

adriaanm closed this as completed in f1cb694 Feb 28, 2014

adriaanm added a commit that referenced this issue Feb 28, 2014

Merge pull request #21 from adriaanm/i20

567cc3c

Fix #20: test locally built scala.xml classes

gourlaysama mentioned this issue Mar 8, 2014

Test locally built scala parser-combinators classes. scala/scala-parser-combinators#13

Merged

adriaanm mentioned this issue Mar 8, 2014

Fork in test to work around classpath issues scala/sbt-scala-module#6

Merged

gourlaysama mentioned this issue Oct 27, 2016

Exception in JUnit test after disabling JVM forking #112

Merged

This was referenced Dec 18, 2016

BasicTransformer exponential complexity #125

Closed

Unable to run scala-xml 1.0.6 on 2.11.8 scala/scala-module-dependency-sample#14

Closed

joescii mentioned this issue Aug 7, 2017

Incorrect classpath precedence for scala-xml sbt/sbt#3405

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests are done against scala-xml dependency and not scala-xml local code !!! #20

tests are done against scala-xml dependency and not scala-xml local code !!! #20

dacr commented Feb 28, 2014

dacr commented Feb 28, 2014

jsuereth commented Feb 28, 2014

jsuereth commented Feb 28, 2014

jsuereth commented Feb 28, 2014

dacr commented Feb 28, 2014

adriaanm commented Feb 28, 2014

adriaanm commented Feb 28, 2014

jsuereth commented Feb 28, 2014

dacr commented Feb 28, 2014

adriaanm commented Feb 28, 2014

jsuereth commented Feb 28, 2014

dacr commented Feb 28, 2014

gourlaysama commented Mar 8, 2014

drewboardman commented Nov 18, 2019

SethTisue commented Nov 19, 2019 •

edited

Loading

tests are done against scala-xml dependency and not scala-xml local code !!! #20

tests are done against scala-xml dependency and not scala-xml local code !!! #20

Comments

dacr commented Feb 28, 2014

dacr commented Feb 28, 2014

jsuereth commented Feb 28, 2014

jsuereth commented Feb 28, 2014

jsuereth commented Feb 28, 2014

dacr commented Feb 28, 2014

adriaanm commented Feb 28, 2014

adriaanm commented Feb 28, 2014

jsuereth commented Feb 28, 2014

dacr commented Feb 28, 2014

adriaanm commented Feb 28, 2014

jsuereth commented Feb 28, 2014

dacr commented Feb 28, 2014

gourlaysama commented Mar 8, 2014

drewboardman commented Nov 18, 2019

SethTisue commented Nov 19, 2019 • edited Loading

SethTisue commented Nov 19, 2019 •

edited

Loading