Transitive dependencies with classifier "test" are not include in the classpath

[samuelsayag]

steps

These steps make use of the hadoop-mini-cluster project.

1. Create a sbt project (I did it with intellij) and modify the sbt by adding:

resolvers ++= Seq(
  "Hortonworks" at "http://repo.hortonworks.com/content/groups/public/",
  Resolver.sonatypeRepo("public")
)

transitiveClassifiers := Seq(Artifact.TestsClassifier, Artifact.SourceClassifier)

libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.1" % Test
libraryDependencies += "com.github.sakserv" % "hadoop-mini-clusters-common" % "0.1.11" % Test 
libraryDependencies += "com.github.sakserv" % "hadoop-mini-clusters-hdfs" % "0.1.11" % Test

2. Create a basic test class (in src/test ) that is supposed to use a "classifier test" artifact

import com.github.sakserv.minicluster.impl.HdfsLocalCluster
import org.apache.hadoop.conf.Configuration
import org.scalatest.{BeforeAndAfterAll, FlatSpec}

class TestHdfsUtility extends FlatSpec  {

  val hdfsLocalCluster = new HdfsLocalCluster.Builder()
    .setHdfsNamenodePort(12345)
    .setHdfsNamenodeHttpPort(12341)
    .setHdfsTempDir("embedded_hdfs")
    .setHdfsNumDatanodes(1)
    .setHdfsEnablePermissions(false)
    .setHdfsFormat(true)
    .setHdfsEnableRunningUserAsProxyUser(true)
    .setHdfsConfig(new Configuration())
    .build()

  "This test" should "start/stop mincluster" in {
    hdfsLocalCluster.start()
    hdfsLocalCluster.stop()
  }
}

problem

It start with the compilation that is successful but issue a warn about a not found resource

~/IdeaProjects/test-miniclusters$ sbt test:compile
[info] Loading project definition from /home/sam/IdeaProjects/test-miniclusters/project
[info] Set current project to test-miniclusters (in build file:/home/sam/IdeaProjects/test-miniclusters/)
[info] Compiling 1 Scala source to /home/sam/IdeaProjects/test-miniclusters/target/scala-2.10/test-classes...
[warn] Class org.apache.hadoop.hdfs.MiniDFSCluster not found - continuing with a stub.
[warn] one warning found
[success] Total time: 3 s, completed Feb 15, 2017 12:28:52 PM

Running the test is failing because of a NoClassDefFoundError

sbt updateClassifiers

(...some update on testing jar...)

sbt test
[info] Loading project definition from /home/sam/IdeaProjects/test-miniclusters/project
[info] Set current project to test-miniclusters (in build file:/home/sam/IdeaProjects/test-miniclusters/)
2017-02-15 13:38:28 INFO  HdfsLocalCluster:171 - HDFS: Starting MiniDfsCluster
[info] TestHdfsUtility:
[info] This test
[info] com.affinytix.util.test.TestHdfsUtility *** ABORTED ***
[info]   java.lang.NoClassDefFoundError: org/apache/hadoop/hdfs/MiniDFSCluster$Builder

Furthermore adding the following dependency:

libraryDependencies += "org.apache.hadoop" % "hadoop-hdfs" % "2.7.3.2.5.3.0-37" % Test classifier "tests"

... just move the problem to anther missing dependency located in a "test-jar"

expectation

In the sbt documentation this can be read:

To obtain particular classifiers for all dependencies transitively, run the updateClassifiers task. By default, this resolves all artifacts with the sources or javadoc classifier. Select the classifiers to obtain by configuring the transitiveClassifiers setting. For example, to only retrieve sources: transitiveClassifiers := Seq("sources")

So I expected (wrongly maybe) that

transitiveClassifiers := Seq(Artifact.TestsClassifier, Artifact.SourceClassifier)

coupled with the updateClassifiers task would add all the test jar to my dependency. But this is not the case.

notes

Maybe this case should be more documented...

[izhangzhihao]

Same issue here, it would be great if this issue will been fixed in next release!
And for @samuelsayag any workaround on that? Thanks!

[michaelahlers]

Been hobbled by this before, and it continues to require awkward project structures where published test artifacts are desired.

[eed3si9n]

I think part of the issue is that

1. most people use Maven repository to publish their artifacts.
2. Maven doesn't have a notion of Ivy configuration.
3. most people do not publish their test JAR as artifacts, so the POM-translated Test configuration is private.

	<configurations>
		<conf name="default" visibility="public" description="runtime dependencies and master artifact can be used with this conf" extends="runtime,master"/>
		<conf name="master" visibility="public" description="contains only the artifact published by this module itself, with no transitive dependencies"/>
		<conf name="compile" visibility="public" description="this is the default scope, used if none is specified. Compile dependencies are available in all classpaths."/>
		<conf name="provided" visibility="public" description="this is much like compile, but indicates you expect the JDK or a container to provide it. It is only available on the compilation classpath, and is not transitive."/>
		<conf name="runtime" visibility="public" description="this scope indicates that the dependency is not required for compilation, but is for execution. It is in the runtime and test classpaths, but not the compile classpath." extends="compile"/>
		<conf name="test" visibility="private" description="this scope indicates that the dependency is not required for normal use of the application, and is only available for the test compilation and execution phases." extends="runtime"/>
		<conf name="system" visibility="public" description="this scope is similar to provided except that you have to provide the JAR which contains it explicitly. The artifact is always available and is not looked up in a repository."/>
		<conf name="sources" visibility="public" description="this configuration contains the source artifact of this module, if any."/>
		<conf name="javadoc" visibility="public" description="this configuration contains the javadoc artifact of this module, if any."/>
		<conf name="optional" visibility="public" description="contains all optional dependencies"/>
	</configurations>

Using classifier to encode this I don't think is good idea either.

Basically, if people want to reuse tests, and still want to use Maven repositories, the module should be in Compile configuration.

[skestle]

Uh, so the recommended path to reusing tests (@eed3si9n) is to ship tests in compile of downstream modules because they need some main code?

Perhaps I'm just too lazy, but I've always had code and relevant test supports shipped from the same module (but am now trying to migrate sbt to maven publishing...)

[eed3si9n]

For clarification, I don't mean that people should include tests directly in some library, but split out the reusable parts and call it "foo-testkit".