Custom dependecy configuration

[David-N-Perkins]

I'm trying to use Scoverage with a Flink job project. Flink jobs are packaged into a 'Fat' or 'Shadow' jar using the com.github.johnrengelman.shadow plugin, but several core Flink libraries needed for compiling and running the code aren't supposed to be included.

So the standard way to do this is to create a separate dependency configuration for the shadow jar. But when I do this, Scoverage doesn't see those dependencies and fails to compile the code.

I've got a more complete write up on Stack Overflow: https://stackoverflow.com/questions/61686702/how-to-use-scoverage-with-flink-shadowjar-custom-gradle-dependency-configuration

[eyalroth]

It seems you're missing a dependencies declaration:

dependencies {
    compile "org.apache.flink:flink-java:${flinkVersion}"
    compile "org.apache.flink:flink-streaming-java_${scalaBinaryVersion}:${flinkVersion}"
}

As in the documentation for Flink: https://ci.apache.org/projects/flink/flink-docs-stable/dev/projectsetup/java_api_quickstart.html

[David-N-Perkins]

I do have those. I didn't post the entire build.gradle, because it's a multi-build project that's complicated. Bellow is a more complete excerpt of the dependencies.

The issue is that the flinkShadowJar dependencies are not getting included in the Scoverage compilation. There's a section that adds flinkShadowJar dependencies to the compile and runtime classpath, but I don't think Scoverage is picking this up.

configurations {
    flinkShadowJar // dependencies which go into the shadowJar

    // always exclude these (also from transitive dependencies) since they are provided by Flink
    flinkShadowJar.exclude group: 'org.apache.flink', module: 'force-shading'
    flinkShadowJar.exclude group: 'com.google.code.findbugs', module: 'jsr305'
    flinkShadowJar.exclude group: 'org.slf4j'
    flinkShadowJar.exclude group: 'log4j'
}

// declare the dependencies for your production and test code
dependencies {
    // Scala lib
    implementation "org.scala-lang:scala-library:${scalaVersion}"

    implementation "org.apache.flink:flink-scala_${scalaBinaryVersion}:${flinkVersion}"
    implementation "org.apache.flink:flink-streaming-scala_${scalaBinaryVersion}:${flinkVersion}"
    implementation "log4j:log4j:${log4jVersion}"
    implementation "org.slf4j:slf4j-log4j12:${slf4jVersion}"

    // --------------------------------------------------------------
    // Dependencies that should be part of the shadow jar, e.g.
    // connectors. These must be in the flinkShadowJar configuration!
    // --------------------------------------------------------------
    flinkShadowJar project(':validator')
    flinkShadowJar "info.picocli:picocli:4.2.0"

    // Add test dependencies here.
    testImplementation "org.scalactic:scalactic_${scalaBinaryVersion}:${scalaTestVersion}"
    testImplementation "org.scalatest:scalatest_${scalaBinaryVersion}:${scalaTestVersion}"
    testImplementation "org.scalamock:scalamock_${scalaBinaryVersion}:4.4.0"
    testImplementation "com.github.stefanbirkner:system-rules:1.19.0"

    testRuntimeOnly "com.vladsch.flexmark:flexmark-all:0.35.10"
}

// make compileOnly dependencies available for tests:
sourceSets {
    main.compileClasspath += configurations.flinkShadowJar
    main.runtimeClasspath += configurations.flinkShadowJar

    test.compileClasspath += configurations.flinkShadowJar
    test.runtimeClasspath += configurations.flinkShadowJar

    javadoc.classpath += configurations.flinkShadowJar
}

[eyalroth]

Looking again at Flink's quickstart example, I don't rightly understand why they configured it that way. It seems you'd want all the dependencies in the compilation, but exclude some from the shadowed jar:

dependencies {
    // Scala lib
    implementation "org.scala-lang:scala-library:${scalaVersion}"

    implementation "org.apache.flink:flink-scala_${scalaBinaryVersion}:${flinkVersion}"
    implementation "org.apache.flink:flink-streaming-scala_${scalaBinaryVersion}:${flinkVersion}"
    implementation "log4j:log4j:${log4jVersion}"
    implementation "org.slf4j:slf4j-log4j12:${slf4jVersion}"
    
    implementation project(':validator')
    implementation "info.picocli:picocli:4.2.0"

    // Add test dependencies here.
    testImplementation "org.scalactic:scalactic_${scalaBinaryVersion}:${scalaTestVersion}"
    testImplementation "org.scalatest:scalatest_${scalaBinaryVersion}:${scalaTestVersion}"
    testImplementation "org.scalamock:scalamock_${scalaBinaryVersion}:4.4.0"
    testImplementation "com.github.stefanbirkner:system-rules:1.19.0"

    testRuntimeOnly "com.vladsch.flexmark:flexmark-all:0.35.10"
}

shadowJar {
   dependencies {
      exclude(dependency('org.apache.flink:force-shading:'))
      exclude(dependency('com.google.code.findbugs:jsr305:'))
      exclude(dependency('org.slf4j:'))
      exclude(dependency('log4j:'))
   }
}

// no need for that part
// sourceSets {}

[David-N-Perkins]

It's because the shadowJar is run on a Flink cluster, which automatically provides the Flink implementation runtime libraries and logging. Several big data processing frameworks work this way. Map Reduce, Spark, etc. Oh wait, I miss read your post.

I think they use the separate configuration, because there's a bug in the shadowJar plugin that won't exclude transitive dependencies. They have a comment about that in their example.

// NOTE: We cannot use "compileOnly" or "shadow" configurations since then we could not run code
// in the IDE or with "gradle run". We also cannot exclude transitive dependencies from the
// shadowJar yet (see GradleUp/shadow#159).
// -> Explicitly define the // libraries we want to be included in the "flinkShadowJar" configuration!

[maiflai]

Could you provide the output of a build with --debug enabled? It might help to show the actual classpaths that are being used.

Where exactly is the missing class expected to be found please?

[eyalroth]

Then how about something like this?

configurations {
    flinkShadowJar

    flinkShadowJar.exclude group: 'org.apache.flink', module: 'force-shading'
    flinkShadowJar.exclude group: 'com.google.code.findbugs', module: 'jsr305'
    flinkShadowJar.exclude group: 'org.slf4j'
    flinkShadowJar.exclude group: 'log4j'
}

dependencies {
    // Scala lib
    implementation "org.scala-lang:scala-library:${scalaVersion}"

    implementation "org.apache.flink:flink-scala_${scalaBinaryVersion}:${flinkVersion}"
    implementation "org.apache.flink:flink-streaming-scala_${scalaBinaryVersion}:${flinkVersion}"
    implementation "log4j:log4j:${log4jVersion}"
    implementation "org.slf4j:slf4j-log4j12:${slf4jVersion}"

    // Add test dependencies here.
    testImplementation "org.scalactic:scalactic_${scalaBinaryVersion}:${scalaTestVersion}"
    testImplementation "org.scalatest:scalatest_${scalaBinaryVersion}:${scalaTestVersion}"
    testImplementation "org.scalamock:scalamock_${scalaBinaryVersion}:4.4.0"
    testImplementation "com.github.stefanbirkner:system-rules:1.19.0"

    testRuntimeOnly "com.vladsch.flexmark:flexmark-all:0.35.10"
}

def configureFlink(configuration) = {
    dependencies {
        configuration project(':validator')
        configuration "info.picocli:picocli:4.2.0"
    }
}

configureFlink(implementation)
configureFlink(flinkShadowJar)

// no need for that part
// sourceSets {}

shadowJar {
    configurations = [project.configurations.flinkShadowJar]
}

I'm not sure this will get the same result, but it seems to me that this is the more correct way of configuring the project. If that doesn't work, you could try making changes to the scoverage source sets:

sourceSets {
    main.compileClasspath += configurations.flinkShadowJar
    main.runtimeClasspath += configurations.flinkShadowJar

    test.compileClasspath += configurations.flinkShadowJar
    test.runtimeClasspath += configurations.flinkShadowJar

    // add this
    scoverage.compileClasspath += configurations.flinkShadowJar
    scoverage.runtimeClasspath += configurations.flinkShadowJar

    javadoc.classpath += configurations.flinkShadowJar
}

[David-N-Perkins]

Your second suggestion worked. Thank you!