Dependencies not downloaded until ktlint task is run with Gradle

[Drarok]

I'm using KTLint for an Android app, on CircleCI, and I don't want to download the binary files every time. I'm successfully downloading and caching deps that are required for building the app, but when the ktlint task runs, Gradle downloads more dependencies.

Is there a way I can force Gradle to download them up-front, and therefore get cached by the CI configuration?

[shyiko]

Hi @Drarok. Any chance you can share your build.gradle? The only case I can think of where ktlint would download anything is when using external 3rd party ruleset (-R flag). But even then caching ~/.m2 should work just fine.

FYI: You can use ktlint as a standalone binary (see https://ktlint.github.io/#getting-started) without the need for Maven or Gradle.

[Drarok]

I may have put things into the wrong gradle file, so here's both! Thanks for looking into this.

Project build.gradle:

// Top-level build file where you can add configuration options common to all sub-projects/modules.

buildscript {
    ext.kotlin_version = '1.1.2-3'
    repositories {
        jcenter()
        mavenCentral()
    }

    dependencies {
        classpath 'com.android.tools.build:gradle:2.3.2'
        classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"

        // NOTE: Do not place your application dependencies here; they belong
        // in the individual module build.gradle files

        // The exception to this rule is compile-time dependencies, such as these:
        classpath 'de.felixschulze.gradle:gradle-hockeyapp-plugin:3.5'
        classpath 'com.github.shyiko:ktlint:0.6.1'
    }
}

allprojects {
    repositories {
        jcenter()
    }
}

task clean(type: Delete) {
    delete rootProject.buildDir
}

App build.gradle:

apply plugin: 'com.android.application'
apply plugin: 'kotlin-android'
apply plugin: 'de.felixschulze.gradle.hockeyapp'

android {
    compileSdkVersion 25
    buildToolsVersion "25.0.0"
    defaultConfig {
        applicationId "<REDACTED>"
        minSdkVersion 19
        targetSdkVersion 25
        versionCode 1
        versionName "1.0"
        testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
    }

    buildTypes {
        release {
            minifyEnabled false
            proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
        }
    }
}

configurations {
    ktlint
}

repositories {
    mavenCentral()
}

dependencies {
    compile fileTree(dir: 'libs', include: ['*.jar'])
    androidTestCompile('com.android.support.test.espresso:espresso-core:2.2.2', {
        exclude group: 'com.android.support', module: 'support-annotations'
    })
    compile "org.jetbrains.kotlin:kotlin-stdlib-jre7:$kotlin_version"
    compile 'com.android.support:appcompat-v7:25.3.1'
    compile 'com.android.support:recyclerview-v7:25.3.1'
    compile 'com.android.support:design:25.3.1'
    compile 'com.android.support:cardview-v7:25.3.1'
    testCompile 'junit:junit:4.12'
    ktlint 'com.github.shyiko:ktlint:0.6.1'
}

task ktlint(type: JavaExec) {
    main = "com.github.shyiko.ktlint.Main"
    classpath = configurations.ktlint
    args "src/**/*.kt"
}

check.dependsOn ktlint

task ktlintFormat(type: JavaExec) {
    main = "com.github.shyiko.ktlint.Main"
    classpath = configurations.ktlint
    args "-F", "src/**/*.kt"
}

hockeyapp {
    apiToken = "<REDACTED>"
}

[shyiko]

@Drarok Just to clarify: you are caching both ~/.gradle and ~/.m2?

[jeremymailen]

@Drarok what gradle target do you currently run in the dependencies section of your circle.yml to pull things to cache? Gradle of course is notoriously lazy/efficient about getting dependencies and of course CircleCI only caches after the dependencies step which is usually ahead of most people's gradle targets that do interesting things.

[Drarok]

I've just double checked, both are cached, yes.

Run: ./gradlew dependencies

Cache: Creating dependency cache from the following paths:
/home/ubuntu/.go_workspace
/home/ubuntu/.gradle
/home/ubuntu/.m2

Run: ./gradlew ktlint

Output (note the downloads from

> Connecting to Daemon> Configuring > 0/2 projects > root project > Resolving dependencies ':classpat> Configuring > 1/2 projects > :appNDK is missing a "platforms" directory.
If you are using NDK, verify the ndk.dir is set to a valid NDK directory.  It is currently set to /usr/local/android-sdk-linux/ndk-bundle.
If you are not using NDK, unset the NDK variable from ANDROID_NDK_HOME or local.properties to remove this warning.

> Configuring > 2/2 projects > Resolving dependencies ':app:_debugApk'> Configuring > 2/2 projectsIncremental java compilation is an incubating feature.
> Configuring > 2/2 projects > Resolving dependencies ':app:_releaseUnitTestApk> Configuring > 2/2 projects > Resolving dependencies ':app:_debugAndroidTestAp> Configuring > 2/2 projects> Building 0% > :app:ktlint > Resolving dependencies ':app:ktlint'> Building 0% > :app:ktlint:app:ktlint
Download https://jcenter.bintray.com/org/jetbrains/kotlin/kotlin-compiler-embeddable/1.1.0/kotlin-compiler-embeddable-1.1.0.jar
> Building 0% > :app:ktlint> Building 0% > :app:ktlint > 833 KB/25.88 MB downloaded> Building 0% > :app:ktlint > 1.87 MB/25.88 MB downloaded> Building 0% > :app:ktlint > 3.06 MB/25.88 MB downloaded> Building 0% > :app:ktlint > 4.50 MB/25.88 MB downloaded> Building 0% > :app:ktlint > 6.25 MB/25.88 MB downloaded> Building 0% > :app:ktlint > 7.14 MB/25.88 MB downloaded> Building 0% > :app:ktlint > 8.92 MB/25.88 MB downloaded> Building 0% > :app:ktlint > 9.64 MB/25.88 MB downloaded> Building 0% > :app:ktlint > 10.87 MB/25.88 MB downloaded> Building 0% > :app:ktlint > 11.70 MB/25.88 MB downloaded> Building 0% > :app:ktlint > 12.78 MB/25.88 MB downloaded> Building 0% > :app:ktlint > 13.99 MB/25.88 MB downloaded> Building 0% > :app:ktlint > 15.26 MB/25.88 MB downloaded> Building 0% > :app:ktlint > 15.92 MB/25.88 MB downloaded> Building 0% > :app:ktlint > 16.50 MB/25.88 MB downloaded> Building 0% > :app:ktlint > 17.23 MB/25.88 MB downloaded> Building 0% > :app:ktlint > 17.77 MB/25.88 MB downloaded> Building 0% > :app:ktlint > 18.54 MB/25.88 MB downloaded> Building 0% > :app:ktlint > 20.03 MB/25.88 MB downloaded> Building 0% > :app:ktlint > 21.31 MB/25.88 MB downloaded> Building 0% > :app:ktlint > 23.57 MB/25.88 MB downloaded> Building 0% > :app:ktlint > 25.88 MB/25.88 MB downloaded
> Building 0% > :app:ktlintDownload https://jcenter.bintray.com/org/apache/httpcomponents/httpclient/4.3.5/httpclient-4.3.5.jar
> Building 0% > :app:ktlint
Download https://jcenter.bintray.com/org/apache/httpcomponents/httpcore/4.3.2/httpcore-4.3.2.jar
> Building 0% > :app:ktlintDownload https://jcenter.bintray.com/commons-codec/commons-codec/1.6/commons-codec-1.6.jar
> Building 0% > :app:ktlint> Building 0%
BUILD SUCCESSFUL

[jeremymailen]

It looks like it only triggers the download when the configurations ktlint is resolved. I don't think the classpath 'com.github.shyiko:ktlint:0.6.1' in the buildscript block is triggering a dependency download since it isn't part of an applied plugin? If you remove it I think you'll find your ktlint task still works 🤔.

[jeremymailen]

Well on 2nd take it looks like having classpath 'com.github.shyiko:ktlint:0.6.1' in buildscript does trigger a download of some dependencies on gradle dependencies but not a few of the transitive ones like kotlin-compiler-embeddable.

[Drarok]

Hmm, so am I stuck having it download its transitive dependencies, or can I force Gradle to download those somehow?

[shyiko]

Sorry @Drarok, got distracted at work. I'll try to provide an answer within 24h.

[Drarok]

Honestly, don't rush! I'm not back in work where this is is only a minor inconvenience until Tuesday anyway.

Have a good weekend! 😄

[shyiko]

Alright, I was able to reproduce aforementioned behavior on gradle@3.5 (worked fine on gradle 2.4.1).
Solution is described at https://discuss.gradle.org/t/download-all-dependencies/6294/4, which boils down to:

1. add to build.gradle

allprojects {
    task dependenciesResolve {
        doLast {
            configurations.all { it.resolve() }
        }
    }
}

2. instead of gradle dependencies use gradle dependenciesResolve.

Tested on chibatching/docker-android-sdk docker image.

🎈

[shyiko]

@Drarok have a great weekend too ;)

[shyiko]

@jeremymailen thank you for trying to help, I appreciate it 🙇‍♂️

[Drarok]

Finally got around to implementing this change, and my latest build didn't download anything extra, so everything's being cached as expected! 🎉

Thanks to everyone that helped.