Make Gradle dependencies available offline

[rjust]

@Greg4cr and @jose I tried to address the "Gradle is downloaded on demand" issue for Mockito. This is an issue for users with a slow internet connection and generally slows down the Mockito builds. Even worse, though, it doesn't allow an offline use of the Mockito defects.

This pull request makes two changes:

1. The init.sh scripts downloads all Gradle versions that are used by any of the commits in the VCS history to a library directory in Defects4J.

2. The post-checkout hook for Mockito modifies the gradle-wrapper.properties file of a checked-out project version and sets the download URL to the proper file in the library directory created by the init script.

The drawback is that the init.sh script in step 1 needs to download ~600MB of gradle distributions, which will take quite some time. Rather than downloading the distributions one by one from the Gradle servers, we could provide an archive with all of the distributions (similar to defects4j-repos.zip) and have the init script download this large archive instead -- still ~600MB.

Alternatively, we could try to build all Mockito versions with the newest version of Gradle used in the VCS history and always point the properties file to this locally available distribution.

What do you guys think?

[jose]

Hi @rjust,

PR looks goods to me. Do you know how many different Gradle versions are required? Either only a few or 38 versions (the number of Mockito bugs), I would be in favour of first trying to build all Mockito bugs "with the newest version of Gradle used in the VCS history" (as you describe as an alternative). It might be the case that only one instance of Gradle is needed.

[Greg4cr]

I agree with @jose. PR looks good, and the idea is clever. However, it would be nice to just be able to use the minimal number of Gradle versions.

…

On Dec 10, 2017 7:06 AM, "José Campos" ***@***.***> wrote: Hi @rjust <https://github.com/rjust>, PR looks goods to me. Do you know how many different Gradle versions are required? Either only a few or 38 versions (the number of Mockito bugs), I would be in favour of first trying to build all Mockito bugs "with the newest version of Gradle used in the VCS history" (as you describe as an alternative). It might be the case that only one instance of Gradle is needed. — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#131 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB0-SshKk2NShgZgrnj6huXrbq0ISE0uks5s-8ksgaJpZM4Q78EJ> .

[rjust]

The pull request includes a little script that extracts all used Gradle versions from a VCS history, which is handy for bug-mining purposes. The number of Gradle vesions for the entire Mockito history is 16 (all between version 1.0-milestone-5 and version 2.4).

We could try to build all 38 Mockito bugs with Gradle version 2.4 and, if successful, only download this dependency in the init script. This would keep the size of the dependency archive small. The downside is that this solution makes future bug mining a bit more complicated -- a user would need to determine a proper version when mining bugs from a Gradle project.

I'll go ahead and try to build all Mockito bugs with Gradle 2.4.

[rjust]

Thinking about the bug-mining scenario further, the bug-mining infrastructure could temporarily download all Gradle dependencies to simplify the process for a user. Then, once a set of bugs is ready to be promoted to the main database, a final sanity check determines how many Gradle versions are necessary to build the entire set of bugs.

[Greg4cr]

Good point. I like your proposed solution, especially if we add further gradle-using projects. You could download all required versions, then have an optional minimize_gradle_versions script for those who want to shrink the size of the install.

…

On Dec 10, 2017 8:45 AM, "René Just" ***@***.***> wrote: Thinking about the bug-mining scenario further, the bug-mining infrastructure could temporarily download all Gradle dependencies to simplify the process for a user. Then, once a set of bugs is ready to be promoted to the main database, a final sanity check determines how many Gradle versions are necessary to build the entire set of bugs. — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#131 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB0-SmfzlNen7VbYlv2zpjlzLKRwMTHVks5s--B8gaJpZM4Q78EJ> .

[Greg4cr]

Thanks for looking into this, @rjust.

Despite my earlier comments, and given Gradle's complexity, we may be better off just distributing all Gradle versions used for each fault rather than minimizing, particularly for future mined faults.

In any case, it's interesting to see the results.

[rjust]

If we download all Gradle versions that are used by any of the Mockito defects (15 distinct versions) in the init.sh script, then the installation time increases because of the additional 600MB download.

While this may take a while with a slow connection, it's probably still better to wait once rather than waiting several minutes every time a Mockito defect is used.

We can still explore options to shrink the size of the install, but the benefits of making Gradle available offline seem to outweigh the drawbacks. Also, this PR should address the issue of resource leaks when batch-processing Mockito defects.

[Greg4cr]

Agreed 100%. I wasn't arguing against offline distribution of Gradle. One 600 MB download is worth performing to remove the need for multiple downloads in the future - especially when running experiments. I just wasn't sure if we should aggressively pursue shrinking the install size, or if that would just introduce headaches. :)

[rjust]

OK, let's download all dependencies for now and explore options to identify a sufficient subset.

[jose]

Ok.