New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Gradle dependencies available offline #131

Merged
merged 46 commits into from Jan 18, 2018

Conversation

Projects
None yet
4 participants
@rjust
Owner

rjust commented Dec 9, 2017

@Greg4cr and @jose I tried to address the "Gradle is downloaded on demand" issue for Mockito. This is an issue for users with a slow internet connection and generally slows down the Mockito builds. Even worse, though, it doesn't allow an offline use of the Mockito defects.

This pull request makes two changes:

  1. The init.sh scripts downloads all Gradle versions that are used by any of the commits in the VCS history to a library directory in Defects4J.

  2. The post-checkout hook for Mockito modifies the gradle-wrapper.properties file of a checked-out project version and sets the download URL to the proper file in the library directory created by the init script.

The drawback is that the init.sh script in step 1 needs to download ~600MB of gradle distributions, which will take quite some time. Rather than downloading the distributions one by one from the Gradle servers, we could provide an archive with all of the distributions (similar to defects4j-repos.zip) and have the init script download this large archive instead -- still ~600MB.

Alternatively, we could try to build all Mockito versions with the newest version of Gradle used in the VCS history and always point the properties file to this locally available distribution.

What do you guys think?

@rjust rjust requested review from jose and Greg4cr Dec 9, 2017

@jose

This comment has been minimized.

Collaborator

jose commented Dec 10, 2017

Hi @rjust,

PR looks goods to me. Do you know how many different Gradle versions are required? Either only a few or 38 versions (the number of Mockito bugs), I would be in favour of first trying to build all Mockito bugs "with the newest version of Gradle used in the VCS history" (as you describe as an alternative). It might be the case that only one instance of Gradle is needed.

@Greg4cr

This comment has been minimized.

Collaborator

Greg4cr commented Dec 10, 2017

@rjust

This comment has been minimized.

Owner

rjust commented Dec 10, 2017

The pull request includes a little script that extracts all used Gradle versions from a VCS history, which is handy for bug-mining purposes. The number of Gradle vesions for the entire Mockito history is 16 (all between version 1.0-milestone-5 and version 2.4).

We could try to build all 38 Mockito bugs with Gradle version 2.4 and, if successful, only download this dependency in the init script. This would keep the size of the dependency archive small. The downside is that this solution makes future bug mining a bit more complicated -- a user would need to determine a proper version when mining bugs from a Gradle project.

I'll go ahead and try to build all Mockito bugs with Gradle 2.4.

@rjust

This comment has been minimized.

Owner

rjust commented Dec 10, 2017

Thinking about the bug-mining scenario further, the bug-mining infrastructure could temporarily download all Gradle dependencies to simplify the process for a user. Then, once a set of bugs is ready to be promoted to the main database, a final sanity check determines how many Gradle versions are necessary to build the entire set of bugs.

@Greg4cr

This comment has been minimized.

Collaborator

Greg4cr commented Dec 10, 2017

@rjust

This comment has been minimized.

Owner

rjust commented Dec 11, 2017

If we download all Gradle versions that are used by any of the Mockito defects (15 distinct versions) in the init.sh script, then the installation time increases because of the additional 600MB download.

While this may take a while with a slow connection, it's probably still better to wait once rather than waiting several minutes every time a Mockito defect is used.

We can still explore options to shrink the size of the install, but the benefits of making Gradle available offline seem to outweigh the drawbacks. Also, this PR should address the issue of resource leaks when batch-processing Mockito defects.

@Greg4cr

This comment has been minimized.

Collaborator

Greg4cr commented Dec 11, 2017

Agreed 100%. I wasn't arguing against offline distribution of Gradle. One 600 MB download is worth performing to remove the need for multiple downloads in the future - especially when running experiments. I just wasn't sure if we should aggressively pursue shrinking the install size, or if that would just introduce headaches. :)

@rjust

This comment has been minimized.

Owner

rjust commented Dec 11, 2017

OK, let's download all dependencies for now and explore options to identify a sufficient subset.

@jose

This comment has been minimized.

Collaborator

jose commented Dec 11, 2017

Ok.

rjust added some commits Dec 11, 2017

@rjust rjust merged commit 107ea67 into master Jan 18, 2018

0 of 2 checks passed

continuous-integration/travis-ci/pr The Travis CI build could not complete due to an error
Details
continuous-integration/travis-ci/push The Travis CI build is in progress
Details

@rjust rjust deleted the offline-gradle branch Jan 18, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment