Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Help wanted to implement combined download+verify task #108

Closed
liblit opened this issue Mar 4, 2018 · 2 comments
Closed

Help wanted to implement combined download+verify task #108

liblit opened this issue Mar 4, 2018 · 2 comments

Comments

@liblit
Copy link

liblit commented Mar 4, 2018

Using Gradle 4.6 and gradle-download-task 3.4.0, the vast majority of my downloads have the same general pattern: download a given URL to a given file, then verify that download using a given checksum. I would like to refactor this recurring pattern into a custom task, but I don‘t know how. This is really more of a support question than a bug report, but I hope that you will lend me some wisdom so that I can use gradle-download-task more awesomely.

The following is a representative example of a typical download task:

task downloadJLex {
  outputs.file 'src/JLex/Main.java'  // varies
  doLast {
    download {
      src 'http://www.cs.princeton.edu/~appel/modern/java/JLex/current/Main.java'  // varies
      dest outputs.files.singleFile
      overwrite true
    }
    verifyChecksum {
      src outputs.files.singleFile
      checksum 'fe0cff5db3e2f0f5d67a153cf6c783af'  // varies
    }
  }
}

Other than the task name, three // varies comments above mark the only parts of this pattern vary throughout my project:

  1. The output file into which the URL should be saved.
  2. The URL to download.
  3. The checksum used to verify the download.

My partial, non-working effort to create a custom class consists of the following Gradle code added directly to a subproject's build.gradle:

class VerifiedDownload extends org.gradle.api.DefaultTask {

  @Input String src
  @Input String checksum
  @OutputFile File dest

  @TaskAction
  downloadAndVerify() {
    download {
      src this.src
      dest this.dest
      overwrite true
      onlyIfModified true
    }
    verifyChecksum {
      src this.dest
      checksum this.checksum
    }
  }
}

Unfortunately, my use of download and verifyChecksum is wrong. If I try to instantiate my custom task...

task downloadJLex(type: VerifiedDownload) {
  src 'http://www.cs.princeton.edu/~appel/modern/java/JLex/current/Main.java'
  checksum 'fe0cff5db3e2f0f5d67a153cf6c783af'
  dest file('src/JLex/Main.java')
}

... then Gradle fails, reporting Could not find method download() for arguments [VerifiedDownload$_downloadAndVerify_closure1@49958194] on task ':com.ibm.wala.cast.java.test.data:downloadJLex' of type VerifiedDownload.

I suspect that my use of this.src, this.dest, and this.checksum in the VerifiedDownload constructor may be bogus as well. I am not sure how to access these fields of the current VerifiedDownload task instance versus invoking the src, dest, and checksum methods that would be available inside the download and verifyChecksum if I were using those blocks correctly.

I hope that my intention is clear from the above examples. I would be grateful for any hints as to the right way to turn that intention into working code.

liblit added a commit to liblit/WALA that referenced this issue Mar 5, 2018
We now download and verify checksums as a single task, rather than as
two separate tasks.  This simplifies other task dependencies, since we
no longer have a checksum-verified "stamp" file separate from the
download itself.  Unfortunately the combined task now has a
significant amount of repeated boilerplate.  I'm hoping to refactor
that all out into a custom task class, but haven't yet figured out the
details:
<michel-kraemer/gradle-download-task#108>.

We now also use ETags to be smarter about when a fresh download is or
is not actually needed.  I think there are still opportunities for
improved caching here, but this is a step in the right direction.
@michel-kraemer
Copy link
Owner

Please try project.download and project.verifyChecksum instead of download and verifyChecksum.

@liblit
Copy link
Author

liblit commented Mar 6, 2018

project.download and project.verifyChecksum worked perfectly. Thank you, @michel-kraemer!

@liblit liblit closed this as completed Mar 6, 2018
liblit added a commit to liblit/WALA that referenced this issue Mar 18, 2018
We now download and verify checksums as a single task, rather than as
two separate tasks.  This simplifies other task dependencies, since we
no longer have a checksum-verified "stamp" file separate from the
download itself.  Unfortunately the combined task now has a
significant amount of repeated boilerplate.  I'm hoping to refactor
that all out into a custom task class, but haven't yet figured out the
details:
<michel-kraemer/gradle-download-task#108>.

We now also use ETags to be smarter about when a fresh download is or
is not actually needed.  I think there are still opportunities for
improved caching here, but this is a step in the right direction.
liblit added a commit to liblit/WALA that referenced this issue Apr 17, 2018
We now download and verify checksums as a single task, rather than as
two separate tasks.  This simplifies other task dependencies, since we
no longer have a checksum-verified "stamp" file separate from the
download itself.  Unfortunately the combined task now has a
significant amount of repeated boilerplate.  I'm hoping to refactor
that all out into a custom task class, but haven't yet figured out the
details:
<michel-kraemer/gradle-download-task#108>.

We now also use ETags to be smarter about when a fresh download is or
is not actually needed.  I think there are still opportunities for
improved caching here, but this is a step in the right direction.
liblit added a commit to liblit/WALA that referenced this issue Apr 18, 2018
We now download and verify checksums as a single task, rather than as
two separate tasks.  This simplifies other task dependencies, since we
no longer have a checksum-verified "stamp" file separate from the
download itself.  Unfortunately the combined task now has a
significant amount of repeated boilerplate.  I'm hoping to refactor
that all out into a custom task class, but haven't yet figured out the
details:
<michel-kraemer/gradle-download-task#108>.

We now also use ETags to be smarter about when a fresh download is or
is not actually needed.  I think there are still opportunities for
improved caching here, but this is a step in the right direction.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants