Assets 3

Added a deadline for downloading any part of a file. If a part cannot be downloaded by this time, the program will attempt to restart downloading the whole part. This should help improve issues if downloading a part hangs for any reason. This version is backwards compatible with previous versions so you should be able to swap it out for an ongoing download.

MD5sum of tarball: 1ca28c56bcc992e6f052d304bf4264b4

@geetduggal geetduggal released this Dec 20, 2018 · 3 commits to master since this release

Assets 3

This release attempts to fix an error caused by a concurrent read and write to a map of download URLs. It is compatible with previous versions so you may swap out an older version of the download agent with this one and restart downloads with no problem.

MD5sum of tarball: ebe066ac5e1447d18c546ef1142ed8e3

@geetduggal geetduggal released this Dec 19, 2018 · 6 commits to master since this release

Assets 3

Resolved an issue where a nil pointer exception tends to appear. This will result in fewer attempts to restart downloading a part for long downloads and cleaner logs. This version is compatible with previous versions of the download agent so you are welcome to interrupt a download and use this executable instead for an improved experience.

MD5sum for this tarball: d00b61803747735029b1a599d31fa59d

@geetduggal geetduggal released this Dec 19, 2018 · 7 commits to master since this release

Assets 3

This release is more graceful when retrying downloads when the entire internet connection is down for minutes or more. The download agent is designed to restart where it left off if the user terminates the connection. If there are any errors or issues you notice, feel free to terminate and restart the process and you should be up and running again.

@geetduggal geetduggal released this Dec 12, 2018 · 12 commits to master since this release

Assets 3

When downloading with many threads, log output for intermittent network failures can be overwhelming. This release contains a modification to prevent many threads from complaining about an intermittent failure simultaneously.

This release also contains a fix to 'ls_manifest' where a '/' was missing between the folder path and file name for subdirectories in a project.

MD5sum for tarball: 8f68f0d4b61ae2ec3e534dfcd5aabf4e

@geetduggal geetduggal released this Dec 12, 2018 · 16 commits to master since this release

Assets 3

Changed the output message to more accurately show "MB/s estimated ver the last 60s".

MD5sum of tarball for this release: e815735b399ecd88c697c84e0234636e

@geetduggal geetduggal released this Dec 11, 2018 · 21 commits to master since this release

Assets 3

Output is consistently using 1024 KB in a MB, etc for the moment. This may change to IEC in the future.
Carriage returns are likely not to interfere with each other.

MD5sum of tarball: 77810eac20730b953660de931f3fa5d1

Note: The output when reporting net throughput written to disk should read something like "XXX MB/s written to disk estimated over the last 60s" as opposed to "MB" (no "per second") as in this release. This will be fixed in the next incremental release.

Assets 3

This release cleans up output of the download agent by placing detailed messages and errors in a log file of the form <manifest-file-name>.download.log. The idea behind this file is that if an error occurs the user can consult this as a historical log (it includes timestamps) and can forward it to DNAnexus support for further debugging. In the future we plan to include more detailed information about the client's networking setup to help further debug issues.

@geetduggal geetduggal released this Nov 28, 2018 · 36 commits to master since this release

Assets 3
  • Obtain a mutex to query database for progress updates (in the future this will be managed with a single connection)
  • Graceful retries of downloading parts are now done perpetually to deal with intermittent cases where retryablehttp obtains a nil pointer (we are investigating why the pointer is appearing nil in this library)
  • Summary output has been cleaned up to be clearer about net throughput in a particular time period and updated every time a part is downloaded

@geetduggal geetduggal released this Nov 26, 2018 · 47 commits to master since this release

Assets 3

This release adds support to perpetually retry on 503s. The wait time to retry uses the retry-after header when available. Otherwise, it exponentially backs off up to 30 minutes in the absence of the retry-after header.