Added a deadline for downloading any part of a file. If a part cannot be downloaded by this time, the program will attempt to restart downloading the whole part. This should help improve issues if downloading a part hangs for any reason. This version is backwards compatible with previous versions so you should be able to swap it out for an ongoing download.
MD5sum of tarball: 1ca28c56bcc992e6f052d304bf4264b4
This release attempts to fix an error caused by a concurrent read and write to a map of download URLs. It is compatible with previous versions so you may swap out an older version of the download agent with this one and restart downloads with no problem.
MD5sum of tarball: ebe066ac5e1447d18c546ef1142ed8e3
Resolved an issue where a nil pointer exception tends to appear. This will result in fewer attempts to restart downloading a part for long downloads and cleaner logs. This version is compatible with previous versions of the download agent so you are welcome to interrupt a download and use this executable instead for an improved experience.
MD5sum for this tarball: d00b61803747735029b1a599d31fa59d
This release is more graceful when retrying downloads when the entire internet connection is down for minutes or more. The download agent is designed to restart where it left off if the user terminates the connection. If there are any errors or issues you notice, feel free to terminate and restart the process and you should be up and running again.
When downloading with many threads, log output for intermittent network failures can be overwhelming. This release contains a modification to prevent many threads from complaining about an intermittent failure simultaneously.
This release also contains a fix to 'ls_manifest' where a '/' was missing between the folder path and file name for subdirectories in a project.
MD5sum for tarball: 8f68f0d4b61ae2ec3e534dfcd5aabf4e
Changed the output message to more accurately show "MB/s estimated ver the last 60s".
MD5sum of tarball for this release: e815735b399ecd88c697c84e0234636e
Output is consistently using 1024 KB in a MB, etc for the moment. This may change to IEC in the future.
Carriage returns are likely not to interfere with each other.
MD5sum of tarball: 77810eac20730b953660de931f3fa5d1
Note: The output when reporting net throughput written to disk should read something like "XXX MB/s written to disk estimated over the last 60s" as opposed to "MB" (no "per second") as in this release. This will be fixed in the next incremental release.
This release cleans up output of the download agent by placing detailed messages and errors in a log file of the form <manifest-file-name>.download.log. The idea behind this file is that if an error occurs the user can consult this as a historical log (it includes timestamps) and can forward it to DNAnexus support for further debugging. In the future we plan to include more detailed information about the client's networking setup to help further debug issues.
- Obtain a mutex to query database for progress updates (in the future this will be managed with a single connection)
- Graceful retries of downloading parts are now done perpetually to deal with intermittent cases where retryablehttp obtains a nil pointer (we are investigating why the pointer is appearing nil in this library)
- Summary output has been cleaned up to be clearer about net throughput in a particular time period and updated every time a part is downloaded
This release adds support to perpetually retry on 503s. The wait time to retry uses the retry-after header when available. Otherwise, it exponentially backs off up to 30 minutes in the absence of the retry-after header.