Package download request resuming #7399
Conversation
This is still a WIP, but I wanted to get feedback at this point. Will update the PR with outstanding TODOs.
| return httpHelpers.request({ | ||
| outputStream, | ||
| headers, | ||
| ...urlOrOptions, |
tmeasday
Jul 13, 2016
Author
Contributor
XXX: should be options
XXX: should be options
benjamn
Jul 13, 2016
Member
If you use options here, the options.headers property will override the headers property, so you might as well update options.headers above, and remove the headers, line here.
If you use options here, the options.headers property will override the headers property, so you might as well update options.headers above, and remove the headers, line here.
tmeasday
Jul 14, 2016
Author
Contributor
👍
| function attempt(triesRemaining) { | ||
| const headers = _.extend({}, urlOrOptions.headers, { | ||
| range: `bytes=${outputStream.size()}-`, | ||
| }); |
benjamn
Jul 13, 2016
•
Member
options.headers = {
...options.headers,
range: `bytes=${outputStream.size()}-`
};
options.headers = {
...options.headers,
range: `bytes=${outputStream.size()}-`
};
tmeasday
Jul 14, 2016
Author
Contributor
👍
| console.log(e); | ||
| if (triesRemaining > 0) { | ||
| console.log(`Request failed, retrying`); | ||
| return attempt(triesRemaining - 1); |
benjamn
Jul 13, 2016
Member
What if the failure only counted against triesRemaining if the download made no progress (or less than some small amount of progress)?
What if the failure only counted against triesRemaining if the download made no progress (or less than some small amount of progress)?
benjamn
Jul 13, 2016
Member
Perhaps there should be a delay of a few seconds between attempts?
Perhaps there should be a delay of a few seconds between attempts?
tmeasday
Jul 14, 2016
Author
Contributor
That's part of the plan, yeah.
That's part of the plan, yeah.
| const url = 'http://warehouse.meteor.com/builds/Pr7L8f6PqXyqNJJn4/1443478653127/aRiirNrp4v/meteor-tool-1.1.9-os.osx.x86_64+web.browser+web.cordova.tgz'; | ||
|
|
||
| const result = require('../utils/http-helpers').getUrlWithResuming({ | ||
| timeout: 1000, |
benjamn
Jul 13, 2016
Member
This should actually test resuming, perhaps by supporting a custom options.outputStream object, and passing a test object whose .write method throws an HTTP error after the first few chunks are written?
This should actually test resuming, perhaps by supporting a custom options.outputStream object, and passing a test object whose .write method throws an HTTP error after the first few chunks are written?
tmeasday
Jul 14, 2016
Author
Contributor
Oooh, that might work, let me try it.
Oooh, that might work, let me try it.
| @@ -14,6 +14,7 @@ var release = require('../packaging/release.js'); | |||
| var Console = require('../console/console.js').Console; | |||
| var timeoutScaleFactor = require('./utils.js').timeoutScaleFactor; | |||
|
|
|||
| import { WritableStreamBuffer } from 'stream-buffers'; | |||
benjamn
Jul 13, 2016
Member
The stream-buffers package needs to be in dev_bundle/lib/node_modules.
The stream-buffers package needs to be in dev_bundle/lib/node_modules.
benjamn
Jul 13, 2016
Member
Ah, sorry, I see that in your to-do list now.
Ah, sorry, I see that in your to-do list now.
tmeasday
Jul 14, 2016
Author
Contributor
Yeah I just wanted to check with you that the concept of using a stream buffer to collect the body made sense before bothering to build a new bundle etc.
Yeah I just wanted to check with you that the concept of using a stream buffer to collect the body made sense before bothering to build a new bundle etc.
| const href = response.request.href; | ||
| throw Error( | ||
| body || | ||
| `Could not get ${href}; server returned [${response.statusCode}]`); |
benjamn
Jul 13, 2016
Member
Maybe this was just for debugging, but I would use the same error message every time, regardless of whether body is defined, and then just return outputStream.getContents() below, if there was no error.
Maybe this was just for debugging, but I would use the same error message every time, regardless of whether body is defined, and then just return outputStream.getContents() below, if there was no error.
tmeasday
Jul 14, 2016
Author
Contributor
Yeah, I was just re-implementing what getUrl does above, but I agree it's sort of weird.
Yeah, I was just re-implementing what getUrl does above, but I agree it's sort of weird.
|
I like the approach of storing partial downloads in memory! |
| const MAX_ATTEMPTS = 5; | ||
| function attempt(triesRemaining) { | ||
| const headers = _.extend({}, urlOrOptions.headers, { | ||
| range: `bytes=${outputStream.size()}-`, |
benjamn
Jul 13, 2016
Member
Should range be capitalized here?
Should range be capitalized here?
tmeasday
Jul 14, 2016
Author
Contributor
👍
Using Console.debug to register messages about retries.
|
@ben I think this should be good to go, awaiting tests |
This is a test that really needs to run and pass every time we run the test suite, so I decided it shouldn't be --slow. If it takes too long, we can always download a smaller test file. Hard-coding the download length was a recipe for brittleness, so now I'm downloading the file without interruptions in parallel with the interrupted/resumed download, so that we can compare the two files when both have finished downloading. Follow-up to #7399.
| @@ -304,7 +304,7 @@ _.extend(exports.Tropohouse.prototype, { | |||
| // it relies on extractTarGz being fast and not reporting any progress. | |||
abernix
Jul 15, 2016
Member
It's just off the top of the diff here, but I think this XXX/TODO would be a huge benefit as it's getting hard to debug when something is stuck downloading or something is stuck extracting.
I'm getting the feeling that people are getting stuck after the actual download, especially on Windows.
It's just off the top of the diff here, but I think this XXX/TODO would be a huge benefit as it's getting hard to debug when something is stuck downloading or something is stuck extracting.
I'm getting the feeling that people are getting stuck after the actual download, especially on Windows.
For #7267
I wanted to get your thoughts on this approach @benjamn.
Todo:
stream-buffersto the dev bundle