Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Package download request resuming #7399

Merged
merged 6 commits into from Jul 14, 2016
Merged

Conversation

tmeasday
Copy link
Contributor

@tmeasday tmeasday commented Jul 13, 2016

For #7267

I wanted to get your thoughts on this approach @benjamn.

Todo:

  • Ensure it actually works with package downloading
  • Figure out if there's a way to make the test meaningful
  • Add a timeout to the retries
  • Better logging
  • Add stream-buffers to the dev bundle

This is still a WIP, but I wanted to get feedback at this point.
Will update the PR with outstanding TODOs.
return httpHelpers.request({
outputStream,
headers,
...urlOrOptions,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

XXX: should be options

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you use options here, the options.headers property will override the headers property, so you might as well update options.headers above, and remove the headers, line here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@benjamn
Copy link
Contributor

benjamn commented Jul 13, 2016

I like the approach of storing partial downloads in memory!

const MAX_ATTEMPTS = 5;
function attempt(triesRemaining) {
const headers = _.extend({}, urlOrOptions.headers, {
range: `bytes=${outputStream.size()}-`,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should range be capitalized here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@tmeasday
Copy link
Contributor Author

@ben I think this should be good to go, awaiting tests

@benjamn benjamn merged commit 09ecd80 into release-1.4 Jul 14, 2016
benjamn added a commit that referenced this pull request Jul 14, 2016
@benjamn benjamn added this to the Release 1.4 milestone Jul 14, 2016
@benjamn benjamn mentioned this pull request Jul 14, 2016
benjamn added a commit that referenced this pull request Jul 14, 2016
benjamn added a commit that referenced this pull request Jul 14, 2016
benjamn added a commit that referenced this pull request Jul 14, 2016
benjamn added a commit that referenced this pull request Jul 14, 2016
@benjamn benjamn mentioned this pull request Jul 14, 2016
benjamn added a commit that referenced this pull request Jul 14, 2016
This is a test that really needs to run and pass every time we run the
test suite, so I decided it shouldn't be --slow. If it takes too long, we
can always download a smaller test file.

Hard-coding the download length was a recipe for brittleness, so now I'm
downloading the file without interruptions in parallel with the
interrupted/resumed download, so that we can compare the two files when
both have finished downloading.

Follow-up to #7399.
@@ -304,7 +304,7 @@ _.extend(exports.Tropohouse.prototype, {
// it relies on extractTarGz being fast and not reporting any progress.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's just off the top of the diff here, but I think this XXX/TODO would be a huge benefit as it's getting hard to debug when something is stuck downloading or something is stuck extracting.

I'm getting the feeling that people are getting stuck after the actual download, especially on Windows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants