Skip to content

pr-pull: fix to fetch extra-large bottles#7343

Merged
jonchang merged 1 commit intoHomebrew:masterfrom
jonchang:pr-pull-jumbo-bottles
Apr 13, 2020
Merged

pr-pull: fix to fetch extra-large bottles#7343
jonchang merged 1 commit intoHomebrew:masterfrom
jonchang:pr-pull-jumbo-bottles

Conversation

@jonchang
Copy link
Copy Markdown
Contributor

  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same change?
  • Have you added an explanation of what your changes do and why you'd like us to include them?
  • Have you written new tests for your changes? Here's an example.
  • Have you successfully run brew style with your changes locally?
  • Have you successfully run brew tests with your changes locally?

This pull request fixes the issue seen in Homebrew/homebrew-core#52984 and other homebrew-core pull requests, where the bottle downloads are too large to be downloaded without timing out.

When retrieving artifacts from GitHub Actions, the artifact download requires an authorization token that is only valid for 1 minute, meaning the download must start within 60 seconds of retrieving that token.

However, the current two-step download strategy, which downloads a single byte to see if the server supports a range request, then resuming the download, means that large bottles often time out before the second request can occur.

This pull request adds a new download strategy, CurlNoResumeDownloadStrategy, which doesn't check if the server supports download continuations, and merely redownloads from scratch if needed. I've confirmed with local tests that if the entire file has been downloaded successfully, curl doesn't re-download the file.

Comment thread Library/Homebrew/download_strategy.rb Outdated
Copy link
Copy Markdown
Member

@MikeMcQuaid MikeMcQuaid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving to unblock you once you're online. I trust you to fiddle with this given my comment. If it merges as-is because you can't find an alternative: no big deal, just give me a shout and I'll see what I can do.

@jonchang
Copy link
Copy Markdown
Contributor Author

Would this instead take different arguments to the CurlDownloadStrategy#initialize? I'm a bit concerned we'll see this pop up for usage in formulae for no good reason. Better still I think this class would live in dev-cmd/pr-pull if it's only used there.

We'd need to override CurlDownloadStrategy#_fetch entirely, and I feel better about subclassing vs adding one-off knobs to CurlDownloadStrategy which could break other things.

I feel safer moving the class to dev-cmd/pr-pull.rb for this reason.

I've changed this to instead pass an argument to GitHub.fetch_artifacts with the download strategy. The strategy can't live in utils/github.rb because then anyone requiring it would bring in the new download strategy that we want to keep contained to pr-pull.

@jonchang jonchang merged commit b2cf3bf into Homebrew:master Apr 13, 2020
@jonchang jonchang deleted the pr-pull-jumbo-bottles branch April 13, 2020 10:29
@jonchang
Copy link
Copy Markdown
Contributor Author

Thanks for the review! Our bottles will fly more comfortably now.

@lock lock bot added the outdated PR was locked due to age label May 20, 2020
@lock lock bot locked as resolved and limited conversation to collaborators May 20, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

outdated PR was locked due to age

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants