Add support for private registry #11766

yahavi · 2021-07-25T07:43:05Z

Have you followed the guidelines in our Contributing document?
Have you checked to ensure there aren't other open Pull Requests for the same change?
Have you added an explanation of what your changes do and why you'd like us to include them?
Have you written new tests for your changes? Here's an example.
Have you successfully run brew style with your changes locally?
Have you successfully run brew typecheck with your changes locally?
Have you successfully run brew tests with your changes locally?

Supersedes #11339
Partially addresses #11287

Allow using a private registry mirroring the common URL for blobs and bottles (https://ghcr.io/v2/homebrew/core). This feature worked before when the Homebrew packages registry was Bintray and this PR makes sure it works as before.

Usage example with Artifactory:

Create a remote Docker repository in Artifactory proxies https://ghcr.io. In my example, I named it brew-remote.
Populate HOMEBREW_DOCKER_REGISTRY_TOKEN environment variable with Artifactory access token and HOMEBREW_ARTIFACT_DOMAIN with a URL to Artifactory repository:

export HOMEBREW_DOCKER_REGISTRY_TOKEN=<artifactory-access-token>
export HOMEBREW_ARTIFACT_DOMAIN=http://127.0.0.1:8081/artifactory/brew-remote

Run brew install:

brew install jfrog-cli

==> Downloading https://ghcr.io/v2/homebrew/core/jfrog-cli/manifests/2.0.1
==> Downloading from http://127.0.0.1:8081/artifactory/brew-remote/v2/homebrew/core/jfrog-cli/manifests/2.0.1
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/jfrog-cli/blobs/sha256:484e41483de267892b3b87aceb84d5040cd66df4db4ced30de419f8e32011a84
==> Downloading from http://127.0.0.1:8081/artifactory/brew-remote/v2/homebrew/core/jfrog-cli/blobs/sha256:484e41483de267892b3b87aceb84d5040cd66df4db4ced30de419f8e32011a84
######################################################################## 100.0%
==> Pouring jfrog-cli--2.0.1.big_sur.bottle.tar.gz
==> Caveats
zsh completions have been installed to:
/Users/yahavi/code/brew/share/zsh/site-functions
==> Summary
🍺 /Users/yahavi/code/brew/Cellar/jfrog-cli/2.0.1: 7 files, 19.2MB

jkenn99 · 2021-07-25T14:53:27Z

How would this work if the private registry doesn't require authentication?
I tried implementing fetching a valid token in #11339.

yahavi · 2021-07-26T07:58:42Z

@jkenn99
Thanks for your feedback! I added another commit to allow anonymous access for private registries.

@MikeMcQuaid
Could you please let me know if you like the design? I can add tests to this PR after your code review.

MikeMcQuaid

A few minor comments but I really like this approach. I like how non-invasive the approach is. Great work 👏🏻

MikeMcQuaid · 2021-07-26T12:29:25Z

Library/Homebrew/download_strategy.rb

@@ -449,7 +449,7 @@ def resolve_url_basename_time_file_size(url, timeout: nil)
    return @resolved_info_cache[url] if @resolved_info_cache.include?(url)

    if (domain = Homebrew::EnvConfig.artifact_domain)
-      url = url.sub(%r{^((ht|f)tps?://)?}, "#{domain.chomp("/")}/")
+      url = url.sub(%r{^((ht|f)tps?://ghcr.io/)?}, "#{domain.chomp("/")}/")


Suggested change

url = url.sub(%r{^((ht|f)tps?://ghcr.io/)?}, "#{domain.chomp("/")}/")

url = url.sub(%r{^(https?://#{GitHubPackages::URL_DOMAIN}/)?}, "#{domain.chomp("/")}/")

Agree, with 2 additional changes:

Added the o flag to handle RuboCop error: https://msp-greg.github.io/rubocop-performance/RuboCop/Cop/Performance/ConstantRegexp.html

URL_DOMAIN constant became public

Library/Homebrew/download_strategy.rb

Library/Homebrew/env_config.rb

MikeMcQuaid · 2021-07-26T12:37:33Z

If the private registry doesn't require authentication this still won't work since it doesn't request a valid token. Please take a look at the changes in #11339.

@jkenn99 Please try to be a little more positive on this PR given that someone else seems likely to implement the functionality that you weren't willing to push to completion. Thanks ❤️

jkenn99 · 2021-07-26T14:11:10Z

@jkenn99 Please try to be a little more positive on this PR given that someone else seems likely to implement the functionality that you weren't willing to push to completion. Thanks ❤️

Good call, my bad.

jkenn99 · 2021-07-26T14:17:03Z

@jkenn99

Thanks for your feedback! I added another commit to allow anonymous access for private registries.

@yahavi Thanks for taking a look at it. Is there any minimum Artifactory version needed to support this?

MikeMcQuaid · 2021-07-26T18:09:59Z

Good call, my bad.

@jkenn99 Thanks for understanding 💖

MikeMcQuaid

This looks good to me! @yahavi can you confirm it all still works for you after the most recent changes and I think I'm good to merge.

Library/Homebrew/download_strategy.rb

yahavi · 2021-07-27T08:00:05Z

@jkenn99 - It works for me on Artifactory 6.9 and above.

@MikeMcQuaid - I confirm. Please merge.

MikeMcQuaid · 2021-07-27T12:52:46Z

Thanks so much for your first contribution! Without people like you submitting PRs we couldn't run this project. You rock, @yahavi!

zie-ent · 2021-07-27T12:57:47Z

I just followed this PR - but thank all of you.
Excited to test this, once you guys release a new version - maybe later today ? 🤔

yahavi · 2021-07-27T12:58:15Z

@MikeMcQuaid thanks for merging this PR and for your kind words :)

SMillerDev · 2021-07-27T13:02:23Z

It seems like this merge has broken core CI, as noticed by @alebcay :

https://github.com/Homebrew/homebrew-core/pull/81954/checks?check_run_id=3171769122
https://github.com/Homebrew/homebrew-core/runs/3171780579?check_suite_focus=true

Error: undefined method `delete_if' for nil:NilClass
/opt/homebrew/Library/Homebrew/download_strategy.rb:394:in `fetch'
/opt/homebrew/Library/Homebrew/resource.rb:142:in `fetch'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/forwardable.rb:230:in `fetch'
/opt/homebrew/Library/Homebrew/formula.rb:2016:in `fetch'
/opt/homebrew/Library/Homebrew/formula_installer.rb:1133:in `fetch'
/opt/homebrew/Library/Homebrew/install.rb:299:in `install_formula'
/opt/homebrew/Library/Homebrew/cmd/install.rb:207:in `block in install'
/opt/homebrew/Library/Homebrew/cmd/install.rb:205:in `each'
/opt/homebrew/Library/Homebrew/cmd/install.rb:205:in `install'
/opt/homebrew/Library/Homebrew/brew.rb:131:in `<main>'

ltm · 2021-07-27T13:17:11Z

I'm getting the same error when trying to install a Cask with Homebrew 3.2.5-5-g250233a:

$ brew install --verbose --debug clion
==> Cask::Installer#install
==> Printing caveats
==> Cask::Installer#fetch
/usr/bin/curl --disable --globoff --show-error --user-agent Homebrew/3.2.5-5-g250233a\ \(Macintosh\;\ Intel\ Mac\ OS\ X\ 11.5\)\ curl/7.64.1 --header Accept-Language:\ en --retry 3 --location --silent --head --request GET https://download.jetbrains.com/cpp/CLion-2021.1.3.dmg
==> Downloading https://download.jetbrains.com/cpp/CLion-2021.1.3.dmg
Error: Download failed on Cask 'clion' with message: undefined method `delete_if' for nil:NilClass
/usr/local/Homebrew/Library/Homebrew/download_strategy.rb:394:in `fetch'
/usr/local/Homebrew/Library/Homebrew/cask/download.rb:25:in `fetch'
/usr/local/Homebrew/Library/Homebrew/cask/installer.rb:171:in `download'
/usr/local/Homebrew/Library/Homebrew/cask/installer.rb:73:in `fetch'
/usr/local/Homebrew/Library/Homebrew/cask/installer.rb:100:in `install'
/usr/local/Homebrew/Library/Homebrew/cask/cmd/install.rb:69:in `block in install_casks'
/usr/local/Homebrew/Library/Homebrew/cask/cmd/install.rb:68:in `each'
/usr/local/Homebrew/Library/Homebrew/cask/cmd/install.rb:68:in `install_casks'
/usr/local/Homebrew/Library/Homebrew/cmd/install.rb:167:in `install'
/usr/local/Homebrew/Library/Homebrew/brew.rb:131:in `<main>'

I don't see an open issue for this yet.

yahavi · 2021-07-27T13:24:37Z

@SMillerDev @ltm
Sorry for this issue. The tests passed for me in the Homebrew/brew repository.
I believe #11782 should fix it.

ltm · 2021-07-27T13:29:21Z

Thanks, @yahavi. I can confirm that #11782 fixes the issue for me:

$ brew install --verbose --debug clion
==> Cask::Installer#install
==> Printing caveats
==> Cask::Installer#fetch
/usr/bin/curl --disable --globoff --show-error --user-agent Homebrew/3.2.5-5-g250233a-dirty\ \(Macintosh\;\ Intel\ Mac\ OS\ X\ 11.5\)\ curl/7.64.1 --header Accept-Language:\ en --retry 3 --location --silent --head --request GET https://download.jetbrains.com/cpp/CLion-2021.1.3.dmg
==> Downloading https://download.jetbrains.com/cpp/CLion-2021.1.3.dmg
==> Downloading from https://download-cdn.jetbrains.com/cpp/CLion-2021.1.3.dmg
/usr/bin/curl --disable --globoff --show-error --user-agent Homebrew/3.2.5-5-g250233a-dirty\ \(Macintosh\;\ Intel\ Mac\ OS\ X\ 11.5\)\ curl/7.64.1 --header Accept-Language:\ en --retry 3 --location --head https://download-cdn.jetbrains.com/cpp/CLion-2021.1.3.dmg
[...]

MikeMcQuaid · 2021-07-28T14:31:36Z

I just followed this PR - but thank all of you.
Excited to test this, once you guys release a new version - maybe later today ? 🤔

@zie-ent We'll likely have a new release within a week (probably Monday).

MikeMcQuaid · 2021-08-03T07:51:29Z

This was tagged in 3.2.6

Add support for private registry

b895403

yahavi force-pushed the private-registry branch from b9e0fd8 to b895403 Compare July 25, 2021 07:50

yahavi mentioned this pull request Jul 25, 2021

Container registry bottle backend with non-ghcr.io URL #11287

Closed

Allow anonymous access in private registries

cc12738

MikeMcQuaid reviewed Jul 26, 2021

View reviewed changes

Fix code review comments + disable authorization on redirections

0335d8c

MikeMcQuaid approved these changes Jul 26, 2021

View reviewed changes

Library/Homebrew/download_strategy.rb Outdated Show resolved Hide resolved

Implement code review comment

1116580

MikeMcQuaid merged commit 250233a into Homebrew:master Jul 27, 2021

yahavi deleted the private-registry branch July 27, 2021 12:54

alebcay mentioned this pull request Jul 27, 2021

Update font-iosevka from 8.0.1 to 8.0.2 Homebrew/homebrew-cask-fonts#4706

Merged

yahavi mentioned this pull request Jul 27, 2021

Error raised when no headers defined in curl download #11782

Merged

7 tasks

yahavi mentioned this pull request Aug 4, 2021

Allow anonymous access in private registries #11822

Merged

7 tasks

github-actions bot added the outdated PR was locked due to age label Sep 4, 2021

github-actions bot locked as resolved and limited conversation to collaborators Sep 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for private registry #11766

Add support for private registry #11766

yahavi commented Jul 25, 2021 •

edited

jkenn99 commented Jul 25, 2021 •

edited

yahavi commented Jul 26, 2021

MikeMcQuaid left a comment

MikeMcQuaid Jul 26, 2021

yahavi Jul 26, 2021

MikeMcQuaid commented Jul 26, 2021

jkenn99 commented Jul 26, 2021 •

edited

jkenn99 commented Jul 26, 2021

MikeMcQuaid commented Jul 26, 2021

MikeMcQuaid left a comment

yahavi commented Jul 27, 2021

MikeMcQuaid commented Jul 27, 2021

zie-ent commented Jul 27, 2021 •

edited

yahavi commented Jul 27, 2021

SMillerDev commented Jul 27, 2021

ltm commented Jul 27, 2021

yahavi commented Jul 27, 2021

ltm commented Jul 27, 2021

MikeMcQuaid commented Jul 28, 2021

MikeMcQuaid commented Aug 3, 2021

	url = url.sub(%r{^((ht\|f)tps?://ghcr.io/)?}, "#{domain.chomp("/")}/")
	url = url.sub(%r{^(https?://#{GitHubPackages::URL_DOMAIN}/)?}, "#{domain.chomp("/")}/")

Add support for private registry #11766

Add support for private registry #11766

Conversation

yahavi commented Jul 25, 2021 • edited

jkenn99 commented Jul 25, 2021 • edited

yahavi commented Jul 26, 2021

MikeMcQuaid left a comment

Choose a reason for hiding this comment

MikeMcQuaid Jul 26, 2021

Choose a reason for hiding this comment

yahavi Jul 26, 2021

Choose a reason for hiding this comment

MikeMcQuaid commented Jul 26, 2021

jkenn99 commented Jul 26, 2021 • edited

jkenn99 commented Jul 26, 2021

MikeMcQuaid commented Jul 26, 2021

MikeMcQuaid left a comment

Choose a reason for hiding this comment

yahavi commented Jul 27, 2021

MikeMcQuaid commented Jul 27, 2021

zie-ent commented Jul 27, 2021 • edited

yahavi commented Jul 27, 2021

SMillerDev commented Jul 27, 2021

ltm commented Jul 27, 2021

yahavi commented Jul 27, 2021

ltm commented Jul 27, 2021

MikeMcQuaid commented Jul 28, 2021

MikeMcQuaid commented Aug 3, 2021

yahavi commented Jul 25, 2021 •

edited

jkenn99 commented Jul 25, 2021 •

edited

jkenn99 commented Jul 26, 2021 •

edited

zie-ent commented Jul 27, 2021 •

edited