Questions on Homebrew's third-party mirroring policy

[taoky]

Hi Homebrew staff,

Since the PR Homebrew/brew#11781 got merged into brew master, it is now impossible for third-party mirrors to properly mirror homebrew/linuxbrew bottles while providing files to users as-is. I'm now curious that how should we mirror homebrew-core, homebrew-cask, etc. , and homebrew/linuxbrew bottles, to speed up brew in certain region or intranet? Or are they not intended to be mirrored?

Thank you.

[SMillerDev]

Have you tried using HOMEBREW_ARTIFACT_DOMAIN as is indicated in that PR?

[taoky]

Hi,

I have rechecked the manifests of bottles and found that it is still possible to have bottles synced. If I'm not understanding it wrong, our mirroring script can get blob url from brew JSON API¹, and manifest file url can be manually constructed from the version number and rebuild count like https://github.com/homebrew/brew/blob/HEAD/Library/Homebrew/software_spec.rb#L405

However, I still have some questions:

1. Is this protocol stabilized (including the format of JSON API response, manifest and blob url, and the corresponding environment variables settings (HOMEBREW_ARTIFACT_DOMAIN, HOMEBREW_BREW_GIT_REMOTE, HOMEBREW_CORE_GIT_REMOTE, etc.) for end-users)? Modifying script and docs every few months is quite a frustrating experience.
2. If it could be stabilized, could you add a page about necessary information for mirroring site in brew documentation? It would be very appreciated to have a detailed documentation like Debian: Setting up a Debian archive mirror and Alpine: How to setup a Alpine Linux mirror.

Thank you.

Footnotes

1. This doc may need updating as it still contains homebrew.bintray.com in example response. ↩

[SMillerDev]

Tagging @Homebrew/brew as the more knowledgeable party here.

[MikeMcQuaid]

@taoky It was not my intention to remove all support for mirroring homebrew-core binaries. Was HOMEBREW_BOTTLE_DOMAIN working with you before I removed it? If so, with what mirror/value (so I can test/verify it).

I removed it because we've had a bunch of reports of it being set and being broken.

[skyzh]

HOMEBREW_BOTTLE_DOMAIN was working in the past year since Homebrew migrated bottles to ghcr.io, simply by doing:

export HOMEBREW_BOTTLE_DOMAIN=https://mirrors.sjtug.sjtu.edu.cn/homebrew-bottles/bottles

But it really takes a lot of effort to (1) enable us to successfully mirror all bottles and (2) let our users download bottles from our mirror site.

• On users' side, they need to change HOMEBREW_BOTTLE_DOMAIN from https://mirrors.sjtug.sjtu.edu.cn/homebrew-bottles to https://mirrors.sjtug.sjtu.edu.cn/homebrew-bottles/bottles (Take mirrors.sjtug.sjtu.edu.cn as example). It seems that HOMEBREW_BOTTLE_DOMAIN was broken in Homebrew/brew@7fc52d2.
• On our side, we have completely rewritten our mirroring script at USTCLUG and at SJTUG several times to adapt to upstream changes.

In fact, we are not everyday users of Homebrew, and we receive frequent reports from mirror users complaining that Homebrew suddenly cannot download bottles from our mirror site, which requires us days to debug and see what happened upstream. I think it's better to provide:

1. a better way for us to mirror and store artifacts

• e.g. fixed mapping between formulas to bottles like homebrew-bottles/jq-1.6.arm64_big_sur.bottle.1.tar.gz, instead of simply expose the package registry URL homebrew-bottles/ghcr.io/v2/homebrew/core/jq/manifests/1.6-1.
• e.g. a stable JSON interface like https://formulae.brew.sh/docs/api/ to get all bottles to mirror and their URLs.

2. have a stable interface to users for configuring mirrors (e.g. not to change how config items like HOMEBREW_BOTTLE_DOMAIN work too frequently).

[MikeMcQuaid]

Thanks for the explanation. I'm reverting Homebrew/brew#11781 in Homebrew/brew#11793

• On users' side, they need to change HOMEBREW_BOTTLE_DOMAIN from https://mirrors.sjtug.sjtu.edu.cn/homebrew-bottles to https://mirrors.sjtug.sjtu.edu.cn/homebrew-bottles/bottles (Take mirrors.sjtug.sjtu.edu.cn as example). It seems that HOMEBREW_BOTTLE_DOMAIN was broken in Homebrew/brew@7fc52d2.

This change was unintentional, I'm sorry. We are happy to revert to the previous behaviour.

Is it expected that https://mirrors.tuna.tsinghua.edu.cn/homebrew-bottles is broken?

• On our side, we have completely rewritten our mirroring script at USTCLUG and at SJTUG several times to adapt to upstream changes.

I'm sorry about this, this was not intentional. The migration to GitHub Packages meant that a lot of the bottling logic was changed (and evidently broke).

1. a better way for us to mirror and store artifacts

Agreed. I think something like HOMEBREW_ARTIFACT_DOMAIN that proxies the requests and caches them may be a better fit?

Given how GitHub Packages is now used, mirroring as a Docker registry probably makes more sense.

• e.g. fixed mapping between formulas to bottles like homebrew-bottles/jq-1.6.arm64_big_sur.bottle.1.tar.gz, instead of simply expose the package registry URL homebrew-bottles/ghcr.io/v2/homebrew/core/jq/manifests/1.6-1.
• e.g. a stable JSON interface like https://formulae.brew.sh/docs/api/ to get all bottles to mirror and their URLs.

I'm not sure I fully understand this request. It's going to be hard to be able to provide those sorts of desired URLs given the GitHub Packages migration.

1. have a stable interface to users for configuring mirrors (e.g. not to change how config items like HOMEBREW_BOTTLE_DOMAIN work too frequently).

Agreed. Unfortunately we have the issue that many of the users of e.g. HOMEBREW_BOTTLE_DOMAIN in e.g. China do not seem to understand when it's a configuration issue, when it's a mirror issue and when it's a Homebrew issue. It's very hard for us to figure out and debug these problems when they occur hence, in this case, my assumption that this behaviour was entirely broken.

I agree that we want to and should maintain a stable interface here. I think what would help with that is:

• the creators of the mirror scripts to submit more regular bug reports to report regressions so we can fix/revert them
• the creators of the mirror scripts and/or Homebrew users in China to create more unit/integration tests so we can catch these regressions before they are merged
• the investigation of different technology solutions for mirroring e.g. Artifactory, Docker registries, etc.

How does that sound?

[skyzh]

Thanks for your response!

Is it expected that https://mirrors.tuna.tsinghua.edu.cn/homebrew-bottles is broken?

Yes, they have not yet adapted to ghcr.io.

Agreed. I think something like HOMEBREW_ARTIFACT_DOMAIN that proxies the requests and caches them may be a better fit?

Mirror maintainer community in China has reached a consensus that a reverse proxy would easily be abused by our users. Some users use exploit tools to send hundreds of requests a second to those endpoints, which will eventually be "mirrored" to source site. Therefore, we always develop our own tools to get full file list, download them to our mirror server, and serve them over static file server.

On the other hand, mirror maintainers are expecting the source to be something on a file system and provide a file list (distros like Ubuntu and Debian provide a extremely convenient rsync endpoint for us), as we will eventually store what we have downloaded to a local SSD or HDD, and serve them directly through nginx or other static web servers. Software repositories nowadays like homebrew and conda are making this difficult for us. But I personally believe that S3-backed or docker-backed software repositories are the future, so we will continually develop tools to mirror and serve them.

It's going to be hard to be able to provide those sorts of desired URLs given the GitHub Packages migration.

I understand this concern, and we need some time to develop new tools to mirror docker repositories.

I agree that we want to and should maintain a stable interface here. I think what would help with that is:

• the creators of the mirror scripts to submit more regular bug reports to report regressions so we can fix/revert them
• the creators of the mirror scripts and/or Homebrew users in China to create more unit/integration tests so we can catch these regressions before they are merged
• the investigation of different technology solutions for mirroring e.g. Artifactory, Docker registries, etc.

Sounds great! As HOMEBREW_BOTTLE_DOMAIN still works now, I suggest leaving it as-is and marking it deprecated, so that both our users and our mirror infrasture do not need to make immediate changes, for now.

In the future, we will eventually migrate to use HOMEBREW_ARTIFACT_DOMAIN. But before that, I would like to make the interface stable enough for our mirror sites, so as to minimize our maintenance efforts. I'm thinking of adding some kind of unit tests in Homebrew, and some RFCs for https://formulae.brew.sh/docs/api/, which could help stabilize how Homebrew interacts with ghcr.io / homebrew-bottles mirrors. Is this possible, and is there anything I could help?

[shankerwangmiao]

Is it expected that https://mirrors.tuna.tsinghua.edu.cn/homebrew-bottles is broken?

Yes. We were occupied by infrastructure upgrading and had no time to follow up with the mentioned updates in Homebrew. So it might be broken now. I'm also interested in a stable mirroring protocol and expect to deploy it soon.

[MikeMcQuaid]

Is it expected that https://mirrors.tuna.tsinghua.edu.cn/homebrew-bottles is broken?
Yes, they have not yet adapted to ghcr.io.
Yes. We were occupied by infrastructure upgrading and had no time to follow up with the mentioned updates in Homebrew. So it might be broken now. I'm also interested in a stable mirroring protocol and expect to deploy it soon.

Cool, that's fine and good to know. We'll need to figure out a way to better communicate this to the users when a bottle mirror is broken so they don't report it to us.

It's going to be hard to be able to provide those sorts of desired URLs given the GitHub Packages migration.

I understand this concern, and we need some time to develop new tools to mirror docker repositories.

👍🏻. Sounds like we're agreed on the approach, it might just take a bit longer.

In the future, we will eventually migrate to use HOMEBREW_ARTIFACT_DOMAIN. But before that, I would like to make the interface stable enough for our mirror sites, so as to minimize our maintenance efforts. I'm thinking of adding some kind of unit tests in Homebrew, and some RFCs for https://formulae.brew.sh/docs/api/, which could help stabilize how Homebrew interacts with ghcr.io / homebrew-bottles mirrors. Is this possible, and is there anything I could help?

This sounds great. Feel free to create a PR (even if it's a bit broken) and I can help you with the rest.

Thanks, all!

[skyzh]

Some follow-ups on this discussion:

• A PR has been created to stabilize the JSON response from Homebrew APIs. docs: stabilize bottle API responses formulae.brew.sh#521
• A PR has been created for issue templates so that users can find the best-fit place to report bugs. .github/ISSUE_TEMPLATE/bug.yml: add instructions for mirror sites bug reporting brew#11797
• A discussion for mirror maintainers to update their mirror information. Setting up mirrors for Homebrew bottles #1917

[byjrack]

Some other context to help with the BOTTLE_DOMAIN vs ARTIFACT_DOMAIN control logic.

I know we have had issues leveraging a caching proxy for ARTIFACT because it needs to handle both OCI registry (ideally) for bottles and arbitrary HTTPS for the casks. Basically our existing solution (Artifactory) works for standard formulas via a OCI registry proxy for ghcr.io and Casks just fail. And this puts aside arbitrary URLs being a security concern unless deny lists and usage auditing are implemented to prevent misuse of the proxy functionality.

Looking into BOTTLE though I have not figured out a good way to steer requests to the correct GithubPackage spec reliably. The BOTTLE var is parsed in software_spec in root_url which I believe occurs before detector is called so it goes to Curl. For ARTIFACT though it is done in download_spec just before download so the root_url remains github up to the last moment I think. I feel a bit uninformed on the design decisions in both cases so missing context to open a good PR.

I know supporting Docker for BOTTLE was attempted as part of #11339, but I recognize it was not merged. Even the URL scheme code and some of the init logic for Authorization header may be all that is needed.

CC @yahavi