Load formulae from the internal API with HOMEBREW_USE_INTERNAL_API

[Rylan12]

This PR is an initial pass at updating how we load Formulae to use the slimmed-down internal API, and fetch the remaining metadata from the OCI image. This is still a work-in-progress, but I wanted to open to get feedback on the approach.

---

One thing I definitely want to hear thoughts on:

How should we go about adding the JSON data to the OCI manifest files? Once this PR is merged, newly created bottles will have the additional data. For now, I've simply added a fallback for "if this fails, use https://formulae.brew.sh/api/formula/wget.json", but I don't think we should be relying on this method long-term, since I imagine many formulae won't get rebottled anytime soon.

Is there a way to mass-update manifest files?

[MikeMcQuaid]

All makes sense to me so far, nice work!

[MikeMcQuaid]

Is there a way to mass-update manifest files?

It's definitely possible. Not sure the easiest way to do so. brew pr-upload --keep-old --upload-only should do at least some of this locally.

May be worth uploading your own test bottle(s) to your own GitHub Packages somewhere and seeing what needs done. After that, we could test on a specific homebrew-core bottle and, eventually, run on all existing bottles.

[Rylan12]

Okay, I wrote a script to easily migrate the manifest to include the new information:

https://github.com/Rylan12/formula-manifest-migration/blob/main/migrate.rb

For now, I have one test package (rylan12/formula-manifest-migration/wget:1.25.0) that this works for.

Anyone can easily view the manifest JSON with:

# Using oras
oras manifest fetch ghcr.io/rylan12/formula-manifest-migration/wget:1.25.0

# Using skopeo
skopeo inspect --raw docker://ghcr.io/rylan12/formula-manifest-migration/wget:1.25.0

Some more thorough testing is probably warranted, and then I'll probably choose one or a few lesser used homebrew/core formulae to try migrating, and test the entire pipeline on those before modifying everything.

[MikeMcQuaid]

@Rylan12 looks good to me so far! I couldn't authenticate but Oras worked as expected and looks good.

skopeo inspect --raw docker://ghcr.io/rylan12/formula-manifest-migration/wget:1.25.0 | jq -r '.annotations["sh.brew.formula.json"]' | jq . showed the relevant added JSON (if IU've understood correctly).

One thought is it might be worth dropping empty hashes/empty arrays/null values from the output to slim it down a bit? We're not as space conscious here but every little helps.

[Rylan12]

@MikeMcQuaid no pressure, but if you'd like to try the script, I added the hello formula and gave you admin access so in theory migrate.rb hello should work

https://github.com/users/Rylan12/packages/container/package/formula-manifest-migration%2Fhello