Why is a full clone of homebrew/core required for brew update?

[tddschn]

Before today, brew update works with a shallow clone of homebrew/core, why is that changed?

And why is a full clone of the core tap important if I only care about using the latest core formulaes?

Running brew update with a shallow clone of the core taps threw the error message:

Error: homebrew-core is a shallow clone. To `brew update` first run:
  git -C "/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core" fetch --unshallow

[gromgit]

See #225.

[Rylan12]

Also, as mentioned in Homebrew/brew#9383, this change was made due to a direct request from GitHub to reduce our shallow clone usage.

[tddschn]

Also, as mentioned in Homebrew/brew#9383, this change was made due to a direct request from GitHub to reduce our shallow clone usage.

I didn't know it's hard on GitHub... What about homebrew/cask{,-versions}? The cask repo is quite large too, will shallow clone of the cask repo(s) be "refused" in the future?

[gromgit]

homebrew/cask is included in Homebrew/brew#9383

[souleater7]

Would it be possible for Homebrew to download the latest tarball of homebrew/core instead of cloning the repository? For example, it could check what is the latest commit hash, store it, and then use that to check if the current "version" is outdated.

[carlocab]

It might be. However, I suspect that requires a complete revamp of

1. brew internals; and,
2. homebrew/core workflows.

If you're able to work out an implementation, please do open the PRs for it.

[tddschn]

There are many downsides of large tap repos:

• The tap repos take up more space on the disk than they needed
• The initial full clone is more likely to fail on low-RAM Mac, and is very time-consuming
• git log / git log -p can be slow on large repos

[gromgit]

It's a consequence of Homebrew being:

• an open project from top to bottom
• an extremely active project
• a Git/GitHub-centric project (other VCSes and hosting solutions will likely have different tradeoffs)

I can't comment on pacman, but yum (RPM) and apt (DEB) repo maintainers generally hide their "working state" from end-users, so the repos can be published as a set of static files, which any web server can serve up without much processing. This is possible because, unlike Homebrew, these repos are actually open-source "walled gardens" which:

• don't allow direct user contributions (every change must be back-channeled to the repo maintainers)
• don't facilitate installing anything except the software (and versions) that the maintainers choose to offer

[iMichka]

Also, I think git was chosen back then to deliver files as it was easier for us to maintain: no need to have a server providing the files (as GitHub did that for us), no need to write complex networking code that is robust against redirects, timeouts and other stuff: git handles the whole networking for us and is quite robust. The files you have on your computer are a direct copy of what is on the git server, so this is also a nice thing.

[jm9151821]

Just wanted to say that as newish user on terminal, this is making it so complicated! I ran the command, but don't know how long it will take to complete.

edit: wait more than 1 minute or 2, then the progress bar will appear.

[Rylan12]

I've modified the error message slightly to mention that the command will take a few minutes to run. This change will be available to users after the next Homebrew release.

[Jimmy-Z]

@gromgit doesn't make sense, Homebrew is not a worldwide writable git repo.

@iMichka it is easier, with consequences.

[galaxy001]

Is this possible to officially fork a new "shadowed" repo with last 50 commits every few months and make brew to always force pull that repo ?

[gromgit]

I'm pretty sure it's not possible to even unofficially maintain a partial commit history in a forked repo. Any gaps in the commit history would lead to some updates not being applied each time, and you'd eventually have to start from scratch with a fresh clone when merging updates becomes impossible.

[SMillerDev]

50 commits happen on a slow day for homebrew-core. So you'd have just the files that released that day.

[alexpirine]

I guess we can just thank Microsoft for that.

[jonchang]

Be sure to read the man page for git gc before doing so as there are a few caveats to be aware of.

[carlocab]

Yes, sorry. I did mention that caveat last time I posted about git gc, but it slipped my mind this time. Please read git help gc before running it.

[luckman212]

@jonchang and @carlocab can either of you TLDR the caveats? I read man git-gc and from what I can tell the main "risk" is running concurrent gc operations within the same repo. As long as one is careful not to do this, is there any real downside?

[gromgit]

The only one that comes to mind: It'll take a LONG time, and it may not be safe to interrupt it, so it's probably best done as a "lights out" maintenance job via your favorite scheduler.

[gromgit]

I guess we can just thank Microsoft for that.

No, you have The Linus to thank for that. His decision to make git do "flexible delta-chaining" is almost certainly the primary reason why shallow cloning (and deep GC) is tolerable in local servers with few clients, but prohibitively expensive at GitHub's scale: https://stackoverflow.com/a/28721047/4432299

[galaxy001]

Can we sync with GitHub pages ?

Like a apt source, with a list and other source files. The client download the list first and check the listed version number or CRC32 of each rb file. Then download only those updated.

[gromgit]

Can we sync with GitHub pages ?

It's technically feasible, but I'm hard-pressed to see why the Homebrew maintainers would want to do this, since:

• it's more work, both up-front and on a recurring basis, to replicate 90% of what's already available for free via a Git repo
• it's harder to debug end-user issues, since it's close to impossible to accurately determine exactly what the user changed (easily done with a Git repo)
• it breaks brew extract, which almost everyone will want to use at one time or another to "backstep" to an earlier version (not an issue with a Git repo)

[carlocab]

The most viable alternative might be a CDN as used by CocoaPods, suggested here.

Even then, I think it would take a whole lot of work on brew internals to implement this.

[iMichka]

Yes, it means synching the code between git and the CDN, or whatever other server we use to ship the data.

And synching data is a non-trivial problem. Problem that git solved for us (though I admit it is slow due to the volume of the repo nowadays).

[carlocab]

I actually think Netlify has existing solutions for the syncing problem; they're the provider that CocoaPods uses as well.

I believe the bigger issue is reworking brew internals. The whole thing is built on the premise that content is delivered through git, so there could be parts of the code that are affected by a switch but aren't easy to identify as such. That said, I'm not at all familiar with brew internals, so I could be completely wrong here.

[dentuzhik]

We have just gotten this error with one of our self-hosted mac machines.
It happened only one one out of three machines, I assume we had an old version of brew there.

After running recommended commands everything seems to work, however by browsing issues and PRs, we couldn't quite understand one thing - is this a one-time issue for some old brew installations, or will it happen in the future too after running consecutively brew update?

Thanks!

[gromgit]

is this a one-time issue

Yes, once you've done the --unshallow. brew update doesn't change the clone depth of the repos it touches.

[dentuzhik]

Thanks @gromgit
Do I understand correctly that for future installations of brew this is also not ever going to happen?
Or do we have to run --unshallow when installing brew when expecting to run brew update regularly on CI?

[SMillerDev]

Nope, will never happen again. It's basically just a change in the defaults

[iamitava]

Just a suggestion: would it be possible for brew update to list both homebrew-core and homebrew-cask as things to update in one go?

The update process went like this for me:

$ brew update
Error: homebrew-core is a shallow clone. To `brew update` first run:
  git -C "/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core" fetch --unshallow
... explanatory text ...
$ git -C "/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core" fetch --unshallow
<<git stuff>>
$ brew update
Error: homebrew-cask is a shallow clone. To `brew update` first run:
  git -C "/usr/local/Homebrew/Library/Taps/homebrew/homebrew-cask" fetch --unshallow
... explanatory text ...
$ git -C "/usr/local/Homebrew/Library/Taps/homebrew/homebrew-cask" fetch --unshallow
<<git stuff>>
... now it's ready ...

The first message doesn't make it clear that there would be more than one command to run, the "call to action" suggests that one would be able to run brew update after running the given git command, which makes the subsequent error a bit surprising. It wasn't easy to spot the difference in the second error message at a glance (that it's -cask not -core), and I think others have had the same confusion.

brew update internally probably checks off core, cask, and probably other repos one-by-one, and halts at the first error. However, since brew update is now the unified interface to update everything, I think in terms of user experience it would be better if the error message could provide all the steps needed to be taken in one place, if possible (i.e., brew update can check if user is using just -core or more and update the git commands accordingly). I'm not sure if this would be possible with one git command or not, but even something like this would be a bit clearer in my opinion:

$ brew update
Error: Homebrew is using shallow clones. To continue, you'll need to use full clones. <<explanatory text>>.
Please run the following commands to do this:
* git -C "/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core" fetch --unshallow
* git -C "/usr/local/Homebrew/Library/Taps/homebrew/homebrew-cask" fetch --unshallow

[gromgit]

That was commited and released 4 days ago.

[iamitava]

Ah, I tried the brew update yesterday, not sure why it didn't pick that up!

[jm9151821]

Oh wow, thank you very much!!!!!!

…

On Wed, Dec 30, 2020, 2:14 PM Rylan Polster ***@***.***> wrote: I've modified the error message slightly to mention that the command will take a few minutes to run. This change will be available to users after the next Homebrew release. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#226 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHXQ654WMESV7FBYV3RWBEDSXOCY3ANCNFSM4UOIB2SA> .

[dkroehan]

Wouldn't the partial clone (blobless / treeless clone) of git be a better solution? Here is a fairly new github blog post that describes it:
https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/

[dkroehan]

Thx for the details!

[melsman]

Should I really setup travis CI to run these commands on every checkput before the script runs a "brew update"?

[SMillerDev]

Until Travis unshallows the brew install in their macOS image, unfortunately yes.

[kruzus]

Hello, I am kinda of new here. Just had a question regarding to this issue, what should I do if i need to update my brew in the future? Do I run these?:
git -C /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core fetch --unshallow
git -C /usr/local/Homebrew/Library/Taps/homebrew/homebrew-cask fetch --unshallow

Until new options arrive?

Thanks!

[SMillerDev]

You run brew update