New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

`brew cleanup` should be run automatically #4760

Closed
MikeMcQuaid opened this Issue Aug 27, 2018 · 31 comments

Comments

9 participants
@MikeMcQuaid
Copy link
Member

MikeMcQuaid commented Aug 27, 2018

One of the most common things we hear on our Twitter account is people saying that they are pleasantly surprised that brew cleanup cleans up so much space. We've held off running it automatically in the past as it's not granular enough to be effectively scoped to a package and because we're concerned about linkage to old versions.

We now have a cached linkage database and the HOMEBREW_UPGRADE_CLEANUP variable which cleans up a bunch more things related to a given formula which means we can start to consider having HOMEBREW_UPGRADE_CLEANUP run on every upgrade/install by default after ensuring that linkage won't be broken.

@chdiza

This comment has been minimized.

Copy link
Contributor

chdiza commented Aug 27, 2018

Will that shellvar still exist and be able to be turned off?

@MikeMcQuaid

This comment has been minimized.

Copy link
Member Author

MikeMcQuaid commented Aug 27, 2018

@chdiza If someone can make a pitch as to why it should be toggleable off: I don't see a reason why not.

@chdiza

This comment has been minimized.

Copy link
Contributor

chdiza commented Aug 27, 2018

Because it is a disaster for anyone who wants to keep previous versions of things around. And keeping previous versions around (at least for really important stuff) is good practice, because breakage is common (here I mainly mean breakage in upstream projects, not HB breakage---though that happens sometimes too). There's a reason why brew switch exists.

To lose this fundmental ability---part of managing packages---because doing normal maintenance doesn't occur to some people would be a real shame.

Come to think of it, if such a non-toggleable change were to occur---and were this to mean that I can't carry e.g., wget 1.19.4 and 1.19.5 for a while until I'm sure that 1.19.5 fixed a problem with 1.19.4, or for something more mission-critical carry rsync 3.1.2 for a while until I'm sure that 3.1.3 won't ruin my life---I think that might mean the end of Homebrew for me.

It's also bad for anyone who tries to keep their HOMEBREW_CACHE in sync across multiple machines, but this is a much smaller concern.

There are ways to keep the aforementioned ability, like creating something akin to pin but that simply means "please don't automatically throw this away", but I'm sure they would involve much more code than adding the toggle would.

@MikeMcQuaid

This comment has been minimized.

Copy link
Member Author

MikeMcQuaid commented Aug 28, 2018

And keeping previous versions around (at least for really important stuff) is good practice

I think it is for you as you file good issues. I don't think it is for the majority of Homebrew users, particularly those who don't even know brew cleanup exists.

To lose this fundmental ability---part of managing packages

Note: the vast majority of package managers don't keep old versions around indefinitely.

but I'm sure they would involve much more code than adding the toggle would.

Agreed, you've convinced me there should be an opt-out toggle for this.

@chdiza

This comment has been minimized.

Copy link
Contributor

chdiza commented Aug 28, 2018

Agreed, you've convinced me there should be an opt-out toggle for this.

Wonderful! :)

Note: the vast majority of package managers don't keep old versions around indefinitely.

I would be surprised if they immediately force removal of old ones with no way to override this. If they do, they're doing it wrong :)

Side-note, now that the toggle will stay: Whether keeping things around (at least for a while) is good practice or not isn't a function of the savviness of the user, but rather of the likelihood that some upstream will break something in a newer version. If e.g., wget 1.19 breaks something, then no matter how good or bad an issue-writer the user is, she will wish she still had the 1.18 keg to brew switch to.

I've been Brewing At Home since late 2012, and I think this has happened to me at least 3 times per year. If I were some sort of Homebrew Instructor, I would tell my students to never run brew cleanup without using the -n flag first, lest they land the aforementioned pickle. If there is something in that output that they're OK with trashing, but also something they don't want to trash yet, they can manually pick out the trashable ones.

@MikeMcQuaid

This comment has been minimized.

Copy link
Member Author

MikeMcQuaid commented Aug 28, 2018

I would be surprised if they immediately force removal of old ones with no way to override this. If they do, they're doing it wrong :)

They do. apt-get dist-upgrade on Debian replaces an old version with a new one.

If e.g., wget 1.19 breaks something, then no matter how good or bad an issue-writer the user is, she will wish she still had the 1.18 keg to brew switch to.

Again, this doesn't apply to you, but I'd argue getting that version update right is the responsibility of the maintainers rather than the user.

@cig0

This comment has been minimized.

Copy link

cig0 commented Sep 19, 2018

In addition to:

@chdiza If someone can make a pitch as to why it should be toggleable off: I don't see a reason why not.

I just hit this issue earlier today: #4942

@MikeMcQuaid

This comment has been minimized.

Copy link
Member Author

MikeMcQuaid commented Sep 20, 2018

@cig0 That's unrelated to this issue.

@maxnordlund

This comment has been minimized.

Copy link
Contributor

maxnordlund commented Sep 24, 2018

I've been running brew update/upgrade/cleanup in a script for years in order to conserve space, and it works like a charm. This would be a nice change 👍

The only time I had to reinstall an old version is when Postgres updates in order to run its database upgrade tool. Now days we have brew postgresql-upgrade-database, which I really appreciate. Having cleanup run automatically shouldn't break that. So perhaps the current, and immediate last version should be kept?

That way the user can switch/pin fairly easily if something goes wrong. Another option would be to cleanup that last version on a second run, to allow for tools like postgresql-update-database, but it might be too confusing behavior.

@MikeMcQuaid

This comment has been minimized.

Copy link
Member Author

MikeMcQuaid commented Sep 24, 2018

The only time I had to reinstall an old version is when Postgres updates in order to run its database upgrade tool. Now days we have brew postgresql-upgrade-database, which I really appreciate. Having cleanup run automatically shouldn't break that.

That script will use the versioned postgresql formulae so should be fine 👍. We may need to make a note to always have the previous version of postgresql around when we do major upgrades for that case alone.

@MikeMcQuaid MikeMcQuaid added this to Help Wanted in Homebrew TODO Oct 3, 2018

@Wojonatior

This comment has been minimized.

Copy link
Contributor

Wojonatior commented Oct 8, 2018

I'd love to help with this! I just wanted to clarify I know what's wanted out of this first. Is the intent of this to run a brew cleanup on install and update unless HOMEBREW_UPGRADE_CLEANUP is set to false?

@sjackman

This comment has been minimized.

Copy link
Member

sjackman commented Oct 8, 2018

Could it be run after brew install, so that the result of brew install happens quickly without holding up the user? So the order of operations is: user runs brew install hello, Homebrew runs brew update automatically, then brew install hello, then brew cleanup.

@Wojonatior

This comment has been minimized.

Copy link
Contributor

Wojonatior commented Oct 8, 2018

Yeah, I follow that. I'll try to get to it this week, and I'll have a status update by the weekend.

@MikeMcQuaid

This comment has been minimized.

Copy link
Member Author

MikeMcQuaid commented Oct 8, 2018

@GauthamGoli was also planning on working on this. @GauthamGoli can you let us know what you've done already (if anything)? @Wojonatior I still encourage you to work on this.

I'd suggest that we run the same cleanup that's currently on HOMEBREW_UPGRADE_CLEANUP for install, upgrade and reinstall if HOMEBREW_INSTALL_CLEANUP it set. This could be extended to do a fuller cleanup with perhaps another variable being set after a certain number of brew installs.

@Wojonatior

This comment has been minimized.

Copy link
Contributor

Wojonatior commented Oct 16, 2018

Hey all, I haven't made any progress on this yet. I have some time set aside this week. But if someone else starts on this, just let me know.

@GauthamGoli

This comment has been minimized.

Copy link
Member

GauthamGoli commented Oct 17, 2018

@Wojonatior Feel free to open a PR. I haven't started working on this YET.

@MikeMcQuaid

This comment has been minimized.

Copy link
Member Author

MikeMcQuaid commented Oct 17, 2018

Folks: in general we don't "assign" issues to anyone until there is an open PR so you don't need to keep us up to date here; just open a PR if you can or don't 😁

@Wojonatior

This comment has been minimized.

Copy link
Contributor

Wojonatior commented Oct 23, 2018

Hey all, I spent some time working on this over the weekend. I want to get a test in place for the current upgrade functionality. I'm unable to get a working test with the environment variables for that. Any guidance?

@MikeMcQuaid

This comment has been minimized.

Copy link
Member Author

MikeMcQuaid commented Oct 23, 2018

@Wojonatior Open a PR as-is; it'll be easier to discuss code than something more abstract.

@Wojonatior

This comment has been minimized.

Copy link
Contributor

Wojonatior commented Oct 25, 2018

@MikeMcQuaid Is the ENV intended to be setup as an opt-out as originally discussed, or as an opt-in, as the previous ENV was setup?

@MikeMcQuaid

This comment has been minimized.

Copy link
Member Author

MikeMcQuaid commented Oct 26, 2018

@Wojonatior Opt-in for now.

MikeMcQuaid added a commit to MikeMcQuaid/brew that referenced this issue Jan 2, 2019

cleanup: run and replace `brew prune`.
It's always seemed a bit pointless to me that we have both of these
commands. Given we're doing more and more to recommend (and eventually,
safely, automatically run (see Homebrew#4760) `brew cleanup` let's roll their
functionality into a single command.

@MikeMcQuaid MikeMcQuaid referenced this issue Jan 2, 2019

Merged

cleanup: run and replace `brew prune`. #5467

6 of 6 tasks complete

MikeMcQuaid added a commit to MikeMcQuaid/brew that referenced this issue Jan 2, 2019

cleanup: run and replace `brew prune`.
It's always seemed a bit pointless to me that we have both of these
commands. Given we're doing more and more to recommend (and eventually,
safely, automatically run (see Homebrew#4760) `brew cleanup` let's roll their
functionality into a single command.
@MikeMcQuaid

This comment has been minimized.

Copy link
Member Author

MikeMcQuaid commented Jan 2, 2019

I've implemented the last piece of this in #5472. HOMEBREW_INSTALL_CLEANUP will opt you in to this behaviour when it is merged and HOMEBREW_NO_INSTALL_CLEANUP will opt you out forever (even after it becomes default/opt-out).

Reanswering previous questions:

Will that shellvar still exist and be able to be turned off?

Yes using HOMEBREW_NO_INSTALL_CLEANUP.

Now days we have brew postgresql-upgrade-database, which I really appreciate. Having cleanup run automatically shouldn't break that.

This script now only uses versioned formulae (whose presence is enforced with an audit) and now old versions (as there were linkage issues) so will not be broken by this behaviour.

Could it be run after brew install, so that the result of brew install happens quickly without holding up the user? So the order of operations is: user runs brew install hello, Homebrew runs brew update automatically, then brew install hello, then brew cleanup.

This is the implemented behaviour.

@knowtheory

This comment has been minimized.

Copy link

knowtheory commented Jan 29, 2019

Hi there,

I seem to have been opted into this automatically and without any notification.

Consequently i'm going to have to spend some time reinstalling a number of formulae which i need to keep and switch between. This has ruined my evening and possibly some deadlines, so i would like to strongly advocate that the behavior is changed to at least prompt users for confirmation the first time this behavior happens.

Also, as ever, thank you for Homebrew, and <3 to the maintainers.

@MikeMcQuaid

This comment has been minimized.

Copy link
Member Author

MikeMcQuaid commented Jan 29, 2019

so i would like to strongly advocate that the behavior is changed to at least prompt users for confirmation the first time this behavior happens.

Sorry, no. Homebrew does not prompt the user. Changing this now would break scripts that rely on this assumption.

This behaviour was announced on our Twitter, our 1.9.0 release notes and man brew. man brew provides details for opting out of brew update going to non-stable tags and this cleanup behaviour.

Also, as ever, thank you for Homebrew, and <3 to the maintainers.

Thanks for the kind words ❤️

@MikeMcQuaid

This comment has been minimized.

Copy link
Member Author

MikeMcQuaid commented Jan 29, 2019

Closing this issue as it has now been implemented for those on the master branch. Set HOMEBREW_NO_INSTALL_CLEANUP to opt-out of this behaviour.

Homebrew TODO automation moved this from Help Wanted to Done Jan 29, 2019

@abraithwaite

This comment has been minimized.

Copy link

abraithwaite commented Jan 29, 2019

Ouch. Guess I should read the release notes. I frequently go back and forth between versions of software in a rolling release model.

Glad there's a flag at least! Would strongly urge a prompt for future changes of this nature though. Even if it does break scripts, I believe it's better than having your entire version cache wiped out.

Perhaps if it were easier to re-install an old version of a formula once it's been removed, then it wouldn't be such an issue. Unless I'm mistaken you still have to do the git song and dance to install older versions of software (unless that version also has a tap). I'd be delighted to learn otherwise. I did not see anything in the man page.

@knowtheory

This comment has been minimized.

Copy link

knowtheory commented Jan 29, 2019

Sorry, no. Homebrew does not prompt the user. Changing this now would break scripts that rely on this assumption.

This behaviour was announced on our Twitter, our 1.9.0 release notes and man brew. man brew provides details for opting out of brew update going to non-stable tags and this cleanup behaviour.

It's unfortunate that the cat's out of the bag, but this is something worth keeping in mind for future. From my perspective as a user, i updated homebrew (which is what i'm supposed to do), installed something, and suddenly the tool's behavior has changed in a way that i didn't expect, runs automatically, and the only way i would have known what's happening is if i followed Homebrew on twitter, or if i happened to be perusing the man pages.

There's no opportunity in there for me to have known to configure my environment differently.

This is obviously all open source software, and maintained by lovely people who enable my life to be much smoother than it'd otherwise be, so, like, i guess i'll just keep a more careful eye out in future. But this feels like a shift from my POV. Before I was making sure to continue to brew update because i needed to keep my formulae up to date (and it happened to be that tooling updates came along with that). This issue has made it clear that before i do anything with a brew, i need to first make sure that i know what's going on in homebrew's release notes, and proceed with the knowledge that any possible brew command could pull down new and different tooling behavior. 🤷‍♂️

❤️

@MikeMcQuaid

This comment has been minimized.

Copy link
Member Author

MikeMcQuaid commented Jan 29, 2019

There's no opportunity in there for me to have known to configure my environment differently.

Well, there was but without you reading the release notes, Twitter or GitHub: no, there was not.

proceed with the knowledge that any possible brew command could pull down new and different tooling behavior

Not any brew command but: yes, brew install, brew upgrade also previously would always run an auto-update and they now also run a cleanup. You are safe in the assumption that brew --prefix will not do that.

@MikeMcQuaid

This comment has been minimized.

Copy link
Member Author

MikeMcQuaid commented Jan 29, 2019

This is obviously all open source software, and maintained by lovely people who enable my life to be much smoother than it'd otherwise be

Thanks for the kind words. They genuinely mean a lot ❤️

@sjackman

This comment has been minimized.

Copy link
Member

sjackman commented Jan 29, 2019

@abraithwaite wrote…

Perhaps if it were easier to re-install an old version of a formula once it's been removed, then it wouldn't be such an issue. Unless I'm mistaken you still have to do the git song and dance to install older versions of software (unless that version also has a tap). I'd be delighted to learn otherwise. I did not see anything in the man page.

You can install older bottles like so:
Find the bottle on Bintray: https://bintray.com/homebrew/bottles/coreutils
Install the bottle:
brew install https://homebrew.bintray.com/bottles/coreutils-8.29.high_sierra.bottle.tar.gz

@abraithwaite

This comment has been minimized.

Copy link

abraithwaite commented Jan 30, 2019

@sj26 Neat, thank you!

@Homebrew Homebrew deleted a comment from jackmaney Feb 4, 2019

@Homebrew Homebrew locked as resolved and limited conversation to collaborators Feb 4, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.