New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove all options from Homebrew/homebrew-core formulae #31510

Closed
MikeMcQuaid opened this Issue Aug 27, 2018 · 65 comments

Comments

@MikeMcQuaid
Copy link
Member

MikeMcQuaid commented Aug 27, 2018

Options in formulae don't produce a good user experience because they have to be built from source, we don't test them in CI and each combination of options provides a new chance for new failures to occur. We should seek to (eventually!) remove all options from formulae in Homebrew/homebrew-core in favour of enabling as much non-exclusive functionality as possible in a given formula for widely used options or encouraging the community to maintain their own custom options in a tap (e.g. https://github.com/denji/homebrew-nginx/blob/master/Formula/nginx-full.rb). As an absolute last resort if we need to depend on the same formula multiple times with different options (e.g.
#13133) we can consider vendoring formulae using resource blocks or even duplicating formulae.

This should be done in multiple phases:

  1. formulae that have build error issues created with particular options should have the options removed immediately (rather than a fix being merged)
  2. less popular formulae should have all their options removed
  3. popular formulae (e.g. https://formulae.brew.sh/analytics/install-on-request/90d/) should be prioritised for having their defaults adjusted (depending on analytics usage) to get the best balance of functionality without options.

I'm open to thoughts on the above and different approaches to solve the problem as well as additional tooling required in Homebrew/brew to accomplish this.

@jhrmnn

This comment has been minimized.

Copy link
Contributor

jhrmnn commented Aug 27, 2018

What about a more general approach akin to Debian's alternatives system? That could handle the issue with options as well as other things.

One thing that I've always missed in Homebrew was being able to pick an implementation of MPI: Open MPI vs Mpich. Currently when one installs Mpich, packages depending on MPI (and so on Open MPI in Homebrew's logic), such as Scalapack, have to be built manually, else Homebrew complains about missing dependencies.

@chdiza

This comment has been minimized.

Copy link
Contributor

chdiza commented Aug 27, 2018

Options in formulae don't produce a good user experience because they have to be built from source, we don't test them in CI and each combination of options provides a new chance for new failures to occur.

Those things do happen with options, but I don't think it follows that eliminating options provides a better user experience. A formula that lacks needed options means the user makes their own tap, in which case...it will be built from source, not tested by CI, and provides new chances for failure. I.e., those three things that an option-using user currently faces will still be faced just about as often once options are eliminated.

@MikeMcQuaid

This comment has been minimized.

Copy link
Member

MikeMcQuaid commented Aug 27, 2018

What about a more general approach akin to Debian's alternatives system? That could handle the issue with options as well as other things.

We can't really do this given our limited CI and options aren't a good approach for this. "Picking an implementation" isn't really something Debian supports for compile-time things; as far as I'm away that only applies for things that can be swapped out underneath (i.e. are API and ABI compatible) without an application change. This is theoretically possible as something we could do but mostly people are requesting the ability to swap out things that aren't necessarily compatible.

Those things do happen with options, but I don't think it follows that eliminating options provides a better user experience. A formula that lacks needed options means the user makes their own tap, in which case...it will be built from source, not tested by CI, and provides new chances for failure. I.e., those three things that an option-using user currently faces will still be faced just about as often once options are eliminated.

I think a less flexible Homebrew/homebrew-core that works for every user every time is superior to a more flexible Homebrew/homebrew-core that works for every user 90% of the time. That's a product/design difference of opinion I realise we don't share.

Essentially, though, you touch upon a good point: for things that have to be built from source or build using options there's no real difference between Homebrew/homebrew-core and a tap that doesn't provide bottles. As a result it's much, much easier for the community to step up there and provide solutions to these problems without adding the negative perception when Homebrew/homebrew-core doesn't work and the issues that result from hard to reproduce, option, from source builds.

@commitay commitay referenced this issue Aug 29, 2018

Closed

carla 1.9.9 (new formula) #31560

4 of 4 tasks complete

tresf added a commit to tresf/homebrew-core that referenced this issue Aug 29, 2018

BrewTestBot added a commit to BrewTestBot/homebrew-core that referenced this issue Aug 29, 2018

BrewTestBot added a commit to BrewTestBot/homebrew-core that referenced this issue Aug 29, 2018

BrewTestBot added a commit to BrewTestBot/homebrew-core that referenced this issue Aug 29, 2018

@sjackman sjackman referenced this issue Aug 29, 2018

Closed

libfuse 2.9.8 #9149

4 of 5 tasks complete
@zbeekman

This comment has been minimized.

Copy link
Member

zbeekman commented Sep 4, 2018

I think a less flexible Homebrew/homebrew-core that works for every user every time is superior to a more flexible Homebrew/homebrew-core that works for every user 90% of the time. That's a product/design difference of opinion I realise we don't share.

I personally tend to agree with @chdiza. (Although I'm willing to concede that I may be in the minority here, and I may not have had the extensive experience that led @MikeMcQuaid to those conclusions.) However, if we go this route, I think we need to greatly improve our documentation for setting up dedicated taps, with all the bells and whistles like bottling etc. I was looking into this a few months ago, and there is no straightforward guide on how to accomplish this. I finally stumbled upon this repository, which seemed like the best example for me to follow. Better documentation for users/contributors of the bottling process would make this type of migration much less of a bitter pill for some (like me!) to swallow.

@MikeMcQuaid

This comment has been minimized.

Copy link
Member

MikeMcQuaid commented Sep 5, 2018

I think we need to greatly improve our documentation for setting up dedicated taps, with all the bells and whistles like bottling etc. I was looking into this a few months ago, and there is no straightforward guide on how to accomplish this.

The current documentation (e.g. https://docs.brew.sh/How-to-Create-and-Maintain-a-Tap and the brew tap-new command in the manpage) is sufficient for many people to setup taps. If you would like to extend that documentation: please feel free.

with all the bells and whistles like bottling etc.

This doesn't really make sense; options are never bottled anyway so to remove them doesn't imply we need to provide an ability for others to bottle them.

Better documentation for users/contributors of the bottling process would make this type of migration much less of a bitter pill for some (like me!) to swallow.

The issue isn't documentation, really, it's setting up free building of macOS packages (on multiple OS versions) with free uploads to a free CDN. I don't think that problem is ours to solve, really, but again: if you're interested in solving it please feel free.

@zbeekman

This comment has been minimized.

Copy link
Member

zbeekman commented Sep 5, 2018

@DomT4

This comment has been minimized.

Copy link
Contributor

DomT4 commented Sep 5, 2018

For the record: I think the mass removal of options is patently ridiculous & achieves nothing except stymying debate about individual options because those PRs are too large to easily discuss just one or two elements.

I also think it is ridiculous to move ahead with this after 10 days of this issue being opened, when precisely two maintainers thus far have weighed in on the subject, and the core team as a whole was essentially rebuilt from the ground up less than one month ago.

Some of these options also have effectively zero maintenance burden, or have never been knowingly problematic in terms of either analytic-logged build failures or human reports, and yet get removed anyway. Other options were the product of days or weeks of discussions about how best to resolve other issues, like Perl binding versioning & incompatibility, and have been casually thrown out the window with zero debate.

I truly tip my hat to the "Ram the change through as rapidly as possible and hope most people, including maintainers, don't notice or object until it's too late" tactic.

@zbeekman

This comment has been minimized.

Copy link
Member

zbeekman commented Sep 5, 2018

@DomT4

This comment has been minimized.

Copy link
Contributor

DomT4 commented Sep 5, 2018

We’re all passionate here and care deeply about Homebrew. As far as I can tell no action has been taken yet.

#31847, #31830 & #31799. That's a lot of "no action".

@zbeekman

This comment has been minimized.

Copy link
Member

zbeekman commented Sep 5, 2018

#31847, #31830 & #31799. That's a lot of "no action".

Fair. I'm on the train and my internet connection is terrible, so I didn't do due diligence before I made that statement.

While I personally like options, and find that they add value to Homebrew for power users or devs who need a certain package built a certain way, I do see that there is a trade off and it's not a one sided issue. If it were put to a vote (I'm NOT saying the it necessarily should be...) I would vote to keep --devel and --HEAD options, and want to evaluate other packages on a case by case basis.

But anyway, I'd be curious to hear what other from @Homebrew/core think, if they have a strong opinion. There's a fairly good chance that they all agree with Mike which is why they didn't feel the need to comment here... 🤷‍♂️

@sjackman

This comment has been minimized.

Copy link
Contributor

sjackman commented Sep 5, 2018

On the whole I'm not a fan of options for the reasons cited by Mike. I do like --HEAD, because it can be useful when troubleshooting upstream bugs. I don't see much benefit in going to the effort of removing options that have caused no documented trouble, either in analytics or issues. I'm in favour of removing options that have documented trouble. I'm in favour of adding no new options to formulae.

BrewTestBot added a commit to BrewTestBot/homebrew-core that referenced this issue Sep 6, 2018

BrewTestBot added a commit to BrewTestBot/homebrew-core that referenced this issue Sep 6, 2018

@fxcoudert

This comment has been minimized.

Copy link
Member

fxcoudert commented Sep 6, 2018

@DomT4

mass removal of options

I'll be holding the open PR, and not opening new ones. I apologise if you felt those were rushed: it clearly was not my intention: I just got a little time to do cleaning-type stuff while I was sat at a conference, and took it. Given that: 1. the PR was open for more than a week, 2. it doesn't look much like a call for discussion, but a plan for action, and 3. no maintainer has objected, I figured it was OK to move forward.

I've tried to remove options from formulas with very few install counts, i.e. options removed should be lower than 0.5% in usage or lower than 5 in absolute install count (per 30 days).

@fxcoudert

This comment has been minimized.

Copy link
Member

fxcoudert commented Sep 6, 2018

As to my personal opinion: apart from a few formulas where they are "natural" (aspell, some video or audio players), options run counter to our main goal, i.e. “just working”, for the arguments given by Mike. If we want to keep many options, I'd like us to consider running all option combinations through CI (which limits their number) or officially declare them “semi-supported” (i.e., we remove them when they break, unless someone patches it).

@claui

This comment has been minimized.

Copy link
Member

claui commented Sep 6, 2018

I feel that keeping things lean and simple is a good thing. Also, existing documentation on maintaining a user tap is just fine.

What we could do better is advertise the user tap feature more often. As in, whenever options pop up as a topic in a PR.

@MikeMcQuaid

This comment has been minimized.

Copy link
Member

MikeMcQuaid commented Sep 6, 2018

I truly tip my hat to the "Ram the change through as rapidly as possible and hope most people, including maintainers, don't notice or object until it's too late" tactic.
I don’t think bristly accusations are warranted

@DomT4 I strongly agree with @zbeekman: this was not an acceptable way to refer to the work of other maintainers such as @fxcoudert who received a PR approval before merging based on an open "help wanted" issue.

I also think it is ridiculous to move ahead with this after 10 days of this issue being opened, when precisely two maintainers thus far have weighed in on the subject
There is a degree of obligation on us maintainers to follow these proposals and issue tickets and voice our opinions.

Again, I agree with @zbeekman: this issue was open for 10 days in which you have been active on this issue tracker and in Slack and you had no comments on it during that time. You've also (re)joined the team more recently than @fxcoudert; he has no obligation to ask you for permission to make a change like this.

I'll be holding from the open PRs. I apologise if you felt those were rushed: it clearly was not my intention: I just got a little time to do cleaning-type stuff while I was sat at a conference, and took it. Given that: 1. the PR was open for more than a week, 2. it doesn't look much like a call for discussion, but a plan for action, and 3. no maintainer has objected, I figured it was OK to move forward.

I've tried to remove options from formulas with very few install counts, i.e. options removed should be lower than 0.5% in usage or lower than 5 in absolute install count (per 30 days).

You have nothing to apologise for @fxcoudert. It's worth noting we've not been accepting options on new formula for a long time now. When we imported formulae from taps (many of which were in our top 1000 and sometimes top 100) we've removed all options in doing so. This has resulted in dramatically fewer build errors and those formulae being easier to support.

To be extra explicit: this has been an undocumented but planned policy for several years.

What we could do better is advertise the user tap feature more often. As in, whenever options pop up as a topic in a PR.

@claui I agree.

Options in formulae don't produce a good user experience because they have to be built from source, we don't test them in CI and each combination of options provides a new chance for new failures to occur.

@fxcoudert Is the only person in this thread to even offer a suggestion of how to resolve any of these issues (run them all through CI). If someone points out an engineering problem like this: suggest an alternative solution.

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

openblas: remove options.
See Homebrew#31510.

Closes Homebrew#36283.

Signed-off-by: FX Coudert <fxcoudert@gmail.com>

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

tesseract: remove options.
See Homebrew#31510.

Closes Homebrew#36293.

Signed-off-by: FX Coudert <fxcoudert@gmail.com>

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

freeswitch: remove options.
See Homebrew#31510.

Closes Homebrew#36266.

Signed-off-by: FX Coudert <fxcoudert@gmail.com>

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

gdal: remove options.
See Homebrew#31510.

Closes Homebrew#36267.

Signed-off-by: FX Coudert <fxcoudert@gmail.com>

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

poppler: remove options.
See Homebrew#31510.

Closes Homebrew#36285.

Signed-off-by: Mike McQuaid <mike@mikemcquaid.com>

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

zim: remove options.
See Homebrew#31510.

Closes Homebrew#36300.

Signed-off-by: Mike McQuaid <mike@mikemcquaid.com>

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

jmeter: remove options.
See Homebrew#31510.

Closes Homebrew#36274.

Signed-off-by: FX Coudert <fxcoudert@gmail.com>

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

vim@7.4: remove options.
See Homebrew#31510.

Closes Homebrew#36295.

Signed-off-by: FX Coudert <fxcoudert@gmail.com>

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

arpack: remove options.
See Homebrew#31510.

Closes Homebrew#36261.

Signed-off-by: Mike McQuaid <mike@mikemcquaid.com>

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

graphviz: remove options.
See Homebrew#31510.

Closes Homebrew#36271.

Signed-off-by: Mike McQuaid <mike@mikemcquaid.com>

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

weechat: remove options.
See Homebrew#31510.

Closes Homebrew#36296.

Signed-off-by: Mike McQuaid <mike@mikemcquaid.com>

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

libav: remove options.
See Homebrew#31510.

Closes Homebrew#36275.

Signed-off-by: FX Coudert <fxcoudert@gmail.com>

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

mplayer: remove options.
See Homebrew#31510.

Closes Homebrew#36280.

Signed-off-by: Mike McQuaid <mike@mikemcquaid.com>

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

graphicsmagick: remove options.
See Homebrew#31510.

Closes Homebrew#36270.

Signed-off-by: Mike McQuaid <mike@mikemcquaid.com>

niheaven added a commit to niheaven/homebrew-core that referenced this issue Jan 23, 2019

fftw: remove options.
See Homebrew#31510.

Closes Homebrew#36265.

Signed-off-by: Mike McQuaid <mike@mikemcquaid.com>
@MikeMcQuaid

This comment has been minimized.

Copy link
Member

MikeMcQuaid commented Jan 23, 2019

I understand your reasons to do this, but there are some formulae which are even harder to use if you don't have options to modify them, for example yarn. Its only option was --without-node which was very helpful for people who are using nvm.

@MekliCZ This applies to many things installed by Homebrew: if /usr/local/bin is first in your path it will override the binaries elsewhere. I'm not sure I understand why putting nvm's nodes before /usr/local/bin in your PATH doesn't fix this?

In short: I'm not claiming that removing options will not require workflow changes for some users. Where we/I can I want to make those workflow changes as obvious an easy as possible.

And as far as I know, the option only removed a dependency which isn't required to successfully install yarn.

It is needed to run or test yarn, though, and it makes us a pretty crappy package manager if we don't install things you need to run the application you install...

It looks like you're not taking any arguments why NOT to remove all options from all formulae

This thread is full of arguments and I unlocked it. What I've said to many people (though) is that they needed to provide solutions to the problems which motivated this change. No-one did provide solutions beyond essentially "just ignore those support requests" so: here we are.

which is the reason I'm moving to macports

I hope it's a better fit for you. If this is your main issue, though, it seems strange. MacPorts also installs node when you install yarn and doesn't provide a variant for this. It's been a while since I used MacPorts so I could be missing something they offer that we don't, here.

@blogabe

This comment has been minimized.

Copy link

blogabe commented Jan 23, 2019

@MikeMcQuaid can't speak for the overall community of course, but a challenge with coming up with ideas that fit the maintainers requirements is that that outside the "lessen the support issues" request, I'm unsure what other constraints we should abide by. I'm using the word 'constraints' specifically to imply maintaining some level of playing well with one another versus going off and doing something completely different.

There are enough people impacting by these changes that it may make sense to work with us and provide some guidance on how we can continue using homebrew and have it support our needs. Creating an independent tap is one of them, and I do not see that as a problem.

What I do see as a problem is now having potentially multiple users creating a tap for a particular formula, say imagemagick, to support options. I would think this is a popular enough formula where we'de want to centralize the hosting rather than have multiple options.

Furthermore, using Google to find imagemagick places an unnecessary burden on the user and may come up with varying results.

This project has started to potentially reach a happy medium: https://github.com/portage-brew

I think a centralized, rather than decentralized, approach to a tap allowing for formula options is a way to go.

Ideally, updates to formulae would be made in one place and carried forward, but I'm unsure how realistic that is. I can say from my own personal experience that as I know host formula w/ options on my own tap, updating the official formula on homebrew is a lesser priority, e.g., adding a revision or bumping the version.

How do you or other maintainers suggest we work together to address? This isn't about maintainers vs power users... it should be about the the overall user community, those that are fine with bottles and those that like to tinker.

@MikeMcQuaid

This comment has been minimized.

Copy link
Member

MikeMcQuaid commented Jan 23, 2019

Creating an independent tap is one of them, and I do not see that as a problem.

Yes, this is the expected model and we want to make that easier.

I would think this is a popular enough formula where we'de want to centralize the hosting rather than have multiple options.

I agree that discovery is a tricky part. The balance is on discovery vs. expectation that things work by default/are supported by the Homebrew maintainers. Perhaps expanding on https://docs.brew.sh/Interesting-Taps-and-Forks would be a start for this? I don't want us to go back to when brew search searches a bunch of community taps because we don't want to be seen to recommend things that we haven't reviewed.

This project has started to potentially reach a happy medium: https://github.com/portage-brew

I'm happy to see projects like this emerge (pun intended).

Ideally, updates to formulae would be made in one place and carried forward, but I'm unsure how realistic that is.

Unfortunately it's not realistic to expect that updates will happen that won't break the options. Most options require passing something to configure or otherwise modifying the install method. I've suggested before there may be a way to do this (subclass the Formula and override install and options but leave url etc. the same) and I'm open to PRs that make that process easier. Even then, though, I'm unconvinced it will be easier than copying-pasting a formula into a tap.

How do you or other maintainers suggest we work together to address?

Beyond what I've said: I don't, I'm afraid. I'm happy to offer personal opinions but kind of the point of this change is that Homebrew maintainers are no longer responsible for how to go forward with this.

it should be about the the overall user community, those that are fine with bottles and those that like to tinker.

I'm afraid I disagree. There's some users whose desire to tinker causes a disproportionate support burden. They want to fiddle with things until they break, file an issue (or even PR) and then a bunch of time is wasted discussing a self-created issue.

It's not unusual for software that is popular with "tinkerers" to outgrow them as their needs and desires for the project become outliers. This is what has happened with Homebrew: a niche, from-source package manager has become a binary package manager relied on by millions of people.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment