Bottle path substitution when HOMEBREW_REPOSITORY == HOMEBREW_PREFIX #9453

fxcoudert · 2020-12-07T19:52:28Z

the bottle you uploaded for freetype in Homebrew/homebrew-core@fe10e86 seems to have a slight issue with Homebrew path substitutions.
It's using HOMEBREW_REPOSITORY in the pkg-config files instead of HOMEBREW_PREFIX. See below.

3:39:18 ⏚ [:~/Downloads/freetype/2.10.4] % ag @@HOMEBREW_
bin/freetype-config
38:  prefix="@@HOMEBREW_REPOSITORY@@/opt/freetype"
39:  exec_prefix="@@HOMEBREW_REPOSITORY@@/opt/freetype"
41:  includedir="@@HOMEBREW_REPOSITORY@@/opt/freetype/include"
42:  libdir="@@HOMEBREW_REPOSITORY@@/opt/freetype/lib"
48:  static_libs="-lfreetype -lz -lbz2 -L@@HOMEBREW_CELLAR@@/libpng/1.6.37/lib -lpng16 -lz"

lib/pkgconfig/freetype2.pc
1:prefix=@@HOMEBREW_REPOSITORY@@/opt/freetype
2:exec_prefix=@@HOMEBREW_REPOSITORY@@/opt/freetype
3:libdir=@@HOMEBREW_REPOSITORY@@/opt/freetype/lib
4:includedir=@@HOMEBREW_REPOSITORY@@/opt/freetype/include
13:Libs.private: -lz -lbz2 -L@@HOMEBREW_CELLAR@@/libpng/1.6.37/lib -lpng16 -lz

Other builds weren't completing on my machine because of this - I have my /opt/homebrew laid out similarly to Mac x86_64 and Linuxbrew, thus HOMEBREW_PREFIX is /opt/homebrew and HOMEBREW_REPOSITORY is /opt/homebrew/Homebrew.
Not sure if that's a supported configuration, but it felt cleaner to me. 🤷🏻‍♂️

The text was updated successfully, but these errors were encountered:

fxcoudert · 2020-12-07T19:54:22Z

It seems like a bug to me: in .pc files, paths should be with respect to HOMEBREW_PREFIX indeed, not HOMEBREW_REPOSITORY. I'm actually not sure when, if ever, we would want HOMEBREW_REPOSITORY recorded instead of HOMEBREW_PREFIX in a bottle?

@Homebrew/brew do you think this is a bug indeed?

sjackman · 2020-12-07T23:30:56Z

Any chance that the native ARM machine had Homebrew installed using git clone … /opt/homebrew rather than git clone … /opt/homebrew/Homebrew? The former will not work for building bottles. I learned this lesson the hard way, and now I share it with you.

williammck · 2020-12-07T23:34:10Z

Any chance that the native ARM machine had Homebrew installed using git clone … /opt/homebrew rather than git clone … /opt/homebrew/Homebrew? The former will not work for building bottles. I learned this lesson the hard way, and now I share it with you.

@sjackman I had my local machine set up the latter way, but @fxcoudert said in #7857 (comment) that we should have HOMEBREW_PREFIX== HOMEBREW_REPOSITORY on ARM. Personally that felt weird, so I put my stuff in /opt/homebrew/Homebrew as well. Which way is the "right" way?

williammck · 2020-12-07T23:58:04Z

FWIW I tracked down where the issue appears to be coming from...

brew/Library/Homebrew/keg_relocate.rb

Line 76 in e416668

    
           changed = s.gsub!(Regexp.union(replacements.keys.sort_by(&:length).reverse), replacements)

It's running replacements.keys.sort_by(&:length).reverse which, since the HOMEBREW_PREFIX and HOMEBREW_REPOSITORY paths are the same length, may not be deterministic in this scenario.

fxcoudert · 2020-12-08T07:06:05Z

@MikeMcQuaid this seems to be a good argument to HOMEBREW_REPOSITORY != HOMEBREW_PREFIX on CI nodes

claui · 2020-12-08T08:56:15Z

I think HOMEBREW_REPOSITORY != HOMEBREW_PREFIX on /opt/homebrew comes with a long-term price tag.

Every single unnecessary subdirectory adds up over the years, even if it’s just half a second of extra typing. On top of that, that double (but differently-cased) homebrew part is going to confuse people even more. I’m convinced that contributor experience will be significantly better if both paths are == to each other.

fxcoudert · 2020-12-08T09:38:59Z

I’m convinced that contributor experience will be significantly better if both paths are == to each other

But then, how do we resolve in bottles between the two paths, when they are equal? Right now, that means we would completely remove support for people who have HOMEBREW_REPOSITORY != HOMEBREW_PREFIX

MikeMcQuaid · 2020-12-08T12:13:54Z

@MikeMcQuaid this seems to be a good argument to HOMEBREW_REPOSITORY != HOMEBREW_PREFIX on CI nodes

@fxcoudert I disagree, this is a bug and we should fix this. This is how we currently direct people to install Homebrew on ARM, too: https://docs.brew.sh/Installation#untar-anywhere

I’m convinced that contributor experience will be significantly better if both paths are == to each other.

I agree. The /usr/local/Homebrew was done just because so many people complained about having e.g. README.md in /usr/local which is used by not-just-Homebrew.

fxcoudert · 2020-12-08T15:19:22Z

I think this means I will have to remove all ARM bottles already committed and build them again. I currently don't have a good way to know which are affected by this issue, but probably all of them. That's 165 files and I don't want to have to remove them from bintray manually… does anyone know the API well enough to do that?

MikeMcQuaid · 2020-12-08T16:03:25Z

I think this means I will have to remove all ARM bottles already committed and build them again.

@fxcoudert I'm sorry about that. I would suggest this is a learning opportunity that we need to take our time more when shipping bottles for a new architecture that's never had them before; there's going to be more issues still that need to be fixed here.

That's 165 files and I don't want to have to remove them from bintray manually… does anyone know the API well enough to do that?

We can't really just remove them as anyone who has them downloading already will then have a broken checksum. We'll need to revision or bump the rebuild on the formula.

claui · 2020-12-08T16:07:18Z

We can't really just remove them as anyone who has them downloading already will then have a broken checksum. We'll need to revision or bump the rebuild on the formula.

A new test-bot invocation should bump rebuild automatically IIRC if both revision and version are unchanged.

fxcoudert · 2020-12-08T16:56:38Z

@MikeMcQuaid I only committed a “small” number of bottles because I wanted to test them out “in the wild”… the problem was not visible in my original testing, and was only made visible because of another user with a system that has HOMEBREW_REPOSITORY != HOMEBREW_PREFIX

We can't really just remove them as anyone who has them downloading already will then have a broken checksum

We never advertised them as supported, anyway, so I'd prefer that option to revision-bumping those formulas for all other users (when there is no content changed).

A new test-bot invocation should bump rebuild automatically IIRC if both revision and version are unchanged

That's another option, of course

MikeMcQuaid · 2020-12-08T17:37:32Z

@MikeMcQuaid I only committed a “small” number of bottles because I wanted to test them out “in the wild”… the problem was not visible in my original testing, and was only made visible because of another user with a system that has HOMEBREW_REPOSITORY != HOMEBREW_PREFIX

@fxcoudert Ok, can you remove the bottle values ASAP then. As for Bintray: you should be able to hack something together using curl or bits of the Bintray module in Homebrew/brew. When we reintroduce them: let's start with ~10 or fewer bottles until we have more confidence.

fxcoudert · 2020-12-08T18:36:09Z

Bottles gone from homebrew-core and bintray. For the bintray part, in case we ever have to do that again:

for i in `cat lists.txt` ; do
  j=$(curl -s -u${HOMEBREW_BINTRAY_USER}:${HOMEBREW_BINTRAY_KEY} -X GET https://api.bintray.com/packages/homebrew/bottles/$i/files | sed 's/^.*"\([^"]*.arm64_big_sur.bottle.[0-9.]*tar.gz\)".*$/\1/')
  echo $j
  curl -s -u${HOMEBREW_BINTRAY_USER}:${HOMEBREW_BINTRAY_KEY} -X DELETE https://api.bintray.com/content/homebrew/bottles/$j
  sleep 2
done

williammck · 2020-12-08T22:54:38Z

Every single unnecessary subdirectory adds up over the years, even if it’s just half a second of extra typing. On top of that, that double (but differently-cased) homebrew part is going to confuse people even more. I’m convinced that contributor experience will be significantly better if both paths are == to each other.

I agree. The /usr/local/Homebrew was done just because so many people complained about having e.g. README.md in /usr/local which is used by not-just-Homebrew.

I'm just a user here voicing my two cents, but it just feels weird to me to have both a repository root and a standard layout of bin etc include lib opt sbin share var in the same directory. I can understand it being less of an issue with /opt/homebrew, still feels messy to me though. 🤷🏻‍♂️

sjackman · 2020-12-09T05:36:40Z

I'm just a user here voicing my two cents, but it just feels weird to me to have both a repository root and a standard layout of bin etc include lib opt sbin share var in the same directory. I can understand it being less of an issue with /opt/homebrew, still feels messy to me though. 🤷🏻‍♂️

I agree with William here. I would prefer to have /opt/homebrew/Homebrew in its own directory. Users shouldn't be needing to poke around in this directory. I think they'd be better served to have /opt/homebrew kept tidy with bin etc include lib opt sbin share var without the added distraction of CHANGELOG.md CONTRIBUTING.md Dockerfile LICENSE.txt Library README.md SECURITY.md bin completions docs manpages, which are not intended for users.

But then, how do we resolve in bottles between the two paths, when they are equal? Right now, that means we would completely remove support for people who have HOMEBREW_REPOSITORY != HOMEBREW_PREFIX

Homebrew on macOS for Intel and Homebrew on Linux both use HOMEBREW_REPOSITORY != HOMEBREW_PREFIX. I feel that it adds an unnecessary burden on Homebrew developers to now switch to HOMEBREW_REPOSITORY == HOMEBREW_PREFIX for Homebrew on macOS on ARM. Bottles on ARM will be different than other architectures, because there will be no way to tell when to use @@HOMEBREW_PREFIX@@ and when to use @@HOMEBREW_REPOSITORY@@. Additionally, these bottles cannot be used if HOMEBREW_PREFIX != HOMEBREW_REPOSITORY, even the cellar :any ones.

I would like to bring this issue to the attention of the @Homebrew/tsc for review. I recommend not building any bottles for Homebrew on macOS for ARM until this issue has been decided.

fxcoudert · 2020-12-09T06:41:30Z

I recommend not building any bottles for Homebrew on macOS for ARM until this issue has been decided.

This is already the case, we have stopped bottling until this is resolved.

Bottles on ARM will be different than other architectures, because there will be no way to tell when to use @@HOMEBREW_PREFIX@@ and when to use @@HOMEBREW_REPOSITORY@@. Additionally, these bottles cannot be used if HOMEBREW_PREFIX != HOMEBREW_REPOSITORY, even the cellar :any ones.

To me this is the key point. @MikeMcQuaid stated it is possible to discriminate, if possible I don't have an objection. (It is probably possible because each subdirectory is unambiguously either part of the prefix or the repo.)

iMichka · 2020-12-09T06:57:01Z

I agree with what @sjackman said above: #9453 (comment)

We should have HOMEBREW_REPOSITORY != HOMEBREW_PREFIX on all platforms, because this is how it is on the existing ones. /opt/homebrew/Homebrew should be it's own repository.

If we are able to fix the bug, all should be good.

claui · 2020-12-09T08:35:15Z

If we absolutely must go that route, please let’s at least pick something more concise like /opt/homebrew/src instead of /opt/homebrew/Homebrew.

MikeMcQuaid · 2020-12-09T09:05:28Z

Homebrew on macOS for Intel and Homebrew on Linux both use HOMEBREW_REPOSITORY != HOMEBREW_PREFIX.

By default the installer installs to this location on macOS and Intel, correct. However:

this was not always the case
this change was made on macOS only because people requested that /usr/local is not cluttered up. I had no input on Linuxbrew's choice here and, if asked, I would have argued against.
the "alternate install" that's currently the default install on ARM doesn't do this https://docs.brew.sh/Installation#untar-anywhere

Given 3. we need to make these changes anyway.

I feel that it adds an unnecessary burden on Homebrew developers to now switch to HOMEBREW_REPOSITORY == HOMEBREW_PREFIX for Homebrew on macOS on ARM.

I am volunteering to do the work to fix this.

Additionally, these bottles cannot be used if HOMEBREW_PREFIX != HOMEBREW_REPOSITORY, even the cellar :any ones.

Yes. This is not a problem provided we add the necessary checks.

I would like to bring this issue to the attention of the @Homebrew/tsc for review. I recommend not building any bottles for Homebrew on macOS for ARM until this issue has been decided.

This is not an objective discussion being brought to the TSC. This is the TSC being brought in to simply overrule me and the work I've been doing.

I was going to ensure that I fixed the relevant issues today but it seems now that I should instead wait for the TSC to make a decision before I do any more work on this, it seems.

This is not a productive way for an open source project powered by volunteer time and motivation to function.

fxcoudert · 2020-12-09T09:19:28Z

I agree this is not a good way to involve the TSC. There was a discussion about prefix choice on ARM, it concluded with a decision that is currently documented and that leads to HOMEBREW_PREFIX == HOMEBREW_REPOSITORY. Those points were not raised and we are now implementing that decision, with significant work already being done in that direction.

Three of the main actors involved in this work (@MikeMcQuaid @claui and I) favour the situation we are now implementing (even though I was not originally entirely convinced by that choice of prefix, I now think it was the right call).

claui · 2020-12-09T09:29:15Z

This is the TSC being brought in to simply overrule me and the work I've been doing.

@MikeMcQuaid Regardless of anyone’s position, including mine:
I’m pretty sure that everyone involved in this thread has been acting in good faith and with the project’s goals in mind. I’m a bit surprised to hear you believe otherwise. Apologies if we made you feel that way.

MikeMcQuaid · 2020-12-09T09:57:30Z

I agree this is not a good way to involve the TSC.

Thanks @fxcoudert.

I’m pretty sure that everyone involved in this thread has been acting in good faith and with the project’s goals in mind.

I don't think anyone has bad intentions here, I agree. I do think that opinions should be as strong as the willingness of the person proposing them to implement them in a timely fashion. This has not been happening consistently and I find this frustrating.

I think individuals can act in good faith and intend to serve the project's goals and end up achieving the opposite.

I’m a bit surprised to hear you believe otherwise.

There has been 6 referrals to the TSC since October. In every case: it’s been to overrule a decision I stated or code I’ve already written. In 5/6 cases: the referrals have been by PLC members.

In the times when there’s been agreed follow-up work that’s needed done: I have done so every time, even when I disagreed with the outcome. This has not consistently been the case otherwise.

Apologies if we made you feel that way.

I do feel this way, yes. Regardless, I appreciate the apology and want to make clear I don't see you as part of the problem here @claui.

MikeMcQuaid · 2020-12-09T09:59:18Z

I was going to ensure that I fixed the relevant issues today but it seems now that I should instead wait for the TSC to make a decision before I do any more work on this, it seems.

I'm not going to do this, after all. If the TSC wants to overrule and make changes otherwise: fine with me. Until then I'm going to keep on doing the actual work rather than talking about it.

iMichka · 2020-12-09T10:06:29Z

Can somebody summarise the differents paths (current and future) on the 3 platforms? I am now completely lost.

From what I understand, we are getting rid of the "Homebrew" subfolder and will have Readme.md and other src files in /usr/local, /home/linuxbrew/.linuxbrew and /opt/homebrew. Will there be a migration script to move old installs to this new layout? But maybe I misunderstood the different options.

MikeMcQuaid · 2020-12-09T10:11:29Z

From what I understand, we are getting rid of the "Homebrew" subfolder and will have Readme.md and other src files in /usr/local, /home/linuxbrew/.linuxbrew and /opt/homebrew. Will there be a migration script to move old installs to this new layout? But maybe I misunderstood the different options.

There is no proposal to change /usr/local or /home/linuxbrew/.linuxbrew installs.

Using https://docs.brew.sh/Installation#untar-anywhere to install into /opt/homebrew so that brew --prefix == brew --repo does not work correctly currently and will be fixed.

iMichka · 2020-12-09T10:22:32Z

I'll summarise myself then. Here is the proposal (with some example paths, so we do not talk past each-other):

Option 1:

Intel mac:
/usr/local/Homebrew/bin
/usr/local/Homebrew/Library
/usr/local/Homebrew/Readme.md
/usr/local/Cellar
/usr/local/lib
/usr/local/bin

ARM mac:
/opt/homebrew/bin
/opt/homebrew/Library
/opt/homebrew/Readme.md
/opt/homebrew/Cellar
/opt/homebrew/lib

Linux
/home/linuxbrew/.linuxbrew/Homebrew/bin
/home/linuxbrew/.linuxbrew/Homebrew/Library
/home/linuxbrew/.linuxbrew/Homebrew/Readme.md
/home/linuxbrew/.linuxbrew/Cellar
/home/linuxbrew/.linuxbrew/lib
/home/linuxbrew/.linuxbrew/bin

Option 2:

Intel mac:
/usr/local/Homebrew/bin
/usr/local/Homebrew/Library
/usr/local/Homebrew/Readme.md
/usr/local/Cellar
/usr/local/lib
/usr/local/bin

ARM mac:
/opt/homebrew/Homebrew/bin
/opt/homebrew/Homebrew/Library
/opt/homebrew/Homebrew/Readme.md
/opt/homebrew/Cellar
/opt/homebrew/lib
/opt/homebrew/bin

Linux
/home/linuxbrew/.linuxbrew/Homebrew/bin
/home/linuxbrew/.linuxbrew/Homebrew/Library
/home/linuxbrew/.linuxbrew/Homebrew/Readme.md
/home/linuxbrew/.linuxbrew/Cellar
/home/linuxbrew/.linuxbrew/lib
/home/linuxbrew/.linuxbrew/bin

And @MikeMcQuaid is in favour of option 1.

One question, the 2 bin folders in option 1 are merged together, will this not cause an issue?

MikeMcQuaid · 2020-12-09T10:28:59Z

@iMichka Yes, those two options are correct. It can be summarised down to just being ARM because that's all we're talking about. I also think Linux shouldn't have the Homebrew directory but I don't care enough to make the change.

One question, the 2 bin folders in option 1 are merged together, will this not cause an issue?

No, it actually removes complexity.

Can someone split this into a new issue? I don't think this (closed, user-submitted) issue is the best place for discussing this.

iMichka · 2020-12-09T10:34:10Z

Ok so I changed my mind, given the above. Actually it's just "/usr/local/" on Intel that is special, and ARM and Linux should be the same. I'm fine with option 2 then, if we keep track somewhere that we want to change this for Linux too one day, for consistency.

Let me call this Option 3 and this is the ideal situation we would end with:

Intel mac:
/usr/local/Homebrew/bin
/usr/local/Homebrew/Library
/usr/local/Homebrew/Readme.md
/usr/local/Cellar
/usr/local/lib
/usr/local/bin

ARM mac:
/opt/homebrew/bin
/opt/homebrew/Library
/opt/homebrew/Readme.md
/opt/homebrew/Cellar
/opt/homebrew/lib

Linux
/home/linuxbrew/.linuxbrew/bin
/home/linuxbrew/.linuxbrew/Library
/home/linuxbrew/.linuxbrew/Readme.md
/home/linuxbrew/.linuxbrew/Cellar
/home/linuxbrew/.linuxbrew/lib

fxcoudert · 2020-12-09T10:59:34Z

I would highlight that “option 2” is the currently agreed-upon, already documented, choice. It's what I've used to install, set up, and test CI nodes on ARM Big Sur. Going back now will cause significant extra work and delay of the ARM availability. The end user consequences (“it will make /opt/homebrew more cluttered for users") are, in my opinion, weak and do not seem to justify disturbing on-going work.

carlocab · 2020-12-09T12:51:37Z

Just a thought, hope it's not unwelcome:

Making sure /opt/homebrew has a clear and intuitive directory structure has advantages more than just in terms of how cluttered it looks. It also helps users better understand and maintain their own installation of Homebrew and the software they've installed using it. This is a benefit that will accumulate over time, so could be worth some additional work and delays right now.

Admittedly, I know I'm not doing any of the actual work here; just thought some input might be helpful.

sjackman · 2020-12-10T02:18:43Z

This decision of whether HOMEBREW_PREFIX == HOMEBREW_REPOSITORY is not one that can be un-made once we start building bottles. It's worth hearing the TSC's opinion before plowing ahead. The PLC can take into consideration who has volunteered to implement each approach. One volunteer to implement one solution doesn't imply there are no volunteers to implement the other solution, which in this case, requires updating the installation documentation.

reitermarkus · 2020-12-10T03:34:34Z

Just from looking at the options, I think option 2 is the cleanest approach, since regardless of platform, HOMEBREW_REPOSITORY is HOMEBREW_PREFIX/Homebrew.

reitermarkus · 2020-12-10T03:40:46Z

I'm actually not sure when, if ever, we would want HOMEBREW_REPOSITORY recorded instead of HOMEBREW_PREFIX in a bottle?

I was wondering the same thing just now. I can't think of any reason.

sjackman · 2020-12-10T04:51:19Z

I'm actually not sure when, if ever, we would want HOMEBREW_REPOSITORY recorded instead of HOMEBREW_PREFIX in a bottle?

Three that come to mind are…

@@HOMEBREW_REPOSITORY@@ is found in INSTALL_RECEIPT.json in the key source.path.

References to the shim compiler and scm scripts:
@@HOMEBREW_REPOSITORY@@/Library/Homebrew/shims/super/cc
@@HOMEBREW_REPOSITORY@@/Library/Homebrew/shims/scm/git

References to the pkg-config directory and files:
@@HOMEBREW_REPOSITORY@@/Library/Homebrew/os/mac/pkgconfig/11.0/

sjackman · 2020-12-10T05:03:39Z

I've confirmed with Homebrew 2.6.1 that when the bottle is built with HOMEBREW_PREFIX == HOMEBREW_REPOSITORY, source.path in INSTALL_RECEIPT.json is @@HOMEBREW_PREFIX@@/Library/Taps/homebrew/homebrew-core/Formula/hello.rb rather than the expected @@HOMEBREW_REPOSITORY@@/Library/Taps/homebrew/homebrew-core/Formula/hello.rb

sjackman · 2020-12-10T05:07:29Z

The bottle for r references @@HOMEBREW_REPOSITORY@@/Library/Homebrew/os/mac/pkgconfig/10.15

sjackman · 2020-12-10T05:35:03Z

One compromise option is available. CI can build bottles using HOMEBREW_PREFIX=/opt/homebrew HOMEBREW_REPOSITORY=/opt/homebrew/Homebrew. Users can use these bottles with either this same configuration or with HOMEBREW_PREFIX=HOMEBREW_REPOSITORY=/opt/homebrew. We can choose either option for the default installation used by install.sh for users.

I would expect an installation with HOMEBREW_PREFIX=HOMEBREW_REPOSITORY=/opt/homebrew to just work with the vast majority of bottles. For the exceptional bottles that do have a hard-coded reference to /opt/homebrew/Homebrew/ in a binary Mach-O, a symlink from /opt/homebrew/Homebrew to /opt/homebrew would resolve those cases, so that for example references to /opt/homebrew/Homebrew/Library/Homebrew/os/mac/pkgconfig/11.0 still work.

Rylan12 · 2020-12-10T05:36:58Z

I don't have a comment on which option to use, but I do have a proposal for how to handle the HOMEBREW_PREFIX == HOMEBREW_REPOSITORY issue:

Right now we have three placeholders that we use:

@@HOMEBREW_PREFIX@@
@@HOMEBREW_CELLAR@@
@@HOMEBREW_REPOSITORY@@

As a hypothetical long-term goal, we may want to remove @@HOMEBREW_REPOSITORY@@ in favor of @@HOMEBREW_LIBRARY@@. I believe HOMEBREW_LIBRARY is always HOMEBREW_REPOSITORY/Library. I can't think of any instances of HOMEBREW_REPOSITORY that don't actually point to HOMEBREW_LIBRARY (that's where the shims, taps, and pkg-config scripts all live.

If we replace the references to @@HOMEBREW_REPOSITORY@@/Library with @@HOMEBREW_LIBRARY@@, this should allow us to continue to offer bottles that work regardless of whether HOMEBREW_PREFIX == HOMEBREW_REPOSITORY (even if we choose to only officially support HOMEBREW_PREFIX != HOMEBREW_REPOSITORY)

Not totally sure exactly what changing to using this would take. I think we could do this while avoiding the need to rebuild bottles for all formulae. If we start to build new bottles that use @@HOMEBREW_LIBRARY@@ instead of @@HOMEBREW_REPOSITORY@@/Library that will solve this issue for the future. Until all instances of @@HOMEBREW_REPOSITORY@@ are eliminated, we could check to see if the @@HOMEBREW_REPOSITORY@@ is followed by /Library. If so, interpret it as @@HOMEBREW_LIBRARY@@. If not, interpret as @@HOMEBREW_PREFIX@@

Hypothetical plan for implementation;

Modify the bottle creation code to use HOMEBREW_LIBRARY instead of HOMEBREW_REPOSITORY for new bottles
Build a bunch of bottles (both through normal version bumps and through mass-bottling for arm)
Wait until lots of formulae have the "new bottles" that use HOMEBREW_LIBRARY instead of HOMEBREW_REPOSITORY/Library
If desired, manually (with a script) update the remaining bottles. This would entail modifying the bottle to replace @@HOMEBREW_REPOSITORY@@/Library with @@HOMEBREW_LIBRARY@@, re-uploading to bintray, replacing checksums in formula bottle blocks. This would not require rebuilding bottles using brew install and brew bottle

It may be that totally removing @@HOMEBREW_REPOSITORY@@ references is not feasible, but if we don't build new bottles with these references, we won't have this issue regardless of our choice on ARM. It won't matter for Intel bottles because we've already established that HOMEBREW_PREFIX != HOMEBREW_REPOSITORY.

I'd be willing to help out with the planning/implementation of this.

If I'm totally off-base with this disregard my suggestion. If this is something we're interested in, we'll need a separate issue for discussion because this one has been resolved and closed.

Edit: looks like Shaun also posted a (much simpler) suggestion for this at the same time 😄

reitermarkus · 2020-12-10T07:03:20Z

As a hypothetical long-term goal, we may want to remove @@HOMEBREW_REPOSITORY@@ in favor of @@HOMEBREW_LIBRARY@@.

In my opinion it would be best to use separate variables for @@HOMEBREW_SHIMS@@, @@HOMEBREW_PKGCONFIG etc. so they are completely decoupled from the repository/directory layout of HOMEBREW_REPOSITORY.

mistydemeo · 2020-12-10T07:45:07Z

Mike, first of all, thank you very much for the fix in #9481. I appreciate your putting the work into resolving this.

At this point it sounds like option 2 has developer support and is what we've already documented.

fxcoudert · 2020-12-10T07:55:31Z

References to the shim compiler and scm scripts:

References to the pkg-config directory and files:

The first one really should not happen (we have an audit for that in place, don't we?). For pkg-config files, I also think it should probably be avoided, shouldn't it?

reitermarkus · 2020-12-10T08:12:50Z

So we have unanimous TSC agreement on “option 2”, which means confirmation of what we have currently been implementing and documenting.

I thought option 2 would be to have HOMEBREW_REPOSITORY != HOMEBREW_PREFIX on every platform?

fxcoudert · 2020-12-10T08:53:12Z

@reitermarkus I've removed my post, I may have gotten the option numbers confused.

fxcoudert · 2020-12-10T09:05:35Z

There is actually an example of why I think this got in the wrong way to involve the TSC, and leads to confusion rather than clarity. There was a choice of prefix discussed already, and we are implementing it, as documented. If it is felt that there is need to change things, then:

Someone needs to actually make a clear proposal of:
a. what the situation is
b. how they want it to change
We use it as a basis for discussion
We see who is getting involved in the practicalities, and how to coordinate
We reach a decision

I'd like to highlight that even now, after we stopped bottling, there are many hours (many dozens, in my case) of work from several maintainers that have been invested into implementing what the consensus was. It is already late in the process, and I would hope that we don't reverse course (and make all that work useless) unless there are very strong incentives.

And to be clear: right now, the arguments that have been discussed are mostly “there are existing bugs for this configuration” (sure, and they can be fixed) or subjective arguments about what is “cleaner” or “less confusing”.

MikeMcQuaid · 2020-12-10T12:17:15Z

Mike, first of all, thank you very much for the fix in #9481. I appreciate your putting the work into resolving this.

Thanks @mistydemeo, I appreciate it.

Repeating myself from yesterday:

Can someone split this into a new issue? I don't think this (closed, user-submitted) issue is the best place for discussing this.

And to be clear: right now, the arguments that have been discussed are mostly “there are existing bugs for this configuration” (sure, and they can be fixed) or subjective arguments about what is “cleaner” or “less confusing”.

It's unclear to me based one the above that anyone besides @fxcoudert and @mistydemeo has looked at (or even noticed) #9481. That fixes the bug referenced in this issue.

If you have other bugs with the current behaviour: please open a reproducible issue.

If you want to propose an alternative case: please open a PR with the code change you propose.

sjackman · 2020-12-10T19:49:27Z

I've opened a PR at #9503

fxcoudert · 2020-12-12T10:13:19Z

For the record, the TSC decision is to proceed with the current choice (leading to HOMEBREW_PREFIX == HOMEBREW_REPOSITORY == /opt/homebrew on Apple Silicon.

fxcoudert mentioned this issue Dec 7, 2020

macOS 11 Big Sur compatibility on Apple Silicon #7857

Closed

9 tasks

MikeMcQuaid self-assigned this Dec 8, 2020

MikeMcQuaid added the bug Reproducible Homebrew/brew bug label Dec 8, 2020

sjackman added the TSC Technical Steering Committee label Dec 9, 2020

MikeMcQuaid mentioned this issue Dec 9, 2020

Fix bottle prefix and repository handling #9481

Merged

MikeMcQuaid closed this as completed in #9481 Dec 9, 2020

sjackman mentioned this issue Dec 10, 2020

Relocate @@HOMEBREW_REPOSITORY@@/Library #9503

Merged

7 tasks

BrewTestBot added the outdated PR was locked due to age label Jan 11, 2021

Homebrew locked as resolved and limited conversation to collaborators Jan 11, 2021

Bottle path substitution when HOMEBREW_REPOSITORY == HOMEBREW_PREFIX #9453

Bottle path substitution when HOMEBREW_REPOSITORY == HOMEBREW_PREFIX #9453

Comments

fxcoudert commented Dec 7, 2020

fxcoudert commented Dec 7, 2020

sjackman commented Dec 7, 2020 • edited

williammck commented Dec 7, 2020

williammck commented Dec 7, 2020

fxcoudert commented Dec 8, 2020

claui commented Dec 8, 2020

fxcoudert commented Dec 8, 2020

MikeMcQuaid commented Dec 8, 2020

fxcoudert commented Dec 8, 2020

MikeMcQuaid commented Dec 8, 2020

claui commented Dec 8, 2020

fxcoudert commented Dec 8, 2020

MikeMcQuaid commented Dec 8, 2020

fxcoudert commented Dec 8, 2020

williammck commented Dec 8, 2020

sjackman commented Dec 9, 2020

fxcoudert commented Dec 9, 2020

iMichka commented Dec 9, 2020

claui commented Dec 9, 2020

MikeMcQuaid commented Dec 9, 2020

fxcoudert commented Dec 9, 2020 • edited

claui commented Dec 9, 2020

MikeMcQuaid commented Dec 9, 2020

MikeMcQuaid commented Dec 9, 2020

iMichka commented Dec 9, 2020

MikeMcQuaid commented Dec 9, 2020

iMichka commented Dec 9, 2020

MikeMcQuaid commented Dec 9, 2020

iMichka commented Dec 9, 2020

fxcoudert commented Dec 9, 2020 • edited

carlocab commented Dec 9, 2020

sjackman commented Dec 10, 2020 • edited

reitermarkus commented Dec 10, 2020

reitermarkus commented Dec 10, 2020

sjackman commented Dec 10, 2020 • edited

sjackman commented Dec 10, 2020 • edited

sjackman commented Dec 10, 2020 • edited

sjackman commented Dec 10, 2020 • edited

Rylan12 commented Dec 10, 2020 • edited

reitermarkus commented Dec 10, 2020

mistydemeo commented Dec 10, 2020

fxcoudert commented Dec 10, 2020

reitermarkus commented Dec 10, 2020

fxcoudert commented Dec 10, 2020

fxcoudert commented Dec 10, 2020

MikeMcQuaid commented Dec 10, 2020

sjackman commented Dec 10, 2020

fxcoudert commented Dec 12, 2020

sjackman commented Dec 7, 2020 •

edited

fxcoudert commented Dec 9, 2020 •

edited

fxcoudert commented Dec 9, 2020 •

edited

sjackman commented Dec 10, 2020 •

edited

sjackman commented Dec 10, 2020 •

edited

sjackman commented Dec 10, 2020 •

edited

sjackman commented Dec 10, 2020 •

edited

sjackman commented Dec 10, 2020 •

edited

Rylan12 commented Dec 10, 2020 •

edited