Consider dropping Envoy's linux build

[codefromthecrypt]

Currently linux for Envoy is incomplete as there is no arm64 support for it. This means many end up using func-e.io anyway because as far as I can tell there's no near term of arm64.

Envoy versions in homebrew are delayed due to perpetual problems with the build. At least the recent ones were linux-specific. Despite a lot of time and energy by maintainers here, it often takes many days to get a version out, and linux blocks os/x. Delays are caused by the complexity and fragility of the build and the lack of ownership in the Envoy project team (ex envoyproxy/envoy#17500)

To add insult to injury, since linux is available elsewhere, even when we do resolve these problems. It is likely linux users who cared to be recent would have gone elsewhere anyway. Meanwhile, os/x users have no other source but homebrew for envoy releases and are blocked by all of this. Also, the build hosts are tied up and more importantly, time of maintainers squandered.

I would suggest we look at the download counts of envoy on linux and seriously consider dropping it. This would allow us to prioritize our time for os/x, tighten release delays and better focus human and CI resources. Another option would be to ask Envoy yet again to own the process, but they'd have to commit IMHO. The status quo here won't do.

[SMillerDev]

It's annoying if things are holding back merging a review, but if we're not doing macOS ARM support, and not doing Linux we might as well remove envoy because clearly it's too hard to package.

Another option of course is just to keep making pull requests and worry less about the speed of releasing because everything is done by volunteers and you can't really force those into a timeline.

Either solution sounds fine to me.

[codefromthecrypt]

FWIW I think envoy will eventually complete macOS/arm64/m1, just it is painfully slow. Linux on envoy does work on arm64 now, just we don't bottle it. I'm not sure anything bottles linux/arm64.

Problem with removing the formula entirely is that it throws the baby out with the bathwater. There's no other build source of macOS for envoy. Upstream doesn't do it, and func-e.io actually doesn't do any builds. It uses homebrew's OCI repository as a source for macOS binaries.

Basically no one would be able to get envoy on macOS anymore if we dropped the formula completely. This is different than linux as because upstream at least provides docker images and maybe one day tarballs.

[SMillerDev]

I'm not sure anything bottles linux/arm64.

It doesn't.

This is different than linux as because upstream at least provides docker images and maybe one day tarballs.

Those aren't really ways of installing things in Linux. They're more ways of running a program under Linux.

[codefromthecrypt]

Those aren't really ways of installing things in Linux. They're more ways of running a program under Linux.

right, technically what happens is https://github.com/tetratelabs/archive-envoy/ makes a one-time copy of the binary from the linux image and re-compresses it into a tarball. Then https://func-e.io uses that to "install" similar to rvm etc, but dissimilar to homebrew or package managers. The lack of linux packages is yet another extremely delayed PR, but again that may eventually complete. When that does, probably many using func-e for linux might use those packages.

The long story short is that there are more options currently in envoy for linux than macOS, which is solely resting on homebrew... and probably should. You could imagine how annoying and difficult it would be for end users to build from source if even we struggle with it!

[carlocab]

I'm not entirely opposed to this, but I'd like to see us explore all other options we have first.

Why doesn't upstream provide builds for Envoy? Do they at least have macOS CI that tests that things work?

[codefromthecrypt]

There used to be a project getenvoy which did packaging, but that wasn't upstream and was told by LF and the Envoy project leads that it violated trademarks. So, that doesn't exist anymore and the overall idea was that upstream would do packaging including tarballs, to reduce brand confusion. However, the step of making distributions of any kind besides Docker hasn't happened, yet.

Currently upstream builds and tests linux, macOS and windows, but only distributes linux and windows and only via docker images. linux builds are tested and then packaged into docker, and macOS is built and tested, but not distributed in any way (no idea why). So both are tested in their CI, though configurations vary. They don't use GHA, rather Azure Pipelines, you can take my word for it or click around.

As to why upstream doesn't provide builds. I think it is cultural. In early June, I formally requested them to republished the builds currently dropped in CI stages as tarballs. As you can see, many people thumbs up and some commented, but the project lead wasn't interested and suggested "We ship ARM docker images right now, and you can of course build your own binaries if you don't like the ones we provide." Obviously, I don't agree that "just build your own" is a good answer, as it echos tech debt and heavy labor across the ecosystem.

In any case, that issue requesting tarballs went no-where. The closest thing was a draft PR for Debian and RPM, but that's been in draft since early July, and even if it were complete, I doubt it would be useful here as you'd need to extract binaries from deb or RPM.

To try to not block the ecosystem entirely, I decided to write a tool car to extract the linux "binary" from the official docker image. So, we could in homebrew use the the same binaries for linux. However, we couldn't use the ones on macOS as they are heh archived from homebrew: chicken/egg problem.

I do hope the issues around upstream publishing anything resolve, but that's what I also meant by "own". Things could be better if upstream owned the success of homebrew (by fixing things broken during their release), or owned success by producing an alternative (such as tarballs).

Right now, that's not the case. Builds fail frequently and we act reactively, often from a position of less skills or influence than upstream has. Even when things are set to improve on Mac, and they do, they take a very long time to resolve. I think that macOS still has a large benefit being served here, and it is unnecessarily difficult, but homebrew is the best place to get things running on macOS, and the policy is usually build from source anyway.

Linux I feel we could be more flexible in order to reduce burden and take advantage of the fact that upstream prioritizes it more than macOS. Especially as linux builds are harder on us: it needs a hosted runner, a complex set of if statements, and more maintenance. I think we could either drop it or move to tarballs if upstream doesn't start actively helping with this.

[codefromthecrypt]

I raised a PR to hopefully resolve this into a decision, even if it is to close my PR Homebrew/homebrew-core#87142

[codefromthecrypt]

I closed the PR as I don't really think the project team here want the formula support for linux removed. Rather, it should be left in just in case someone wants the two hours of laptop burning one day. If in the future that position changes, it should be very easy for someone to remove it, but I've done all I can and no longer happy to continue this effort.