general: support MacPorts as an installation source

[myitcv]

Raising off the back of #1121, to capture discussion about whether we add MacPorts support or not.

I know nothing about MacPorts, so will flesh out this description based on replies below.

Is your feature request related to a problem? Please describe.

TBC

Describe the solution you'd like

Support for MacPorts as an installation source:

sudo port selfupdate
sudo port install cue

Describe alternatives you've considered

TBC

Additional context

n/a

@herbygillot - I would be grateful if you could help to flesh out the points above.

[myitcv]

Thanks for raising this @herbygillot.

We already maintain a homebrew setup at https://github.com/cue-lang/homebrew-tap, managed via GoReleaser.

I'm not super familiar with MacPorts, so forgive the basic questions:

• How many people use MacPorts?
• Is the Homebrew setup insufficient/inappropriate in some way?
• How do you propose to maintain the MacPorts config file, which I think lives at https://github.com/macports/macports-ports/blob/master/devel/cue/Portfile

[herbygillot]

So I'm a volunteer maintainer with the MacPorts project.

In a sense, MacPorts is an alternative package manager for macOS. It has a different philosophy and mode of operation compared to Homebrew. Software projects and packages are added to MacPorts as ports, which are defined by Portfiles. You are correct that this is the Portfile for cue.

How many people use MacPorts?

I can't say for sure - there are statistics, but those are from advanced users who knowingly opt-in to have their stats uploaded.

Is the Homebrew setup insufficient/inappropriate in some way?

As mentioned before, MacPorts is an alternative to Homebrew. Some folks use Homebrew to manage open source software on their Mac, others use MacPorts, and still some other adventurous folk use both. Adding cue to the MacPorts project in no way implies some insufficiency with Homebrew, it's just making cue available to people who prefer or primarily use MacPorts.

How do you propose to maintain the MacPorts config file

Portfiles are kept up-to-date by maintainers. In most cases, Postfiles are marked openmaintainer, meaning that anyone can open a pull request to update a Portfile, which is then approved by the maintainer or MacPorts administrators. This is indicated within the maintainers section in the Portfile. But the primary responsibility of keeping a port current lies with the maintainer.

[myitcv]

Thanks very much for taking the time to explain, @herbygillot.

How many people use MacPorts?

I can't say for sure - there are statistics, but those are from advanced users who knowingly opt-in to have their stats uploaded.

To my knowledge (searching Slack and GitHub) we haven't yet had anyone request support for MacPorts. I'm not sure how much to read into that however.

Portfiles are kept up-to-date by maintainers

One of the things we really like about goreleaser is that it keeps us in control of the different target configurations. Is there a way we can do that with MacPorts?

[herbygillot]

To my knowledge (searching Slack and GitHub) we haven't yet had anyone request support for MacPorts. I'm not sure how much to read into that however.

I think that makes sense. I would imagine that the number of Homebrew users dwarf MacPorts users by a quite a margin. Usually if folks want to see a particular software project added to the MacPorts catalog, they'll put in a request in the MacPorts Trac or send a request through the mailing lists.

Portfiles are kept up-to-date by maintainers

One of the things we really like about goreleaser is that it keeps us in control of the different target configurations. Is there a way we can do that with MacPorts?

Which makes complete sense. Unfortunately goreleaser does not have any integration with MacPorts today, and there are unfortunately not a lot of automated options at the moment. What I've seen many projects do is explicitly mark sources like MacPorts and other repos as 3rd-party sources so that folks are aware that it's not the same as the official source.

[b4nst]

Hey @herbygillot @myitcv. Just wanted to mention that I'm also using MacPorts over Homebrew for the vast majority of my packages. I'm also willing to maintain the Portfile manually until we find an automatic tool to tackle that.

JFYI I opened goreleaser/goreleaser#1556 a while ago but it has been closed for not enough votes.

[myitcv]

Thanks @b4nst.

@herbygillot a couple of follow-up questions:

How are port files tested? Is there a way in which a port file can be tested prior to it going live?

The current portfile appears to actually build cmd/cue. One thing that concerns me with this sort of approach is that we end up with variations on how CUE is built. The recent goreleaser change switched to build in a mode which resolves cuelang.org/go as a dependency, hence we get full version information for cuelang.org/go and all its dependencies in the resulting binary. Is there perhaps a way that the port file doesn't build at all and instead, like the HomeBrew Tap delegates to the GitHub release assets?

[b4nst]

You can create a Portfile that rely on pre-build binaries, tho it's not a recommended practice. If there is a way to build in the same mode on the end user machine, I strongly suspect a push back from the MacPorts community. It's more seen as an exception rather than an alternative.

Regarding tests, before the Portfile being merged (= going live) it has to pass 3 pipelines (2 GithubAction for macos 10.15 and 11, one AzurePipeline for macos 10.14) and be reviewed and merged by a MacPorts maintainer. During deployment or update off the Portfile you can also test it locally.

[myitcv]

Thanks for the detail, @b4nst.

It's more seen as an exception rather than an alternative.

Can you provide some more detail on this?

The main reason for me asking is that the official CUE binaries distributed via GitHub will be fully verifiable, built in a controlled environment. So from an integrity perspective, people will be able to precisely verify what binaries they are using.

Regarding tests..

Thanks, this is useful. Ideally we'd move to a model whereby the source of truth for this file is the main CUE repo, and automation ensures that file is then vendored elsewhere. Hence I was wondering if to help shorten that development cycle there is some way we could test the port file standalone from MacPorts, here in the CUE repo?

[b4nst]

I understand your point and honestly I quite agree with it. I do not have many details on why it has been that way on MacPorts, but my first PR (when I was not aware of that) on MacPorts was refused because it was using pre-built binaries (actually the ones created by goreleaser) instead of sources. They asked me to switch to end-user build before merging. I guess they're trying precisely to avoid relying on external party building the binary, so you always have a way to check the source.

From an integrity perspective, the source are often (it's the case for GitHub backed ports) downloaded from an archive and checksum checked. My understanding is that if you don't set openmaintainers on the Portfile, you're the only entity authorized to create the pull request. @herbygillot has probably a better understanding of that than me tho.

Hence I was wondering if to help shorten that development cycle there is some way we could test the port file standalone from MacPorts, here in the CUE repo?

Yup, I don't see why it would not be technically feasible. You can build and test a Portfile locally, so you can also do it on your CI Pipeline (like with a dedicated GithubAction Workflow for example). And that workflow could be the one in charge of creating the PR on MacPorts when everything has been tested and validated "locally".

[myitcv]

I guess they're trying precisely to avoid relying on external party building the binary, so you always have a way to check the source.

Thanks, it's exactly for this reason that I'd like to probe this a bit more.

We're absolutely not against third parties building CUE, but it needs to be built consistently, i.e. resolving cuelang.org/go as a dependency of proxy.golang.org and sum.golang.org. Because otherwise we don't have verifiability etc.

Indeed if third parties build in the same way as goreleaser-produced release assets, then we will get byte-for-byte identical builds.

[herbygillot]

Hello @b4nst and thank you for breaking things down here. I think adding MacPorts support to GoReleaser is definitely an important discussion in its own right. I will chime in on the issue you opened, but it may merit discussion in the MacPorts mailing lists.

So @myitcv, MacPorts does have a preferred philosophy around building software:

• Assets should be source code, and not pre-built binaries
• Source code should be downloadable, checksummed archive files (like zip or tar files), and not git checkouts
• Source archives should be specified, manifest-stye, in the Portfile with checksums
• Downloads and verification should only be done by MacPorts itself, and not by the compiler or interpreter (Go)
• Verified/checksummed assets that are listed in the Portfile can then be distributed & mirrored by the MacPorts mirrors

Basically MacPorts wants to be the one doing the download, verification and distribution of both project and dependency assets, for a number of reasons, including trust, and also the ability to continue to build the port even if the original upstream dissapears (because it can mirror all the dependency assets to MacPorts mirrors, and then MacPorts can download from these mirrors when a user requests the given port to build the target software, even if upstream isn't available).

MacPorts defines phases for building a piece of software: fetch, extract, patch, build, destroot... etc.

Canonically, Go software is supposed to use what's called the golang portgroup and go.vendors to work in the way that MacPorts prefers:

• Required Go modules are specified using go.vendors so that MacPorts can track, download, distribute and mirror them, then set them up in a way that Go expects
• Environment vars are used to prohibit Go from fetching whatever it wants from the Internet, and makes it only build using the modules that MacPorts provides

This is how the current cue Portfile works: https://github.com/macports/macports-ports/blob/1109917e97baa8ff554f91d154ecf78d00b388ff/devel/cue/Portfile

All this is done for reasons of reproducibility, consistency, security and more.

However there are definite exceptions - many Go ports are too onerous to make work with go.vendors, so modifications are made so that Go can work as it naturally does, taking MacPorts out of the picture for module dependencies.

Compared to having MacPorts distribute a pre-built official cue binary, I think the preference would definitely be to use goreleaser, because then at least we are still building source code, which is the strong preference.

The downside with goreleaser is that it requires a git checkout, and now so we no longer have downloadable checksummed assets that can be archived, verified with checksums, mirrored, and known to provide a consistent and reproducible build.

That said, I have added goreleaser to MacPorts, and using it, I now have an example pull request that builds cue with it: macports/macports-ports#11783

Here's what the Portfile looks like from this PR: https://github.com/macports/macports-ports/blob/ed35bd934e1e0c9c7f1a7050abec002266c832c5/devel/cue/Portfile

[verdverm]

Off topic: How does MacPorts deal with retracted modules? (https://golang.org/ref/mod#go-mod-file-retract)

With all of the modules being listed and mirrored, would information like this be potentially missing? Does this mean they replicate a common behavior checking for this type of thing across languages?

[herbygillot]

Off topic: How does MacPorts deal with retracted modules? (https://golang.org/ref/mod#go-mod-file-retract)

With all of the modules being listed and mirrored, would information like this be potentially missing? Does this mean they replicate a common behavior checking for this type of thing across languages?

To be clear, go.mod is still used when compiling Go software in MacPorts. As a matter of fact, go.vendors is usually generated from a project's go.mod.

The modules specified in the Portfile's go.vendors section are downloaded by MacPorts and extracted into the Go root, and then MacPorts has Go compile as normal (just only using whats been extracted, and not fetching from the Internet). Go continues to compile through and reference the go.mod file.

So if a new version of something is released that retracts one or more module versions, everything still works as expected. You would update go.vendors from the new version's go.mod. The retracted modules may still be on the MacPorts mirrors, but they won't be used when the newer version of the software builds as per the newer version's go.mod.

[herbygillot]

macports/macports-ports#11783 has been merged. cue will now build via goreleaser within MacPorts going forward.

[myitcv]

Compared to having MacPorts distribute a pre-built official cue binary, I think the preference would definitely be to use goreleaser, because then at least we are still building source code, which is the strong preference.

I think we're slightly talking past each other here :) Which is unsurprising, because there is a good deal of nuance in all these sorts of discussions. I'll try and explain my point a slightly different way.

On a related point, how is Go itself installed via MacPorts?

All this is done for reasons of reproducibility, consistency, security and more.

My main concern with this approach is that following the MacPorts philosophy establishes a different mode of reproducibility, consistency etc to the officially distributed binaries. As I mentioned in #1125 (comment), we're absolutely not against third parties building CUE, but I'm trying to push for everyone to do it consistently.

cue will now build via goreleaser within MacPorts going forward.

goreleaser does not need to be a dependency of the CUE portfile: only Go is required. The only requirement is that MacPorts builds CUE in the same way that goreleaser does, i.e. resolving dependencies via the public proxy proxy.golang.org and public checksum db sum.golang.org etc. Because that's the only way the resulting binary can be verified against other builds of cmd/cue. Incidentally, the way that goreleaser builds is identical to the way the official install mechanism works: go install cuelang.org/go/cmd/cue@v0.4.0. So having MacPorts build in the same way, means we are comparing apples with apples. If every package manager had a different method of managing dependencies, building and installing CUE we can never be totally clear/sure we are comparing apples with apples.

Go development of publicly accessible code now almost exclusively relies on the public proxy and checksum db. It seems sensible therefore for builds of Go modules like CUE to depend on that infrastructure, infrastructure that is then consistent between a user's development environment and the setup used to build/install development tools.

That seems easiest done by installing the officially distributed CUE binaries, but if MacPorts prefers to install from source, then following the same build approach as the official install mechanism, goreleaser and others... seems most sensible.

The acid test of whether the build result is correct is to confirm the output from go version -m /path/to/cue is the same as the official binaries. Hence why I was asking above whether we can test the MacPorts install via our CI/release infrastructure to ensure the "correct" binary is built.

If jumping on a call is the easiest way to talk this through I'd be happy to setup some time.

[herbygillot]

On a related point, how is Go itself installed via MacPorts?

MacPorts builds Go from source using the officially documented build process into the MacPorts prefix (/opt/local).

A user then installs Go using the same MacPorts port command: sudo port install go

Go will then be available at /opt/local/bin/go:

➜  ~ which go
/opt/local/bin/go
➜  ~ go version
go version go1.16.7 darwin/amd64

goreleaser does not need to be a dependency of the CUE portfile: only Go is required.

goreleaser is specified in the depends_build section of cue's Portfile, which tells MacPorts that goreleaser is only needed when building cue, and not when installing it, or in any other capacity.

When a user does sudo port install cue, goreleaser will not be downloaded unless we are missing cue in the MacPorts mirrors (meaning MacPorts CI failed to build cue for some reason), in which case goreleaser will be downloaded to build cue from source on the user's machine.

If every package manager had a different method of managing dependencies, building and installing CUE we can never be totally clear/sure we are comparing apples with apples.

...but if MacPorts prefers to install from source, then following the same build approach as the official install mechanism, goreleaser and others... seems most sensible.

This is exactly what we are doing now. We're building cue using goreleaser to ensure that it is being built as close to the official way as possible.

The acid test of whether the build result is correct is to confirm the output from go version -m /path/to/cue is the same as the official binaries.

As requested:

➜  ~ go version -m /opt/local/bin/cue
/opt/local/bin/cue: go1.16.7
	path	cuelang.org/go/cmd/cue
	mod	cuelang.org/go	(devel)
	dep	github.com/cockroachdb/apd/v2	v2.0.1	h1:y1Rh3tEU89D+7Tgbw+lp52T6p/GJLpDmNvr10UWqLTE=
	dep	github.com/emicklei/proto	v1.6.15	h1:XbpwxmuOPrdES97FrSfpyy67SSCV/wBIKXqgJzh6hNw=
	dep	github.com/golang/glog	v0.0.0-20160126235308-23def4e6c14b	h1:VKtxabqXZkF25pY9ekfRL6a582T4P37/31XEstQ5p58=
	dep	github.com/google/uuid	v1.2.0	h1:qJYtXnJRWmpe7m/3XlyhrsLrEURqHRM2kxzoxXqyUDs=
	dep	github.com/mpvl/unique	v0.0.0-20150818121801-cbe035fff7de	h1:D5x39vF5KCwKQaw+OC9ZPiLVHXz3UFw2+psEX+gYcto=
	dep	github.com/pkg/errors	v0.8.1	h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I=
	dep	github.com/protocolbuffers/txtpbfmt	v0.0.0-20201118171849-f6a6b3f636fc	h1:gSVONBi2HWMFXCa9jFdYvYk7IwW/mTLxWOF7rXS4LO0=
	dep	github.com/spf13/cobra	v1.0.0	h1:6m/oheQuQ13N9ks4hubMG6BnvwOeaJrqSPLahSnczz8=
	dep	github.com/spf13/pflag	v1.0.3	h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg=
	dep	golang.org/x/mod	v0.3.1-0.20200828183125-ce943fd02449	h1:xUIPaMhvROX9dhPvRCenIJtU78+lbEenGbgqB5hfHCQ=
	dep	golang.org/x/net	v0.0.0-20200226121028-0de0cce0169b	h1:0mm1VjtFUOIlE1SbDlwjYaDxZVDP2S5ou6y0gSgXHu8=
	dep	golang.org/x/text	v0.3.2	h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs=
	dep	golang.org/x/tools	v0.0.0-20200612220849-54c614fe050c	h1:g6oFfz6Cmw68izP3xsdud3Oxu145IPkeFzyRg58AKHM=
	dep	golang.org/x/xerrors	v0.0.0-20191204190536-9bdfabe68543	h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
	dep	gopkg.in/yaml.v3	v3.0.0-20200121175148-a6ecf24a6d71	h1:Xe2gvTZUJpsvOWUnvmL/tmhVBZUmHSvLbMjRj6NUUKo=

Hence why I was asking above whether we can test the MacPorts install via our CI/release infrastructure to ensure the "correct" binary is built.

This may be possible, but I'm not 100% sure, as MacPorts requires a macOS base image + special tooling to run in CI. I don't have enough experience setting that up to give guidance here at this current moment.

[herbygillot]

Compared go version -m output between the official Darwin amd64 binary and the one MacPorts has built:

❯ diff <( go version -m ./cue-official ) <( go version -m /opt/local/bin/cue )
1c1
< ./cue-official: go1.15.12
---
> /opt/local/bin/cue: go1.16.7

They're identical, with the only difference being the version of Go each was built with; MacPorts is using the latest version of Go (1.16.7) as of this writing).

[verdverm]

Running MacPorts from a GitHub action seems doable fwiu. GitHub has macos runners and there look to be (outdated) setup-macport actions around.

Compiling with different Go versions produces different binaries (and code) due to differences in the std library (at a minimum). I would imagine we need to ensure the Go version is the same as well.

$ diff ./cue-official $(which cue)
Binary files ./cue-official and /home/tony/go/bin/cue differ

$ go version
go version go1.17rc1 linux/amd64

[myitcv]

@herbygillot apologies for the delay here.

Whilst not totally insignificant (because of potential bugs, security issues, performance etc), the Go version difference can be ignored. Out of interest, if a user bumps their Go version, are all packages that build Go binaries rebuilt?

Thanks for including the output. The significant line in what I quote below is the last one:

➜  ~ go version -m /opt/local/bin/cue
/opt/local/bin/cue: go1.16.7
	path	cuelang.org/go/cmd/cue
	mod	cuelang.org/go	(devel)

i.e. we have no version information for the CUE module itself, and that's the critical bit.

goreleaser is specified in the depends_build section of cue's Portfile, which tells MacPorts that goreleaser is only needed when building cue, and not when installing it, or in any other capacity.

I suspect I have confused the discussion here. My initial reference to goreleaser was to suggest MacPorts could use the pre-built binaries that are built by goreleaser and published as assets on GitHub. I totally understand the reasons for preferring to build from source. My further reference to goreleaser was then to explain that it uses the Go proxy and sumdb as part of building assets.

But just to reemphasise, building CUE from source in the official way does not require goreleaser, only Go. For example, here are the steps that allow CUE to be built, using proxy and sumdb resolved assets, in the official way:

$ cd $(mktemp -d)
$ export GOBIN=$PWD/bin
$ go install -trimpath cuelang.org/go/cmd/cue@v0.4.0
$ go version -m $PWD/bin/cue
/tmp/tmp.Y57aCpvlPR/bin/cue: devel go1.18-ad97d204f0 Sun Sep 12 16:46:58 2021 +0000
        path    cuelang.org/go/cmd/cue
        mod     cuelang.org/go  v0.4.0  h1:GLJblw6m2WGGCA3k1v6Wbk9gTOt2qto48ahO2MmSd6I=
        dep     github.com/cockroachdb/apd/v2   v2.0.1  h1:y1Rh3tEU89D+7Tgbw+lp52T6p/GJLpDmNvr10UWqLTE=
        dep     github.com/emicklei/proto       v1.6.15 h1:XbpwxmuOPrdES97FrSfpyy67SSCV/wBIKXqgJzh6hNw=
        dep     github.com/golang/glog  v0.0.0-20160126235308-23def4e6c14b      h1:VKtxabqXZkF25pY9ekfRL6a582T4P37/31XEstQ5p58=
        dep     github.com/google/uuid  v1.2.0  h1:qJYtXnJRWmpe7m/3XlyhrsLrEURqHRM2kxzoxXqyUDs=
        dep     github.com/mpvl/unique  v0.0.0-20150818121801-cbe035fff7de      h1:D5x39vF5KCwKQaw+OC9ZPiLVHXz3UFw2+psEX+gYcto=
        dep     github.com/pkg/errors   v0.8.1  h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I=
        dep     github.com/protocolbuffers/txtpbfmt     v0.0.0-20201118171849-f6a6b3f636fc      h1:gSVONBi2HWMFXCa9jFdYvYk7IwW/mTLxWOF7rXS4LO0=
        dep     github.com/spf13/cobra  v1.0.0  h1:6m/oheQuQ13N9ks4hubMG6BnvwOeaJrqSPLahSnczz8=
        dep     github.com/spf13/pflag  v1.0.3  h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg=
        dep     golang.org/x/mod        v0.3.1-0.20200828183125-ce943fd02449    h1:xUIPaMhvROX9dhPvRCenIJtU78+lbEenGbgqB5hfHCQ=
        dep     golang.org/x/net        v0.0.0-20200226121028-0de0cce0169b      h1:0mm1VjtFUOIlE1SbDlwjYaDxZVDP2S5ou6y0gSgXHu8=
        dep     golang.org/x/text       v0.3.2  h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs=
        dep     golang.org/x/tools      v0.0.0-20200612220849-54c614fe050c      h1:g6oFfz6Cmw68izP3xsdud3Oxu145IPkeFzyRg58AKHM=
        dep     golang.org/x/xerrors    v0.0.0-20191204190536-9bdfabe68543      h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
        dep     gopkg.in/yaml.v3        v3.0.0-20200121175148-a6ecf24a6d71      h1:Xe2gvTZUJpsvOWUnvmL/tmhVBZUmHSvLbMjRj6NUUKo=

Notice the version and checksum information associated with cuelang.org/go.

Recent changes to the CUE project's goreleaser setup should ensure the cuelang.org/go module appears as versioned in the output (not this will not affect historic release artefacts). I have just noticed that those changes did not include the setting of -trimpath, which we will fix.

Hence my suggestion would be that goreleaser can be removed from the MacPorts portfile for building CUE, and plain cmd/go commands used instead.

Regarding updating of the portfile with new releases. Is there any tooling we can leverage to automate that process? I'm thinking something along the following lines: https://github.com/mislav/bump-homebrew-formula-action. How are changes to the portfile controlled?