New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: vgo and the OpenBSD packager's dilemma #26852

Open
qbit opened this Issue Aug 7, 2018 · 11 comments

Comments

Projects
None yet
4 participants
@qbit

qbit commented Aug 7, 2018

OpenBSD's constraints

Packaging on OpenBSD follows a fairly strict workflow:

  1. The build process has no network access.
  2. Distribution files must be fetched using tooling in OpenBSD base. In this case using ftp(1) (a tool similar to wget), which has extra security measures like pledge(2) and is run as a user with reduced privileges.
  3. Packaging of binaries is prohibited. This is largely because of how OpenBSD is developed; -current is where ports and the base system are worked on. At any point ABI breakage can occur, rendering binaries built pre-ABI-change potentially broken.

The current state

Currently packaging Go applications is a trivial task given any of the following (listed in order of ease from the packager's perspective)

  • Upstreams include a populated vendor directory in their source tree.
  • Upstreams provide source tarballs that contain a vendor directory even though they don't track it in their repository.
  • Upstreams use tooling such as dep which allow packagers to re-roll distribution files that include the vendor directory.

The dilemma

Though it hasn't been explicitly stated, one can infer[1] that the vendor directory is on its way out. If this is the case, it will dramatically complicate the build process for packages distributed using frameworks like the OpenBSD ports tree (likely all BSDs and probably a few Linux distributions will be impacted as well).

At first glance, the Proxy Server mentioned in the module doc seems like it might be a decent workaround, however, sticking to the OpenBSD workflow would necessitate rebuilding large portions of the "fetch" mechanism (remember the requirement for fetching with base tooling). It would also require adding infrastructure (and administration overhead) to store the source files.

[1]: https://research.swtch.com/vgo-module - The End of Vendoring

The questions

  • Will the concept of the vendor directory be going away entirely?
  • Is there a plan for official, public facing "Proxy Servers" that could be used to grab versioned tarballs?
  • If the above are false, is there some mechanism planned that will allow for the constraints listed above to be met?

@bradfitz bradfitz added the modules label Aug 7, 2018

@bradfitz

This comment has been minimized.

Member

bradfitz commented Aug 7, 2018

Will the concept of the vendor directory be going away entirely?

I don't think vendor is going away anytime soon. Its usage might drop in popularity over time, though.

Is there a plan for official, public facing "Proxy Servers" that could be used to grab versioned tarballs?

There is the https://github.com/gomods/athens effort.

I will defer to @bcmills @rsc on actual answers to your questions, though.

@bradfitz bradfitz added this to the Unplanned milestone Aug 7, 2018

@bcmills

This comment has been minimized.

Member

bcmills commented Aug 7, 2018

Is there a plan for official, public facing "Proxy Servers" that could be used to grab versioned tarballs?

You'll soon be able to fetch zip files of sources using the go tool itself: see #26610. And note that the GOPROXY protocol supports static file paths, including arbitrary filesystem directories, so once you've downloaded those files you can plop them down wherever you like and start building.

Are those features sufficient for your requirements?

@qbit

This comment has been minimized.

qbit commented Aug 7, 2018

You'll soon be able to fetch zip files of sources using the go tool itself

If the versioned zip's can be fetched outside of the go tool, most of our issues would be resolved. As I mentioned previously, fetching is to be done using in-base tools only, and there is no network access during the actual build phase.

Once we have the zips, we could then use the GOPROXY environment variable to do the actual build after the zip files have been extracted.

@bcmills

This comment has been minimized.

Member

bcmills commented Aug 7, 2018

Put the zips in a tarball on an FTP server?

(Specifically, run go clean -modcache; go mod download to prime the module cache, then tar that up and plop it on an FTP server somewhere.)

@qbit

This comment has been minimized.

qbit commented Aug 7, 2018

If the zips are just compressed versions of a given dependencies source tree, we would just need to extract them into the proper location in GOPROXY (correct?), no need to re-compress them with tar.

We do something similar for Rust applications that use Cargo. During the fetch phase dep source is downloaded from crates.io (using the ftp(1) command) and extracted into the ports build env. Then the offline build phase happens.

@qbit

This comment has been minimized.

qbit commented Aug 7, 2018

Also, just to clarify - ftp on OpenBSD does http requests. There is no requirement on ftp servers :)

@bcmills

This comment has been minimized.

Member

bcmills commented Aug 8, 2018

I don't think you even need to unzip them. Just put the raw zips (and metadata files) in the GOPROXY tree and I think you're good to go.

@rasky

This comment has been minimized.

Member

rasky commented Aug 8, 2018

Why is "a tarball of binary zip files" mentioned as an alternative to vendoring, which already works well, has a very simple semantic, and it's easy to handle?

I'm a little worried about Go's team stance on vendoring. Vendoring is a very important tool for many workflows that might not exist within Google but are very common in the world outside. It's easy to explain, easy to use, easy to version. I'm not sure how side-loading a tarball of binary files into a different directory and then pointing a variable called GOPROXY that counterintuitively supports direct filesystem paths can ever be a good alternative to vendoring.

@qbit

This comment has been minimized.

qbit commented Aug 8, 2018

Why is "a tarball of binary zip files" mentioned as an alternative to vendoring, which already works well, has a very simple semantic, and it's easy to handle?

I think this was mentioned as a OpenBSD specific solution to the packaging problem.

I'm not sure how side-loading a tarball of binary files into a different directory and then pointing a variable called GOPROXY that counterintuitively supports direct filesystem paths can ever be a good alternative to vendoring.

Again, this is something that is specific to 3rd party packagers building packages for their respective systems (in this case, OpenBSD). Not a solution for general consumption - or a proposal for how vendoring should be.

@bcmills I am going to start playing around with athens to see if I can get a working build env inside the OpenBSD ports tree. Assuming that all works fine (it seems like it will based off the features you have mentioned), the only remaining question will be: will there be a publicly available "proxy instance" to pull down the zip files.

@rasky

This comment has been minimized.

Member

rasky commented Aug 9, 2018

I think this was mentioned as a OpenBSD specific solution to the packaging problem.

OpenBSD problem can be solved using go mod vendor, which creates/updates the vendor directory, and thus makes sure that go build -mod=vendor works without ever touching the network.

If the vendor directory is not going away, this is by far the easiest solution: it doesn't require external scripts, directories, environment variables, binary files. The fact that @bcmills is suggesting a far more complicated solution that allows to avoid using vendoring is what worries me.

@qbit

This comment has been minimized.

qbit commented Aug 9, 2018

So from my point of view, having something that allows us to download distfiles (versioned sources packaged as a zip or tarball) for each component (dep) of a given package is ideal. Then there is no need to hand roll a new tarball that contains the vendor directory.

I agree go mod vendor would work, but then we are back to hand-rolling distfiles for every go application in the ports tree.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment