proposal: embed: Allow remote resources to be embedded

[malt3]

Currently, go embed only allows for local files to be embedded in the final binary. While I understand that this beneficial for security, reproducibility and running go build in airgapped environments, it also has a major drawback:
Every file embedded into the go binary has to either be checked in with your VCS or be downloaded dynamically before using go build (and other go commands).

Checking every file into your VCS is not feasible for binary blobs of certain sizes as file sizes are often limited and binary blobs are prone to creating huge diffs.
Adding extra required steps before running go build is problematic as well. I like the fact that I don't need special Makefiles and build tools to use go.

My proposal is as follows:

• Allow go embed with a url and a hash
• Can either point to a single file or tar
• Embedded files can be cached and downloaded in advance (very similar to the handling of go modules)

This has the following advantages:

• Fully deterministic by specifying expected hash
• Lightweight (only url and hash are part of the embed directive)
• Cachable using hash
• Optional early download step allows compilation in air-gapped systems

[seankhliao]

This opens up a massive can of worms on remote resource access: protocol, proxies, authentication, caching, as well as all the issues of tar.

[ianlancetaylor]

I think that people understand that commands like go get access the network. I think people would be very surprised if commands like go build access the network.

[rittneje]

You can use go generate instead to download these artifacts (via curl or similar). Then your build process is essentially go generate && go build.

[malt3]

This opens up a massive can of worms on remote resource access: protocol, proxies, authentication, caching, as well as all the issues of tar.

This is a valid concern and I would understand if a proposal is rejected due to the overall complexity of the implementation.

I think that people understand that commands like go get access the network. I think people would be very surprised if commands like go build access the network

I believe (depending on your proxy configuration, vendoring and previous go build and go mod download invocations), go build will in fact access the network already to download go modules.

You can use go generate instead to download these artifacts (via curl or similar). Then your build process is essentially go generate && go build.

This is a possible workaround if the download should occur in the module I am building directly. This will not work if a module I depend on wants to download + embed a file.
This approach also prevents CI environments and IDEs from working out of the box. Many tools have the assumption of:
git clone the source (or have it as a dependency in go.mod + go.sum), then perform any action directly like: go build, go vet ./..., go test ./... and more. This is why it is recommended to add go generated files into VCS so the assumption holds.
Not having to add every embedded file to the VCS is the entire goal of my proposal.

[mvdan]

Checking every file into your VCS is not feasible for binary blobs of certain sizes as file sizes are often limited and binary blobs are prone to creating huge diffs.

Sounds like you could look into git-lfs then: #47308

Also note that you don't even need to use a VCS to publish a Go module. You can publish versions as zip files on an HTTP server: https://go.dev/ref/mod#serving-from-proxy

Or, more simply, if we're talking about a Go module that you will just build locally without a need to go get it, users could download a zip archive and run go build . directly. Ultimately, the source files for building your Go module belong in one place.

Finally, note that there is currently a limit of about 500 MiB on module sizes, both when compressed as a zip and when uncompressed. I would argue that embedding more than a few tens of megabytes of files is already a bad idea in general, but you'd also run into this limit pretty easily with large files.

[rsc]

This contradicts most of the design goals for modules. We really really want reproducible, self-contained builds that don't change from day to day. The hash helps, but the build can still fail today when it worked yesterday.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[rsc]

This proposal has been declined as infeasible.
— rsc for the proposal review group