proposal: build configuration file

[henryas]

Summary

This proposal suggests a “build.config” file inside a go package. The build configuration file is optional and used to specify custom build scripts for the package. It is similar to makefile but with cross-platform path and file manipulation commands. Go build tools should be aware of this build file and build the package accordingly.

Overview

Currently, Go has various tools to allow custom build operations: +build tag, +file tag, go generate, file suffixes, etc. It is possible that more custom build mechanisms may be added to the language in the future.

The problem with the current approaches is that the build specifications are scattered all over the place. It is difficult to see what is going on at a glance. Do you find the go generate in this file or that file?

Another problem is that go generate requires manual invocation. If the developer forgets to call go generate, the files and the resources may not get updated. There is also no intelligent incremental build with go generate.

There is no centralized place for future build-related expansion. Without a centralized place, future build-related expansion will only clutter the language specification (eg. more tags?).

In my Go projects, I often have to resort to makefile to pack resources, auto-generate files, and finally build the project. So far the solution is adequate but makefile is not very portable. There are workarounds, but they are in a weird, makefile-kind of way.

Proposal

This proposed solution is to allow a build configuration file in a package. The build file functions as a custom script to go build tool. There has to be a fixed name for it (eg. build.config) so that people would know where to look. The build file is optional and each package can contain a maximum of one build file.

Go build tool processes the build file first before carrying out the normal build. Processing the build file should be similar to makefile. Check for the specified dependencies and build them if they are out-of-date using the specified custom recipes.

The build file should be portable and work across platforms without any change. It should supports the following cross-platform commands:
• File and directory manipulation
• Filepath utilities

Benefits

There is a centralized place for build configuration. Future build-related expansion can be added here instead of adding more syntax or tags to the language. More supported commands can be added to the build file.

Go build now automatically ensures all resources and files are up-to-date.

Specifics

I envision the build file to have similar components to a typical makefile: targets, dependencies, and recipes, possibly with variable substitution for long names and paths.

Syntax-wise it should be similar to makefile for reduced learning curve, but json or yaml (or maybe Go?) is also fine.

Since the build file is meant for go tools and not manual invocation by the user, there is no need for phony targets. It should simplify the syntax a bit since all targets and dependencies are real actual files.

I don’t think it should support all makefile features. Probably we should just keep it simple with: targets, dependencies, recipes, and variables for the time being.

Let me know what you think.

Thanks.

Henry

[mvdan]

Does this mean go build would run arbitrary code? Because this seems directly against its design. Running arbitrary code will slow down builds and add security problems, especially with third party code.

Also see #39005.

[henryas]

Does this mean go build would run arbitrary code? Because this seems directly against its design. Running arbitrary code will slow down builds and add security problems, especially with third party code.

Also see #39005.

It should work in a similar fashion to makefile. It supports incremental build and rebuilds only the obsolete files. If building a project involves packing resources and updating the relevant source files, should you discount those processes in your overall build time? In my work, it is common to use some kind of config file which gets auto-generated into go source files. It saves time and prevent error when certain requirements change. Or when the data schema changes, it should update the relevant database access code. I am pretty sure other people may use a similar mechanism.

Does makefile prevent running arbitrary code? How are projects that use makefile over the years managing this? Does 'go generate' prevent running arbitrary code? If a third party wants to cause harm, they have more than one way to cause harm. When you use a third party library, you place a certain amount of trust. There is no foolproof way to fully prevent a third party author from causing harms other than to audit the third party code yourself.

One way to bolster security is by restricting the commands available to the build file. Perhaps file manipulation commands should be restricted to the package directory and its children?

[ianlancetaylor]

Makefiles do not prevent running arbitrary code, but go build is not make. In particular go get will download and build third party code from the Internet. We do not want that code to run arbitrary third party code. It's reasonably obvious that running a program built with third party code can be a security risk. It's not obvious at all that running go get will run arbitrary third party code and become a security risk. In particular, go generate is separated from go get and go build to avoid just this risk.

There is no chance that this proposal as written will be accepted, so closing the issue. Please comment if you disagree. Thanks.