Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
cmd/go: [modules + integration] provide foreign content extension points #31326
This report is part of a series, filled at the request of @mdempsky, focused at making Go modules integrator-friendly.
Please do not close or mark it as duplicate before making sure you’ve read and understood the general context. A lot of work went into identifying problems points precisely.
Go needs to provide system extension points, either in go
The metadata file should allow declaring the replacement of an in-module:
… by some system-specific path or value.
Source code is not sufficient to build complex software. It needs other files (documentation, legal files, content files, protobuf files, other language source files…).
At the last stages of system integration, when the integrator has fine knowledge of the target OS, and the necessary tooling to coordinate Go modules with other content providers, it is possible to relocate some of this content to the correct places of the filesystem, or even replace it with better versions. However this is difficult to do without module surgery, if Go modules do not provide an official mechanism for this kind of extension/replacement.
Some of those elements are quite bulky and a system replace will usually happen at the same time of a removal from the
Just a directive that says to the compiler “while compiling with this module, replace references to,
This way Go projects can continue to ignore system-specific file locations while coding, and system integrators can redirect them to the correct place at system integration time.
It's not a replace because replace deals with Go package names and foreign resources do not obey Go package name rules.
The system filename or upstream filename of a foreign resource will not necessarily match the one inside the Go module (some upstreams are notorious for changing their naming regularly).
Other foreign resources can be split in many more files than the layout embedded in the Go module (full Noto, for example, will use a lot more files than a cut-down module-embedded version).
Other foreign resources, like protobuf, use
So basically, you need to replace variable not paths in some cases, and when you do need to replace paths the replacing may require replacing the whole filename, not just the directory name.
At minima, the metadata file should permit a module author to declare a list of variables that can be replaced at integration time with another value (typically, protobuf PATHs)
Allowing to replace foreign resource directories and filenames is only interesting in non-variable mode, if it results in something like python monkey-patching (ie does not require an explicit declaration by the module author).
@rsc You can indeed do everything with patches, but that’s not saying much.
It is laborious to maintain a series of patches at a high quality level over a long span of versions and time. Sometimes, changing a location or a
That’s why Linux distributions have a strong preference for getting fixes merged upstream as soon as possible, and change at little as possible in the upstream dependency graph in the meanwhile.
However, many Go upstreams, while friendly and aware than
This issue is an attempt to define a simple standard extension/handover mechanism, which is convenient for both parties:
While this sounds useful for distribution maintainers, maybe it is a good idea if you first develop a tool that does what you want, and then see if it can become part of go officially, much like happened with vgo? Op vr 12 apr. 2019 16:37 schreef nim-nim <email@example.com>:…
And to give honor where honor is due: it’s a massively simplified variant of the foreign depends suggested by @perillo <https://github.com/perillo> on golang-dev <https://groups.google.com/d/msg/golang-dev/DD88cds-LuI/wBY6rksfCwAJ> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#31326 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEWeec6keLYtbbR6_ZbqymtZ_xEHzLnks5vgJorgaJpZM4cgv1b> .
And the feature would not be useful just to distribution maintainers. Whenever we encountered problems managing the non Go code parts of Go projects, the Go upstreams I interfaced with lamented, that there was no built-in Go feature to handover foreign material. They wanted to be relieved of the burden of worrying, how to manage non-Go material within Go tooling.
I would like to have a
There also should be a standard
Finally the license file should always be defined in a know file.
Alternatively a Go module can have an associated manifest file to declare where these files are stored.
A possible tool that will access this data is
Of course, we have this standard directory setup system-side (at least each system evolves one after a while), and it would be nice to have the pendant within Go modules, to map one to the other easily. This request is only about the mapping part, I didn't want to get into the business of prescribing any particular Go module internal layout.
As described in one of the messages, some things like protobuf
For things like documentation and licensing, those seem like conventions that transcend languages and that it's reinventing the wheel for each language to redefine conventions. For example, https://github.com/licensee/licensee is a programming language agnostic solution to identify licenses within a package.
For things like tweaking file paths or protobufs, those seem very package-specific. I'm having a hard time imagining a solution general and flexible enough to handle those use cases that's substantially different from just patching the source, like @rsc suggested earlier.
Maybe you can give examples of how those problems would be addressed if the packages written in a programming language other than Go.
Most other languages use one or several
Go is relatively unusual in removing the
One way to read this proposal is just the migration of conditionalizing options inside Go module metadata.
Because setting a huge number of options all the time is not fun, Linux systems have consolidated on common filesystem standards (FHS, XDG…). A lot of things can be assumed to be in a standard place and do not need explicit location passing. The existence of this standard directory structure is one reason devs dropped proprietary Unixes like hot potatoes as soon as Linux x86 systems were powerful enough. Proprietary Unixes never achieved this level of standardization.
One thing we will need to define Fedora-side, BTW, is the default location of the system GOPROXY directory (#31304); upstream guidance is welcome.
That is, however, a drag in porting software to systems less normalized (ie Windows, though Microsoft has been steadily fleshing out its own default directory structure in past years).
To limit even further the amount of build variables that need a manual set the C/C++ guys have defined the pkgconfig system. That allows a component to drop metadata in a standard place, with the variables needed to build against it, and their local value. A lot of standard metadata fields are
One way to handle all this would be to make the "build option" layer a pure metadata override of the Go module content, with no change it its payload zip. Though that keeps an unused embedded copy of the foreign content inside the zip payload, which may be confusing, and is definitely inefficient for very bulky foreign content.
Another option would be to have
I believe that GNU autoconf is used for this, not make. And I'm glad that Go does not use autoconf. Optional parts should, ideally, be supported by plugings.
Note that Cgo do support pkg-config. The problem is that many Go projects don't use it because:
In the absence of
One of the persistent nightmares of the C world was how much OSes/distros messed with what goes where and what's done how. The less of that we carry over to Go, the better. I for one consider e.g. Debian's attempt to package Go libraries as "golang-xxx-dev" just miserable.
There's a lot of Go/Node/Rust/etc code being written these days that is typically not installed via the old school linux distro way of putting some files in /usr/share, etc. From my end user/power user/ex-DebianDeveloper perspective, that fresh start seems to have really made many things simpler and more comprehensible. (I'd personally much rather take a distro with pervasive containers for system services than yet another "we have a global package database and everything in /usr" design.)
It would really help sell this issue if you split out specific use cases into their own issues, for example:
Right now the whole thing just comes across as big ball of vague "you are not autoconf, make, or C!". To which my personal answer is: Good!
One of the persistent nightmare of deploying software in production is the way foo language devs feel entitled to copy non-foo language elements in their projects and leave those elements in a dismal state because they are foo language devs and do not really understand non-foo things.
Anyone who had to audit the usual pile of CVE-ridden legal-questionnable stuff project devs like to accumulate will react the same way as distribution. I don't care if the dev feels convenient to copy things and let them rot. I am the person deploying the software in production. I own the systems that will get holed via those security holes. I can get dragged to the judge if I deploy fonts or other things without clear licensing.
So by all means let devs cut and paste mountains of things they do not care triaging correctly in their project code. But let us replace those things with checked and trusted elements before going into production.
devs don't want to deal with the complexity of checking things properly (because that's where the complexity is, not inside distro packaging tech that any IT student can master in a month). That's why distributors exist. The day devs do the correct thing by default there will be no market for distributions.
But I would absolutely like to have better, more uniform, community standards for the above. (Doesn't sound like a compiler change, but just conventions for source layout.)
If you were to split that concrete thing from this vague issue, I'd support the concrete thing. I don't see what the above has to do with extracting licenses.