proposal: cmd/go: allow fixing all GODEBUG settings at compile time

[apparentlymart]

The GODEBUG environment variable has grown to become the lynchpin of Go's approach to making subtle breaking changes, and at the time I'm writing these there are 50 possible such settings that each causes some part of the Go standard library to behave in a slightly different way than it would by default.

Being able to change these settings dynamically at runtime based on an environment variable is presumably useful for the case where an organization is building a Go program purely for their own internal use, such as running a service that others access over a network, since I imagine this provides some desirable operational flexibility for responding quickly to regressions in production without having to completely rebuild and redeploy.

However, runtime control of these divergent behaviors is less ideal for those building programs that are shipped to third parties to run on their own computers. In that case, the maintainers of the software tend to receive bug reports about behaviors they cannot directly observe and so must attempt to reproduce in a separate environment. The more potential variations in the runtime context, the harder it can be to successfully recreate a similar enough environment to reproduce a reported problem.

Having 50 different settings that can each subtly change the runtime behavior is therefore quite concerning. Even if each of those settings were only binary flags that are either on or off (which is not the case), that represents roughly a million billion possible variations in addition to all of the unavoidable variations of current operating system version, position in the network, etc, etc. Exactly what each of these settings actually does is also not immediately intuitive without studying the associated code carefully, which makes it hard to know that it's even worth trying to enable one to see if that varies the behavior.

For example, running software in an environment where GODEBUG=fips140=on is set can apparently cause subtle interop problems. Of course, I happen to have seen that issue and so now that's something I have on my radar when I'm thinking about how to reproduce a TLS-related bug report, but if I were not aware of it and someone reported that the software I maintain cannot connect to some specific server then it's less likely I would think to try reproducing that with different fips140 settings just to see if that makes it more or less reproducible.

Proposal

I'd love to have an opt-in way to specify that a particular Go program should have a fixed set of "godebug" settings decided entirely at compile time, with no support for overriding them on program startup using an environment variable.

I'm not particularly set on any specific way to represent that, but one possibility would be to add a special new permanent godebug setting that effectively disables the runtime handling of godebug settings, and so can be specified only in the godebug directive in the go.mod of the application's main module:

godebug (
    runtimegodebug=off
)

If any other settings are included in the godebug directive then those would still be compiled into the program, so that the application developer (rather than the end-user) can still use this to disable any behavior changes that the application has not yet been updated to handle.

In the implementation details, I think that what I'm asking for is for the second parsegodebug call here to be skipped if the first one has caused runtimegodebug to be set to off:

go/src/runtime/runtime1.go

Lines 430 to 434 in d50a571

	// apply compile-time GODEBUG settings
	parsegodebug(godebugDefault, nil)
	// apply environment settings
	parsegodebug(godebug, nil)

...though of course exactly how this is implemented is not my main concern; I'm sharing this just in the hope it clarifies what behavior I'm hoping for.

My motivation here is focused on constraining the number of possible runtime behaviors that can vary outside of my program's control, and so it's not important to me for this to make the runtime handling of godebug settings any faster just because they were fixed at compile time, or for this to disable any of the various hooks the runtime has for if the GODEBUG environment variable is changed or unset by the program itself at runtime.

I just want to be able to run automated tests for a specific fixed configuration in our build environment and ensure that everyone using the resulting executables is running the same codepaths we tested against, and thus also the same codepaths I'd be running if I attempted to reproduce a problem using the same executables on my own system.

[ianlancetaylor]

I understand the problem. I also see that the fix you are suggesting is not always going to work. The problem is that different GODEBUG values are for different kinds of situations.

For a GODEBUG like panicnil or updatemaxprocs what you say makes sense. The program can be tested with the desired settings. Fixing them at compile time makes sense. Changing them at runtime would be an odd choice. It would be entirely reasonable for these sorts of GODEBUG setting to only be changed via a godebug line in go.mod or a //go:debug comment in the main package.

But there are also GODEBUG options like netedns0. This GODEBUG rarely needs to be set, but it is required when the local DNS server is some old cable modems used as WiFi routers, as can happen in some home setups. If that GODEBUG is fixed at compile time, then a user running the program in an unfortunate environment won't be able to tell the program to not use the normally desirable EDNS0 DNS feature.

[seankhliao]

#60943 proposed similar but for performance reasons

[seankhliao]

It seems as though rather than blindly trying "a million billion possible variations", it would be more productive for you to build tooling that can capture the environment relevant to your program? It would be able to cover a lot more than just godebugs, including os, external tools, your application's config, etc.

go has go env / go bug, a lot of other tools have similar functionality.

[apparentlymart]

Thanks for the consideration!

But there are also GODEBUG options like netedns0. This GODEBUG rarely needs to be set, but it is required when the local DNS server is some old cable modems used as WiFi routers, as can happen in some home setups. If that GODEBUG is fixed at compile time, then a user running the program in an unfortunate environment won't be able to tell the program to not use the normally desirable EDNS0 DNS feature.

I think the crux of my disagreement with this is not that there aren't cases where such a dynamic switch is required, but that I want the availability of that setting to be something under my application's control, rather than something done by the Go runtime in spite of my application.

The usual way that settings like this are modeled in our application is in a configuration file rather than in environment variables, and certainly not in environment variables whose names have a GO prefix. So I read the above as a good argument for retaining some ability for the application to modify non-immutable GODEBUG settings itself at runtime, rather than for the runtime unilaterally making that decision before application code begins to run.

Currently the way applications can modify GODEBUG settings at runtime is by using os.Setenv("GODEBUG", "...") and thus activating the runtime's special hooks for changes to that specific environment variable name. While I think that's pretty idiosyncratic in the hypothetical world where my proposal had been accepted and so that environment variable would not have been used on process startup, it also just seems okay as a solution that already exists and works, and so keeping that means of dynamic reconfiguration working under this proposal would be a good enough solution for our needs.

To be concrete about it, the way I would want this EDNS0-failure-related discussion to happen would be:

• An end-user of our application encounters a problem where DNS isn't working reliably, and opens a bug report with us about it.

• Our team learns about the EDNS0 feature flag somehow, and we make the tradeoff about whether to either force netedns0 off for all users, whether to make it dynamically-selectable based on a setting in our application's own configuration file, or to decide that running this application in an affected environment is not supported at all, using our knowledge about what contexts our application tends to be used in and our capacity for supporting more functionality variants.

  (this does admittedly have the same problem that the exact effect of each of these GODEBUG settings is not very discoverable, and so it's debatable whether we'd actually even realize this was the cause and that there was a GODEBUG option to change it, but it's more likely that we'd figure it out than our end-user would because our end-users are not typically thinking about the fact that our application is written in Go and seeking out Go-runtime-based solutions to their problems.)

• We make a new release based on that decision. If we decided to either force it on for everyone or to make it opt-in, we also document how long we expect to keep that fallback available so that end-users can plan to fix the problem on their end before the fallback is removed.

  Our application may choose to remove the fallback at a different time than Go does. If we decide to remove it earlier then that's great: we just fix that GODEBUG as disabled in our go.mod file and move on.

  If we decide that we need to keep a fix around for longer than the Go library does then of course that's harder because we'd need to implement our own local mitigation somehow¹, but at least if this opt-in had been through our application's configuration file then we can make our setting activate our own local mitigation instead of the Go runtime's mitigation so that the transition is invisible to end-users².

Overall my position here is that, from the perspective of our end-users, the fact that our application is written in Go is an implementation detail and not something they should ever need to be aware of to successfully use our product, nor something they should ever rely on to achieve functionality beyond what we explicitly decided to support.

---

it would be more productive for you to build tooling that can capture the environment relevant to your program

Our application is under compatibility promises of a similar style than those of the Go project, and so we need to be able to know and constrain what exactly we're including in those promises, and to ensure we have tests covering everything we've historically promised so that we can catch regressions. We don't have the resources to proactively review all newly-added GODEBUG settings, determine how they might affect the behavior of our product, and add regression tests for each one to make sure it stays working in future versions.

I agree that it's important to be able to capture such contextual information and we already do that, but for me that's an entirely separate question from what different behaviors are supported and thus whether our end-users share our understanding of what our compatibility promises are actually covering.

If some of our users need a certain behavior that is activated by a non-default GODEBUG setting then we want to know about that and make our own decision about whether, how, and when to enable it, rather than having end-users quietly tweak behavior using GODEBUG settings we don't even know about and then be retroactively upset if those options vanish or change behavior later (e.g. because we upgraded to a newer version of Go), or if those tweaks quietly cause secondary unwanted behavior which they then perceive as a bug in our application.

(Side-question: is there an API in the standard library somewhere for enumerating all of the currently-supported GODEBUG settings and their effective values at runtime? The codepaths I know of related to GODEBUG are in src/internal and so not accessible to our application, but if there is some way to fully enumerate them all in today's Go then I'd consider including all of that in our collected debug information as a workaround for now. I'd still prefer to be able to disable the environment variable completely, but being able to comprehensively capture all of the effective settings would at least give us better clues for reproduction. 50 different variations is still a lot of possibilities to consider when you aren't already very familiar with what exactly they all do, but it'd be slightly better than the current situation.)

---

#60943 proposed similar but for performance reasons

I'm sorry I didn't find that one in my initial search. If you'd like to close this one as a duplicate then I have no objection.

I do note that the specific implementation idea I shared in my writeup would not address the performance-related concern, and I expect that satisfying the other proposal would be more invasive to the existing godebug checks throughout the standard library than what I asked for here, so I can see an argument for deciding each of these proposals separately even though they overlap with one another.

Footnotes

1. While not applicable to the EDNS0 case in particular, I note that we already have a couple cases where we have effectively our own vendored fork of some older code from the Go standard library where we concluded we needed to retain certain legacy behavior for longer than the Go project did. Of course that's not ideal, but it's a tradeoff we accept that we'll sometimes need to make. ↩

2. It seems that the Go runtime fails with an error in early startup if GODEBUG is set to something unsupported, in which case it would presumably not be possible for our application to "fake" support for an older GODEBUG setting to activate a local mitigation instead. Not that I'd want to do that anyway, but I wanted to address that potential workaround immediately. ↩

[mateusz834]

Wouldn't adding:

func main() {
    os.Setenv("GODEBUG", "")
    // ....
}

solve that?

If not, what are the problems with that currently?

[apparentlymart]

If forcefully setting that environment variable to empty either at the start of main or in an init in the main package were effective in disabling all of the settings then that would be good enough for me, indeed.

As far as I know, currently resetting GODEBUG early in main can work for many of the settings but not for all of them. In particular, the fips140 setting is considered "immutable" and so whatever is chosen by the runtime at startup is frozen for the life of the program.

As far as I know that's the only one that's currently immutable and so a more focused change dealing with just that setting (and any other immutable settings added in future) could be sufficient in principle. I don't have a concrete idea for how that would work, though.

[mateusz834]

As far as I know, currently resetting GODEBUG early in main can work for many of the settings but not for all of them. In particular, the fips140 setting is considered "immutable" and so whatever is chosen by the runtime at startup is frozen for the life of the program.

You can hack that with CGO, since _cgo_init is called before even the runtime gets control, so:

package main

/*
#include <stdlib.h>
#include <stdio.h>

__attribute__((constructor))
static void call_init_env() {
    setenv("GODEBUG", "", 1);
}
*/
import "C"

func main() {
}

go build . && GODEBUG=inittrace=1 ./test

Try commenting the setenv out and see it.

This only works in case you can use CGO.

Of course not ideal, but can be done.

[seankhliao]

runetime/metrics will get you a list of all compatibility godebugs

[rittneje]

If the end user does not know your application is a Go binary, then why would they have GODEBUG set in the first place?

Meanwhile, if they know it is a Go binary, and have intentionally set GODEBUG to get it to work, it seems unhelpful for your application to overrule that. (And if they do that, they should understand it is "out of warranty" so to speak.)

If you are really concerned, can't you just make your application abort if GODEBUG is set?

but if there is some way to fully enumerate them all in today's Go then I'd consider including all of that in our collected debug information as a workaround for now.

Why not just grab the value of the GODEBUG environment variable?

[mknyszek]

As far as I know that's the only one that's currently immutable and so a more focused change dealing with just that setting

There are plenty of immutable GODEBUG keys but they're mostly for debugging the runtime.

Just to be abundantly clear, this discussion should perhaps separate the Go runtime GODEBUG keys that exist to debug the runtime from the ones used for compatibility, and (separately to this discussion) perhaps we should consider separating the Go-runtime-debugging GODEBUG values out from the others in some way.

[apparentlymart]

If you are really concerned, can't you just make your application abort if GODEBUG is set?

That is an alternative I would consider if this proposal were rejected. We are already considering introducing a hard failure if fips140.Enabled returns true, since we already know our application currently malfunctions in that case and so it's likely better to just fail up front rather than failing at some arbitrary point during execution. Rejecting the environment completely would be a generalization of that.

However, I think it's typically good manners to ignore unexpected/unwanted/invalid environment variables rather than failing hard, since the default of environment variables being passed from parent process to child process tends to cause them to end up reaching more programs than they were intended to reach.

I would much prefer to have the variable be completely ignored than to fail with an error, so that it can be set for the benefit of some other Go program that someone might also want to run in the same shell session. Our typical practice -- for the environment variables we intentionally respond to, rather than the ones the Go runtime handles itself -- is to use them only if they contain a value we consider valid and to silently ignore them if they are not, in case they were set for the benefit of other software. (Where possible, we generally prefer to use a configuration file rather than environment variables so that it's unambiguous whether a particular setting was intended for our application's use.)